[IPython-user] ipython1 and farm tasking
Brian Granger
ellisonbg.net@gmail....
Thu Feb 28 13:33:39 CST 2008
On Thu, Feb 28, 2008 at 11:20 AM, Alexandre Gillet <gillet@scripps.edu> wrote:
>
>
>
> Hi Brian,
>
> Thanks for your answer it help get to do what I needed and understand
> better
> the code.
> I have another question.
> I am going to run my task in a non-blocking mode. My task are going to be
> running for few hours. How do I get to check the status of a task? I want
> to
> know if it is done. Do you use tc.get_task_results(id,block=False) ? It
> will
> return None if the task is not done.
Yes, this should work. Let us know if it doesn't.
> How will you find the number of task still in the queue that need to be
> run?
This is a feature we are still lacking. I might have time to
implement this over the weekend (we are having a coding sprint). I
will post to the list if I get it done. Otherwise, you will have to
simply ask if individual tasks are done. Something like:
def remaining(task_ids):
n=0
for tid in task_ids:
if tc.get_task_result(tid,False) is not None:
n +=1
return n
Should work. Let us know if there are other things in the Task
interface that you would like to see.
Brian
>
> By the way what a great package. I am so glad I found it for my
> development.
> Thanks
> Alex
>
>
>
>
> On 2/27/08 2:29 PM, "Brian Granger" <ellisonbg.net@gmail.com> wrote:
>
> > Alex,
> >
> > First, I would suggest updating your ipython1 install from our svn
> > repository. We are about to push out a major new version and the
> > documentation is _much_ better. Also, there are many new features
> > that will hopefully help you. Here is a simple example (using the
> > latest svn of ipython1):
> >
> > In [1]: from ipython1.kernel import client
> >
> > In [2]: mec = client.MultiEngineClient(('127.0.0.1',10105))
> >
> > In [3]: tc = client.TaskClient(('127.0.0.1',10113))
> >
> > In [4]: def fold_package(x):
> > ...: return 2.0*x
> > ...:
> >
> > In [5]: mec.push_function(dict(fold_package=fold_package))
> > Out[5]: [None, None, None, None]
> >
> > In [6]: tasks =
> > [client.Task("y=fold_package(x)",push={'x':x},pull=('y',)) for x in
> > range(128)]
> >
> > In [7]: task_ids = [tc.run(t) for t in tasks]
> >
> > In [8]: tc.barrier(task_ids)
> >
> > In [9]: task_results = [tc.get_task_result(tid) for tid in task_ids]
> >
> > In [10]: results = [tr.ns.y for tr in task_results]
> >
> > In [11]: print results
> > [0.0, 2.0, 4.0, 6.0, 8.0, 10.0, 12.0, 14.0, 16.0, 18.0, 20.0, 22.0,
> > 24.0, 26.0, 28.0, 30.0, 32.0, 34.0, 36.0, 38.0, 40.0, 42.0, 44.0,
> > 46.0, 48.0, 50.0, 52.0, 54.0, 56.0, 58.0, 60.0, 62.0, 64.0, 66.0,
> > 68.0, 70.0, 72.0, 74.0, 76.0, 78.0, 80.0, 82.0, 84.0, 86.0, 88.0,
> > 90.0, 92.0, 94.0, 96.0, 98.0, 100.0, 102.0, 104.0, 106.0, 108.0,
> > 110.0, 112.0, 114.0, 116.0, 118.0, 120.0, 122.0, 124.0, 126.0, 128.0,
> > 130.0, 132.0, 134.0, 136.0, 138.0, 140.0, 142.0, 144.0, 146.0, 148.0,
> > 150.0, 152.0, 154.0, 156.0, 158.0, 160.0, 162.0, 164.0, 166.0, 168.0,
> > 170.0, 172.0, 174.0, 176.0, 178.0, 180.0, 182.0, 184.0, 186.0, 188.0,
> > 190.0, 192.0, 194.0, 196.0, 198.0, 200.0, 202.0, 204.0, 206.0, 208.0,
> > 210.0, 212.0, 214.0, 216.0, 218.0, 220.0, 222.0, 224.0, 226.0, 228.0,
> > 230.0, 232.0, 234.0, 236.0, 238.0, 240.0, 242.0, 244.0, 246.0, 248.0,
> > 250.0, 252.0, 254.0]
> >
> > Or if you don't need load balancing:
> >
> > # This sends the fold_package function for you!
> > results = mec.map(fold_package, range(128))
> >
> > Let us know if you run into other problems.
> >
> > Cheers,
> >
> > Brian
> >
> > On Mon, Feb 25, 2008 at 7:44 PM, Alexandre Gillet <gillet@scripps.edu>
> wrote:
> >> Hi,
> >>
> >> I just started using ipython1 do to distribute job on multiple cpu. I
> >> am having some issue and I am not sure how it works.
> >> I want to pass a function to be run by each task on each client.
> >> In the following code, the function fold_package need to be run on each
> >> client.
> >>
> >> packages_list=[ '3114', '3115','3116']
> >> # create a remote controller instance
> >> rc = kernel.RemoteController(('127.0.0.1',10105))
> >> # create task controller instance
> >> tc = kernel.TaskController(('127.0.0.1', 10113))
> >> # commands won't block by default
> >> rc.block = False
> >> # get id of available engine
> >> engines_id = rc.getIDs()
> >> # process the list of packages by dispatching them to different
> computer
> >> # create the task list
> >> tasks = [kernel.Task("fold_package(%s)"%t) for t in packages_list]
> >> # test task controller
> >> taskIDs = [tc.run(t) for t in tasks]
> >>
> >>
> >> when I run that code I get:
> >> NameError: name 'fold_package' is not defined
> >>
> >> My questions are;
> >> How do you pass a function define in my script to the client engine?
> >> Or Do I have to create a package that will contains my function and
> >> installed it on each client?
> >>
> >> Thanks for any advices and answers.
> >> Alex
> >>
> >> --
> >> o Alexandre Gillet Ph.D. email: gillet@scripps.edu
> >> / The Scripps Research Institute,
> >> o Dept. Molecular Biology, MB-5,
> >> \ 10550 North Torrey Pines Road,
> >> o La Jolla, CA 92037-1000, USA.
> >> / tel: (858) 784-2053
> >> o fax: (858) 784-2860
> >> web: http://mgl.scripps.edu/projects/tangible_models/
> >> _______________________________________________
> >> IPython-user mailing list
> >> IPython-user@scipy.org
> >> http://lists.ipython.scipy.org/mailman/listinfo/ipython-user
> >>
>
> --
> o Alexandre Gillet Ph.D. email: gillet (-at-) scripps.edu
>
> / The Scripps Research Institute,
> o Dept. Molecular Biology, MB-5,
> \ 10550 North Torrey Pines Road,
> o La Jolla, CA 92037-1000, USA.
> / tel: (858) 784-2053
> o fax: (858) 784-2860
> web: http://mgl.scripps.edu/projects/tangible_models/
>
>
More information about the IPython-user
mailing list