[IPython-dev] Update on IPython recent developments

[Stefan van der Walt]

Hi Brian

On Wed, Jan 24, 2007 at 08:45:51PM -0700, Brian Granger wrote:
> > - Is there a limit on the size of data sent from the controller to an
> >   engine?  When working on one machine (with more than one engine), is
> >   data being sent around exactly as on a network (i.e. are there no
> >   optimisations made for working on a local machine?)  Would a UNIX
> >   socket be a viable alternative (to a socket) in this case?
> 
> Absolutely.  There are many limits that come into play:
> 
> 1.  The RAM on each involved machine.
> 
> 2.  The number of engines a controller is controlling.
> 
> 3.  The network protocol being used.  If you  are using the
> perspective broker protocols in the "saw" branch, the size limit for
> this protocol is set in pbconfig.py.  Just a warning though, if you
> exceed this limit on a pull/gather currently, things will hang.  On a
> push you will get a sensible exception raised.  We know how to solve
> the problem on pull/gather of too large objects, but haven't done that
> yet.

In order to solve the problem of sending big heaps of data, I was
thinking of breaking up the data, and providing it piece-wise to an
engine when free, i.e.

while len(data_queue) != 0:
    id = rc.sendToFree({'x':data_queue.pop()})
    rc.execute(id,'compute_longtime("x")',block=False)

which is slightly different from scatter, which immediately sends off
all the data to the different nodes.  But this method will make the
'gather'-operation harder to implement.  Clearly, I've got some more
thinking to do.

Thanks for taking the time to answer all my questions!

Regards
Stéfan