[IPython-dev] pushing python class definition to engines

[Brian Granger]

Andrew,

As Robert mentioned, this is really a limitation in how Python
serializes classes and instances.  I have dealt with this in a couple
of ways:

1.  Don't use classes :)  More, seriously though, functions, even
those defined interactively can be pushed and pulled using
push_function and pull_function.  There are only a few limitations,
like no closures.

2.  Define your classes in a standalone module that can be imported by
the engines, controller and client.  Then, you can push and pull
instances just fine using push and pull.  This is mostly how I work -
I develop classes in serial first, put them in a module and then
simply import.

Ironically, functions work better for this than classes.  Until Python
has a different way of serializing classes and instances, we are
stuck.

Cheers,

Brian


On Tue, Jul 15, 2008 at 6:01 PM, Andrew Straw <strawman@astraw.com> wrote:
> After seeing Brian Granger's great demo of the parallel abilities in
> IPython last week at the SIAM conference, I'm playing with ipython trunk
> which I just pulled out of launchpad (for this reason, I'm posting on
> the -dev list).
>
> Anyhow, if I start a client in the usual way:
>
> from IPython.kernel import client
> mec = client.MultiEngineClient()
>
> I can easily push Python integers:
>
> a=5
> mec['a'] = a
> print mec.gather('a')
>
> But pushing classes doesn't work:
>
> class MyObj(object):
>    def __init__(self,x):
>        print 'PID',os.getpid()
>        self.x=x
>
> mec['MyObj'] = MyObj
> print mec.gather('MyObj')
>
> That gives the following error. How can I push a class definition to my
> engines?
>
> Traceback (most recent call last):
>  File "flydra/test_ipython.py", line 14, in <module>
>    mec.push_function(dict(MyObj= MyObj,))
>  File
> "/home/astraw/PY_ipython/lib/python2.5/site-packages/IPython/kernel/multiengineclient.py",
> line 606, in push_function
>    return self._blockFromThread(self.smultiengine.push_function,
> namespace, targets=targets, block=block)
>  File
> "/home/astraw/PY_ipython/lib/python2.5/site-packages/IPython/kernel/multiengineclient.py",
> line 456, in _blockFromThread
>    result = blockingCallFromThread(function, *args, **kwargs)
>  File
> "/home/astraw/PY_ipython/lib/python2.5/site-packages/IPython/kernel/twistedutil.py",
> line 69, in blockingCallFromThread
>    return twisted.internet.threads.blockingCallFromThread(reactor, f,
> *a, **kw)
>  File
> "/home/astraw/PY_ipython/lib/python2.5/site-packages/Twisted-8.1.0-py2.5-linux-x86_64.egg/twisted/internet/threads.py",
> line 83, in blockingCallFromThread
>    result.raiseException()
>  File
> "/home/astraw/PY_ipython/lib/python2.5/site-packages/Twisted-8.1.0-py2.5-linux-x86_64.egg/twisted/python/failure.py",
> line 319, in raiseException
>    raise self.type, self.value, self.tb
> AttributeError: 'module' object has no attribute 'MyObj'
>
> _______________________________________________
> IPython-dev mailing list
> IPython-dev@scipy.org
> http://lists.ipython.scipy.org/mailman/listinfo/ipython-dev
>