[Ipython-tickets] [IPython] #231: ipython1 ipcontroller's memory usage

[IPython]

#231: ipython1 ipcontroller's memory usage
--------------------------+-------------------------------------------------
 Reporter:  yichun        |        Owner:  bgranger
     Type:  defect        |       Status:  new     
 Priority:  normal        |    Milestone:          
Component:  ipython1      |      Version:          
 Severity:  normal        |   Resolution:          
 Keywords:  ipcontroller  |  
--------------------------+-------------------------------------------------
Comment (by yichun):

 I am doing embarrassingly parallel tasks in large number (441 task for a
 bunch and about 5*5*5*7 bunches -- on 22 to 30 nodes, so these add up to
 quite some memory, and maybe I did not restart the ipcontroller before re-
 launching my tasks. Each task supposedly only transmit back one float
 number to the controller, in total this will result a about 6M data in my
 controller in some configurations.

 I used to write data file to disks but eventually changed to do as much as
 possible in engines.

 It would be great if use can control some memory usage of the
 ipcontroller, say, to ask it forget (from memory) results of tasks n to
 n+1000, etc, etc..

 Thanks for your answer. I might soon switch to optimization in the
 parameter space which will require much less memory.

-- 
Ticket URL: <http://ipython.scipy.org/ipython/ipython/ticket/231#comment:2>
IPython <http://ipython.scipy.org>
The IPython interactive Python system