[Ipython-tickets] [IPython] #231: ipython1 ipcontroller's memory usage

[IPython]

#231: ipython1 ipcontroller's memory usage
--------------------------+-------------------------------------------------
 Reporter:  yichun        |        Owner:  bgranger
     Type:  defect        |       Status:  new     
 Priority:  normal        |    Milestone:          
Component:  ipython1      |      Version:          
 Severity:  normal        |   Resolution:          
 Keywords:  ipcontroller  |  
--------------------------+-------------------------------------------------
Comment (by bgranger):

 There are a number of things that control the memory consumption of the
 Controller:

 1) The number of pending tasks/commands that are not yet being worked on.
 If you have 1000s of pending tasks, these will surely take up memory.

 2) The number of completed tasks - in the task controller, all completed
 tasks are kept around currently.  This is obviously a design problem.
 Eventually, we will move to a system that keeps the most recent N results
 in memory, and stores the rest to disk.  But, do you really need to bring
 the result back through the controller?  Can you save it to disk?  Do you
 see this when you run many such tasks?

 3) The number of engines connected and being used.  We have seen cases
 where a user was pulling a 10 MB array from 64 engines.  While 10MB seems
 small, 640 MB is not.  Furthermore, if you are running the engines on the
 same machine as the controller, there is a short period of time where the
 arrays is both on the controller and the engine -> then your memory
 consumption is over a GB.  The moral of the story is that you do have to
 be careful about moving data around.

 Can you give us more information about exactly what you are doing - the
 controller should not be consuming 400 MB unless you are doing something
 that is memory intensive.

-- 
Ticket URL: <http://ipython.scipy.org/ipython/ipython/ticket/231#comment:1>
IPython <http://ipython.scipy.org>
The IPython interactive Python system