[IPython-user] Colorizing lines in debuggers and post-mortem

[R. Bernstein]

Fernando Perez writes:
 > Sure, I don't see a problem with that, and I'd be happy to apply such
 > a patch.  

Actually, I don't think I'll be able to look into this for a while. I
just wanted to put it out there though (in case there's someone
else). But as I think about it and scan the Internet for an existing
incremental tokenizer/colorizer. I came across this:
http://tom.idealog.info/blog/20031205-1070600658.shtml

The two relevant observations there that I think are:

1. You don't really need parsing, just lexical analysis.

2. The tokenizer object might have a "state" variable so that when it
   is called successively it picks up where it was before. (This I
   gleaned looking at the code). Clearly one might want a reset()
   method.  In Python, the state across lines is simple andreally
   amounts to only "inside a string" or not. Comments and tokens can't
   span lines.


 > As long as you test it with things like
 > 
 > 
 > """foo.....
 > 
 > being the last line of input (a typical case that would give
 > tokenization problems, being an unterminated string), we'll gladly
 > apply it.
 > 
 > The debugging support in ipython is admittedly rather minimal, and has
 > really only improved significantly thanks to user contributions (such
 > as yours).

Thanks for your kind words.

 > 
 > I'll also try to catch up on the recent patches in a couple of days.

Thanks.