[IPython-user] Chinese char width problem in pyreadline

[Jörgen Stenarson]

I have made some changes since the version I think you are using. Can
you update and try again?

I use two functions to go from unicode->str and str->unicode to get a 
uniform treatment everywhere. We may have to change this at some point
but I think it is a good starting point to have a uniform treatment.

These functions are in pyreadline.unicode_helper
ensure_unicode converts str->unicode assuming pyreadline_encoding
ensure_str     converts unicode->str assuming pyreadline_encoding
pyreadline_encoding  contains the current encoding by default 
sys.stdout.encoding

/Jörgen


pan skrev:
> Sorry for the delayed reply. The attached png shows some encoding 
> problems(but the problems may be not due to pyreadline):
> 1. 0.png shows the varying encodings(utf8, cp936, unicode) of the same 
> Chinese characters.
> 2. 1.png shows the problems occurs when using python console with pyreadline
> 3. 2.png shows the problems occurs when using ipython console with 
> pyreadline
> 
> The reason(I thought) of "using utf8 results in garbage being printed" 
> is that:
> 1. readline_hook returns to the paser of python with the string of 
> consolecodepage(utf8) encoding: 
> readline_hook(prompt).encode(consolecodepage).
> 2. but the parser of python thinks the text is encoded as cp1252(maybe 
> it determine by using locale.getdefaultlocale()[1] or 
> sys.stdout.encoding), so the parser decodes the text using cp1252, and 
> encodes the text using utf8, and parser it.
> 3. finally the result of the paser, will be converted to cp1252, and 
> return to console.
> 
> The problem of paths(or files) encoding is that all the string inputed 
> in ipython will be converted to utf8 automatically(as showing in 2.png). 
> When "cd <non-ascii>", the <non-ascii> strings will be encoded as utf8, 
> and passed to os.chdir, but os.chdir only supports unicode or 
> locale.getdefaultlocale()[1] or sys.stdout.encoding.
> 
> And the attached patch fixed cursor flicking and some Asian IME problem.
> 
> 
> ------------------------------------------------------------------------
> 
> 
> ------------------------------------------------------------------------
> 
> 
> ------------------------------------------------------------------------
>