[IPython-user] problem with unicode and the compile function in ipython

[pan]

2007/12/13, Robert Kern <robert.kern@gmail.com>:
>
>
> sys.setdefaultencoding() should not be used under any circumstances.
>
>
If can not use sys.setdefaultencoding(), have to implement the behavior of
setdefaultencoding by self.

Something like this, just as pyreadline does in unicode_helper.py, now can
use ensure_unicode as unicode and ensure_str as str.

import sys

try:
    # Get system encoding at startup time.  Certain terminals (like Emacs
    # under Win32 have it set to None, and we need to have a known valid
    # encoding to use in the raw_input() method
    codepage = sys.stdin.encoding or 'ascii'
except AttributeError:        # This error occurs when pdb imports readline
and doctest has replaced
                              # stdout with stdout collector
    codepage = "ascii"   # assume ascii codepage


def ensure_unicode(text, codepage=codepage):
    # unicode, object does not have '__str__'
    if isinstance(text, unicode) or not hasattr(text, '__str__'):
        return unicode(text)

    # str, object
    return str(text).decode(codepage, "replace")


def ensure_str(text, codepage=codepage):
    # unicode
    if isinstance(text, unicode):
        return text.encode(codepage, "replace")

    # object, str, object does not have '__str__'
    return str(text)

But you still must be careful in some circumstances, like this:

    If s1 is unicode, s2 is str:
        1. '%s, %s' % (s1, s2)
        2. ' '.join([s1, s2])

The result will be unicode, so s2 will be decoded as 'ascii'.
So you have to ensure_str or ensure_unicode s1 and s2 before.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.ipython.scipy.org/pipermail/ipython-user/attachments/20071213/042a5d3e/attachment.html