我有这个字符串，它已使用电子邮件模块从 Quoted-printable 解码为 ISO-8859-1。这给了我像“\xC4pple”这样的字符串，它对应于“Äpple”(瑞典语中的 Apple)。 但是，我无法将这些字符串转换为 UTF-8。

>>> apple = "\xC4pple"
>>> apple
'\xc4pple'
>>> apple.encode("UTF-8")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc4 in position 0: ordinal not in     range(128)

我该怎么办？

最佳答案

这是一个常见的问题，所以这里有一个比较彻底的说明。

对于非 unicode 字符串(即那些没有 u 前缀的字符串，例如 u'\xc4pple')，必须从 native 编码(iso8859-1 /latin1，除非 modified with the enigmatic sys.setdefaultencoding 函数)到 unicode , 然后编码成可以显示你想要的字符的字符集，在这种情况下我推荐UTF-8 .

首先，这是一个方便的实用函数，它有助于阐明 Python 2.7 字符串和 unicode 的模式:

>>> def tell_me_about(s): return (type(s), s)

纯字符串

>>> v = "\xC4pple" # iso-8859-1 aka latin1 encoded string

>>> tell_me_about(v)
(<type 'str'>, '\xc4pple')

>>> v
'\xc4pple'        # representation in memory

>>> print v
?pple             # map the iso-8859-1 in-memory to iso-8859-1 chars
                  # note that '\xc4' has no representation in iso-8859-1, 
                  # so is printed as "?".

解码 iso8859-1 字符串 - 将纯字符串转换为 unicode

>>> uv = v.decode("iso-8859-1")
>>> uv
u'\xc4pple'       # decoding iso-8859-1 becomes unicode, in memory

>>> tell_me_about(uv)
(<type 'unicode'>, u'\xc4pple')

>>> print v.decode("iso-8859-1")
Äpple             # convert unicode to the default character set
                  # (utf-8, based on sys.stdout.encoding)

>>> v.decode('iso-8859-1') == u'\xc4pple'
True              # one could have just used a unicode representation 
                  # from the start

更多的插图 — 带有“Ä”

>>> u"Ä" == u"\xc4"
True              # the native unicode char and escaped versions are the same

>>> "Ä" == u"\xc4"  
False             # the native unicode char is '\xc3\x84' in latin1

>>> "Ä".decode('utf8') == u"\xc4"
True              # one can decode the string to get unicode

>>> "Ä" == "\xc4"
False             # the native character and the escaped string are
                  # of course not equal ('\xc3\x84' != '\xc4').

编码为 UTF

>>> u8 = v.decode("iso-8859-1").encode("utf-8")
>>> u8
'\xc3\x84pple'    # convert iso-8859-1 to unicode to utf-8

>>> tell_me_about(u8)
(<type 'str'>, '\xc3\x84pple')

>>> u16 = v.decode('iso-8859-1').encode('utf-16')
>>> tell_me_about(u16)
(<type 'str'>, '\xff\xfe\xc4\x00p\x00p\x00l\x00e\x00')

>>> tell_me_about(u8.decode('utf8'))
(<type 'unicode'>, u'\xc4pple')

>>> tell_me_about(u16.decode('utf16'))
(<type 'unicode'>, u'\xc4pple')

unicode 与 UTF 和 latin1 的关系

>>> print u8
Äpple             # printing utf-8 - because of the encoding we now know
                  # how to print the characters

>>> print u8.decode('utf-8') # printing unicode
Äpple

>>> print u16     # printing 'bytes' of u16
���pple

>>> print u16.decode('utf16')
Äpple             # printing unicode

>>> v == u8
False             # v is a iso8859-1 string; u8 is a utf-8 string

>>> v.decode('iso8859-1') == u8
False             # v.decode(...) returns unicode

>>> u8.decode('utf-8') == v.decode('latin1') == u16.decode('utf-16')
True              # all decode to the same unicode memory representation
                  # (latin1 is iso-8859-1)

Unicode 异常

 >>> u8.encode('iso8859-1')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0:
  ordinal not in range(128)

>>> u16.encode('iso8859-1')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0:
  ordinal not in range(128)

>>> v.encode('iso8859-1')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc4 in position 0:
  ordinal not in range(128)

可以通过从特定编码(latin-1、utf8、utf16)转换为 unicode 来解决这些问题，例如u8.decode('utf8').encode('latin1').

所以也许可以得出以下原则和概括:

• str 类型是一组字节，它可能具有多种编码之一，例如 Latin-1、UTF-8 和 UTF-16
• unicode 类型是一组字节，可以转换为任意数量的编码，最常见的是 UTF-8 和 latin-1 (iso8859-1)
• print 命令有 its own logic for encoding ，设置为 sys.stdout.encoding 并默认为 UTF-8
• 在转换为另一种编码之前，必须将 str 解码为 un​​icode。

当然，所有这些都在 Python 3.x 中发生了变化。

希望这是有启发性的。

进一步阅读

• Characters vs. Bytes , 蒂姆·布雷 (Tim Bray)。

还有 Armin Ronacher 非常具有说明性的咆哮:

• The Updated Guide to Unicode on Python (July 2, 2013)
• More About Unicode in Python 2 and 3 (January 5, 2014)
• UCS vs UTF-8 as Internal String Encoding (January 9, 2014)
• Everything you did not want to know about Unicode in Python 3 (May 12, 2014)

关于Python:从 ISO-8859-1/latin1 转换为 UTF-8，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/6539881/