text:
PEP: 100 Title: Python Unicode Integration Author: Marc-André Lemburg
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 10-Mar-2000 Python-Version: 2.0 Post-History:

Historical Note

This document was first written by Marc-Andre in the pre-PEP days, and
was originally distributed as Misc/unicode.txt in Python distributions
up to and included Python 2.1. The last revision of the proposal in that
location was labeled version 1.7 (CVS revision 3.10). Because the
document clearly serves the purpose of an informational PEP in the
post-PEP era, it has been moved here and reformatted to comply with PEP
guidelines. Future revisions will be made to this document, while
Misc/unicode.txt will contain a pointer to this PEP.

-Barry Warsaw, PEP editor

Introduction

The idea of this proposal is to add native Unicode 3.0 support to Python
in a way that makes use of Unicode strings as simple as possible without
introducing too many pitfalls along the way.

Since this goal is not easy to achieve -- strings being one of the most
fundamental objects in Python -- we expect this proposal to undergo some
significant refinements.

Note that the current version of this proposal is still a bit unsorted
due to the many different aspects of the Unicode-Python integration.

The latest version of this document is always available at:
http://starship.python.net/~lemburg/unicode-proposal.txt

Older versions are available as:
http://starship.python.net/~lemburg/unicode-proposal-X.X.txt

[ed. note: new revisions should be made to this PEP document, while the
historical record previous to version 1.7 should be retrieved from MAL's
url, or Misc/unicode.txt]

Conventions

-   In examples we use u = Unicode object and s = Python string
-   'XXX' markings indicate points of discussion (PODs)

General Remarks

-   Unicode encoding names should be lower case on output and
    case-insensitive on input (they will be converted to lower case by
    all APIs taking an encoding name as input).
-   Encoding names should follow the name conventions as used by the
    Unicode Consortium: spaces are converted to hyphens, e.g. 'utf 16'
    is written as 'utf-16'.
-   Codec modules should use the same names, but with hyphens converted
    to underscores, e.g. utf_8, utf_16, iso_8859_1.

Unicode Default Encoding

The Unicode implementation has to make some assumption about the
encoding of 8-bit strings passed to it for coercion and about the
encoding to as default for conversion of Unicode to strings when no
specific encoding is given. This encoding is called <default encoding>
throughout this text.

For this, the implementation maintains a global which can be set in the
site.py Python startup script. Subsequent changes are not possible. The
<default encoding> can be set and queried using the two sys module APIs:

sys.setdefaultencoding(encoding)

    Sets the <default encoding> used by the Unicode implementation.
    encoding has to be an encoding which is supported by the Python
    installation, otherwise, a LookupError is raised.

    Note: This API is only available in site.py! It is removed from the
    sys module by site.py after usage.

sys.getdefaultencoding()

    Returns the current <default encoding>.

If not otherwise defined or set, the <default encoding> defaults to
'ascii'. This encoding is also the startup default of Python (and in
effect before site.py is executed).

Note that the default site.py startup module contains disabled optional
code which can set the <default encoding> according to the encoding
defined by the current locale. The locale module is used to extract the
encoding from the locale default settings defined by the OS environment
(see locale.py). If the encoding cannot be determined, is unknown or
unsupported, the code defaults to setting the <default encoding> to
'ascii'. To enable this code, edit the site.py file or place the
appropriate code into the sitecustomize.py module of your Python
installation.

Unicode Constructors

Python should provide a built-in constructor for Unicode strings which
is available through __builtins__:

    u = unicode(encoded_string[,encoding=<default encoding>][,errors="strict"])

    u = u'<unicode-escape encoded Python string>'

    u = ur'<raw-unicode-escape encoded Python string>'

With the 'unicode-escape' encoding being defined as:

-   all non-escape characters represent themselves as Unicode ordinal
    (e.g. 'a' -> U+0061).
-   all existing defined Python escape sequences are interpreted as
    Unicode ordinals; note that \xXXXX can represent all Unicode
    ordinals, and \OOO (octal) can represent Unicode ordinals up to
    U+01FF.
-   a new escape sequence, \uXXXX, represents U+XXXX; it is a syntax
    error to have fewer than 4 digits after \u.

For an explanation of possible values for errors see the Codec section
below.

Examples:

    u'abc'          -> U+0061 U+0062 U+0063
    u'\u1234'       -> U+1234
    u'abc\u1234\n'  -> U+0061 U+0062 U+0063 U+1234 U+005c

The 'raw-unicode-escape' encoding is defined as follows:

-   \uXXXX sequence represent the U+XXXX Unicode character if and only
    if the number of leading backslashes is odd
-   all other characters represent themselves as Unicode ordinal (e.g.
    'b' -> U+0062)

Note that you should provide some hint to the encoding you used to write
your programs as pragma line in one the first few comment lines of the
source file (e.g. '# source file encoding: latin-1'). If you only use
7-bit ASCII then everything is fine and no such notice is needed, but if
you include Latin-1 characters not defined in ASCII, it may well be
worthwhile including a hint since people in other countries will want to
be able to read your source strings too.

Unicode Type Object

Unicode objects should have the type UnicodeType with type name
'unicode', made available through the standard types module.

Unicode Output

Unicode objects have a method .encode([encoding=<default encoding>])
which returns a Python string encoding the Unicode string using the
given scheme (see Codecs).

    print u := print u.encode()   # using the <default encoding>

    str(u)  := u.encode()         # using the <default encoding>

    repr(u) := "u%s" % repr(u.encode('unicode-escape'))

Also see Internal Argument Parsing and Buffer Interface for details on
how other APIs written in C will treat Unicode objects.

Unicode Ordinals

Since Unicode 3.0 has a 32-bit ordinal character set, the implementation
should provide 32-bit aware ordinal conversion APIs:

    ord(u[:1]) (this is the standard ord() extended to work with Unicode
                objects)
      --> Unicode ordinal number (32-bit)

    unichr(i)
        --> Unicode object for character i (provided it is 32-bit);
            ValueError otherwise

Both APIs should go into __builtins__ just like their string
counterparts ord() and chr().

Note that Unicode provides space for private encodings. Usage of these
can cause different output representations on different machines. This
problem is not a Python or Unicode problem, but a machine setup and
maintenance one.

Comparison & Hash Value

Unicode objects should compare equal to other objects after these other
objects have been coerced to Unicode. For strings this means that they
are interpreted as Unicode string using the <default encoding>.

Unicode objects should return the same hash value as their ASCII
equivalent strings. Unicode strings holding non-ASCII values are not
guaranteed to return the same hash values as the default encoded
equivalent string representation.

When compared using cmp() (or PyObject_Compare()) the implementation
should mask TypeErrors raised during the conversion to remain in synch
with the string behavior. All other errors such as ValueErrors raised
during coercion of strings to Unicode should not be masked and passed
through to the user.

In containment tests ('a' in u'abc' and u'a' in 'abc') both sides should
be coerced to Unicode before applying the test. Errors occurring during
coercion (e.g. None in u'abc') should not be masked.

Coercion

Using Python strings and Unicode objects to form new objects should
always coerce to the more precise format, i.e. Unicode objects.

    u + s := u + unicode(s)

    s + u := unicode(s) + u

All string methods should delegate the call to an equivalent Unicode
object method call by converting all involved strings to Unicode and
then applying the arguments to the Unicode method of the same name, e.g.

    string.join((s,u),sep) := (s + sep) + u

    sep.join((s,u)) := (s + sep) + u

For a discussion of %-formatting w/r to Unicode objects, see Formatting
Markers.

Exceptions

UnicodeError is defined in the exceptions module as a subclass of
ValueError. It is available at the C level via PyExc_UnicodeError. All
exceptions related to Unicode encoding/decoding should be subclasses of
UnicodeError.

Codecs (Coder/Decoders) Lookup

A Codec (see Codec Interface Definition) search registry should be
implemented by a module "codecs":

    codecs.register(search_function)

Search functions are expected to take one argument, the encoding name in
all lower case letters and with hyphens and spaces converted to
underscores, and return a tuple of functions (encoder, decoder,
stream_reader, stream_writer) taking the following arguments:

encoder and decoder

    These must be functions or methods which have the same interface as
    the .encode/.decode methods of Codec instances (see Codec
    Interface). The functions/methods are expected to work in a
    stateless mode.

stream_reader and stream_writer

    These need to be factory functions with the following interface:

        factory(stream,errors='strict')

    The factory functions must return objects providing the interfaces
    defined by StreamWriter/StreamReader resp. (see Codec Interface).
    Stream codecs can maintain state.

    Possible values for errors are defined in the Codec section below.

In case a search function cannot find a given encoding, it should return
None.

Aliasing support for encodings is left to the search functions to
implement.

The codecs module will maintain an encoding cache for performance
reasons. Encodings are first looked up in the cache. If not found, the
list of registered search functions is scanned. If no codecs tuple is
found, a LookupError is raised. Otherwise, the codecs tuple is stored in
the cache and returned to the caller.

To query the Codec instance the following API should be used:

    codecs.lookup(encoding)

This will either return the found codecs tuple or raise a LookupError.

Standard Codecs

Standard codecs should live inside an encodings/ package directory in
the Standard Python Code Library. The __init__.py file of that directory
should include a Codec Lookup compatible search function implementing a
lazy module based codec lookup.

Python should provide a few standard codecs for the most relevant
encodings, e.g.

    'utf-8':              8-bit variable length encoding
    'utf-16':             16-bit variable length encoding (little/big endian)
    'utf-16-le':          utf-16 but explicitly little endian
    'utf-16-be':          utf-16 but explicitly big endian
    'ascii':              7-bit ASCII codepage
    'iso-8859-1':         ISO 8859-1 (Latin 1) codepage
    'unicode-escape':     See Unicode Constructors for a definition
    'raw-unicode-escape': See Unicode Constructors for a definition
    'native':             Dump of the Internal Format used by Python

Common aliases should also be provided per default, e.g. 'latin-1' for
'iso-8859-1'.

Note: 'utf-16' should be implemented by using and requiring byte order
marks (BOM) for file input/output.

All other encodings such as the CJK ones to support Asian scripts should
be implemented in separate packages which do not get included in the
core Python distribution and are not a part of this proposal.

Codecs Interface Definition

The following base class should be defined in the module "codecs". They
provide not only templates for use by encoding module implementors, but
also define the interface which is expected by the Unicode
implementation.

Note that the Codec Interface defined here is well suitable for a larger
range of applications. The Unicode implementation expects Unicode
objects on input for .encode() and .write() and character buffer
compatible objects on input for .decode(). Output of .encode() and
.read() should be a Python string and .decode() must return an Unicode
object.

First, we have the stateless encoders/decoders. These do not work in
chunks as the stream codecs (see below) do, because all components are
expected to be available in memory.

    class Codec:

        """Defines the interface for stateless encoders/decoders.

           The .encode()/.decode() methods may implement different
           error handling schemes by providing the errors argument.
           These string values are defined:

             'strict'  - raise an error (or a subclass)
             'ignore'  - ignore the character and continue with the next
             'replace' - replace with a suitable replacement character;
                         Python will use the official U+FFFD
                         REPLACEMENT CHARACTER for the builtin Unicode
                         codecs.
        """

        def encode(self,input,errors='strict'):

            """Encodes the object input and returns a tuple (output
               object, length consumed).

               errors defines the error handling to apply.  It
               defaults to 'strict' handling.

               The method may not store state in the Codec instance.
               Use StreamCodec for codecs which have to keep state in
               order to make encoding/decoding efficient.
            """

        def decode(self,input,errors='strict'):

            """Decodes the object input and returns a tuple (output
               object, length consumed).

               input must be an object which provides the
               bf_getreadbuf buffer slot.  Python strings, buffer
               objects and memory mapped files are examples of objects
               providing this slot.

               errors defines the error handling to apply.  It
               defaults to 'strict' handling.

               The method may not store state in the Codec instance.
               Use StreamCodec for codecs which have to keep state in
               order to make encoding/decoding efficient.

            """

StreamWriter and StreamReader define the interface for stateful
encoders/decoders which work on streams. These allow processing of the
data in chunks to efficiently use memory. If you have large strings in
memory, you may want to wrap them with cStringIO objects and then use
these codecs on them to be able to do chunk processing as well, e.g. to
provide progress information to the user.

    class StreamWriter(Codec):

        def __init__(self,stream,errors='strict'):

            """Creates a StreamWriter instance.

               stream must be a file-like object open for writing
               (binary) data.

               The StreamWriter may implement different error handling
               schemes by providing the errors keyword argument.
               These parameters are defined:

                 'strict' - raise a ValueError (or a subclass)
                 'ignore' - ignore the character and continue with the next
                 'replace'- replace with a suitable replacement character
            """
            self.stream = stream
            self.errors = errors

        def write(self,object):

            """Writes the object's contents encoded to self.stream.
            """
            data, consumed = self.encode(object,self.errors)
            self.stream.write(data)

        def writelines(self, list):

            """Writes the concatenated list of strings to the stream
               using .write().
            """
            self.write(''.join(list))

        def reset(self):

            """Flushes and resets the codec buffers used for keeping state.

               Calling this method should ensure that the data on the
               output is put into a clean state, that allows appending
               of new fresh data without having to rescan the whole
               stream to recover state.
            """
            pass

        def __getattr__(self,name, getattr=getattr):

            """Inherit all other methods from the underlying stream.
            """
            return getattr(self.stream,name)

    class StreamReader(Codec):

        def __init__(self,stream,errors='strict'):

            """Creates a StreamReader instance.

               stream must be a file-like object open for reading
               (binary) data.

               The StreamReader may implement different error handling
               schemes by providing the errors keyword argument.
               These parameters are defined:

                 'strict' - raise a ValueError (or a subclass)
                 'ignore' - ignore the character and continue with the next
                 'replace'- replace with a suitable replacement character;
            """
            self.stream = stream
            self.errors = errors

        def read(self,size=-1):

            """Decodes data from the stream self.stream and returns the
               resulting object.

               size indicates the approximate maximum number of bytes
               to read from the stream for decoding purposes.  The
               decoder can modify this setting as appropriate.  The
               default value -1 indicates to read and decode as much
               as possible.  size is intended to prevent having to
               decode huge files in one step.

               The method should use a greedy read strategy meaning
               that it should read as much data as is allowed within
               the definition of the encoding and the given size, e.g.
               if optional encoding endings or state markers are
               available on the stream, these should be read too.
            """
            # Unsliced reading:
            if size < 0:
                return self.decode(self.stream.read())[0]

            # Sliced reading:
            read = self.stream.read
            decode = self.decode
            data = read(size)
            i = 0
            while 1:
                try:
                    object, decodedbytes = decode(data)
                except ValueError,why:
                    # This method is slow but should work under pretty
                    # much all conditions; at most 10 tries are made
                    i = i + 1
                    newdata = read(1)
                    if not newdata or i > 10:
                        raise
                    data = data + newdata
                else:
                    return object

        def readline(self, size=None):

            """Read one line from the input stream and return the
               decoded data.

               Note: Unlike the .readlines() method, this method
               inherits the line breaking knowledge from the
               underlying stream's .readline() method -- there is
               currently no support for line breaking using the codec
               decoder due to lack of line buffering.  Subclasses
               should however, if possible, try to implement this
               method using their own knowledge of line breaking.

               size, if given, is passed as size argument to the
               stream's .readline() method.
            """
            if size is None:
                line = self.stream.readline()
            else:
                line = self.stream.readline(size)
            return self.decode(line)[0]

        def readlines(self, sizehint=0):

            """Read all lines available on the input stream
               and return them as list of lines.

               Line breaks are implemented using the codec's decoder
               method and are included in the list entries.

               sizehint, if given, is passed as size argument to the
               stream's .read() method.
            """
            if sizehint is None:
                data = self.stream.read()
            else:
                data = self.stream.read(sizehint)
            return self.decode(data)[0].splitlines(1)

        def reset(self):

            """Resets the codec buffers used for keeping state.

               Note that no stream repositioning should take place.
               This method is primarily intended to be able to recover
               from decoding errors.

            """
            pass

        def __getattr__(self,name, getattr=getattr):

            """ Inherit all other methods from the underlying stream.
            """
            return getattr(self.stream,name)

Stream codec implementors are free to combine the StreamWriter and
StreamReader interfaces into one class. Even combining all these with
the Codec class should be possible.

Implementors are free to add additional methods to enhance the codec
functionality or provide extra state information needed for them to
work. The internal codec implementation will only use the above
interfaces, though.

It is not required by the Unicode implementation to use these base
classes, only the interfaces must match; this allows writing Codecs as
extension types.

As guideline, large mapping tables should be implemented using static C
data in separate (shared) extension modules. That way multiple processes
can share the same data.

A tool to auto-convert Unicode mapping files to mapping modules should
be provided to simplify support for additional mappings (see
References).

Whitespace

The .split() method will have to know about what is considered
whitespace in Unicode.

Case Conversion

Case conversion is rather complicated with Unicode data, since there are
many different conditions to respect. See

  http://www.unicode.org/unicode/reports/tr13/

for some guidelines on implementing case conversion.

For Python, we should only implement the 1-1 conversions included in
Unicode. Locale dependent and other special case conversions (see the
Unicode standard file SpecialCasing.txt) should be left to user land
routines and not go into the core interpreter.

The methods .capitalize() and .iscapitalized() should follow the case
mapping algorithm defined in the above technical report as closely as
possible.

Line Breaks

Line breaking should be done for all Unicode characters having the B
property as well as the combinations CRLF, CR, LF (interpreted in that
order) and other special line separators defined by the standard.

The Unicode type should provide a .splitlines() method which returns a
list of lines according to the above specification. See Unicode Methods.

Unicode Character Properties

A separate module "unicodedata" should provide a compact interface to
all Unicode character properties defined in the standard's
UnicodeData.txt file.

Among other things, these properties provide ways to recognize numbers,
digits, spaces, whitespace, etc.

Since this module will have to provide access to all Unicode characters,
it will eventually have to contain the data from UnicodeData.txt which
takes up around 600kB. For this reason, the data should be stored in
static C data. This enables compilation as shared module which the
underlying OS can shared between processes (unlike normal Python code
modules).

There should be a standard Python interface for accessing this
information so that other implementors can plug in their own possibly
enhanced versions, e.g. ones that do decompressing of the data
on-the-fly.

Private Code Point Areas

Support for these is left to user land Codecs and not explicitly
integrated into the core. Note that due to the Internal Format being
implemented, only the area between \uE000 and \uF8FF is usable for
private encodings.

Internal Format

The internal format for Unicode objects should use a Python specific
fixed format <PythonUnicode> implemented as 'unsigned short' (or another
unsigned numeric type having 16 bits). Byte order is platform dependent.

This format will hold UTF-16 encodings of the corresponding Unicode
ordinals. The Python Unicode implementation will address these values as
if they were UCS-2 values. UCS-2 and UTF-16 are the same for all
currently defined Unicode character points. UTF-16 without surrogates
provides access to about 64k characters and covers all characters in the
Basic Multilingual Plane (BMP) of Unicode.

It is the Codec's responsibility to ensure that the data they pass to
the Unicode object constructor respects this assumption. The constructor
does not check the data for Unicode compliance or use of surrogates.

Future implementations can extend the 32 bit restriction to the full set
of all UTF-16 addressable characters (around 1M characters).

The Unicode API should provide interface routines from <PythonUnicode>
to the compiler's wchar_t which can be 16 or 32 bit depending on the
compiler/libc/platform being used.

Unicode objects should have a pointer to a cached Python string object
<defenc> holding the object's value using the <default encoding>. This
is needed for performance and internal parsing (see Internal Argument
Parsing) reasons. The buffer is filled when the first conversion request
to the <default encoding> is issued on the object.

Interning is not needed (for now), since Python identifiers are defined
as being ASCII only.

codecs.BOM should return the byte order mark (BOM) for the format used
internally. The codecs module should provide the following additional
constants for convenience and reference (codecs.BOM will either be
BOM_BE or BOM_LE depending on the platform):

    BOM_BE: '\376\377'
      (corresponds to Unicode U+0000FEFF in UTF-16 on big endian
       platforms == ZERO WIDTH NO-BREAK SPACE)

    BOM_LE: '\377\376'
      (corresponds to Unicode U+0000FFFE in UTF-16 on little endian
       platforms == defined as being an illegal Unicode character)

    BOM4_BE: '\000\000\376\377'
      (corresponds to Unicode U+0000FEFF in UCS-4)

    BOM4_LE: '\377\376\000\000'
      (corresponds to Unicode U+0000FFFE in UCS-4)

Note that Unicode sees big endian byte order as being "correct". The
swapped order is taken to be an indicator for a "wrong" format, hence
the illegal character definition.

The configure script should provide aid in deciding whether Python can
use the native wchar_t type or not (it has to be a 16-bit unsigned
type).

Buffer Interface

Implement the buffer interface using the <defenc> Python string object
as basis for bf_getcharbuf and the internal buffer for bf_getreadbuf. If
bf_getcharbuf is requested and the <defenc> object does not yet exist,
it is created first.

Note that as special case, the parser marker "s#" will not return raw
Unicode UTF-16 data (which the bf_getreadbuf returns), but instead tries
to encode the Unicode object using the default encoding and then returns
a pointer to the resulting string object (or raises an exception in case
the conversion fails). This was done in order to prevent accidentally
writing binary data to an output stream which the other end might not
recognize.

This has the advantage of being able to write to output streams (which
typically use this interface) without additional specification of the
encoding to use.

If you need to access the read buffer interface of Unicode objects, use
the PyObject_AsReadBuffer() interface.

The internal format can also be accessed using the 'unicode-internal'
codec, e.g. via u.encode('unicode-internal').

Pickle/Marshalling

Should have native Unicode object support. The objects should be encoded
using platform independent encodings.

Marshal should use UTF-8 and Pickle should either choose
Raw-Unicode-Escape (in text mode) or UTF-8 (in binary mode) as encoding.
Using UTF-8 instead of UTF-16 has the advantage of eliminating the need
to store a BOM mark.

Regular Expressions

Secret Labs AB is working on a Unicode-aware regular expression
machinery. It works on plain 8-bit, UCS-2, and (optionally) UCS-4
internal character buffers.

Also see

  http://www.unicode.org/unicode/reports/tr18/

for some remarks on how to treat Unicode REs.

Formatting Markers

Format markers are used in Python format strings. If Python strings are
used as format strings, the following interpretations should be in
effect:

    '%s': For Unicode objects this will cause coercion of the
          whole format string to Unicode.  Note that you should use
          a Unicode format string to start with for performance
          reasons.

In case the format string is an Unicode object, all parameters are
coerced to Unicode first and then put together and formatted according
to the format string. Numbers are first converted to strings and then to
Unicode.

    '%s': Python strings are interpreted as Unicode
          string using the <default encoding>.  Unicode objects are
          taken as is.

All other string formatters should work accordingly.

Example:

    u"%s %s" % (u"abc", "abc")  ==  u"abc abc"

Internal Argument Parsing

These markers are used by the PyArg_ParseTuple() APIs:

"U"

    Check for Unicode object and return a pointer to it

"s"

    For Unicode objects: return a pointer to the object's <defenc>
    buffer (which uses the <default encoding>).

"s#"

    Access to the default encoded version of the Unicode object (see
    Buffer Interface); note that the length relates to the length of the
    default encoded string rather than the Unicode object length.

"t#"

    Same as "s#".

"es"

    Takes two parameters: encoding (const char *) and buffer (char **).

    The input object is first coerced to Unicode in the usual way and
    then encoded into a string using the given encoding.

    On output, a buffer of the needed size is allocated and returned
    through *buffer as NULL-terminated string. The encoded may not
    contain embedded NULL characters. The caller is responsible for
    calling PyMem_Free() to free the allocated *buffer after usage.

"es#"

    Takes three parameters: encoding (const char *), buffer (char **)
    and buffer_len (int *).

    The input object is first coerced to Unicode in the usual way and
    then encoded into a string using the given encoding.

    If *buffer is non-NULL, *buffer_len must be set to sizeof(buffer) on
    input. Output is then copied to *buffer.

    If *buffer is NULL, a buffer of the needed size is allocated and
    output copied into it. *buffer is then updated to point to the
    allocated memory area. The caller is responsible for calling
    PyMem_Free() to free the allocated *buffer after usage.

    In both cases *buffer_len is updated to the number of characters
    written (excluding the trailing NULL-byte). The output buffer is
    assured to be NULL-terminated.

Examples:

Using "es#" with auto-allocation:

    static PyObject *
    test_parser(PyObject *self,
                PyObject *args)
    {
        PyObject *str;
        const char *encoding = "latin-1";
        char *buffer = NULL;
        int buffer_len = 0;

        if (!PyArg_ParseTuple(args, "es#:test_parser",
                              encoding, &buffer, &buffer_len))
            return NULL;
        if (!buffer) {
            PyErr_SetString(PyExc_SystemError,
                            "buffer is NULL");
            return NULL;
        }
        str = PyString_FromStringAndSize(buffer, buffer_len);
        PyMem_Free(buffer);
        return str;
    }

Using "es" with auto-allocation returning a NULL-terminated string:

    static PyObject *
    test_parser(PyObject *self,
                PyObject *args)
    {
        PyObject *str;
        const char *encoding = "latin-1";
        char *buffer = NULL;

        if (!PyArg_ParseTuple(args, "es:test_parser",
                              encoding, &buffer))
            return NULL;
        if (!buffer) {
            PyErr_SetString(PyExc_SystemError,
                            "buffer is NULL");
            return NULL;
        }
        str = PyString_FromString(buffer);
        PyMem_Free(buffer);
        return str;
    }

Using "es#" with a pre-allocated buffer:

    static PyObject *
    test_parser(PyObject *self,
                PyObject *args)
    {
        PyObject *str;
        const char *encoding = "latin-1";
        char _buffer[10];
        char *buffer = _buffer;
        int buffer_len = sizeof(_buffer);

        if (!PyArg_ParseTuple(args, "es#:test_parser",
                              encoding, &buffer, &buffer_len))
            return NULL;
        if (!buffer) {
            PyErr_SetString(PyExc_SystemError,
                            "buffer is NULL");
            return NULL;
        }
        str = PyString_FromStringAndSize(buffer, buffer_len);
        return str;
    }

File/Stream Output

Since file.write(object) and most other stream writers use the "s#" or
"t#" argument parsing marker for querying the data to write, the default
encoded string version of the Unicode object will be written to the
streams (see Buffer Interface).

For explicit handling of files using Unicode, the standard stream codecs
as available through the codecs module should be used.

The codecs module should provide a short-cut
open(filename,mode,encoding) available which also assures that mode
contains the 'b' character when needed.

File/Stream Input

Only the user knows what encoding the input data uses, so no special
magic is applied. The user will have to explicitly convert the string
data to Unicode objects as needed or use the file wrappers defined in
the codecs module (see File/Stream Output).

Unicode Methods & Attributes

All Python string methods, plus:

    .encode([encoding=<default encoding>][,errors="strict"])
       --> see Unicode Output

    .splitlines([include_breaks=0])
       --> breaks the Unicode string into a list of (Unicode) lines;
           returns the lines with line breaks included, if
           include_breaks is true.  See Line Breaks for a
           specification of how line breaking is done.

Code Base

We should use Fredrik Lundh's Unicode object implementation as basis. It
already implements most of the string methods needed and provides a well
written code base which we can build upon.

The object sharing implemented in Fredrik's implementation should be
dropped.

Test Cases

Test cases should follow those in Lib/test/test_string.py and include
additional checks for the Codec Registry and the Standard Codecs.

References

-   Unicode Consortium: http://www.unicode.org/
-   Unicode FAQ: http://www.unicode.org/unicode/faq/
-   Unicode 3.0:
    http://www.unicode.org/unicode/standard/versions/Unicode3.0.html
-   Unicode-TechReports:
    http://www.unicode.org/unicode/reports/techreports.html
-   Unicode-Mappings: ftp://ftp.unicode.org/Public/MAPPINGS/
-   Introduction to Unicode (a little outdated by still nice to read):
    http://www.nada.kth.se/i18n/ucs/unicode-iso10646-oview.html
-   For comparison: Introducing Unicode to ECMAScript (aka JavaScript)
    --http://www-4.ibm.com/software/developer/library/internationalization-support.html
-   IANA Character Set Names:
    ftp://ftp.isi.edu/in-notes/iana/assignments/character-sets
-   Discussion of UTF-8 and Unicode support for POSIX and Linux:
    http://www.cl.cam.ac.uk/~mgk25/unicode.html
-   Encodings:
    -   Overview: http://czyborra.com/utf/
    -   UCS-2: http://www.uazone.org/multiling/unicode/ucs2.html
    -   UTF-7: Defined in 2152
    -   UTF-8: Defined in 2279
    -   UTF-16: http://www.uazone.org/multiling/unicode/wg2n1035.html

History of this Proposal

[ed. note: revisions prior to 1.7 are available in the CVS history of
Misc/unicode.txt from the standard Python distribution. All subsequent
history is available via the CVS revisions on this file.]

1.7

-   Added note about the changed behaviour of "s#".

1.6

-   Changed <defencstr> to <defenc> since this is the name used in the
    implementation.
-   Added notes about the usage of <defenc> in the buffer protocol
    implementation.

1.5

-   Added notes about setting the <default encoding>.
-   Fixed some typos (thanks to Andrew Kuchling).
-   Changed <defencstr> to <utf8str>.

1.4

-   Added note about mixed type comparisons and contains tests.
-   Changed treating of Unicode objects in format strings (if used with
    '%s' % u they will now cause the format string to be coerced to
    Unicode, thus producing a Unicode object on return).
-   Added link to IANA charset names (thanks to Lars Marius Garshol).
-   Added new codec methods .readline(), .readlines() and .writelines().

1.3

-   Added new "es" and "es#" parser markers

1.2

-   Removed POD about codecs.open()

1.1

-   Added note about comparisons and hash values.
-   Added note about case mapping algorithms.
-   Changed stream codecs .read() and .write() method to match the
    standard file-like object methods (bytes consumed information is no
    longer returned by the methods)

1.0

-   changed encode Codec method to be symmetric to the decode method
    (they both return (object, data consumed) now and thus become
    interchangeable);
-   removed __init__ method of Codec class (the methods are stateless)
    and moved the errors argument down to the methods;
-   made the Codec design more generic w/r to type of input and output
    objects;
-   changed StreamWriter.flush to StreamWriter.reset in order to avoid
    overriding the stream's .flush() method;
-   renamed .breaklines() to .splitlines();
-   renamed the module unicodec to codecs;
-   modified the File I/O section to refer to the stream codecs.

0.9

-   changed errors keyword argument definition;
-   added 'replace' error handling;
-   changed the codec APIs to accept buffer like objects on input;
-   some minor typo fixes;
-   added Whitespace section and included references for Unicode
    characters that have the whitespace and the line break
    characteristic;
-   added note that search functions can expect lower-case encoding
    names;
-   dropped slicing and offsets in the codec APIs

0.8

-   added encodings package and raw unicode escape encoding;
-   untabified the proposal;
-   added notes on Unicode format strings;
-   added .breaklines() method

0.7

-   added a whole new set of codec APIs;
-   added a different encoder lookup scheme;
-   fixed some names

0.6

-   changed "s#" to "t#";
-   changed <defencbuf> to <defencstr> holding a real Python string
    object;
-   changed Buffer Interface to delegate requests to <defencstr>'s
    buffer interface;
-   removed the explicit reference to the unicodec.codecs dictionary
    (the module can implement this in way fit for the purpose);
-   removed the settable default encoding;
-   move UnicodeError from unicodec to exceptions;
-   "s#" not returns the internal data;
-   passed the UCS-2/UTF-16 checking from the Unicode constructor to the
    Codecs

0.5

-   moved sys.bom to unicodec.BOM;
-   added sections on case mapping,
-   private use encodings and Unicode character properties

0.4

-   added Codec interface, notes on %-formatting,
-   changed some encoding details,
-   added comments on stream wrappers,
-   fixed some discussion points (most important: Internal Format),
-   clarified the 'unicode-escape' encoding, added encoding references

0.3

-   added references, comments on codec modules, the internal format,
    bf_getcharbuffer and the RE engine;
-   added 'unicode-escape' encoding proposed by Tim Peters and fixed
    repr(u) accordingly

0.2

-   integrated Guido's suggestions, added stream codecs and file
    wrapping

0.1

-   first version


text:
PEP: 3115 Title: Metaclasses in Python 3000 Author: Talin
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 07-Mar-2007 Python-Version: 3.0 Post-History:
11-Mar-2007, 14-Mar-2007

Abstract

This PEP proposes changing the syntax for declaring metaclasses, and
alters the semantics for how classes with metaclasses are constructed.

Rationale

There are two rationales for this PEP, both of which are somewhat
subtle.

The primary reason for changing the way metaclasses work, is that there
are a number of interesting use cases that require the metaclass to get
involved earlier in the class construction process than is currently
possible. Currently, the metaclass mechanism is essentially a
post-processing step. With the advent of class decorators, much of these
post-processing chores can be taken over by the decorator mechanism.

In particular, there is an important body of use cases where it would be
useful to preserve the order in which a class members are declared.
Ordinary Python objects store their members in a dictionary, in which
ordering is unimportant, and members are accessed strictly by name.
However, Python is often used to interface with external systems in
which the members are organized according to an implicit ordering.
Examples include declaration of C structs; COM objects; Automatic
translation of Python classes into IDL or database schemas, such as used
in an ORM; and so on.

In such cases, it would be useful for a Python programmer to specify
such ordering directly using the declaration order of class members.
Currently, such orderings must be specified explicitly, using some other
mechanism (see the ctypes module for an example.)

Unfortunately, the current method for declaring a metaclass does not
allow for this, since the ordering information has already been lost by
the time the metaclass comes into play. By allowing the metaclass to get
involved in the class construction process earlier, the new system
allows the ordering or other early artifacts of construction to be
preserved and examined.

There proposed metaclass mechanism also supports a number of other
interesting use cases beyond preserving the ordering of declarations.
One use case is to insert symbols into the namespace of the class body
which are only valid during class construction. An example of this might
be "field constructors", small functions that are used in the creation
of class members. Another interesting possibility is supporting forward
references, i.e. references to Python symbols that are declared further
down in the class body.

The other, weaker, rationale is purely cosmetic: The current method for
specifying a metaclass is by assignment to the special variable
__metaclass__, which is considered by some to be aesthetically less than
ideal. Others disagree strongly with that opinion. This PEP will not
address this issue, other than to note it, since aesthetic debates
cannot be resolved via logical proofs.

Specification

In the new model, the syntax for specifying a metaclass is via a keyword
argument in the list of base classes:

    class Foo(base1, base2, metaclass=mymeta):
        ...

Additional keywords will also be allowed here, and will be passed to the
metaclass, as in the following example:

    class Foo(base1, base2, metaclass=mymeta, private=True):
        ...

Note that this PEP makes no attempt to define what these other keywords
might be - that is up to metaclass implementors to determine.

More generally, the parameter list passed to a class definition will now
support all of the features of a function call, meaning that you can now
use *args and **kwargs-style arguments in the class base list:

    class Foo(*bases, **kwds):
        ...

Invoking the Metaclass

In the current metaclass system, the metaclass object can be any
callable type. This does not change, however in order to fully exploit
all of the new features the metaclass will need to have an extra
attribute which is used during class pre-construction.

This attribute is named __prepare__, which is invoked as a function
before the evaluation of the class body. The __prepare__ function takes
two positional arguments, and an arbitrary number of keyword arguments.
The two positional arguments are:

  ------- --------------------------------------
  name    the name of the class being created.
  bases   the list of base classes.
  ------- --------------------------------------

The interpreter always tests for the existence of __prepare__ before
calling it; If it is not present, then a regular dictionary is used, as
illustrated in the following Python snippet.

    def prepare_class(name, *bases, metaclass=None, **kwargs):
        if metaclass is None:
            metaclass = compute_default_metaclass(bases)
        prepare = getattr(metaclass, '__prepare__', None)
        if prepare is not None:
            return prepare(name, bases, **kwargs)
        else:
            return dict()

The example above illustrates how the arguments to 'class' are
interpreted. The class name is the first argument, followed by an
arbitrary length list of base classes. After the base classes, there may
be one or more keyword arguments, one of which can be metaclass. Note
that the metaclass argument is not included in kwargs, since it is
filtered out by the normal parameter assignment algorithm. (Note also
that metaclass is a keyword-only argument as per PEP 3102.)

Even though __prepare__ is not required, the default metaclass ('type')
implements it, for the convenience of subclasses calling it via super().

__prepare__ returns a dictionary-like object which is used to store the
class member definitions during evaluation of the class body. In other
words, the class body is evaluated as a function block (just like it is
now), except that the local variables dictionary is replaced by the
dictionary returned from __prepare__. This dictionary object can be a
regular dictionary or a custom mapping type.

This dictionary-like object is not required to support the full
dictionary interface. A dictionary which supports a limited set of
dictionary operations will restrict what kinds of actions can occur
during evaluation of the class body. A minimal implementation might only
support adding and retrieving values from the dictionary - most class
bodies will do no more than that during evaluation. For some classes, it
may be desirable to support deletion as well. Many metaclasses will need
to make a copy of this dictionary afterwards, so iteration or other
means for reading out the dictionary contents may also be useful.

The __prepare__ method will most often be implemented as a class method
rather than an instance method because it is called before the metaclass
instance (i.e. the class itself) is created.

Once the class body has finished evaluating, the metaclass will be
called (as a callable) with the class dictionary, which is no different
from the current metaclass mechanism.

Typically, a metaclass will create a custom dictionary - either a
subclass of dict, or a wrapper around it - that will contain additional
properties that are set either before or during the evaluation of the
class body. Then in the second phase, the metaclass can use these
additional properties to further customize the class.

An example would be a metaclass that uses information about the ordering
of member declarations to create a C struct. The metaclass would provide
a custom dictionary that simply keeps a record of the order of
insertions. This does not need to be a full 'ordered dict'
implementation, but rather just a Python list of (key,value) pairs that
is appended to for each insertion.

Note that in such a case, the metaclass would be required to deal with
the possibility of duplicate keys, but in most cases that is trivial.
The metaclass can use the first declaration, the last, combine them in
some fashion, or simply throw an exception. It's up to the metaclass to
decide how it wants to handle that case.

Example

Here's a simple example of a metaclass which creates a list of the names
of all class members, in the order that they were declared:

    # The custom dictionary
    class member_table(dict):
        def __init__(self):
            self.member_names = []

        def __setitem__(self, key, value):
            # if the key is not already defined, add to the
            # list of keys.
            if key not in self:
                self.member_names.append(key)

            # Call superclass
            dict.__setitem__(self, key, value)

    # The metaclass
    class OrderedClass(type):

        # The prepare function
        @classmethod
        def __prepare__(metacls, name, bases): # No keywords in this case
            return member_table()

        # The metaclass invocation
        def __new__(cls, name, bases, classdict):
            # Note that we replace the classdict with a regular
            # dict before passing it to the superclass, so that we
            # don't continue to record member names after the class
            # has been created.
            result = type.__new__(cls, name, bases, dict(classdict))
            result.member_names = classdict.member_names
            return result

    class MyClass(metaclass=OrderedClass):
        # method1 goes in array element 0
        def method1(self):
            pass

        # method2 goes in array element 1
        def method2(self):
            pass

Sample Implementation

Guido van Rossum has created a patch which implements the new
functionality: https://bugs.python.org/issue1681101

Alternate Proposals

Josiah Carlson proposed using the name 'type' instead of 'metaclass', on
the theory that what is really being specified is the type of the type.
While this is technically correct, it is also confusing from the point
of view of a programmer creating a new class. From the application
programmer's point of view, the 'type' that they are interested in is
the class that they are writing; the type of that type is the metaclass.

There were some objections in the discussion to the 'two-phase' creation
process, where the metaclass is invoked twice, once to create the class
dictionary and once to 'finish' the class. Some people felt that these
two phases should be completely separate, in that there ought to be
separate syntax for specifying the custom dict as for specifying the
metaclass. However, in most cases, the two will be intimately tied
together, and the metaclass will most likely have an intimate knowledge
of the internal details of the class dict. Requiring the programmer to
insure that the correct dict type and the correct metaclass type are
used together creates an additional and unneeded burden on the
programmer.

Another good suggestion was to simply use an ordered dict for all
classes, and skip the whole 'custom dict' mechanism. This was based on
the observation that most use cases for a custom dict were for the
purposes of preserving order information. However, this idea has several
drawbacks, first because it means that an ordered dict implementation
would have to be added to the set of built-in types in Python, and
second because it would impose a slight speed (and complexity) penalty
on all class declarations. Later, several people came up with ideas for
use cases for custom dictionaries other than preserving field orderings,
so this idea was dropped.

Backwards Compatibility

It would be possible to leave the existing __metaclass__ syntax in
place. Alternatively, it would not be too difficult to modify the syntax
rules of the Py3K translation tool to convert from the old to the new
syntax.

References

[1] [Python-3000] Metaclasses in Py3K (original proposal)
https://mail.python.org/pipermail/python-3000/2006-December/005030.html

[2] [Python-3000] Metaclasses in Py3K (Guido's suggested syntax)
https://mail.python.org/pipermail/python-3000/2006-December/005033.html

[3] [Python-3000] Metaclasses in Py3K (Objections to two-phase init)
https://mail.python.org/pipermail/python-3000/2006-December/005108.html

[4] [Python-3000] Metaclasses in Py3K (Always use an ordered dict)
https://mail.python.org/pipermail/python-3000/2006-December/005118.html

Copyright

This document has been placed in the public domain.

license: Public Domain


text:
PEP: 686 Title: Make UTF-8 mode default Author: Inada Naoki
<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/14737 Status: Accepted Type: Standards
Track Content-Type: text/x-rst Created: 18-Mar-2022 Python-Version: 3.15
Post-History: 18-Mar-2022, 31-Mar-2022 Resolution:
https://discuss.python.org/t/14737/9

Abstract

This PEP proposes enabling UTF-8 mode <540> by default.

With this change, Python consistently uses UTF-8 for default encoding of
files, stdio, and pipes.

Motivation

UTF-8 becomes de facto standard text encoding.

-   The default encoding of Python source files is UTF-8.
-   JSON, TOML, YAML use UTF-8.
-   Most text editors, including Visual Studio Code and Windows Notepad
    use UTF-8 by default.
-   Most websites and text data on the internet use UTF-8.
-   And many other popular programming languages, including Node.js, Go,
    Rust, and Java uses UTF-8 by default.

Changing the default encoding to UTF-8 makes it easier for Python to
interoperate with them.

Additionally, many Python developers using Unix forget that the default
encoding is platform dependent. They omit to specify encoding="utf-8"
when they read text files encoded in UTF-8 (e.g. JSON, TOML, Markdown,
and Python source files). Inconsistent default encoding causes many
bugs.

Specification

Enable UTF-8 mode by default

Python will enable UTF-8 mode by default from Python 3.15.

Users can still disable UTF-8 mode by setting PYTHONUTF8=0 or -X utf8=0.

locale.getencoding()

Since UTF-8 mode affects locale.getpreferredencoding(False), we need an
API to get locale encoding regardless of UTF-8 mode.

locale.getencoding() will be added for this purpose. It returns locale
encoding too, but ignores UTF-8 mode.

When warn_default_encoding option is specified,
locale.getpreferredencoding() will emit EncodingWarning like open() (see
also PEP 597).

This API was added in Python 3.11.

Fixing encoding="locale" option

PEP 597 added the encoding="locale" option to the TextIOWrapper. This
option is used to specify the locale encoding explicitly. TextIOWrapper
should use locale encoding when the option is specified, regardless of
default text encoding.

But TextIOWrapper uses "UTF-8" in UTF-8 mode even if encoding="locale"
is specified for now. This behavior is inconsistent with the PEP 597
motivation. It is because we didn't expect making UTF-8 mode default
when Python changes its default text encoding.

This inconsistency should be fixed before making UTF-8 mode default.
TextIOWrapper should use locale encoding when encoding="locale" is
passed even in UTF-8 mode.

This issue was fixed in Python 3.11.

Backward Compatibility

Most Unix systems use UTF-8 locale and Python enables UTF-8 mode when
its locale is C or POSIX. So this change mostly affects Windows users.

When a Python program depends on the default encoding, this change may
cause UnicodeError, mojibake, or even silent data corruption. So this
change should be announced loudly.

This is the guideline to fix this backward compatibility issue:

1.  Disable UTF-8 mode.
2.  Use EncodingWarning (PEP 597) to find every places UTF-8 mode
    affects.
    -   If encoding option is omitted, consider using encoding="utf-8"
        or encoding="locale".
    -   If locale.getpreferredencoding() is used, consider using "utf-8"
        or locale.getencoding().
3.  Test the application with UTF-8 mode.

Preceding examples

-   Ruby changed the default external_encoding to UTF-8 on Windows in
    Ruby 3.0 (2020).
-   Java changed the default text encoding to UTF-8 in JDK 18. (2022).

Both Ruby and Java have an option for backward compatibility. They don't
provide any warning like PEP 597's EncodingWarning in Python for use of
the default encoding.

Rejected Alternative

Deprecate implicit encoding

Deprecating the use of the default encoding is considered.

But there are many cases that the default encoding is used for
reading/writing only ASCII text. Additionally, such warnings are not
useful for non-cross platform applications run on Unix.

So forcing users to specify the encoding everywhere is too painful.
Emitting a lot of DeprecationWarning will lead users ignore warnings.

PEP 387 requires adding a warning for backward incompatible changes. But
it doesn't require using DeprecationWarning. So using optional
EncodingWarning doesn't violate the PEP 387.

Java also rejected this idea in JEP 400.

Use PYTHONIOENCODING for PIPEs

To ease backward compatibility issue, using PYTHONIOENCODING as the
default encoding of PIPEs in the subprocess module is considered.

With this idea, users can use legacy encoding for
subprocess.Popen(text=True) even in UTF-8 mode.

But this idea makes "default encoding" complicated. And this idea is
also backward incompatible.

So this idea is rejected. Users can disable UTF-8 mode until they
replace text=True with encoding="utf-8" or encoding="locale".

How to teach this

For new users, this change reduces things that need to teach. Users
don't need to learn about text encoding in their first year. They should
learn it when they need to use non-UTF-8 text files.

For existing users, see the Backward compatibility section.

References

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 3103 Title: A Switch/Case Statement Author: Guido van Rossum
<EMAIL_ADDRESS>Status: Rejected Type: Standards Track Content-Type:
text/x-rst Created: 25-Jun-2006 Python-Version: 3.0 Post-History:
26-Jun-2006

Rejection Notice

A quick poll during my keynote presentation at PyCon 2007 shows this
proposal has no popular support. I therefore reject it.

Abstract

Python-dev has recently seen a flurry of discussion on adding a switch
statement. In this PEP I'm trying to extract my own preferences from the
smorgasbord of proposals, discussing alternatives and explaining my
choices where I can. I'll also indicate how strongly I feel about
alternatives I discuss.

This PEP should be seen as an alternative to PEP 275. My views are
somewhat different from that PEP's author, but I'm grateful for the work
done in that PEP.

This PEP introduces canonical names for the many variants that have been
discussed for different aspects of the syntax and semantics, such as
"alternative 1", "school II", "option 3" and so on. Hopefully these
names will help the discussion.

Rationale

A common programming idiom is to consider an expression and do different
things depending on its value. This is usually done with a chain of
if/elif tests; I'll refer to this form as the "if/elif chain". There are
two main motivations to want to introduce new syntax for this idiom:

-   It is repetitive: the variable and the test operator, usually '=='
    or 'in', are repeated in each if/elif branch.
-   It is inefficient: when an expression matches the last test value
    (or no test value at all) it is compared to each of the preceding
    test values.

Both of these complaints are relatively mild; there isn't a lot of
readability or performance to be gained by writing this differently.
Yet, some kind of switch statement is found in many languages and it is
not unreasonable to expect that its addition to Python will allow us to
write up certain code more cleanly and efficiently than before.

There are forms of dispatch that are not suitable for the proposed
switch statement; for example, when the number of cases is not
statically known, or when it is desirable to place the code for
different cases in different classes or files.

Basic Syntax

I'm considering several variants of the syntax first proposed in PEP 275
here. There are lots of other possibilities, but I don't see that they
add anything.

I've recently been converted to alternative 1.

I should note that all alternatives here have the "implicit break"
property: at the end of the suite for a particular case, the control
flow jumps to the end of the whole switch statement. There is no way to
pass control from one case to another. This in contrast to C, where an
explicit 'break' statement is required to prevent falling through to the
next case.

In all alternatives, the else-suite is optional. It is more Pythonic to
use 'else' here rather than introducing a new reserved word, 'default',
as in C.

Semantics are discussed in the next top-level section.

Alternative 1

This is the preferred form in PEP 275:

    switch EXPR:
        case EXPR:
            SUITE
        case EXPR:
            SUITE
        ...
        else:
            SUITE

The main downside is that the suites where all the action is are
indented two levels deep; this can be remedied by indenting the cases
"half a level" (e.g. 2 spaces if the general indentation level is 4).

Alternative 2

This is Fredrik Lundh's preferred form; it differs by not indenting the
cases:

    switch EXPR:
    case EXPR:
        SUITE
    case EXPR:
        SUITE
    ....
    else:
        SUITE

Some reasons not to choose this include expected difficulties for
auto-indenting editors, folding editors, and the like; and confused
users. There are no situations currently in Python where a line ending
in a colon is followed by an unindented line.

Alternative 3

This is the same as alternative 2 but leaves out the colon after the
switch:

    switch EXPR
    case EXPR:
        SUITE
    case EXPR:
        SUITE
    ....
    else:
        SUITE

The hope of this alternative is that it will not upset the auto-indent
logic of the average Python-aware text editor less. But it looks strange
to me.

Alternative 4

This leaves out the 'case' keyword on the basis that it is redundant:

    switch EXPR:
        EXPR:
            SUITE
        EXPR:
            SUITE
        ...
        else:
            SUITE

Unfortunately now we are forced to indent the case expressions, because
otherwise (at least in the absence of an 'else' keyword) the parser
would have a hard time distinguishing between an unindented case
expression (which continues the switch statement) or an unrelated
statement that starts like an expression (such as an assignment or a
procedure call). The parser is not smart enough to backtrack once it
sees the colon. This is my least favorite alternative.

Extended Syntax

There is one additional concern that needs to be addressed
syntactically. Often two or more values need to be treated the same. In
C, this done by writing multiple case labels together without any code
between them. The "fall through" semantics then mean that these are all
handled by the same code. Since the Python switch will not have
fall-through semantics (which have yet to find a champion) we need
another solution. Here are some alternatives.

Alternative A

Use:

    case EXPR:

to match on a single expression; use:

    case EXPR, EXPR, ...:

to match on multiple expressions. The is interpreted so that if EXPR is
a parenthesized tuple or another expression whose value is a tuple, the
switch expression must equal that tuple, not one of its elements. This
means that we cannot use a variable to indicate multiple cases. While
this is also true in C's switch statement, it is a relatively common
occurrence in Python (see for example sre_compile.py).

Alternative B

Use:

    case EXPR:

to match on a single expression; use:

    case in EXPR_LIST:

to match on multiple expressions. If EXPR_LIST is a single expression,
the 'in' forces its interpretation as an iterable (or something
supporting __contains__, in a minority semantics alternative). If it is
multiple expressions, each of those is considered for a match.

Alternative C

Use:

    case EXPR:

to match on a single expression; use:

    case EXPR, EXPR, ...:

to match on multiple expressions (as in alternative A); and use:

    case *EXPR:

to match on the elements of an expression whose value is an iterable.
The latter two cases can be combined, so that the true syntax is more
like this:

    case [*]EXPR, [*]EXPR, ...:

The * notation is similar to the use of prefix * already in use for
variable-length parameter lists and for passing computed argument lists,
and often proposed for value-unpacking (e.g. a, b, *c = X as an
alternative to (a, b), c = X[:2], X[2:]).

Alternative D

This is a mixture of alternatives B and C; the syntax is like
alternative B but instead of the 'in' keyword it uses '*'. This is more
limited, but still allows the same flexibility. It uses:

    case EXPR:

to match on a single expression and:

    case *EXPR:

to match on the elements of an iterable. If one wants to specify
multiple matches in one case, one can write this:

    case *(EXPR, EXPR, ...):

or perhaps this (although it's a bit strange because the relative
priority of '*' and ',' is different than elsewhere):

    case * EXPR, EXPR, ...:

Discussion

Alternatives B, C and D are motivated by the desire to specify multiple
cases with the same treatment using a variable representing a set
(usually a tuple) rather than spelling them out. The motivation for this
is usually that if one has several switches over the same set of cases
it's a shame to have to spell out all the alternatives each time. An
additional motivation is to be able to specify ranges to be matched
easily and efficiently, similar to Pascal's "1..1000:" notation. At the
same time we want to prevent the kind of mistake that is common in
exception handling (and which will be addressed in Python 3000 by
changing the syntax of the except clause): writing "case 1, 2:" where
"case (1, 2):" was meant, or vice versa.

The case could be made that the need is insufficient for the added
complexity; C doesn't have a way to express ranges either, and it's used
a lot more than Pascal these days. Also, if a dispatch method based on
dict lookup is chosen as the semantics, large ranges could be
inefficient (consider range(1, sys.maxint)).

All in all my preferences are (from most to least favorite) B, A, D', C,
where D' is D without the third possibility.

Semantics

There are several issues to review before we can choose the right
semantics.

If/Elif Chain vs. Dict-based Dispatch

There are several main schools of thought about the switch statement's
semantics:

-   School I wants to define the switch statement in term of an
    equivalent if/elif chain (possibly with some optimization thrown
    in).
-   School II prefers to think of it as a dispatch on a precomputed
    dict. There are different choices for when the precomputation
    happens.
-   There's also school III, which agrees with school I that the
    definition of a switch statement should be in terms of an equivalent
    if/elif chain, but concedes to the optimization camp that all
    expressions involved must be hashable.

We need to further separate school I into school Ia and school Ib:

-   School Ia has a simple position: a switch statement is translated to
    an equivalent if/elif chain, and that's that. It should not be
    linked to optimization at all. That is also my main objection
    against this school: without any hint of optimization, the switch
    statement isn't attractive enough to warrant new syntax.
-   School Ib has a more complex position: it agrees with school II that
    optimization is important, and is willing to concede the compiler
    certain liberties to allow this. (For example, PEP 275 Solution 1.)
    In particular, hash() of the switch and case expressions may or may
    not be called (so it should be side-effect-free); and the case
    expressions may not be evaluated each time as expected by the
    if/elif chain behavior, so the case expressions should also be
    side-effect free. My objection to this (elaborated below) is that if
    either the hash() or the case expressions aren't side-effect-free,
    optimized and unoptimized code may behave differently.

School II grew out of the realization that optimization of commonly
found cases isn't so easy, and that it's better to face this head on.
This will become clear below.

The differences between school I (mostly school Ib) and school II are
threefold:

-   When optimizing using a dispatch dict, if either the switch
    expression or the case expressions are unhashable (in which case
    hash() raises an exception), school Ib requires catching the hash()
    failure and falling back to an if/elif chain. School II simply lets
    the exception happen. The problem with catching an exception in
    hash() as required by school Ib, is that this may hide a genuine
    bug. A possible way out is to only use a dispatch dict if all case
    expressions are ints, strings or other built-ins with known good
    hash behavior, and to only attempt to hash the switch expression if
    it is also one of those types. Type objects should probably also be
    supported here. This is the (only) problem that school III
    addresses.
-   When optimizing using a dispatch dict, if the hash() function of any
    expression involved returns an incorrect value, under school Ib,
    optimized code will not behave the same as unoptimized code. This is
    a well-known problem with optimization-related bugs, and waste lots
    of developer time. Under school II, in this situation incorrect
    results are produced at least consistently, which should make
    debugging a bit easier. The way out proposed for the previous bullet
    would also help here.
-   School Ib doesn't have a good optimization strategy if the case
    expressions are named constants. The compiler cannot know their
    values for sure, and it cannot know whether they are truly constant.
    As a way out, it has been proposed to re-evaluate the expression
    corresponding to the case once the dict has identified which case
    should be taken, to verify that the value of the expression didn't
    change. But strictly speaking, all the case expressions occurring
    before that case would also have to be checked, in order to preserve
    the true if/elif chain semantics, thereby completely killing the
    optimization. Another proposed solution is to have callbacks
    notifying the dispatch dict of changes in the value of variables or
    attributes involved in the case expressions. But this is not likely
    implementable in the general case, and would require many namespaces
    to bear the burden of supporting such callbacks, which currently
    don't exist at all.
-   Finally, there's a difference of opinion regarding the treatment of
    duplicate cases (i.e. two or more cases with match expressions that
    evaluates to the same value). School I wants to treat this the same
    is an if/elif chain would treat it (i.e. the first match wins and
    the code for the second match is silently unreachable); school II
    wants this to be an error at the time the dispatch dict is frozen
    (so dead code doesn't go undiagnosed).

School I sees trouble in school II's approach of pre-freezing a dispatch
dict because it places a new and unusual burden on programmers to
understand exactly what kinds of case values are allowed to be frozen
and when the case values will be frozen, or they might be surprised by
the switch statement's behavior.

School II doesn't believe that school Ia's unoptimized switch is worth
the effort, and it sees trouble in school Ib's proposal for
optimization, which can cause optimized and unoptimized code to behave
differently.

In addition, school II sees little value in allowing cases involving
unhashable values; after all if the user expects such values, they can
just as easily write an if/elif chain. School II also doesn't believe
that it's right to allow dead code due to overlapping cases to occur
unflagged, when the dict-based dispatch implementation makes it so easy
to trap this.

However, there are some use cases for overlapping/duplicate cases.
Suppose you're switching on some OS-specific constants (e.g. exported by
the os module or some module like that). You have a case for each. But
on some OS, two different constants have the same value (since on that
OS they are implemented the same way -- like O_TEXT and O_BINARY on
Unix). If duplicate cases are flagged as errors, your switch wouldn't
work at all on that OS. It would be much better if you could arrange the
cases so that one case has preference over another.

There's also the (more likely) use case where you have a set of cases to
be treated the same, but one member of the set must be treated
differently. It would be convenient to put the exception in an earlier
case and be done with it.

(Yes, it seems a shame not to be able to diagnose dead code due to
accidental case duplication. Maybe that's less important, and pychecker
can deal with it? After all we don't diagnose duplicate method
definitions either.)

This suggests school IIb: like school II but redundant cases must be
resolved by choosing the first match. This is trivial to implement when
building the dispatch dict (skip keys already present).

(An alternative would be to introduce new syntax to indicate "okay to
have overlapping cases" or "ok if this case is dead code" but I find
that overkill.)

Personally, I'm in school II: I believe that the dict-based dispatch is
the one true implementation for switch statements and that we should
face the limitations up front, so that we can reap maximal benefits. I'm
leaning towards school IIb -- duplicate cases should be resolved by the
ordering of the cases instead of flagged as errors.

When to Freeze the Dispatch Dict

For the supporters of school II (dict-based dispatch), the next big
dividing issue is when to create the dict used for switching. I call
this "freezing the dict".

The main problem that makes this interesting is the observation that
Python doesn't have named compile-time constants. What is conceptually a
constant, such as re.IGNORECASE, is a variable to the compiler, and
there's nothing to stop crooked code from modifying its value.

Option 1

The most limiting option is to freeze the dict in the compiler. This
would require that the case expressions are all literals or compile-time
expressions involving only literals and operators whose semantics are
known to the compiler, since with the current state of Python's dynamic
semantics and single-module compilation, there is no hope for the
compiler to know with sufficient certainty the values of any variables
occurring in such expressions. This is widely though not universally
considered too restrictive.

Raymond Hettinger is the main advocate of this approach. He proposes a
syntax where only a single literal of certain types is allowed as the
case expression. It has the advantage of being unambiguous and easy to
implement.

My main complaint about this is that by disallowing "named constants" we
force programmers to give up good habits. Named constants are introduced
in most languages to solve the problem of "magic numbers" occurring in
the source code. For example, sys.maxint is a lot more readable than
2147483647. Raymond proposes to use string literals instead of named
"enums", observing that the string literal's content can be the name
that the constant would otherwise have. Thus, we could write "case
'IGNORECASE':" instead of "case re.IGNORECASE:". However, if there is a
spelling error in the string literal, the case will silently be ignored,
and who knows when the bug is detected. If there is a spelling error in
a NAME, however, the error will be caught as soon as it is evaluated.
Also, sometimes the constants are externally defined (e.g. when parsing
a file format like JPEG) and we can't easily choose appropriate string
values. Using an explicit mapping dict sounds like a poor hack.

Option 2

The oldest proposal to deal with this is to freeze the dispatch dict the
first time the switch is executed. At this point we can assume that all
the named "constants" (constant in the programmer's mind, though not to
the compiler) used as case expressions are defined --otherwise an
if/elif chain would have little chance of success either. Assuming the
switch will be executed many times, doing some extra work the first time
pays back quickly by very quick dispatch times later.

An objection to this option is that there is no obvious object where the
dispatch dict can be stored. It can't be stored on the code object,
which is supposed to be immutable; it can't be stored on the function
object, since many function objects may be created for the same function
(e.g. for nested functions). In practice, I'm sure that something can be
found; it could be stored in a section of the code object that's not
considered when comparing two code objects or when pickling or
marshalling a code object; or all switches could be stored in a dict
indexed by weak references to code objects. The solution should also be
careful not to leak switch dicts between multiple interpreters.

Another objection is that the first-use rule allows obfuscated code like
this:

    def foo(x, y):
        switch x:
        case y:
            print 42

To the untrained eye (not familiar with Python) this code would be
equivalent to this:

    def foo(x, y):
        if x == y:
            print 42

but that's not what it does (unless it is always called with the same
value as the second argument). This has been addressed by suggesting
that the case expressions should not be allowed to reference local
variables, but this is somewhat arbitrary.

A final objection is that in a multi-threaded application, the first-use
rule requires intricate locking in order to guarantee the correct
semantics. (The first-use rule suggests a promise that side effects of
case expressions are incurred exactly once.) This may be as tricky as
the import lock has proved to be, since the lock has to be held while
all the case expressions are being evaluated.

Option 3

A proposal that has been winning support (including mine) is to freeze a
switch's dict when the innermost function containing it is defined. The
switch dict is stored on the function object, just as parameter defaults
are, and in fact the case expressions are evaluated at the same time and
in the same scope as the parameter defaults (i.e. in the scope
containing the function definition).

This option has the advantage of avoiding many of the finesses needed to
make option 2 work: there's no need for locking, no worry about
immutable code objects or multiple interpreters. It also provides a
clear explanation for why locals can't be referenced in case
expressions.

This option works just as well for situations where one would typically
use a switch; case expressions involving imported or global named
constants work exactly the same way as in option 2, as long as they are
imported or defined before the function definition is encountered.

A downside however is that the dispatch dict for a switch inside a
nested function must be recomputed each time the nested function is
defined. For certain "functional" styles of programming this may make
switch unattractive in nested functions. (Unless all case expressions
are compile-time constants; then the compiler is of course free to
optimize away the switch freezing code and make the dispatch table part
of the code object.)

Another downside is that under this option, there's no clear moment when
the dispatch dict is frozen for a switch that doesn't occur inside a
function. There are a few pragmatic choices for how to treat a switch
outside a function:

(a) Disallow it.
(b) Translate it into an if/elif chain.
(c) Allow only compile-time constant expressions.
(d) Compute the dispatch dict each time the switch is reached.
(e) Like (b) but tests that all expressions evaluated are hashable.

Of these, (a) seems too restrictive: it's uniformly worse than (c); and
(d) has poor performance for little or no benefits compared to (b). It
doesn't make sense to have a performance-critical inner loop at the
module level, as all local variable references are slow there; hence (b)
is my (weak) favorite. Perhaps I should favor (e), which attempts to
prevent atypical use of a switch; examples that work interactively but
not in a function are annoying. In the end I don't think this issue is
all that important (except it must be resolved somehow) and am willing
to leave it up to whoever ends up implementing it.

When a switch occurs in a class but not in a function, we can freeze the
dispatch dict at the same time the temporary function object
representing the class body is created. This means the case expressions
can reference module globals but not class variables. Alternatively, if
we choose (b) above, we could choose this implementation inside a class
definition as well.

Option 4

There are a number of proposals to add a construct to the language that
makes the concept of a value pre-computed at function definition time
generally available, without tying it either to parameter default values
or case expressions. Some keywords proposed include 'const', 'static',
'only' or 'cached'. The associated syntax and semantics vary.

These proposals are out of scope for this PEP, except to suggest that if
such a proposal is accepted, there are two ways for the switch to
benefit: we could require case expressions to be either compile-time
constants or pre-computed values; or we could make pre-computed values
the default (and only) evaluation mode for case expressions. The latter
would be my preference, since I don't see a use for more dynamic case
expressions that isn't addressed adequately by writing an explicit
if/elif chain.

Conclusion

It is too early to decide. I'd like to see at least one completed
proposal for pre-computed values before deciding. In the meantime,
Python is fine without a switch statement, and perhaps those who claim
it would be a mistake to add one are right.

Copyright

This document has been placed in the public domain.


text:
PEP: 396 Title: Module Version Numbers Version: $Revision: 65628 $
Last-Modified: $Date: 2008-08-10 09:59:20 -0400 (Sun, 10 Aug 2008) $
Author: Barry Warsaw<EMAIL_ADDRESS>Status: Rejected Type:
Informational Topic: Packaging Content-Type: text/x-rst Created:
16-Mar-2011 Post-History: 05-Apr-2011

Abstract

Given that it is useful and common to specify version numbers for Python
modules, and given that different ways of doing this have grown
organically within the Python community, it is useful to establish
standard conventions for module authors to adhere to and reference. This
informational PEP describes best practices for Python module authors who
want to define the version number of their Python module.

Conformance with this PEP is optional, however other Python tools (such
as distutils2[1]) may be adapted to use the conventions defined here.

PEP Rejection

This PEP was formally rejected on 2021-04-14. The packaging ecosystem
has changed significantly in the intervening years since this PEP was
first written, and APIs such as importlib.metadata.version()[2] provide
for a much better experience.

User Stories

Alice is writing a new module, called alice, which she wants to share
with other Python developers. alice is a simple module and lives in one
file, alice.py. Alice wants to specify a version number so that her
users can tell which version they are using. Because her module lives
entirely in one file, she wants to add the version number to that file.

Bob has written a module called bob which he has shared with many users.
bob.py contains a version number for the convenience of his users. Bob
learns about the Cheeseshop[3], and adds some simple packaging using
classic distutils so that he can upload The Bob Bundle to the
Cheeseshop. Because bob.py already specifies a version number which his
users can access programmatically, he wants the same API to continue to
work even though his users now get it from the Cheeseshop.

Carol maintains several namespace packages, each of which are
independently developed and distributed. In order for her users to
properly specify dependencies on the right versions of her packages, she
specifies the version numbers in the namespace package's setup.py file.
Because Carol wants to have to update one version number per package,
she specifies the version number in her module and has the setup.py
extract the module version number when she builds the sdist archive.

David maintains a package in the standard library, and also produces
standalone versions for other versions of Python. The standard library
copy defines the version number in the module, and this same version
number is used for the standalone distributions as well.

Rationale

Python modules, both in the standard library and available from third
parties, have long included version numbers. There are established de
facto standards for describing version numbers, and many ad-hoc ways
have grown organically over the years. Often, version numbers can be
retrieved from a module programmatically, by importing the module and
inspecting an attribute. Classic Python distutils setup() functions[4]
describe a version argument where the release's version number can be
specified. PEP 8 describes the use of a module attribute called
__version__ for recording "Subversion, CVS, or RCS" version strings
using keyword expansion. In the PEP author's own email archives, the
earliest example of the use of an __version__ module attribute by
independent module developers dates back to 1995.

Another example of version information is the sqlite3[5] module with its
sqlite_version_info, version, and version_info attributes. It may not be
immediately obvious which attribute contains a version number for the
module, and which contains a version number for the underlying SQLite3
library.

This informational PEP codifies established practice, and recommends
standard ways of describing module version numbers, along with some use
cases for when -- and when not -- to include them. Its adoption by
module authors is purely voluntary; packaging tools in the standard
library will provide optional support for the standards defined herein,
and other tools in the Python universe may comply as well.

Specification

1.  In general, modules in the standard library SHOULD NOT have version
    numbers. They implicitly carry the version number of the Python
    release they are included in.
2.  On a case-by-case basis, standard library modules which are also
    released in standalone form for other Python versions MAY include a
    module version number when included in the standard library, and
    SHOULD include a version number when packaged separately.
3.  When a module (or package) includes a version number, the version
    SHOULD be available in the __version__ attribute.
4.  For modules which live inside a namespace package, the module SHOULD
    include the __version__ attribute. The namespace package itself
    SHOULD NOT include its own __version__ attribute.
5.  The __version__ attribute's value SHOULD be a string.
6.  Module version numbers SHOULD conform to the normalized version
    format specified in PEP 386.
7.  Module version numbers SHOULD NOT contain version control system
    supplied revision numbers, or any other semantically different
    version numbers (e.g. underlying library version number).
8.  The version attribute in a classic distutils setup.py file, or the
    PEP 345 Version metadata field SHOULD be derived from the
    __version__ field, or vice versa.

Examples

Retrieving the version number from a third party package:

    >>> import bzrlib
    >>> bzrlib.__version__
    '2.3.0'

Retrieving the version number from a standard library package that is
also distributed as a standalone module:

    >>> import email
    >>> email.__version__
    '5.1.0'

Version numbers for namespace packages:

    >>> import flufl.i18n
    >>> import flufl.enum
    >>> import flufl.lock

    >>> print flufl.i18n.__version__
    1.0.4
    >>> print flufl.enum.__version__
    3.1
    >>> print flufl.lock.__version__
    2.1

    >>> import flufl
    >>> flufl.__version__
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    AttributeError: 'module' object has no attribute '__version__'
    >>>

Deriving

Module version numbers can appear in at least two places, and sometimes
more. For example, in accordance with this PEP, they are available
programmatically on the module's __version__ attribute. In a classic
distutils setup.py file, the setup() function takes a version argument,
while the distutils2 setup.cfg file has a version key. The version
number must also get into the PEP 345 metadata, preferably when the
sdist archive is built. It's desirable for module authors to only have
to specify the version number once, and have all the other uses derive
from this single definition.

This could be done in any number of ways, a few of which are outlined
below. These are included for illustrative purposes only and are not
intended to be definitive, complete, or all-encompassing. Other
approaches are possible, and some included below may have limitations
that prevent their use in some situations.

Let's say Elle adds this attribute to her module file elle.py:

    __version__ = '3.1.1'

Classic distutils

In classic distutils, the simplest way to add the version string to the
setup() function in setup.py is to do something like this:

    from elle import __version__
    setup(name='elle', version=__version__)

In the PEP author's experience however, this can fail in some cases,
such as when the module uses automatic Python 3 conversion via the 2to3
program (because setup.py is executed by Python 3 before the elle module
has been converted).

In that case, it's not much more difficult to write a little code to
parse the __version__ from the file rather than importing it. Without
providing too much detail, it's likely that modules such as distutils2
will provide a way to parse version strings from files. E.g.:

    from distutils2 import get_version
    setup(name='elle', version=get_version('elle.py'))

Distutils2

Because the distutils2 style setup.cfg is declarative, we can't run any
code to extract the __version__ attribute, either via import or via
parsing.

In consultation with the distutils-sig[6], two options are proposed.
Both entail containing the version number in a file, and declaring that
file in the setup.cfg. When the entire contents of the file contains the
version number, the version-file key will be used:

    [metadata]
    version-file: version.txt

When the version number is contained within a larger file, e.g. of
Python code, such that the file must be parsed to extract the version,
the key version-from-file will be used:

    [metadata]
    version-from-file: elle.py

A parsing method similar to that described above will be performed on
the file named after the colon. The exact recipe for doing this will be
discussed in the appropriate distutils2 development forum.

An alternative is to only define the version number in setup.cfg and use
the pkgutil module[7] to make it available programmatically. E.g. in
elle.py:

    from distutils2._backport import pkgutil
    __version__ = pkgutil.get_distribution('elle').metadata['version']

PEP 376 metadata

PEP 376 defines a standard for static metadata, but doesn't describe the
process by which this metadata gets created. It is highly desirable for
the derived version information to be placed into the PEP 376 .dist-info
metadata at build-time rather than install-time. This way, the metadata
will be available for introspection even when the code is not installed.

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] Distutils2 documentation (http://distutils2.notmyidea.org/)

[2] importlib.metadata
(https://docs.python.org/3/library/importlib.metadata.html#distribution-versions)

[3] The Cheeseshop (Python Package Index) (http://pypi.python.org)

[4] http://docs.python.org/distutils/setupscript.html

[5] sqlite3 module documentation
(http://docs.python.org/library/sqlite3.html)

[6] https://mail.python.org/pipermail/distutils-sig/2011-June/017862.html

[7] pkgutil - Package utilities
(http://distutils2.notmyidea.org/library/pkgutil.html)