text:
PEP: 538 Title: Coercing the legacy C locale to a UTF-8 based locale
Version: $Revision$ Last-Modified: $Date$ Author: Alyssa Coghlan
<EMAIL_ADDRESS>BDFL-Delegate: INADA Naoki Status: Final Type:
Standards Track Content-Type: text/x-rst Created: 28-Dec-2016
Python-Version: 3.7 Post-History: 03-Jan-2017, 07-Jan-2017, 05-Mar-2017,
09-May-2017 Resolution:
https://mail.python.org/pipermail/python-dev/2017-May/148035.html

Abstract

An ongoing challenge with Python 3 on *nix systems is the conflict
between needing to use the configured locale encoding by default for
consistency with other locale-aware components in the same process or
subprocesses, and the fact that the standard C locale (as defined in
POSIX:2001) typically implies a default text encoding of ASCII, which is
entirely inadequate for the development of networked services and client
applications in a multilingual world.

PEP 540 proposes a change to CPython's handling of the legacy C locale
such that CPython will assume the use of UTF-8 in such environments,
rather than persisting with the demonstrably problematic assumption of
ASCII as an appropriate encoding for communicating with operating system
interfaces. This is a good approach for cases where network encoding
interoperability is a more important concern than local encoding
interoperability.

However, it comes at the cost of making CPython's encoding assumptions
diverge from those of other locale-aware components in the same process,
as well as those of components running in subprocesses that share the
same environment.

This can cause interoperability problems with some extension modules
(such as GNU readline's command line history editing), as well as with
components running in subprocesses (such as older Python runtimes).

It also requires non-trivial changes to the internals of how CPython
itself works, rather than relying primarily on existing configuration
settings that are supported by Python versions prior to Python 3.7.

Accordingly, this PEP proposes that independently of the UTF-8 mode
proposed in PEP 540, the way the CPython implementation handles the
default C locale be changed to be roughly equivalent to the following
existing configuration settings (supported since Python 3.1):

    LC_CTYPE=C.UTF-8
    PYTHONIOENCODING=utf-8:surrogateescape

The exact target locale for coercion will be chosen from a predefined
list at runtime based on the actually available locales.

The reinterpreted locale settings will be written back to the
environment so they're visible to other components in the same process
and in subprocesses, but the changed PYTHONIOENCODING default will be
made implicit in order to avoid causing compatibility problems with
Python 2 subprocesses that don't provide the surrogateescape error
handler.

The new legacy locale coercion behavior can be disabled either by
setting LC_ALL (which may still lead to a Unicode compatibility warning)
or by setting the new PYTHONCOERCECLOCALE environment variable to 0.

With this change, any *nix platform that does not offer at least one of
the C.UTF-8, C.utf8 or UTF-8 locales as part of its standard
configuration would only be considered a fully supported platform for
CPython 3.7+ deployments when a suitable locale other than the default C
locale is configured explicitly (e.g. en_AU.UTF-8, zh_CN.gb18030). If
PEP 540 is accepted in addition to this PEP, then pure Python modules
would also be supported when using the proposed PYTHONUTF8 mode, but
expectations for full Unicode compatibility in extension modules would
continue to be limited to the platforms covered by this PEP.

As it only reflects a change in default settings rather than a
fundamentally new capability, redistributors (such as Linux
distributions) with a narrower target audience than the upstream CPython
development team may also choose to opt in to this locale coercion
behaviour for the Python 3.6.x series by applying the necessary changes
as a downstream patch.

Implementation Notes

Attempting to implement the PEP as originally accepted showed that the
proposal to emit locale coercion and compatibility warnings by default
simply wasn't practical (there were too many cases where previously
working code failed because of the warnings, rather than because of
latent locale handling defects in the affected code).

As a result, the PY_WARN_ON_C_LOCALE config flag was removed, and
replaced with a runtime PYTHONCOERCECLOCALE=warn environment variable
setting that allows developers and system integrators to opt-in to
receiving locale coercion and compatibility warnings, without emitting
them by default.

The output examples in the PEP itself have also been updated to remove
the warnings and make them easier to read.

Background

While the CPython interpreter is starting up, it may need to convert
from the char * format to the wchar_t * format, or from one of those
formats to PyUnicodeObject *, in a way that's consistent with the locale
settings of the overall system. It handles these cases by relying on the
operating system to do the conversion and then ensuring that the text
encoding name reported by sys.getfilesystemencoding() matches the
encoding used during this early bootstrapping process.

On Windows, the limitations of the mbcs format used by default in these
conversions proved sufficiently problematic that PEP 528 and PEP 529
were implemented to bypass the operating system supplied interfaces for
binary data handling and force the use of UTF-8 instead.

On Mac OS X, iOS, and Android, many components, including CPython,
already assume the use of UTF-8 as the system encoding, regardless of
the locale setting. However, this isn't the case for all components, and
the discrepancy can cause problems in some situations (for example, when
using the GNU readline module [16]).

On non-Apple and non-Android *nix systems, these operations are handled
using the C locale system in glibc, which has the following
characteristics[1]:

-   by default, all processes start in the C locale, which uses ASCII
    for these conversions. This is almost never what anyone doing
    multilingual text processing actually wants (including CPython and
    C/C++ GUI frameworks).
-   calling setlocale(LC_ALL, "") reconfigures the active locale based
    on the locale categories configured in the current process
    environment
-   if the locale requested by the current environment is unknown, or no
    specific locale is configured, then the default C locale will remain
    active

The specific locale category that covers the APIs that CPython depends
on is LC_CTYPE, which applies to "classification and conversion of
characters, and to multibyte and wide characters"[2]. Accordingly,
CPython includes the following key calls to setlocale:

-   in the main python binary, CPython calls setlocale(LC_ALL, "") to
    configure the entire C locale subsystem according to the process
    environment. It does this prior to making any calls into the shared
    CPython library
-   in Py_Initialize, CPython calls setlocale(LC_CTYPE, ""), such that
    the configured locale settings for that category always match those
    set in the environment. It does this unconditionally, and it doesn't
    revert the process state change in Py_Finalize

(This summary of the locale handling omits several technical details
related to exactly where and when the text encoding declared as part of
the locale settings is used - see PEP 540 for further discussion, as
these particular details matter more when decoupling CPython from the
declared C locale than they do when overriding the locale with one based
on UTF-8)

These calls are usually sufficient to provide sensible behaviour, but
they can still fail in the following cases:

-   SSH environment forwarding means that SSH clients may sometimes
    forward client locale settings to servers that don't have that
    locale installed. This leads to CPython running in the default
    ASCII-based C locale
-   some process environments (such as Linux containers) may not have
    any explicit locale configured at all. As with unknown locales, this
    leads to CPython running in the default ASCII-based C locale
-   on Android, rather than configuring the locale based on environment
    variables, the empty locale "" is treated as specifically requesting
    the "C" locale

The simplest way to deal with this problem for currently released
versions of CPython is to explicitly set a more sensible locale when
launching the application. For example:

    LC_CTYPE=C.UTF-8 python3 ...

The C.UTF-8 locale is a full locale definition that uses UTF-8 for the
LC_CTYPE category, and the same settings as the C locale for all other
categories (including LC_COLLATE). It is offered by a number of Linux
distributions (including Debian, Ubuntu, Fedora, Alpine and Android) as
an alternative to the ASCII-based C locale. Some other platforms (such
as HP-UX) offer an equivalent locale definition under the name C.utf8.

Mac OS X and other *BSD systems have taken a different approach: instead
of offering a C.UTF-8 locale, they offer a partial UTF-8 locale that
only defines the LC_CTYPE category. On such systems, the preferred
environmental locale adjustment is to set LC_CTYPE=UTF-8 rather than to
set LC_ALL or LANG.[3]

In the specific case of Docker containers and similar technologies, the
appropriate locale setting can be specified directly in the container
image definition.

Another common failure case is developers specifying LANG=C in order to
see otherwise translated user interface messages in English, rather than
the more narrowly scoped LC_MESSAGES=C or LANGUAGE=en.

Relationship with other PEPs

This PEP shares a common problem statement with PEP 540 (improving
Python 3's behaviour in the default C locale), but diverges markedly in
the proposed solution:

-   PEP 540 proposes to entirely decouple CPython's default text
    encoding from the C locale system in that case, allowing text
    handling inconsistencies to arise between CPython and other
    locale-aware components running in the same process and in
    subprocesses. This approach aims to make CPython behave less like a
    locale-aware application, and more like locale-independent language
    runtimes like those for Go, Node.js (V8), and Rust
-   this PEP proposes to override the legacy C locale with a more
    recently defined locale that uses UTF-8 as its default text
    encoding. This means that the text encoding override will apply not
    only to CPython, but also to any locale-aware extension modules
    loaded into the current process, as well as to locale-aware
    applications invoked in subprocesses that inherit their environment
    from the parent process. This approach aims to retain CPython's
    traditional strong support for integration with other locale-aware
    components while also actively helping to push forward the adoption
    and standardisation of the C.UTF-8 locale as a Unicode-aware
    replacement for the legacy C locale in the wider C/C++ ecosystem

After reviewing both PEPs, it became clear that they didn't actually
conflict at a technical level, and the proposal in PEP 540 offered a
superior option in cases where no suitable locale was available, as well
as offering a better reference behaviour for platforms where the notion
of a "locale encoding" doesn't make sense (for example, embedded systems
running MicroPython rather than the CPython reference interpreter).

Meanwhile, this PEP offered improved compatibility with other
locale-aware components, and an approach more amenable to being
backported to Python 3.6 by downstream redistributors.

As a result, this PEP was amended to refer to PEP 540 as a complementary
solution that offered improved behaviour when none of the standard UTF-8
based locales were available, as well as extending the changes in the
default settings to APIs that aren't currently independently
configurable (such as the default encoding and error handler for
open()).

The availability of PEP 540 also meant that the LC_CTYPE=en_US.UTF-8
legacy fallback was removed from the list of UTF-8 locales tried as a
coercion target, with the expectation being that CPython will instead
rely solely on the proposed PYTHONUTF8 mode in such cases.

Motivation

While Linux container technologies like Docker, Kubernetes, and
OpenShift are best known for their use in web service development, the
related container formats and execution models are also being adopted
for Linux command line application development. Technologies like Gnome
Flatpak[4] and Ubuntu Snappy[5] further aim to bring these same
techniques to Linux GUI application development.

When using Python 3 for application development in these contexts, it
isn't uncommon to see text encoding related errors akin to the
following:

    $ docker run --rm fedora:25 python3 -c 'print("ℙƴ☂ℌøἤ")'
    Unable to decode the command from the command line:
    UnicodeEncodeError: 'utf-8' codec can't encode character '\udce2' in position 7: surrogates not allowed
    $ docker run --rm ncoghlan/debian-python python3 -c 'print("ℙƴ☂ℌøἤ")'
    Unable to decode the command from the command line:
    UnicodeEncodeError: 'utf-8' codec can't encode character '\udce2' in position 7: surrogates not allowed

Even though the same command is likely to work fine when run locally:

    $ python3 -c 'print("ℙƴ☂ℌøἤ")'
    ℙƴ☂ℌøἤ

The source of the problem can be seen by instead running the locale
command in the three environments:

    $ locale | grep -E 'LC_ALL|LC_CTYPE|LANG'
    LANG=en_AU.UTF-8
    LC_CTYPE="en_AU.UTF-8"
    LC_ALL=
    $ docker run --rm fedora:25 locale | grep -E 'LC_ALL|LC_CTYPE|LANG'
    LANG=
    LC_CTYPE="POSIX"
    LC_ALL=
    $ docker run --rm ncoghlan/debian-python locale | grep -E 'LC_ALL|LC_CTYPE|LANG'
    LANG=
    LANGUAGE=
    LC_CTYPE="POSIX"
    LC_ALL=

In this particular example, we can see that the host system locale is
set to "en_AU.UTF-8", so CPython uses UTF-8 as the default text
encoding. By contrast, the base Docker images for Fedora and Debian
don't have any specific locale set, so they use the POSIX locale by
default, which is an alias for the ASCII-based default C locale.

The simplest way to get Python 3 (regardless of the exact version) to
behave sensibly in Fedora and Debian based containers is to run it in
the C.UTF-8 locale that both distros provide:

    $ docker run --rm -e LC_CTYPE=C.UTF-8 fedora:25 python3 -c 'print("ℙƴ☂ℌøἤ")'
    ℙƴ☂ℌøἤ
    $ docker run --rm -e LC_CTYPE=C.UTF-8 ncoghlan/debian-python python3 -c 'print("ℙƴ☂ℌøἤ")'
    ℙƴ☂ℌøἤ

    $ docker run --rm -e LC_CTYPE=C.UTF-8 fedora:25 locale | grep -E 'LC_ALL|LC_CTYPE|LANG'
    LANG=
    LC_CTYPE=C.UTF-8
    LC_ALL=
    $ docker run --rm -e LC_CTYPE=C.UTF-8 ncoghlan/debian-python locale | grep -E 'LC_ALL|LC_CTYPE|LANG'
    LANG=
    LANGUAGE=
    LC_CTYPE=C.UTF-8
    LC_ALL=

The Alpine Linux based Python images provided by Docker, Inc. already
use the C.UTF-8 locale by default:

    $ docker run --rm python:3 python3 -c 'print("ℙƴ☂ℌøἤ")'
    ℙƴ☂ℌøἤ
    $ docker run --rm python:3 locale | grep -E 'LC_ALL|LC_CTYPE|LANG'
    LANG=C.UTF-8
    LANGUAGE=
    LC_CTYPE="C.UTF-8"
    LC_ALL=

Similarly, for custom container images (i.e. those adding additional
content on top of a base distro image), a more suitable locale can be
set in the image definition so everything just works by default.
However, it would provide a much nicer and more consistent user
experience if CPython were able to just deal with this problem
automatically rather than relying on redistributors or end users to
handle it through system configuration changes.

While the glibc developers are working towards making the C.UTF-8 locale
universally available for use by glibc based applications like
CPython[6], this unfortunately doesn't help on platforms that ship older
versions of glibc without that feature, and also don't provide C.UTF-8
(or an equivalent) as an on-disk locale the way Debian and Fedora do.
These platforms are considered out of scope for this PEP - see PEP 540
for further discussion of possible options for improving CPython's
default behaviour in such environments.

Design Principles

The above motivation leads to the following core design principles for
the proposed solution:

-   if a locale other than the default C locale is explicitly
    configured, we'll continue to respect it
-   as far as is feasible, any changes made will use existing
    configuration options
-   Python's runtime behaviour in potential coercion target locales
    should be identical regardless of whether the locale was set
    explicitly in the environment or implicitly as a locale coercion
    target
-   for Python 3.7, if we're changing the locale setting without an
    explicit config option, we'll emit a warning on stderr that we're
    doing so rather than silently changing the process configuration.
    This will alert application and system integrators to the change,
    even if they don't closely follow the PEP process or Python release
    announcements. However, to minimize the chance of introducing new
    problems for end users, we'll do this without using the warnings
    system, so even running with -Werror won't turn it into a runtime
    exception. (Note: these warnings ended up being silenced by default.
    See the Implementation Note above for more details)
-   for Python 3.7, any changed defaults will offer some form of
    explicit "off" switch at build time, runtime, or both

Minimizing the negative impact on systems currently correctly configured
to use GB-18030 or another partially ASCII compatible universal encoding
leads to the following design principle:

-   if a UTF-8 based Linux container is run on a host that is explicitly
    configured to use a non-UTF-8 encoding, and tries to exchange
    locally encoded data with that host rather than exchanging
    explicitly UTF-8 encoded data, CPython will endeavour to correctly
    round-trip host provided data that is concatenated or split solely
    at common ASCII compatible code points, but may otherwise emit
    nonsensical results.

Minimizing the negative impact on systems and programs correctly
configured to use an explicit locale category like LC_TIME, LC_MONETARY
or LC_NUMERIC while otherwise running in the legacy C locale gives the
following design principles:

-   don't make any environmental changes that would alter any existing
    settings for locale categories other than LC_CTYPE (most notably:
    don't set LC_ALL or LANG)

Finally, maintaining compatibility with running arbitrary subprocesses
in orchestration use cases leads to the following design principle:

-   don't make any Python-specific environmental changes that might be
    incompatible with any still supported version of CPython (including
    CPython 2.7)

Specification

To better handle the cases where CPython would otherwise end up
attempting to operate in the C locale, this PEP proposes that CPython
automatically attempt to coerce the legacy C locale to a UTF-8 based
locale for the LC_CTYPE category when it is run as a standalone command
line application.

It further proposes to emit a warning on stderr if the legacy C locale
is in effect for the LC_CTYPE category at the point where the language
runtime itself is initialized, and the explicit environmental flag to
disable locale coercion is not set, in order to warn system and
application integrators that they're running CPython in an unsupported
configuration.

In addition to these general changes, some additional Android-specific
changes are proposed to handle the differences in the behaviour of
setlocale on that platform.

Legacy C locale coercion in the standalone Python interpreter binary

When run as a standalone application, CPython has the opportunity to
reconfigure the C locale before any locale dependent operations are
executed in the process.

This means that it can change the locale settings not only for the
CPython runtime, but also for any other locale-aware components running
in the current process (e.g. as part of extension modules), as well as
in subprocesses that inherit their environment from the current process.

After calling setlocale(LC_ALL, "") to initialize the locale settings in
the current process, the main interpreter binary will be updated to
include the following call:

    const char *ctype_loc = setlocale(LC_CTYPE, NULL);

This cryptic invocation is the API that C provides to query the current
locale setting without changing it. Given that query, it is possible to
check for exactly the C locale with strcmp:

    ctype_loc != NULL && strcmp(ctype_loc, "C") == 0 # true only in the C locale

This call also returns "C" when either no particular locale is set, or
the nominal locale is set to an alias for the C locale (such as POSIX).

Given this information, CPython can then attempt to coerce the locale to
one that uses UTF-8 rather than ASCII as the default encoding.

Three such locales will be tried:

-   C.UTF-8 (available at least in Debian, Ubuntu, Alpine, and Fedora
    25+, and expected to be available by default in a future version of
    glibc)
-   C.utf8 (available at least in HP-UX)
-   UTF-8 (available in at least some *BSD variants, including Mac OS X)

The coercion will be implemented by setting the LC_CTYPE environment
variable to the candidate locale name, such that future calls to
setlocale() will see it, as will other components looking for those
settings (such as GUI development frameworks and Python's own locale
module).

To allow for better cross-platform binary portability and to adjust
automatically to future changes in locale availability, these checks
will be implemented at runtime on all platforms other than Windows,
rather than attempting to determine which locales to try at compile
time.

When this locale coercion is activated, the following warning will be
printed on stderr, with the warning containing whichever locale was
successfully configured:

    Python detected LC_CTYPE=C: LC_CTYPE coerced to C.UTF-8 (set another
    locale or PYTHONCOERCECLOCALE=0 to disable this locale coercion behaviour).

(Note: this warning ended up being silenced by default. See the
Implementation Note above for more details)

As long as the current platform provides at least one of the candidate
UTF-8 based environments, this locale coercion will mean that the
standard Python binary and locale-aware extensions should once again
"just work" in the three main failure cases we're aware of (missing
locale settings, SSH forwarding of unknown locales via LANG or LC_CTYPE,
and developers explicitly requesting LANG=C).

The one case where failures may still occur is when stderr is
specifically being checked for no output, which can be resolved either
by configuring a locale other than the C locale, or else by using a
mechanism other than "there was no output on stderr" to check for
subprocess errors (e.g. checking process return codes).

If none of the candidate locales are successfully configured, or the
LC_ALL, locale override is defined in the current process environment,
then initialization will continue in the C locale and the Unicode
compatibility warning described in the next section will be emitted just
as it would for any other application.

If PYTHONCOERCECLOCALE=0 is explicitly set, initialization will continue
in the C locale and the Unicode compatibility warning described in the
next section will be automatically suppressed.

The interpreter will always check for the PYTHONCOERCECLOCALE
environment variable at startup (even when running under the -E or -I
switches), as the locale coercion check necessarily takes place before
any command line argument processing. For consistency, the runtime check
to determine whether or not to suppress the locale compatibility warning
will be similarly independent of these settings.

Legacy C locale warning during runtime initialization

By the time that Py_Initialize is called, arbitrary locale-dependent
operations may have taken place in the current process. This means that
by the time it is called, it is too late to reliably switch to a
different locale - doing so would introduce inconsistencies in decoded
text, even in the context of the standalone Python interpreter binary.

Accordingly, when Py_Initialize is called and CPython detects that the
configured locale is still the default C locale and
PYTHONCOERCECLOCALE=0 is not set, the following warning will be issued:

    Python runtime initialized with LC_CTYPE=C (a locale with default ASCII
    encoding), which may cause Unicode compatibility problems. Using C.UTF-8,
    C.utf8, or UTF-8 (if available) as alternative Unicode-compatible
    locales is recommended.

(Note: this warning ended up being silenced by default. See the
Implementation Note above for more details)

In this case, no actual change will be made to the locale settings.

Instead, the warning informs both system and application integrators
that they're running Python 3 in a configuration that we don't expect to
work properly.

The second sentence providing recommendations may eventually be
conditionally compiled based on the operating system (e.g. recommending
LC_CTYPE=UTF-8 on *BSD systems), but the initial implementation will
just use the common generic message shown above.

New build-time configuration options

While both of the above behaviours would be enabled by default, they
would also have new associated configuration options and preprocessor
definitions for the benefit of redistributors that want to override
those default settings.

The locale coercion behaviour would be controlled by the flag
--with[out]-c-locale-coercion, which would set the PY_COERCE_C_LOCALE
preprocessor definition.

The locale warning behaviour would be controlled by the flag
--with[out]-c-locale-warning, which would set the PY_WARN_ON_C_LOCALE
preprocessor definition.

(Note: this compile time warning option ended up being replaced by a
runtime PYTHONCOERCECLOCALE=warn option. See the Implementation Note
above for more details)

On platforms which don't use the autotools based build system (i.e.
Windows) these preprocessor variables would always be undefined.

Changes to the default error handling on the standard streams

Since Python 3.5, CPython has defaulted to using surrogateescape on the
standard streams (sys.stdin, sys.stdout) when it detects that the
current locale is C and no specific error handled has been set using
either the PYTHONIOENCODING environment variable or the
Py_setStandardStreamEncoding API. For other locales, the default error
handler for the standard streams is strict.

In order to preserve this behaviour without introducing any behavioural
discrepancies between locale coercion and explicitly configuring a
locale, the coercion target locales (C.UTF-8, C.utf8, and UTF-8) will be
added to the list of locales that use surrogateescape as their default
error handler for the standard streams.

No changes are proposed to the default error handler for sys.stderr:
that will continue to be backslashreplace.

Changes to locale settings on Android

Independently of the other changes in this PEP, CPython on Android
systems will be updated to call setlocale(LC_ALL, "C.UTF-8") where it
currently calls setlocale(LC_ALL, "") and setlocale(LC_CTYPE, "C.UTF-8")
where it currently calls setlocale(LC_CTYPE, "").

This Android-specific behaviour is being introduced due to the following
Android-specific details:

-   on Android, passing "" to setlocale is equivalent to passing "C"
-   the C.UTF-8 locale is always available

Platform Support Changes

A new "Legacy C Locale" section will be added to PEP 11 that states:

-   as of CPython 3.7, *nix platforms are expected to provide at least
    one of C.UTF-8 (full locale), C.utf8 (full locale) or UTF-8 (
    LC_CTYPE-only locale) as an alternative to the legacy C locale. Any
    Unicode related integration problems that occur only in the legacy C
    locale and cannot be reproduced in an appropriately configured
    non-ASCII locale will be closed as "won't fix".

Rationale

Improving the handling of the C locale

It has been clear for some time that the C locale's default encoding of
ASCII is entirely the wrong choice for development of modern networked
services. Newer languages like Rust and Go have eschewed that default
entirely, and instead made it a deployment requirement that systems be
configured to use UTF-8 as the text encoding for operating system
interfaces. Similarly, Node.js assumes UTF-8 by default (a behaviour
inherited from the V8 JavaScript engine) and requires custom build
settings to indicate it should use the system locale settings for
locale-aware operations. Both the JVM and the .NET CLR use UTF-16-LE as
their primary encoding for passing text between applications and the
application runtime (i.e. the JVM/CLR, not the host operating system).

The challenge for CPython has been the fact that in addition to being
used for network service development, it is also extensively used as an
embedded scripting language in larger applications, and as a desktop
application development language, where it is more important to be
consistent with other locale-aware components sharing the same process,
as well as with the user's desktop locale settings, than it is with the
emergent conventions of modern network service development.

The core premise of this PEP is that for all of these use cases, the
assumption of ASCII implied by the default "C" locale is the wrong
choice, and furthermore that the following assumptions are valid:

-   in desktop application use cases, the process locale will already be
    configured appropriately, and if it isn't, then that is an operating
    system or embedding application level problem that needs to be
    reported to and resolved by the operating system provider or
    application developer
-   in network service development use cases (especially those based on
    Linux containers), the process locale may not be configured at all,
    and if it isn't, then the expectation is that components will impose
    their own default encoding the way Rust, Go and Node.js do, rather
    than trusting the legacy C default encoding of ASCII the way CPython
    currently does

Defaulting to "surrogateescape" error handling on the standard IO streams

By coercing the locale away from the legacy C default and its assumption
of ASCII as the preferred text encoding, this PEP also disables the
implicit use of the "surrogateescape" error handler on the standard IO
streams that was introduced in Python 3.5 ([7]), as well as the
automatic use of surrogateescape when operating in PEP 540's proposed
UTF-8 mode.

Rather than introducing yet another configuration option to adjust that
behaviour, this PEP instead proposes to extend the "surrogateescape"
default for stdin and stderr error handling to also apply to the three
potential coercion target locales.

The aim of this behaviour is to attempt to ensure that operating system
provided text values are typically able to be transparently passed
through a Python 3 application even if it is incorrect in assuming that
that text has been encoded as UTF-8.

In particular, GB 18030[8] is a Chinese national text encoding standard
that handles all Unicode code points, that is formally incompatible with
both ASCII and UTF-8, but will nevertheless often tolerate processing as
surrogate escaped data - the points where GB 18030 reuses ASCII byte
values in an incompatible way are likely to be invalid in UTF-8, and
will therefore be escaped and opaque to string processing operations
that split on or search for the relevant ASCII code points. Operations
that don't involve splitting on or searching for particular ASCII or
Unicode code point values are almost certain to work correctly.

Similarly, Shift-JIS[9] and ISO-2022-JP[10] remain in widespread use in
Japan, and are incompatible with both ASCII and UTF-8, but will tolerate
text processing operations that don't involve splitting on or searching
for particular ASCII or Unicode code point values.

As an example, consider two files, one encoded with UTF-8 (the default
encoding for en_AU.UTF-8), and one encoded with GB-18030 (the default
encoding for zh_CN.gb18030):

    $ python3 -c 'open("utf8.txt", "wb").write("ℙƴ☂ℌøἤ\n".encode("utf-8"))'
    $ python3 -c 'open("gb18030.txt", "wb").write("ℙƴ☂ℌøἤ\n".encode("gb18030"))'

On disk, we can see that these are two very different files:

    $ python3 -c 'print("UTF-8:  ", open("utf8.txt", "rb").read().strip()); \
                  print("GB18030:", open("gb18030.txt", "rb").read().strip())'
    UTF-8:   b'\xe2\x84\x99\xc6\xb4\xe2\x98\x82\xe2\x84\x8c\xc3\xb8\xe1\xbc\xa4\n'
    GB18030: b'\x816\xbd6\x810\x9d0\x817\xa29\x816\xbc4\x810\x8b3\x816\x8d6\n'

That nevertheless can both be rendered correctly to the terminal as long
as they're decoded prior to printing:

    $ python3 -c 'print("UTF-8:  ", open("utf8.txt", "r", encoding="utf-8").read().strip()); \
                  print("GB18030:", open("gb18030.txt", "r", encoding="gb18030").read().strip())'
    UTF-8:   ℙƴ☂ℌøἤ
    GB18030: ℙƴ☂ℌøἤ

By contrast, if we just pass along the raw bytes, as cat and similar
C/C++ utilities will tend to do:

    $ LANG=en_AU.UTF-8 cat utf8.txt gb18030.txt
    ℙƴ☂ℌøἤ
    �6�6�0�0�7�9�6�4�0�3�6�6

Even setting a specifically Chinese locale won't help in getting the
GB-18030 encoded file rendered correctly:

    $ LANG=zh_CN.gb18030 cat utf8.txt gb18030.txt
    ℙƴ☂ℌøἤ
    �6�6�0�0�7�9�6�4�0�3�6�6

The problem is that the terminal encoding setting remains UTF-8,
regardless of the nominal locale. A GB18030 terminal can be emulated
using the iconv utility:

    $ cat utf8.txt gb18030.txt | iconv -f GB18030 -t UTF-8
    鈩櫰粹槀鈩屆羔激
    ℙƴ☂ℌøἤ

This reverses the problem, such that the GB18030 file is rendered
correctly, but the UTF-8 file has been converted to unrelated hanzi
characters, rather than the expected rendering of "Python" as non-ASCII
characters.

With the emulated GB18030 terminal encoding, assuming UTF-8 in Python
results in both files being displayed incorrectly:

    $ python3 -c 'print("UTF-8:  ", open("utf8.txt", "r", encoding="utf-8").read().strip()); \
                  print("GB18030:", open("gb18030.txt", "r", encoding="gb18030").read().strip())' \
      | iconv -f GB18030 -t UTF-8
    UTF-8:   鈩櫰粹槀鈩屆羔激
    GB18030: 鈩櫰粹槀鈩屆羔激

However, setting the locale correctly means that the emulated GB18030
terminal now displays both files as originally intended:

    $ LANG=zh_CN.gb18030 \
      python3 -c 'print("UTF-8:  ", open("utf8.txt", "r", encoding="utf-8").read().strip()); \
                  print("GB18030:", open("gb18030.txt", "r", encoding="gb18030").read().strip())' \
      | iconv -f GB18030 -t UTF-8
    UTF-8:   ℙƴ☂ℌøἤ
    GB18030: ℙƴ☂ℌøἤ

The rationale for retaining surrogateescape as the default IO encoding
is that it will preserve the following helpful behaviour in the C
locale:

    $ cat gb18030.txt \
      | LANG=C python3 -c "import sys; print(sys.stdin.read())" \
      | iconv -f GB18030 -t UTF-8
    ℙƴ☂ℌøἤ

Rather than reverting to the exception currently seen when a UTF-8 based
locale is explicitly configured:

    $ cat gb18030.txt \
      | python3 -c "import sys; print(sys.stdin.read())" \
      | iconv -f GB18030 -t UTF-8
    Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/usr/lib64/python3.5/codecs.py", line 321, in decode
        (result, consumed) = self._buffer_decode(data, self.errors, final)
    UnicodeDecodeError: 'utf-8' codec can't decode byte 0x81 in position 0: invalid start byte

As an added benefit, environments explicitly configured to use one of
the coercion target locales will implicitly gain the encoding
transparency behaviour currently enabled by default in the C locale.

Avoiding setting PYTHONIOENCODING during UTF-8 locale coercion

Rather than changing the default handling of the standard streams during
interpreter initialization, earlier versions of this PEP proposed
setting PYTHONIOENCODING to utf-8:surrogateescape. This turned out to
create a significant compatibility problem: since the surrogateescape
handler only exists in Python 3.1+, running Python 2.7 processes in
subprocesses could potentially break in a confusing way with that
configuration.

The current design means that earlier Python versions will instead
retain their default strict error handling on the standard streams,
while Python 3.7+ will consistently use the more permissive
surrogateescape handler even when these locales are explicitly
configured (rather than being reached through locale coercion).

Dropping official support for ASCII based text handling in the legacy C locale

We've been trying to get strict bytes/text separation to work reliably
in the legacy C locale for over a decade at this point. Not only haven't
we been able to get it to work, neither has anyone else - the only
viable alternatives identified have been to pass the bytes along
verbatim without eagerly decoding them to text (C/C++, Python 2.x, Ruby,
etc), or else to largely ignore the nominal C/C++ locale encoding and
assume the use of either UTF-8 (PEP 540, Rust, Go, Node.js, etc) or
UTF-16-LE (JVM, .NET CLR).

While this PEP ensures that developers that genuinely need to do so can
still opt-in to running their Python code in the legacy C locale (by
setting LC_ALL=C, PYTHONCOERCECLOCALE=0, or running a custom build that
sets --without-c-locale-coercion), it also makes it clear that we don't
expect Python 3's Unicode handling to be completely reliable in that
configuration, and the recommended alternative is to use a more
appropriate locale setting (potentially in combination with PEP 540's
UTF-8 mode, if that is available).

Providing implicit locale coercion only when running standalone

The major downside of the proposed design in this PEP is that it
introduces a potential discrepancy between the behaviour of the CPython
runtime when it is run as a standalone application and when it is run as
an embedded component inside a larger system (e.g. mod_wsgi running
inside Apache httpd).

Over the course of Python 3.x development, multiple attempts have been
made to improve the handling of incorrect locale settings at the point
where the Python interpreter is initialised. The problem that emerged is
that this is ultimately too late in the interpreter startup process -
data such as command line arguments and the contents of environment
variables may have already been retrieved from the operating system and
processed under the incorrect ASCII text encoding assumption well before
Py_Initialize is called.

The problems created by those inconsistencies were then even harder to
diagnose and debug than those created by believing the operating
system's claim that ASCII was a suitable encoding to use for operating
system interfaces. This was the case even for the default CPython
binary, let alone larger C/C++ applications that embed CPython as a
scripting engine.

The approach proposed in this PEP handles that problem by moving the
locale coercion as early as possible in the interpreter startup sequence
when running standalone: it takes place directly in the C-level main()
function, even before calling in to the Py_Main() library function that
implements the features of the CPython interpreter CLI.

The Py_Initialize API then only gains an explicit warning (emitted on
stderr) when it detects use of the C locale, and relies on the embedding
application to specify something more reasonable.

That said, the reference implementation for this PEP adds most of the
functionality to the shared library, with the CLI being updated to
unconditionally call two new private APIs:

    if (_Py_LegacyLocaleDetected()) {
        _Py_CoerceLegacyLocale();
    }

These are similar to other "pre-configuration" APIs intended for
embedding applications: they're designed to be called before
Py_Initialize, and hence change the way the interpreter gets
initialized.

If these were made public (either as part of this PEP or in a subsequent
RFE), then it would be straightforward for other embedding applications
to recreate the same behaviour as is proposed for the CPython CLI.

Allowing restoration of the legacy behaviour

The CPython command line interpreter is often used to investigate faults
that occur in other applications that embed CPython, and those
applications may still be using the C locale even after this PEP is
implemented.

Providing a simple on/off switch for the locale coercion behaviour makes
it much easier to reproduce the behaviour of such applications for
debugging purposes, as well as making it easier to reproduce the
behaviour of older 3.x runtimes even when running a version with this
change applied.

Querying LC_CTYPE for C locale detection

LC_CTYPE is the actual locale category that CPython relies on to drive
the implicit decoding of environment variables, command line arguments,
and other text values received from the operating system.

As such, it makes sense to check it specifically when attempting to
determine whether or not the current locale configuration is likely to
cause Unicode handling problems.

Explicitly setting LC_CTYPE for UTF-8 locale coercion

Python is often used as a glue language, integrating other C/C++ ABI
compatible components in the current process, and components written in
arbitrary languages in subprocesses.

Setting LC_CTYPE to C.UTF-8 is important to handle cases where the
problem has arisen from a setting like LC_CTYPE=UTF-8 being provided on
a system where no UTF-8 locale is defined (e.g. when a Mac OS X ssh
client is configured to forward locale settings, and the user logs into
a Linux server).

This should be sufficient to ensure that when the locale coercion is
activated, the switch to the UTF-8 based locale will be applied
consistently across the current process and any subprocesses that
inherit the current environment.

Avoiding setting LANG for UTF-8 locale coercion

Earlier versions of this PEP proposed setting the LANG category
independent default locale, in addition to setting LC_CTYPE.

This was later removed on the grounds that setting only LC_CTYPE is
sufficient to handle all of the problematic scenarios that the PEP aimed
to resolve, while setting LANG as well would break cases where LANG was
set correctly, and the locale problems were solely due to an incorrect
LC_CTYPE setting ([11]).

For example, consider a Python application that called the Linux date
utility in a subprocess rather than doing its own date formatting:

    $ LANG=ja_JP.UTF-8 LC_CTYPE=C date
    2017年  5月 23日 火曜日 17:31:03 JST

    $ LANG=ja_JP.UTF-8 LC_CTYPE=C.UTF-8 date  # Coercing only LC_CTYPE
    2017年  5月 23日 火曜日 17:32:58 JST

    $ LANG=C.UTF-8 LC_CTYPE=C.UTF-8 date  # Coercing both of LC_CTYPE and LANG
    Tue May 23 17:31:10 JST 2017

With only LC_CTYPE updated in the Python process, the subprocess would
continue to behave as expected. However, if LANG was updated as well,
that would effectively override the LC_TIME setting and use the wrong
date formatting conventions.

Avoiding setting LC_ALL for UTF-8 locale coercion

Earlier versions of this PEP proposed setting the LC_ALL locale
override, in addition to setting LC_CTYPE.

This was changed after it was determined that just setting LC_CTYPE and
LANG should be sufficient to handle all the scenarios the PEP aims to
cover, as it avoids causing any problems in cases like the following:

    $ LANG=C LC_MONETARY=ja_JP.utf8 ./python -c \
      "from locale import setlocale, LC_ALL, currency; setlocale(LC_ALL, ''); print(currency(1e6))"
    ￥1000000

Skipping locale coercion if LC_ALL is set in the current environment

With locale coercion now only setting LC_CTYPE and LANG, it will have no
effect if LC_ALL is also set. To avoid emitting a spurious locale
coercion notice in that case, coercion is instead skipped entirely.

Considering locale coercion independently of "UTF-8 mode"

With both this PEP's locale coercion and PEP 540's UTF-8 mode under
consideration for Python 3.7, it makes sense to ask whether or not we
can limit ourselves to only doing one or the other, rather than making
both changes.

The UTF-8 mode proposed in PEP 540 has two major limitations that make
it a potential complement to this PEP rather than a potential
replacement.

First, unlike this PEP, PEP 540's UTF-8 mode makes it possible to change
default behaviours that are not currently configurable at all. While
that's exactly what makes the proposal interesting, it's also what makes
it an entirely unproven approach. By contrast, the approach proposed in
this PEP builds directly atop existing configuration settings for the C
locale system ( LC_CTYPE, LANG) and Python's standard streams
(PYTHONIOENCODING) that have already been in use for years to handle the
kinds of compatibility problems discussed in this PEP.

Secondly, one of the things we know based on that experience is that the
proposed locale coercion can resolve problems not only in CPython
itself, but also in extension modules that interact with the standard
streams, like GNU readline. As an example, consider the following
interactive session from a PEP 538 enabled CPython build, where each
line after the first is executed by doing "up-arrow, left-arrow x4,
delete, enter":

    $ LANG=C ./python
    Python 3.7.0a0 (heads/pep538-coerce-c-locale:188e780, May  7 2017, 00:21:13)
    [GCC 6.3.1 20161221 (Red Hat 6.3.1-1)] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> print("ℙƴ☂ℌøἤ")
    ℙƴ☂ℌøἤ
    >>> print("ℙƴ☂ℌἤ")
    ℙƴ☂ℌἤ
    >>> print("ℙƴ☂ἤ")
    ℙƴ☂ἤ
    >>> print("ℙƴἤ")
    ℙƴἤ
    >>> print("ℙἤ")
    ℙἤ
    >>> print("ἤ")
    ἤ
    >>>

This is exactly what we'd expect from a well-behaved command history
editor.

By contrast, the following is what currently happens on an older release
if you only change the Python level stream encoding settings without
updating the locale settings:

    $ LANG=C PYTHONIOENCODING=utf-8:surrogateescape python3
    Python 3.5.3 (default, Apr 24 2017, 13:32:13)
    [GCC 6.3.1 20161221 (Red Hat 6.3.1-1)] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> print("ℙƴ☂ℌøἤ")
    ℙƴ☂ℌøἤ
    >>> print("ℙƴ☂ℌ�")
     File "<stdin>", line 0

       ^
    SyntaxError: 'utf-8' codec can't decode bytes in position 20-21:
    invalid continuation byte

That particular misbehaviour is coming from GNU readline, not CPython
-because the command history editing wasn't UTF-8 aware, it corrupted
the history buffer and fed such nonsense to stdin that even the
surrogateescape error handler was bypassed. While PEP 540's UTF-8 mode
could technically be updated to also reconfigure readline, that's just
one extension module that might be interacting with the standard streams
without going through the CPython C API, and any change made by CPython
would only apply when readline is running directly as part of Python 3.7
rather than in a separate subprocess.

However, if we actually change the configured locale, GNU readline
starts behaving itself, without requiring any changes to the embedding
application:

    $ LANG=C.UTF-8 python3
    Python 3.5.3 (default, Apr 24 2017, 13:32:13)
    [GCC 6.3.1 20161221 (Red Hat 6.3.1-1)] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> print("ℙƴ☂ℌøἤ")
    ℙƴ☂ℌøἤ
    >>> print("ℙƴ☂ℌἤ")
    ℙƴ☂ℌἤ
    >>> print("ℙƴ☂ἤ")
    ℙƴ☂ἤ
    >>> print("ℙƴἤ")
    ℙƴἤ
    >>> print("ℙἤ")
    ℙἤ
    >>> print("ἤ")
    ἤ
    >>>
    $ LC_CTYPE=C.UTF-8 python3
    Python 3.5.3 (default, Apr 24 2017, 13:32:13)
    [GCC 6.3.1 20161221 (Red Hat 6.3.1-1)] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> print("ℙƴ☂ℌøἤ")
    ℙƴ☂ℌøἤ
    >>> print("ℙƴ☂ℌἤ")
    ℙƴ☂ℌἤ
    >>> print("ℙƴ☂ἤ")
    ℙƴ☂ἤ
    >>> print("ℙƴἤ")
    ℙƴἤ
    >>> print("ℙἤ")
    ℙἤ
    >>> print("ἤ")
    ἤ
    >>>

Enabling C locale coercion and warnings on Mac OS X, iOS and Android

On Mac OS X, iOS, and Android, CPython already assumes the use of UTF-8
for system interfaces, and we expect most other locale-aware components
to do the same.

Accordingly, this PEP originally proposed to disable locale coercion and
warnings at build time for these platforms, on the assumption that it
would be entirely redundant.

However, that assumption turned out to be incorrect, as subsequent
investigations showed that if you explicitly configure LANG=C on these
platforms, extension modules like GNU readline will misbehave in much
the same way as they do on other *nix systems.[12]

In addition, Mac OS X is also frequently used as a development and
testing platform for Python software intended for deployment to other
*nix environments (such as Linux or Android), and Linux is similarly
often used as a development and testing platform for mobile and Mac OS X
applications.

Accordingly, this PEP enables the locale coercion and warning features
by default on all platforms that use CPython's autotools based build
toolchain (i.e. everywhere other than Windows).

Implementation

The reference implementation is being developed in the
pep538-coerce-c-locale feature branch[13] in Alyssa Coghlan's fork of
the CPython repository on GitHub. A work-in-progress PR is available
at[14].

This reference implementation covers not only the enhancement request in
issue 28180[15], but also the Android compatibility fixes needed to
resolve issue 28997[16].

Backporting to earlier Python 3 releases

Backporting to Python 3.6.x

If this PEP is accepted for Python 3.7, redistributors backporting the
change specifically to their initial Python 3.6.x release will be both
allowed and encouraged. However, such backports should only be
undertaken either in conjunction with the changes needed to also provide
a suitable locale by default, or else specifically for platforms where
such a locale is already consistently available.

At least the Fedora project is planning to pursue this approach for the
upcoming Fedora 26 release[17].

Backporting to other 3.x releases

While the proposed behavioural change is seen primarily as a bug fix
addressing Python 3's current misbehaviour in the default ASCII-based C
locale, it still represents a reasonably significant change in the way
CPython interacts with the C locale system. As such, while some
redistributors may still choose to backport it to even earlier Python
3.x releases based on the needs and interests of their particular user
base, this wouldn't be encouraged as a general practice.

However, configuring Python 3 environments (such as base container
images) to use these configuration settings by default is both allowed
and recommended.

Acknowledgements

The locale coercion approach proposed in this PEP is inspired directly
by Armin Ronacher's handling of this problem in the click command line
utility development framework[18]:

    $ LANG=C python3 -c 'import click; cli = click.command()(lambda:None); cli()'
    Traceback (most recent call last):
      ...
    RuntimeError: Click will abort further execution because Python 3 was
    configured to use ASCII as encoding for the environment.  Either run this
    under Python 2 or consult http://click.pocoo.org/python3/ for mitigation
    steps.

    This system supports the C.UTF-8 locale which is recommended.
    You might be able to resolve your issue by exporting the
    following environment variables:

        export LC_ALL=C.UTF-8
        export LANG=C.UTF-8

The change was originally proposed as a downstream patch for Fedora's
system Python 3.6 package[19], and then reformulated as a PEP for Python
3.7 with a section allowing for backports to earlier versions by
redistributors. In parallel with the development of the upstream patch,
Charalampos Stratakis has been working on the Fedora 26 backport and
providing feedback on the practical viability of the proposed changes.

The initial draft was posted to the Python Linux SIG for discussion[20]
and then amended based on both that discussion and Victor Stinner's work
in PEP 540[21].

The "ℙƴ☂ℌøἤ" string used in the Unicode handling examples throughout
this PEP is taken from Ned Batchelder's excellent "Pragmatic Unicode"
presentation[22].

Stephen Turnbull has long provided valuable insight into the text
encoding handling challenges he regularly encounters at the University
of Tsukuba (筑波大学).

References

Copyright

This document has been placed in the public domain under the terms of
the CC0 1.0 license: https://creativecommons.org/publicdomain/zero/1.0/

[1] GNU C: How Programs Set the Locale
(https://www.gnu.org/software/libc/manual/html_node/Setting-the-Locale.html)

[2] GNU C: Locale Categories
(https://www.gnu.org/software/libc/manual/html_node/Locale-Categories.html)

[3] UTF-8 locale discussion on "locale.getdefaultlocale() fails on Mac
OS X with default language set to English"
(https://bugs.python.org/issue18378#msg215215)

[4] GNOME Flatpak (https://flatpak.org/)

[5] Ubuntu Snappy (https://www.ubuntu.com/desktop/snappy)

[6] glibc C.UTF-8 locale proposal
(https://sourceware.org/glibc/wiki/Proposals/C.UTF-8)

[7] Use "surrogateescape" error handler for sys.stdin and sys.stdout on
UNIX for the C locale (https://bugs.python.org/issue19977)

[8] GB 18030 (https://en.wikipedia.org/wiki/GB_18030)

[9] Shift-JIS (https://en.wikipedia.org/wiki/Shift_JIS)

[10] ISO-2022 (https://en.wikipedia.org/wiki/ISO/IEC_2022)

[11] Potential problems when setting LANG in addition to setting
LC_CTYPE
(https://mail.python.org/pipermail/python-dev/2017-May/147968.html)

[12] GNU readline misbehaviour on Mac OS X with LANG=C
(https://mail.python.org/pipermail/python-dev/2017-May/147897.html)

[13] GitHub branch diff for ncoghlan:pep538-coerce-c-locale
(https://github.com/python/cpython/compare/master...ncoghlan:pep538-coerce-c-locale)

[14] GitHub pull request for the reference implementation
(https://github.com/python/cpython/pull/659)

[15] CPython: sys.getfilesystemencoding() should default to utf-8
(https://bugs.python.org/issue28180)

[16] test_readline.test_nonascii fails on Android
(https://bugs.python.org/issue28997)

[17] Fedora 26 change proposal for locale coercion backport
(https://fedoraproject.org/wiki/Changes/python3_c.utf-8_locale)

[18] Locale configuration required for click applications under Python 3
(https://click.palletsprojects.com/en/5.x/python3/#python-3-surrogate-handling)

[19] Fedora: force C.UTF-8 when Python 3 is run under the C locale
(https://bugzilla.redhat.com/show_bug.cgi?id=1404918)

[20] linux-sig discussion of initial PEP draft
(https://mail.python.org/pipermail/linux-sig/2017-January/000014.html)

[21] Feedback notes from linux-sig discussion and PEP 540
(https://github.com/python/peps/issues/171)

[22] Pragmatic Unicode (https://nedbatchelder.com/text/unipain.html)


text:
PEP: 513 Title: A Platform Tag for Portable Linux Built Distributions
Version: $Revision$ Last-Modified: $Date$ Author: Robert T. McGibbon
<EMAIL_ADDRESS>Nathaniel J. Smith<EMAIL_ADDRESS>BDFL-Delegate:
Alyssa Coghlan<EMAIL_ADDRESS>Discussions-To:
<EMAIL_ADDRESS>Status: Superseded Type: Informational Topic:
Packaging Content-Type: text/x-rst Created: 19-Jan-2016 Post-History:
19-Jan-2016, 25-Jan-2016, 29-Jan-2016 Superseded-By: 600 Resolution:
https://mail.python.org/pipermail/distutils-sig/2016-January/028211.html

Abstract

This PEP proposes the creation of a new platform tag for Python package
built distributions, such as wheels, called manylinux1_{x86_64,i686}
with external dependencies limited to a standardized, restricted subset
of the Linux kernel and core userspace ABI. It proposes that PyPI
support uploading and distributing wheels with this platform tag, and
that pip support downloading and installing these packages on compatible
platforms.

Rationale

Currently, distribution of binary Python extensions for Windows and OS X
is straightforward. Developers and packagers build wheels (PEP 427, PEP
491), which are assigned platform tags such as win32 or
macosx_10_6_intel, and upload these wheels to PyPI. Users can download
and install these wheels using tools such as pip.

For Linux, the situation is much more delicate. In general, compiled
Python extension modules built on one Linux distribution will not work
on other Linux distributions, or even on different machines running the
same Linux distribution with different system libraries installed.

Build tools using PEP 425 platform tags do not track information about
the particular Linux distribution or installed system libraries, and
instead assign all wheels the too-vague linux_i686 or linux_x86_64 tags.
Because of this ambiguity, there is no expectation that linux-tagged
built distributions compiled on one machine will work properly on
another, and for this reason, PyPI has not permitted the uploading of
wheels for Linux.

It would be ideal if wheel packages could be compiled that would work on
any linux system. But, because of the incredible diversity of Linux
systems -- from PCs to Android to embedded systems with custom libcs --
this cannot be guaranteed in general.

Instead, we define a standard subset of the kernel+core userspace ABI
that, in practice, is compatible enough that packages conforming to this
standard will work on many linux systems, including essentially all of
the desktop and server distributions in common use. We know this because
there are companies who have been distributing such widely-portable
pre-compiled Python extension modules for Linux -- e.g. Enthought with
Canopy[1] and Continuum Analytics with Anaconda[2].

Building on the compatibility lessons learned from these companies, we
thus define a baseline manylinux1 platform tag for use by binary Python
wheels, and introduce the implementation of preliminary tools to aid in
the construction of these manylinux1 wheels.

Key Causes of Inter-Linux Binary Incompatibility

To properly define a standard that will guarantee that wheel packages
meeting this specification will operate on many linux platforms, it is
necessary to understand the root causes which often prevent portability
of pre-compiled binaries on Linux. The two key causes are dependencies
on shared libraries which are not present on users' systems, and
dependencies on particular versions of certain core libraries like
glibc.

External Shared Libraries

Most desktop and server linux distributions come with a system package
manager (examples include APT on Debian-based systems, yum on RPM-based
systems, and pacman on Arch linux) that manages, among other
responsibilities, the installation of shared libraries installed to
system directories such as /usr/lib. Most non-trivial Python extensions
will depend on one or more of these shared libraries, and thus function
properly only on systems where the user has the proper libraries (and
the proper versions thereof), either installed using their package
manager, or installed manually by setting certain environment variables
such as LD_LIBRARY_PATH to notify the runtime linker of the location of
the depended-upon shared libraries.

Versioning of Core Shared Libraries

Even if the developers a Python extension module wish to use no external
shared libraries, the modules will generally have a dynamic runtime
dependency on the GNU C library, glibc. While it is possible, statically
linking glibc is usually a bad idea because certain important C
functions like dlopen() cannot be called from code that statically links
glibc. A runtime shared library dependency on a system-provided glibc is
unavoidable in practice.

The maintainers of the GNU C library follow a strict symbol versioning
scheme for backward compatibility. This ensures that binaries compiled
against an older version of glibc can run on systems that have a newer
glibc. The opposite is generally not true -- binaries compiled on newer
Linux distributions tend to rely upon versioned functions in glibc that
are not available on older systems.

This generally prevents wheels compiled on the latest Linux
distributions from being portable.

The manylinux1 policy

For these reasons, to achieve broad portability, Python wheels

-   should depend only on an extremely limited set of external shared
    libraries; and
-   should depend only on "old" symbol versions in those external shared
    libraries; and
-   should depend only on a widely-compatible kernel ABI.

To be eligible for the manylinux1 platform tag, a Python wheel must
therefore both (a) contain binary executables and compiled code that
links only to libraries with SONAMEs included in the following list: :

    libpanelw.so.5
    libncursesw.so.5
    libgcc_s.so.1
    libstdc++.so.6
    libm.so.6
    libdl.so.2
    librt.so.1
    libc.so.6
    libnsl.so.1
    libutil.so.1
    libpthread.so.0
    libresolv.so.2
    libX11.so.6
    libXext.so.6
    libXrender.so.1
    libICE.so.6
    libSM.so.6
    libGL.so.1
    libgobject-2.0.so.0
    libgthread-2.0.so.0
    libglib-2.0.so.0

and, (b) work on a stock CentOS 5.11[3] system that contains the system
package manager's provided versions of these libraries.

libcrypt.so.1 was retrospectively removed from the whitelist after
Fedora 30 was released with libcrypt.so.2 instead.

Because CentOS 5 is only available for x86_64 and i686 architectures,
these are the only architectures currently supported by the manylinux1
policy.

On Debian-based systems, these libraries are provided by the packages :

    libncurses5 libgcc1 libstdc++6 libc6 libx11-6 libxext6
    libxrender1 libice6 libsm6 libgl1-mesa-glx libglib2.0-0

On RPM-based systems, these libraries are provided by the packages :

    ncurses libgcc libstdc++ glibc libXext libXrender
    libICE libSM mesa-libGL glib2

This list was compiled by checking the external shared library
dependencies of the Canopy[4] and Anaconda[5] distributions, which both
include a wide array of the most popular Python modules and have been
confirmed in practice to work across a wide swath of Linux systems in
the wild.

Many of the permitted system libraries listed above use symbol
versioning schemes for backward compatibility. The latest symbol
versions provided with the CentOS 5.11 versions of these libraries are:
:

    GLIBC_2.5
    CXXABI_3.4.8
    GLIBCXX_3.4.9
    GCC_4.2.0

Therefore, as a consequence of requirement (b), any wheel that depends
on versioned symbols from the above shared libraries may depend only on
symbols with the following versions: :

    GLIBC <= 2.5
    CXXABI <= 3.4.8
    GLIBCXX <= 3.4.9
    GCC <= 4.2.0

These recommendations are the outcome of the relevant discussions in
January 2016[6],[7].

Note that in our recommendations below, we do not suggest that pip or
PyPI should attempt to check for and enforce the details of this policy
(just as they don't check for and enforce the details of existing
platform tags like win32). The text above is provided (a) as advice to
package builders, and (b) as a method for allocating blame if a given
wheel doesn't work on some system: if it satisfies the policy above,
then this is a bug in the spec or the installation tool; if it does not
satisfy the policy above, then it's a bug in the wheel. One useful
consequence of this approach is that it leaves open the possibility of
further updates and tweaks as we gain more experience, e.g., we could
have a "manylinux 1.1" policy which targets the same systems and uses
the same manylinux1 platform tag (and thus requires no further changes
to pip or PyPI), but that adjusts the list above to remove libraries
that have turned out to be problematic or add libraries that have turned
out to be safe.

libpythonX.Y.so.1

Note that libpythonX.Y.so.1 is not on the list of libraries that a
manylinux1 extension is allowed to link to. Explicitly linking to
libpythonX.Y.so.1 is unnecessary in almost all cases: the way ELF
linking works, extension modules that are loaded into the interpreter
automatically get access to all of the interpreter's symbols, regardless
of whether or not the extension itself is explicitly linked against
libpython. Furthermore, explicit linking to libpython creates problems
in the common configuration where Python is not built with
--enable-shared. In particular, on Debian and Ubuntu systems,
apt install pythonX.Y does not even install libpythonX.Y.so.1, meaning
that any wheel that did depend on libpythonX.Y.so.1 could fail to
import.

There is one situation where extensions that are linked in this way can
fail to work: if a host program (e.g., apache2) uses dlopen() to load a
module (e.g., mod_wsgi) that embeds the CPython interpreter, and the
host program does not pass the RTLD_GLOBAL flag to dlopen(), then the
embedded CPython will be unable to load any extension modules that do
not themselves link explicitly to libpythonX.Y.so.1. Fortunately,
apache2 does set the RTLD_GLOBAL flag, as do all the other programs that
embed-CPython-via-a-dlopened-plugin that we could locate, so this does
not seem to be a serious problem in practice. The incompatibility with
Debian/Ubuntu is more of an issue than the theoretical incompatibility
with a rather obscure corner case.

This is a rather complex and subtle issue that extends beyond the scope
of manylinux1; for more discussion see:[8],[9], [10].

UCS-2 vs UCS-4 builds

All versions of CPython 2.x, plus CPython 3.0-3.2 inclusive, can be
built in two ABI-incompatible modes: builds using the
--enable-unicode=ucs2 configure flag store Unicode data in UCS-2 (or
really UTF-16) format, while builds using the --enable-unicode=ucs4
configure flag store Unicode data in UCS-4. (CPython 3.3 and greater use
a different storage method that always supports UCS-4.) If we want to
make sure ucs2 wheels don't get installed into ucs4 CPythons and
vice-versa, then something must be done.

An earlier version of this PEP included a requirement that manylinux1
wheels targeting these older CPython versions should always use the ucs4
ABI. But then, in between the PEP's initial acceptance and its
implementation, pip and wheel gained first-class support for tracking
and checking this aspect of ABI compatibility for the relevant CPython
versions, which is a better solution. So we now allow the manylinux1
platform tags to be used in combination with any ABI tag. However, to
maintain compatibility it is crucial to ensure that all manylinux1
wheels include a non-trivial abi tag. For example, a wheel built against
a ucs4 CPython might have a name like:

    PKG-VERSION-cp27-cp27mu-manylinux1_x86_64.whl
                     ^^^^^^ Good!

While a wheel built against the ucs2 ABI might have a name like:

    PKG-VERSION-cp27-cp27m-manylinux1_x86_64.whl
                     ^^^^^ Okay!

But you should never have a wheel with a name like:

    PKG-VERSION-cp27-none-manylinux1_x86_64.whl
                     ^^^^ BAD! Don't do this!

This wheel claims to be simultaneously compatible with both ucs2 and
ucs4 builds, which is bad.

We note for information that the ucs4 ABI appears to be much more
widespread among Linux CPython distributors.

fpectl builds vs. no fpectl builds

All extant versions of CPython can be built either with or without the
--with-fpectl flag to configure. It turns out that this changes the
CPython ABI: extensions that are built against a no-fpectl CPython are
always compatible with yes-fpectl CPython, but the reverse is not
necessarily true. (Symptom: errors at import time complaining about
undefined symbol: PyFPE_jbuf.) See: [11].

For maximum compatibility, therefore, the CPython used to build
manylinux1 wheels must be compiled without the --with-fpectl flag, and
manylinux1 extensions must not reference the symbol PyFPE_jbuf.

Compilation of Compliant Wheels

The way glibc, libgcc, and libstdc++ manage their symbol versioning
means that in practice, the compiler toolchains that most developers use
to do their daily work are incapable of building manylinux1-compliant
wheels. Therefore, we do not attempt to change the default behavior of
pip wheel / bdist_wheel: they will continue to generate regular linux_*
platform tags, and developers who wish to use them to generate
manylinux1-tagged wheels will have to change the tag as a second
post-processing step.

To support the compilation of wheels meeting the manylinux1 standard, we
provide initial drafts of two tools.

Docker Image

The first tool is a Docker image based on CentOS 5.11, which is
recommended as an easy to use self-contained build box for compiling
manylinux1 wheels [12]. Compiling on a more recently-released linux
distribution will generally introduce dependencies on too-new versioned
symbols. The image comes with a full compiler suite installed (gcc, g++,
and gfortran 4.8.2) as well as the latest releases of Python and pip.

Auditwheel

The second tool is a command line executable called auditwheel[13] that
may aid in package maintainers in dealing with third-party external
dependencies.

There are at least three methods for building wheels that use
third-party external libraries in a way that meets the above policy.

1.  The third-party libraries can be statically linked.
2.  The third-party shared libraries can be distributed in separate
    packages on PyPI which are depended upon by the wheel.
3.  The third-party shared libraries can be bundled inside the wheel
    libraries, linked with a relative path.

All of these are valid option which may be effectively used by different
packages and communities. Statically linking generally requires
package-specific modifications to the build system, and distributing
third-party dependencies on PyPI may require some coordination of the
community of users of the package.

As an often-automatic alternative to these options, we introduce
auditwheel. The tool inspects all of the ELF files inside a wheel to
check for dependencies on versioned symbols or external shared
libraries, and verifies conformance with the manylinux1 policy. This
includes the ability to add the new platform tag to conforming wheels.
More importantly, auditwheel has the ability to automatically modify
wheels that depend on external shared libraries by copying those shared
libraries from the system into the wheel itself, and modifying the
appropriate RPATH entries such that these libraries will be picked up at
runtime. This accomplishes a similar result as if the libraries had been
statically linked without requiring changes to the build system.
Packagers are advised that bundling, like static linking, may implicate
copyright concerns.

Bundled Wheels on Linux

While we acknowledge many approaches for dealing with third-party
library dependencies within manylinux1 wheels, we recognize that the
manylinux1 policy encourages bundling external dependencies, a practice
which runs counter to the package management policies of many linux
distributions' system package managers[14],[15]. The primary purpose of
this is cross-distro compatibility. Furthermore, manylinux1 wheels on
PyPI occupy a different niche than the Python packages available through
the system package manager.

The decision in this PEP to encourage departure from general Linux
distribution unbundling policies is informed by the following concerns:

1.  In these days of automated continuous integration and deployment
    pipelines, publishing new versions and updating dependencies is
    easier than it was when those policies were defined.
2.  pip users remain free to use the "--no-binary" option if they want
    to force local builds rather than using pre-built wheel files.
3.  The popularity of modern container based deployment and "immutable
    infrastructure" models involve substantial bundling at the
    application layer anyway.
4.  Distribution of bundled wheels through PyPI is currently the norm
    for Windows and OS X.
5.  This PEP doesn't rule out the idea of offering more targeted
    binaries for particular Linux distributions in the future.

The model described in this PEP is most ideally suited for
cross-platform Python packages, because it means they can reuse much of
the work that they're already doing to make static Windows and OS X
wheels. We recognize that it is less optimal for Linux-specific packages
that might prefer to interact more closely with Linux's unique package
management functionality and only care about targeting a small set of
particular distos.

Security Implications

One of the advantages of dependencies on centralized libraries in Linux
is that bugfixes and security updates can be deployed system-wide, and
applications which depend on these libraries will automatically feel the
effects of these patches when the underlying libraries are updated. This
can be particularly important for security updates in packages engaged
in communication across the network or cryptography.

manylinux1 wheels distributed through PyPI that bundle security-critical
libraries like OpenSSL will thus assume responsibility for prompt
updates in response disclosed vulnerabilities and patches. This closely
parallels the security implications of the distribution of binary wheels
on Windows that, because the platform lacks a system package manager,
generally bundle their dependencies. In particular, because it lacks a
stable ABI, OpenSSL cannot be included in the manylinux1 profile.

Platform Detection for Installers

Above, we defined what it means for a wheel to be manylinux1-compatible.
Here we discuss what it means for a Python installation to be
manylinux1-compatible. In particular, this is important for tools like
pip to know when deciding whether or not they should consider
manylinux1-tagged wheels for installation.

Because the manylinux1 profile is already known to work for the many
thousands of users of popular commercial Python distributions, we
suggest that installation tools should error on the side of assuming
that a system is compatible, unless there is specific reason to think
otherwise.

We know of four main sources of potential incompatibility that are
likely to arise in practice:

-   Eventually, in the future, there may exist distributions that break
    compatibility with this profile (e.g., if one of the libraries in
    the profile changes its ABI in a backwards-incompatible way)
-   A linux distribution that is too old (e.g. RHEL 4)
-   A linux distribution that does not use glibc (e.g. Alpine Linux,
    which is based on musl libc, or Android)

To address these we propose a two-pronged approach. To handle potential
future incompatibilities, we standardize a mechanism for a Python
distributor to signal that a particular Python install definitely is or
is not compatible with manylinux1: this is done by installing a module
named _manylinux, and setting its manylinux1_compatible attribute. We do
not propose adding any such module to the standard library -- this is
merely a well-known name by which distributors and installation tools
can rendezvous. However, if a distributor does add this module, they
should add it to the standard library rather than to a site-packages/
directory, because the standard library is inherited by virtualenvs
(which we want), and site-packages/ in general is not.

Then, to handle the last two cases for existing Python distributions, we
suggest a simple and reliable method to check for the presence and
version of glibc (basically using it as a "clock" for the overall age of
the distribution).

Specifically, the algorithm we propose is:

    def is_manylinux1_compatible():
        # Only Linux, and only x86-64 / i686
        from distutils.util import get_platform
        if get_platform() not in ["linux-x86_64", "linux-i686"]:
            return False

        # Check for presence of _manylinux module
        try:
            import _manylinux
            return bool(_manylinux.manylinux1_compatible)
        except (ImportError, AttributeError):
            # Fall through to heuristic check below
            pass

        # Check glibc version. CentOS 5 uses glibc 2.5.
        return have_compatible_glibc(2, 5)

    def have_compatible_glibc(major, minimum_minor):
        import ctypes

        process_namespace = ctypes.CDLL(None)
        try:
            gnu_get_libc_version = process_namespace.gnu_get_libc_version
        except AttributeError:
            # Symbol doesn't exist -> therefore, we are not linked to
            # glibc.
            return False

        # Call gnu_get_libc_version, which returns a string like "2.5".
        gnu_get_libc_version.restype = ctypes.c_char_p
        version_str = gnu_get_libc_version()
        # py2 / py3 compatibility:
        if not isinstance(version_str, str):
            version_str = version_str.decode("ascii")

        # Parse string and check against requested version.
        version = [int(piece) for piece in version_str.split(".")]
        assert len(version) == 2
        if major != version[0]:
            return False
        if minimum_minor > version[1]:
            return False
        return True

Rejected alternatives: We also considered using a configuration file,
e.g. /etc/python/compatibility.cfg. The problem with this is that a
single filesystem might contain many different interpreter environments,
each with their own ABI profile -- the manylinux1 compatibility of a
system-installed x86_64 CPython might not tell us much about the
manylinux1 compatibility of a user-installed i686 PyPy. Locating this
configuration information within the Python environment itself ensures
that it remains attached to the correct binary, and dramatically
simplifies lookup code.

We also considered using a more elaborate structure, like a list of all
platform tags that should be considered compatible, together with their
preference ordering, for example:
_binary_compat.compatible = ["manylinux1_x86_64", "centos5_x86_64", "linux_x86_64"].
However, this introduces several complications. For example, we want to
be able to distinguish between the state of "doesn't support manylinux1"
(or eventually manylinux2, etc.) versus "doesn't specify either way
whether it supports manylinux1", which is not entirely obvious in the
above representation; and, it's not at all clear what features are
really needed vis a vis preference ordering given that right now the
only possible platform tags are manylinux1 and linux. So we're deferring
a more complete solution here for a separate PEP, when / if Linux gets
more platform tags.

For the library compatibility check, we also considered much more
elaborate checks (e.g. checking the kernel version, searching for and
checking the versions of all the individual libraries listed in the
manylinux1 profile, etc.), but ultimately decided that this would be
more likely to introduce confusing bugs than actually help the user.
(For example: different distributions vary in where they actually put
these libraries, and if our checking code failed to use the correct path
search then it could easily return incorrect answers.)

PyPI Support

PyPI should permit wheels containing the manylinux1 platform tag to be
uploaded. PyPI should not attempt to formally verify that wheels
containing the manylinux1 platform tag adhere to the manylinux1 policy
described in this document. This verification tasks should be left to
other tools, like auditwheel, that are developed separately.

Rejected Alternatives

One alternative would be to provide separate platform tags for each
Linux distribution (and each version thereof), e.g. RHEL6, ubuntu14_10,
debian_jessie, etc. Nothing in this proposal rules out the possibility
of adding such platform tags in the future, or of further extensions to
wheel metadata that would allow wheels to declare dependencies on
external system-installed packages. However, such extensions would
require substantially more work than this proposal, and still might not
be appreciated by package developers who would prefer not to have to
maintain multiple build environments and build multiple wheels in order
to cover all the common Linux distributions. Therefore, we consider such
proposals to be out-of-scope for this PEP.

Future updates

We anticipate that at some point in the future there will be a
manylinux2 specifying a more modern baseline environment (perhaps based
on CentOS 6), and someday a manylinux3 and so forth, but we defer
specifying these until we have more experience with the initial
manylinux1 proposal.

References

Copyright

This document has been placed into the public domain.

[1] Enthought Canopy Python Distribution
(https://store.enthought.com/downloads/)

[2] Continuum Analytics Anaconda Python Distribution
(https://www.continuum.io/downloads)

[3] CentOS 5.11 Release Notes
(https://wiki.centos.org/Manuals/ReleaseNotes/CentOS5.11)

[4] Enthought Canopy Python Distribution
(https://store.enthought.com/downloads/)

[5] Continuum Analytics Anaconda Python Distribution
(https://www.continuum.io/downloads)

[6] manylinux-discuss mailing list discussion
(https://groups.google.com/forum/#!topic/manylinux-discuss/-4l3rrjfr9U)

[7] distutils-sig discussion
(https://mail.python.org/pipermail/distutils-sig/2016-January/027997.html)

[8] distutils-sig discussion
(https://mail.python.org/pipermail/distutils-sig/2016-February/028275.html)

[9] github issue discussion
(https://github.com/pypa/manylinux/issues/30)

[10] python bug tracker discussion (https://bugs.python.org/issue21536)

[11] numpy bug report:
https://github.com/numpy/numpy/issues/8415#issuecomment-269095235

[12] manylinux1 docker images (Source:
https://github.com/pypa/manylinux; x86-64:
https://quay.io/repository/pypa/manylinux1_x86_64; x86-32:
https://quay.io/repository/pypa/manylinux1_i686)

[13] auditwheel tool (https://pypi.python.org/pypi/auditwheel)

[14] Fedora Bundled Software Policy
(https://fedoraproject.org/wiki/Bundled_Software_policy)

[15] Debian Policy Manual -- 4.13: Convenience copies of code
(https://www.debian.org/doc/debian-policy/ch-source.html#s-embeddedfiles)

license: Public Domain

authors:
Nathaniel J. Smith
Robert T. McGibbon