text:
PEP: 493 Title: HTTPS verification migration tools for Python 2.7
Version: $Revision$ Last-Modified: $Date$ Author: Alyssa Coghlan
<EMAIL_ADDRESS>Robert Kuska<EMAIL_ADDRESS>Marc-André
Lemburg<EMAIL_ADDRESS>BDFL-Delegate: Barry Warsaw Status: Final
Type: Standards Track Content-Type: text/x-rst Created: 10-May-2015
Python-Version: 2.7.12 Post-History: 06-Jul-2015, 11-Nov-2015,
24-Nov-2015, 24-Feb-2016 Resolution:
https://mail.python.org/pipermail/python-dev/2016-March/143450.html

Abstract

PEP 476 updated Python's default handling of HTTPS certificates in
client modules to align with certificate handling in web browsers, by
validating that the certificates received belonged to the server the
client was attempting to contact. The Python 2.7 long term maintenance
series was judged to be in scope for this change, with the new behaviour
introduced in the Python 2.7.9 maintenance release.

This has created a non-trivial barrier to adoption for affected Python
2.7 maintenance releases, so this PEP proposes additional Python 2.7
specific features that allow system administrators and other users to
more easily decouple the decision to verify server certificates in HTTPS
client modules from the decision to update to newer Python 2.7
maintenance releases.

Rationale

PEP 476 changed Python's default behaviour to align with expectations
established by web browsers in regards to the semantics of HTTPS URLs:
starting with Python 2.7.9 and 3.4.3, HTTPS clients in the standard
library validate server certificates by default.

However, it is also the case that this change does cause problems for
infrastructure administrators operating private intranets that rely on
self-signed certificates, or otherwise encounter problems with the new
default certificate verification settings.

To manage these kinds of situations, web browsers provide users with
"click through" warnings that allow the user to add the server's
certificate to the browser's certificate store. Network client tools
like curl and wget offer options to switch off certificate checking
entirely (by way of curl --insecure and wget --no-check-certificate,
respectively).

At a different layer of the technology stack, Linux security modules
like SELinux and AppArmor, while enabled by default by distribution
vendors, offer relatively straightforward mechanisms for turning them
off.

At the moment, no such convenient mechanisms exist to disable Python's
default certificate checking for a whole process.

PEP 476 did attempt to address this question, by covering how to revert
to the old settings process wide by monkeypatching the ssl module to
restore the old behaviour. Unfortunately, the sitecustomize.py based
technique proposed to allow system administrators to disable the feature
by default in their Standard Operating Environment definition has been
determined to be insufficient in at least some cases. The specific case
that led to the initial creation of this PEP is the one where a Linux
distributor aims to provide their users with a smoother migration path
than the standard one provided by consuming upstream CPython 2.7
releases directly, but other potential challenges have also been pointed
out with updating embedded Python runtimes and other user level
installations of Python.

Rather than allowing a plethora of mutually incompatible migration
techniques to bloom, this PEP proposes an additional feature to be added
to Python 2.7.12 to make it easier to revert a process to the past
behaviour of skipping certificate validation in HTTPS client modules. It
also provides additional recommendations to redistributors backporting
these features to versions of Python prior to Python 2.7.9.

Alternatives

In the absence of clear upstream guidance and recommendations,
commercial redistributors will still make their own design decisions in
the interests of their customers. The main approaches available are:

-   Continuing to rebase on new Python 2.7.x releases, while providing
    no additional assistance beyond the mechanisms defined in PEP 476 in
    migrating from unchecked to checked hostnames in standard library
    HTTPS clients
-   Gating availability of the changes in default handling of HTTPS
    connections on upgrading from Python 2 to Python 3
-   For Linux distribution vendors, gating availability of the changes
    in default handling of HTTPS connections on upgrading to a new
    operating system version
-   Implementing one or both of the backport suggestions described in
    this PEP, regardless of the formal status of the PEP

Scope Limitations

These changes are being proposed purely as tools for helping to manage
the transition to the new default certificate handling behaviour in the
context of Python 2.7. They are not being proposed as new features for
Python 3, as it is expected that the vast majority of client
applications affected by this problem without the ability to update the
application itself will be Python 2 applications.

It would likely be desirable for a future version of Python 3 to allow
the default certificate handling for secure protocols to be configurable
on a per-protocol basis, but that question is beyond the scope of this
PEP.

Requirements for capability detection

As the proposals in this PEP aim to facilitate backports to earlier
Python versions, the Python version number cannot be used as a reliable
means for detecting them. Instead, they are designed to allow the
presence or absence of the feature to be determined using the following
technique:

    python -c "import ssl; ssl.<_relevant_attribute>"

This will fail with AttributeError (and hence a non-zero return code) if
the relevant capability is not available.

The feature detection attributes defined by this PEP are:

-   ssl._https_verify_certificates: runtime configuration API
-   ssl._https_verify_envvar: environment based configuration
-   ssl._cert_verification_config: file based configuration (PEP 476
    opt-in)

The marker attributes are prefixed with an underscore to indicate the
implementation dependent and security sensitive nature of these
capabilities.

Feature: Configuration API

This change is proposed for inclusion in CPython 2.7.12 and later
CPython 2.7.x releases. It consists of a new
ssl._https_verify_certificates() to specify the default handling of
HTTPS certificates in standard library client libraries.

It is not proposed to forward port this change to Python 3, so Python 3
applications that need to support skipping certificate verification will
still need to define their own suitable security context.

Feature detection

The marker attribute on the ssl module related to this feature is the
ssl._https_verify_certificates function itself.

Specification

The ssl._https_verify_certificates function will work as follows:

    def _https_verify_certificates(enable=True):
        """Verify server HTTPS certificates by default?"""
        global _create_default_https_context
        if enable:
            _create_default_https_context = create_default_context
        else:
            _create_default_https_context = _create_unverified_context

If called without arguments, or with enable set to a true value, then
standard library client modules will subsequently verify HTTPS
certificates by default, otherwise they will skip verification.

If called with enable set to a false value, then standard library client
modules will subsequently skip verifying HTTPS certificates by default.

Security Considerations

The inclusion of this feature will allow security sensitive applications
to include the following forward-compatible snippet in their code:

    if hasattr(ssl, "_https_verify_certificates"):
        ssl._https_verify_certificates()

Some developers may also choose to opt out of certificate checking using
ssl._https_verify_certificates(enable=False). This doesn't introduce any
major new security concerns, as monkeypatching the affected internal
APIs was already possible.

Feature: environment based configuration

This change is proposed for inclusion in CPython 2.7.12 and later
CPython 2.7.x releases. It consists of a new PYTHONHTTPSVERIFY
environment variable that can be set to '0' to disable the default
verification without modifying the application source code (which may
not even be available in cases of bytecode-only application
distribution)

It is not proposed to forward port this change to Python 3, so Python 3
applications that need to support skipping certificate verification will
still need to define their own suitable security context.

Feature detection

The marker attribute on the ssl module related to this feature is:

-   the ssl._https_verify_envvar attribute, giving the name of
    environment variable affecting the default behaviour

This not only makes it straightforward to detect the presence (or
absence) of the capability, it also makes it possible to
programmatically determine the relevant environment variable name.

Specification

Rather than always defaulting to the use of ssl.create_default_context,
the ssl module will be modified to:

-   read the PYTHONHTTPSVERIFY environment variable when the module is
    first imported into a Python process
-   set the ssl._create_default_https_context function to be an alias
    for ssl._create_unverified_context if this environment variable is
    present and set to '0'
-   otherwise, set the ssl._create_default_https_context function to be
    an alias for ssl.create_default_context as usual

Example implementation

    _https_verify_envvar = 'PYTHONHTTPSVERIFY'

    def _get_https_context_factory():
        if not sys.flags.ignore_environment:
            config_setting = os.environ.get(_https_verify_envvar)
            if config_setting == '0':
                return _create_unverified_context
        return create_default_context

    _create_default_https_context = _get_https_context_factory()

Security Considerations

Relative to the behaviour in Python 3.4.3+ and Python 2.7.9->2.7.11,
this approach does introduce a new downgrade attack against the default
security settings that potentially allows a sufficiently determined
attacker to revert Python to the default behaviour used in CPython 2.7.8
and earlier releases.

This slight increase in the available attack surface is a key reason
why:

-   security sensitive applications should still define their own SSL
    context
-   the migration features described in this PEP are not being added to
    Python 3

However, it's also worth keeping in mind that carrying out such an
attack requires the ability to modify the execution environment of a
Python process prior to the import of the ssl module. In combination
with the ability to write to any part of the filesystem (such as /tmp),
any attacker with such access would already be able to modify the
behaviour of the underlying OpenSSL implementation, the dynamic library
loader, and other potentially security sensitive components.

Interaction with Python virtual environments

The default setting is read directly from the process environment, and
hence works the same way regardless of whether or not the interpreter is
being run inside an activated Python virtual environment.

Reference Implementation

A patch for Python 2.7 implementing the above two features is attached
to the relevant tracker issue.

Backporting this PEP to earlier Python versions

If this PEP is accepted, then commercial Python redistributors may
choose to backport the per-process configuration mechanisms defined in
this PEP to base versions older than Python 2.7.9, without also
backporting PEP 476's change to the default behaviour of the overall
Python installation.

Such a backport would differ from the mechanism proposed in this PEP
solely in the default behaviour when PYTHONHTTPSVERIFY was not set at
all: it would continue to default to skipping certificate validation.

In this case, if the PYTHONHTTPSVERIFY environment variable is defined,
and set to anything other than '0', then HTTPS certificate verification
should be enabled.

Feature detection

There's no specific attribute indicating that this situation applies.
Rather, it is indicated by the ssl._https_verify_certificates and
ssl._https_verify_envvar attributes being present in a Python version
that is nominally older than Python 2.7.12.

Specification

Implementing this backport involves backporting the changes in PEP 466,
476 and this PEP, with the following change to the handling of the
PYTHONHTTPSVERIFY environment variable in the ssl module:

-   read the PYTHONHTTPSVERIFY environment variable when the module is
    first imported into a Python process
-   set the ssl._create_default_https_context function to be an alias
    for ssl.create_default_context if this environment variable is
    present and set to any value other than '0'
-   otherwise, set the ssl._create_default_https_context function to be
    an alias for ssl._create_unverified_context

Example implementation

    _https_verify_envvar = 'PYTHONHTTPSVERIFY'

    def _get_https_context_factory():
        if not sys.flags.ignore_environment:
            config_setting = os.environ.get(_https_verify_envvar)
            if config_setting != '0':
                return create_default_context
        return _create_unverified_context

    _create_default_https_context = _get_https_context_factory()

    def _disable_https_default_verification():
        """Skip verification of HTTPS certificates by default"""
        global _create_default_https_context
        _create_default_https_context = _create_unverified_context

Security Considerations

This change would be a strict security upgrade for any Python version
that currently defaults to skipping certificate validation in standard
library HTTPS clients. The technical trade-offs to be taken into account
relate largely to the magnitude of the PEP 466 backport also required
rather than to anything security related.

Interaction with Python virtual environments

The default setting is read directly from the process environment, and
hence works the same way regardless of whether or not the interpreter is
being run inside an activated Python virtual environment.

Backporting PEP 476 to earlier Python versions

The backporting approach described above leaves the default HTTPS
certificate verification behaviour of a Python 2.7 installation
unmodified: verifying certificates still needs to be opted into on a
per-connection or per-process basis.

To allow the default behaviour of the entire installation to be modified
without breaking backwards compatibility, Red Hat designed a
configuration mechanism for the system Python 2.7 installation in Red
Hat Enterprise Linux 7.2+ that provides:

-   an opt-in model that allows the decision to enable HTTPS certificate
    verification to be made independently of the decision to upgrade to
    the operating system version where the feature was first backported
-   the ability for system administrators to set the default behaviour
    of Python applications and scripts run directly in the system Python
    installation
-   the ability for the redistributor to consider changing the default
    behaviour of new installations at some point in the future without
    impacting existing installations that have been explicitly
    configured to skip verifying HTTPS certificates by default

As it only affects backports to earlier releases of Python 2.7, this
change is not proposed for inclusion in upstream CPython, but rather is
offered as a recommendation to other redistributors that choose to offer
a similar feature to their users.

This PEP doesn't take a position on whether or not this particular
change is a good idea - rather, it suggests that if a redistributor
chooses to go down the path of making the default behaviour configurable
in a version of Python older than Python 2.7.9, then maintaining a
consistent approach across redistributors would be beneficial for users.

However, this approach SHOULD NOT be used for any Python installation
that advertises itself as providing Python 2.7.9 or later, as most
Python users will have the reasonable expectation that all such
environments will verify HTTPS certificates by default.

Feature detection

The marker attribute on the ssl module related to this feature is:

    _cert_verification_config = '<path to configuration file>'

This not only makes it straightforward to detect the presence (or
absence) of the capability, it also makes it possible to
programmatically determine the relevant configuration file name.

Recommended modifications to the Python standard library

The recommended approach to backporting the PEP 476 modifications to an
earlier point release is to implement the following changes relative to
the default PEP 476 behaviour implemented in Python 2.7.9+:

-   modify the ssl module to read a system wide configuration file when
    the module is first imported into a Python process
-   define a platform default behaviour (either verifying or not
    verifying HTTPS certificates) to be used if this configuration file
    is not present
-   support selection between the following three modes of operation:
    -   ensure HTTPS certificate verification is enabled
    -   ensure HTTPS certificate verification is disabled
    -   delegate the decision to the redistributor providing this Python
        version
-   set the ssl._create_default_https_context function to be an alias
    for either ssl.create_default_context or
    ssl._create_unverified_context based on the given configuration
    setting.

Recommended file location

As the PEP authors are not aware of any vendors providing long-term
support releases targeting Windows, Mac OS X or *BSD systems, this
approach is currently only specifically defined for Linux system Python
installations.

The recommended configuration file name on Linux systems is
/etc/python/cert-verification.cfg.

The .cfg filename extension is recommended for consistency with the
pyvenv.cfg used by the venv module in Python 3's standard library.

Recommended file format

The configuration file should use a ConfigParser ini-style format with a
single section named [https] containing one required setting verify.

The suggested section name is taken from the "https" URL schema passed
to affected client APIs.

Permitted values for verify are:

-   enable: ensure HTTPS certificate verification is enabled by default
-   disable: ensure HTTPS certificate verification is disabled by
    default
-   platform_default: delegate the decision to the redistributor
    providing this particular Python version

If the [https] section or the verify setting are missing, or if the
verify setting is set to an unknown value, it should be treated as if
the configuration file is not present.

Example implementation

    _cert_verification_config = '/etc/python/cert-verification.cfg'

    def _get_https_context_factory():
        # Check for a system-wide override of the default behaviour
        context_factories = {
            'enable': create_default_context,
            'disable': _create_unverified_context,
            'platform_default': _create_unverified_context, # For now :)
        }
        import ConfigParser
        config = ConfigParser.RawConfigParser()
        config.read(_cert_verification_config)
        try:
            verify_mode = config.get('https', 'verify')
        except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
            verify_mode = 'platform_default'
        default_factory = context_factories.get('platform_default')
        return context_factories.get(verify_mode, default_factory)

    _create_default_https_context = _get_https_context_factory()

Security Considerations

The specific recommendations for this backporting case are designed to
work for privileged, security sensitive processes, even those being run
in the following locked down configuration:

-   run from a locked down administrator controlled directory rather
    than a normal user directory (preventing sys.path[0] based privilege
    escalation attacks)
-   run using the -E switch (preventing PYTHON* environment variable
    based privilege escalation attacks)
-   run using the -s switch (preventing user site directory based
    privilege escalation attacks)
-   run using the -S switch (preventing sitecustomize based privilege
    escalation attacks)

The intent is that the only reason HTTPS verification should be getting
turned off installation wide when using this approach is because:

-   an end user is running a redistributor provided version of CPython
    rather than running upstream CPython directly
-   that redistributor has decided to provide a smoother migration path
    to verifying HTTPS certificates by default than that being provided
    by the upstream project
-   either the redistributor or the local infrastructure administrator
    has determined that it is appropriate to retain the default
    pre-2.7.9 behaviour (at least for the time being)

Using an administrator controlled configuration file rather than an
environment variable has the essential feature of providing a smoother
migration path, even for applications being run with the -E switch.

Interaction with Python virtual environments

This setting is scoped by the interpreter installation and affects all
Python processes using that interpreter, regardless of whether or not
the interpreter is being run inside an activated Python virtual
environment.

Origins of this recommendation

This recommendation is based on the backporting approach adopted for Red
Hat Enterprise Linux 7.2, as published in the original July 2015 draft
of this PEP and described in detail in this KnowledgeBase article. Red
Hat's patches implementing this backport for Python 2.7.5 can be found
in the CentOS git repository.

Recommendation for combined feature backports

If a redistributor chooses to backport the environment variable based
configuration setting from this PEP to a modified Python version that
also implements the configuration file based PEP 476 backport, then the
environment variable should take precedence over the system-wide
configuration setting. This allows the setting to be changed for a given
user or application, regardless of the installation-wide default
behaviour.

Example implementation

    _https_verify_envvar = 'PYTHONHTTPSVERIFY'
    _cert_verification_config = '/etc/python/cert-verification.cfg'

    def _get_https_context_factory():
        # Check for an environmental override of the default behaviour
        if not sys.flags.ignore_environment:
            config_setting = os.environ.get(_https_verify_envvar)
            if config_setting is not None:
                if config_setting == '0':
                    return _create_unverified_context
                return create_default_context

        # Check for a system-wide override of the default behaviour
        context_factories = {
            'enable': create_default_context,
            'disable': _create_unverified_context,
            'platform_default': _create_unverified_context, # For now :)
        }
        import ConfigParser
        config = ConfigParser.RawConfigParser()
        config.read(_cert_verification_config)
        try:
            verify_mode = config.get('https', 'verify')
        except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
            verify_mode = 'platform_default'
        default_factory = context_factories.get('platform_default')
        return context_factories.get(verify_mode, default_factory)

    _create_default_https_context = _get_https_context_factory()

Copyright

This document has been placed into the public domain.


text:
PEP: 261 Title: Support for "wide" Unicode characters Version:
$Revision$ Last-Modified: $Date$ Author: Paul Prescod<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type: text/x-rst Created:
27-Jun-2001 Python-Version: 2.2 Post-History: 27-Jun-2001

Abstract

Python 2.1 unicode characters can have ordinals only up to 2**16 - 1.
This range corresponds to a range in Unicode known as the Basic
Multilingual Plane. There are now characters in Unicode that live on
other "planes". The largest addressable character in Unicode has the
ordinal 17 * 2**16 - 1 (0x10ffff). For readability, we will call this
TOPCHAR and call characters in this range "wide characters".

Glossary

Character

    Used by itself, means the addressable units of a Python Unicode
    string.

Code point

    A code point is an integer between 0 and TOPCHAR. If you imagine
    Unicode as a mapping from integers to characters, each integer is a
    code point. But the integers between 0 and TOPCHAR that do not map
    to characters are also code points. Some will someday be used for
    characters. Some are guaranteed never to be used for characters.

Codec

    A set of functions for translating between physical encodings (e.g.
    on disk or coming in from a network) into logical Python objects.

Encoding

    Mechanism for representing abstract characters in terms of physical
    bits and bytes. Encodings allow us to store Unicode characters on
    disk and transmit them over networks in a manner that is compatible
    with other Unicode software.

Surrogate pair

    Two physical characters that represent a single logical character.
    Part of a convention for representing 32-bit code points in terms of
    two 16-bit code points.

Unicode string

    A Python type representing a sequence of code points with "string
    semantics" (e.g. case conversions, regular expression compatibility,
    etc.) Constructed with the unicode() function.

Proposed Solution

One solution would be to merely increase the maximum ordinal to a larger
value. Unfortunately the only straightforward implementation of this
idea is to use 4 bytes per character. This has the effect of doubling
the size of most Unicode strings. In order to avoid imposing this cost
on every user, Python 2.2 will allow the 4-byte implementation as a
build-time option. Users can choose whether they care about wide
characters or prefer to preserve memory.

The 4-byte option is called "wide Py_UNICODE". The 2-byte option is
called "narrow Py_UNICODE".

Most things will behave identically in the wide and narrow worlds.

-   unichr(i) for 0 <= i < 2**16 (0x10000) always returns a length-one
    string.

-   unichr(i) for 2**16 <= i <= TOPCHAR will return a length-one string
    on wide Python builds. On narrow builds it will raise ValueError.

    ISSUE

      Python currently allows \U literals that cannot be represented as
      a single Python character. It generates two Python characters
      known as a "surrogate pair". Should this be disallowed on future
      narrow Python builds?

    Pro:

      Python already the construction of a surrogate pair for a large
      unicode literal character escape sequence. This is basically
      designed as a simple way to construct "wide characters" even in a
      narrow Python build. It is also somewhat logical considering that
      the Unicode-literal syntax is basically a short-form way of
      invoking the unicode-escape codec.

    Con:

      Surrogates could be easily created this way but the user still
      needs to be careful about slicing, indexing, printing etc.
      Therefore, some have suggested that Unicode literals should not
      support surrogates.

    ISSUE

      Should Python allow the construction of characters that do not
      correspond to Unicode code points? Unassigned Unicode code points
      should obviously be legal (because they could be assigned at any
      time). But code points above TOPCHAR are guaranteed never to be
      used by Unicode. Should we allow access to them anyhow?

    Pro:

      If a Python user thinks they know what they're doing why should we
      try to prevent them from violating the Unicode spec? After all, we
      don't stop 8-bit strings from containing non-ASCII characters.

    Con:

      Codecs and other Unicode-consuming code will have to be careful of
      these characters which are disallowed by the Unicode
      specification.

-   ord() is always the inverse of unichr()

-   There is an integer value in the sys module that describes the
    largest ordinal for a character in a Unicode string on the current
    interpreter. sys.maxunicode is 2**16-1 (0xffff) on narrow builds of
    Python and TOPCHAR on wide builds.

    ISSUE:

      Should there be distinct constants for accessing TOPCHAR and the
      real upper bound for the domain of unichr (if they differ)? There
      has also been a suggestion of sys.unicodewidth which can take the
      values 'wide' and 'narrow'.

-   every Python Unicode character represents exactly one Unicode code
    point (i.e. Python Unicode Character = Abstract Unicode character).

-   codecs will be upgraded to support "wide characters" (represented
    directly in UCS-4, and as variable-length sequences in UTF-8 and
    UTF-16). This is the main part of the implementation left to be
    done.

-   There is a convention in the Unicode world for encoding a 32-bit
    code point in terms of two 16-bit code points. These are known as
    "surrogate pairs". Python's codecs will adopt this convention and
    encode 32-bit code points as surrogate pairs on narrow Python
    builds.

    ISSUE

      Should there be a way to tell codecs not to generate surrogates
      and instead treat wide characters as errors?

    Pro:

      I might want to write code that works only with fixed-width
      characters and does not have to worry about surrogates.

    Con:

      No clear proposal of how to communicate this to codecs.

-   there are no restrictions on constructing strings that use code
    points "reserved for surrogates" improperly. These are called
    "isolated surrogates". The codecs should disallow reading these from
    files, but you could construct them using string literals or
    unichr().

Implementation

There is a new define:

    #define Py_UNICODE_SIZE 2

To test whether UCS2 or UCS4 is in use, the derived macro
Py_UNICODE_WIDE should be used, which is defined when UCS-4 is in use.

There is a new configure option:

  ----------------------- -------------------------------------------------------------
  --enable-unicode=ucs2   configures a narrow Py_UNICODE, and uses wchar_t if it fits
  --enable-unicode=ucs4   configures a wide Py_UNICODE, and uses wchar_t if it fits
  --enable-unicode        same as "=ucs2"
  --disable-unicode       entirely remove the Unicode functionality.
  ----------------------- -------------------------------------------------------------

It is also proposed that one day --enable-unicode will just default to
the width of your platforms wchar_t.

Windows builds will be narrow for a while based on the fact that there
have been few requests for wide characters, those requests are mostly
from hard-core programmers with the ability to buy their own Python and
Windows itself is strongly biased towards 16-bit characters.

Notes

This PEP does NOT imply that people using Unicode need to use a 4-byte
encoding for their files on disk or sent over the network. It only
allows them to do so. For example, ASCII is still a legitimate (7-bit)
Unicode-encoding.

It has been proposed that there should be a module that handles
surrogates in narrow Python builds for programmers. If someone wants to
implement that, it will be another PEP. It might also be combined with
features that allow other kinds of character-, word- and line- based
indexing.

Rejected Suggestions

More or less the status-quo

  We could officially say that Python characters are 16-bit and require
  programmers to implement wide characters in their application logic by
  combining surrogate pairs. This is a heavy burden because emulating
  32-bit characters is likely to be very inefficient if it is coded
  entirely in Python. Plus these abstracted pseudo-strings would not be
  legal as input to the regular expression engine.

"Space-efficient Unicode" type

  Another class of solution is to use some efficient storage internally
  but present an abstraction of wide characters to the programmer. Any
  of these would require a much more complex implementation than the
  accepted solution. For instance consider the impact on the regular
  expression engine. In theory, we could move to this implementation in
  the future without breaking Python code. A future Python could
  "emulate" wide Python semantics on narrow Python. Guido is not willing
  to undertake the implementation right now.

Two types

  We could introduce a 32-bit Unicode type alongside the 16-bit type.
  There is a lot of code that expects there to be only a single Unicode
  type.

This PEP represents the least-effort solution. Over the next several
years, 32-bit Unicode characters will become more common and that may
either convince us that we need a more sophisticated solution or (on the
other hand) convince us that simply mandating wide Unicode characters is
an appropriate solution. Right now the two options on the table are do
nothing or do this.

References

Unicode Glossary: http://www.unicode.org/glossary/

Copyright

This document has been placed in the public domain.

license: Public Domain


text:
PEP: 517 Title: A build-system independent format for source trees
Version: $Revision$ Last-Modified: $Date$ Author: Nathaniel J. Smith
<EMAIL_ADDRESS>Thomas Kluyver<EMAIL_ADDRESS>BDFL-Delegate:
Alyssa Coghlan<EMAIL_ADDRESS>Discussions-To:
<EMAIL_ADDRESS>Status: Final Type: Standards Track Topic:
Packaging Content-Type: text/x-rst Created: 30-Sep-2015 Post-History:
01-Oct-2015, 25-Oct-2015, 19-May-2017, 11-Sep-2017 Resolution:
https://mail.python.org/pipermail/distutils-sig/2017-September/031548.html

Abstract

While distutils / setuptools have taken us a long way, they suffer from
three serious problems: (a) they're missing important features like
usable build-time dependency declaration, autoconfiguration, and even
basic ergonomic niceties like DRY-compliant version number management,
and (b) extending them is difficult, so while there do exist various
solutions to the above problems, they're often quirky, fragile, and
expensive to maintain, and yet (c) it's very difficult to use anything
else, because distutils/setuptools provide the standard interface for
installing packages expected by both users and installation tools like
pip.

Previous efforts (e.g. distutils2 or setuptools itself) have attempted
to solve problems (a) and/or (b). This proposal aims to solve (c).

The goal of this PEP is get distutils-sig out of the business of being a
gatekeeper for Python build systems. If you want to use distutils,
great; if you want to use something else, then that should be easy to do
using standardized methods. The difficulty of interfacing with distutils
means that there aren't many such systems right now, but to give a sense
of what we're thinking about see flit or bento. Fortunately, wheels have
now solved many of the hard problems here -- e.g. it's no longer
necessary that a build system also know about every possible
installation configuration -- so pretty much all we really need from a
build system is that it have some way to spit out standard-compliant
wheels and sdists.

We therefore propose a new, relatively minimal interface for
installation tools like pip to interact with package source trees and
source distributions.

Terminology and goals

A source tree is something like a VCS checkout. We need a standard
interface for installing from this format, to support usages like
pip install some-directory/.

A source distribution is a static snapshot representing a particular
release of some source code, like lxml-3.4.4.tar.gz. Source
distributions serve many purposes: they form an archival record of
releases, they provide a stupid-simple de facto standard for tools that
want to ingest and process large corpora of code, possibly written in
many languages (e.g. code search), they act as the input to downstream
packaging systems like Debian/Fedora/Conda/..., and so forth. In the
Python ecosystem they additionally have a particularly important role to
play, because packaging tools like pip are able to use source
distributions to fulfill binary dependencies, e.g. if there is a
distribution foo.whl which declares a dependency on bar, then we need to
support the case where pip install bar or pip install foo automatically
locates the sdist for bar, downloads it, builds it, and installs the
resulting package.

Source distributions are also known as sdists for short.

A build frontend is a tool that users might run that takes arbitrary
source trees or source distributions and builds wheels from them. The
actual building is done by each source tree's build backend. In a
command like pip wheel some-directory/, pip is acting as a build
frontend.

An integration frontend is a tool that users might run that takes a set
of package requirements (e.g. a requirements.txt file) and attempts to
update a working environment to satisfy those requirements. This may
require locating, building, and installing a combination of wheels and
sdists. In a command like pip install lxml==2.4.0, pip is acting as an
integration frontend.

Source trees

There is an existing, legacy source tree format involving setup.py. We
don't try to specify it further; its de facto specification is encoded
in the source code and documentation of distutils, setuptools, pip, and
other tools. We'll refer to it as the setup.py-style.

Here we define a new style of source tree based around the
pyproject.toml file defined in PEP 518, extending the [build-system]
table in that file with one additional key, build-backend. Here's an
example of how it would look:

    [build-system]
    # Defined by PEP 518:
    requires = ["flit"]
    # Defined by this PEP:
    build-backend = "flit.api:main"

build-backend is a string naming a Python object that will be used to
perform the build (see below for details). This is formatted following
the same module:object syntax as a setuptools entry point. For instance,
if the string is "flit.api:main" as in the example above, this object
would be looked up by executing the equivalent of:

    import flit.api
    backend = flit.api.main

It's also legal to leave out the :object part, e.g. :

    build-backend = "flit.api"

which acts like:

    import flit.api
    backend = flit.api

Formally, the string should satisfy this grammar:

    identifier = (letter | '_') (letter | '_' | digit)*
    module_path = identifier ('.' identifier)*
    object_path = identifier ('.' identifier)*
    entry_point = module_path (':' object_path)?

And we import module_path and then lookup module_path.object_path (or
just module_path if object_path is missing).

When importing the module path, we do not look in the directory
containing the source tree, unless that would be on sys.path anyway
(e.g. because it is specified in PYTHONPATH). Although Python
automatically adds the working directory to sys.path in some situations,
code to resolve the backend should not be affected by this.

If the pyproject.toml file is absent, or the build-backend key is
missing, the source tree is not using this specification, and tools
should revert to the legacy behaviour of running setup.py (either
directly, or by implicitly invoking the setuptools.build_meta:__legacy__
backend).

Where the build-backend key exists, this takes precedence and the source
tree follows the format and conventions of the specified backend (as
such no setup.py is needed unless the backend requires it). Projects may
still wish to include a setup.py for compatibility with tools that do
not use this spec.

This PEP also defines a backend-path key for use in pyproject.toml, see
the "In-Tree Build Backends" section below. This key would be used as
follows:

    [build-system]
    # Defined by PEP 518:
    requires = ["flit"]
    # Defined by this PEP:
    build-backend = "local_backend"
    backend-path = ["backend"]

Build requirements

This PEP places a number of additional requirements on the "build
requirements" section of pyproject.toml. These are intended to ensure
that projects do not create impossible to satisfy conditions with their
build requirements.

-   Project build requirements will define a directed graph of
    requirements (project A needs B to build, B needs C and D, etc.)
    This graph MUST NOT contain cycles. If (due to lack of co-ordination
    between projects, for example) a cycle is present, front ends MAY
    refuse to build the project.
-   Where build requirements are available as wheels, front ends SHOULD
    use these where practical, to avoid deeply nested builds. However
    front ends MAY have modes where they do not consider wheels when
    locating build requirements, and so projects MUST NOT assume that
    publishing wheels is sufficient to break a requirement cycle.
-   Front ends SHOULD check explicitly for requirement cycles, and
    terminate the build with an informative message if one is found.

Note in particular that the requirement for no requirement cycles means
that backends wishing to self-host (i.e., building a wheel for a backend
uses that backend for the build) need to make special provision to avoid
causing cycles. Typically this will involve specifying themselves as an
in-tree backend, and avoiding external build dependencies (usually by
vendoring them).

Build backend interface

The build backend object is expected to have attributes which provide
some or all of the following hooks. The common config_settings argument
is described after the individual hooks.

Mandatory hooks

build_wheel

    def build_wheel(wheel_directory, config_settings=None, metadata_directory=None):
        ...

Must build a .whl file, and place it in the specified wheel_directory.
It must return the basename (not the full path) of the .whl file it
creates, as a unicode string.

If the build frontend has previously called
prepare_metadata_for_build_wheel and depends on the wheel resulting from
this call to have metadata matching this earlier call, then it should
provide the path to the created .dist-info directory as the
metadata_directory argument. If this argument is provided, then
build_wheel MUST produce a wheel with identical metadata. The directory
passed in by the build frontend MUST be identical to the directory
created by prepare_metadata_for_build_wheel, including any unrecognized
files it created.

Backends which do not provide the prepare_metadata_for_build_wheel hook
may either silently ignore the metadata_directory parameter to
build_wheel, or else raise an exception when it is set to anything other
than None.

To ensure that wheels from different sources are built the same way,
frontends may call build_sdist first, and then call build_wheel in the
unpacked sdist. But if the backend indicates that it is missing some
requirements for creating an sdist (see below), the frontend will fall
back to calling build_wheel in the source directory.

The source directory may be read-only. Backends should therefore be
prepared to build without creating or modifying any files in the source
directory, but they may opt not to handle this case, in which case
failures will be visible to the user. Frontends are not responsible for
any special handling of read-only source directories.

The backend may store intermediate artifacts in cache locations or
temporary directories. The presence or absence of any caches should not
make a material difference to the final result of the build.

build_sdist

    def build_sdist(sdist_directory, config_settings=None):
        ...

Must build a .tar.gz source distribution and place it in the specified
sdist_directory. It must return the basename (not the full path) of the
.tar.gz file it creates, as a unicode string.

A .tar.gz source distribution (sdist) contains a single top-level
directory called {name}-{version} (e.g. foo-1.0), containing the source
files of the package. This directory must also contain the
pyproject.toml from the build directory, and a PKG-INFO file containing
metadata in the format described in PEP 345. Although historically zip
files have also been used as sdists, this hook should produce a gzipped
tarball. This is already the more common format for sdists, and having a
consistent format makes for simpler tooling.

The generated tarball should use the modern POSIX.1-2001 pax tar format,
which specifies UTF-8 based file names. This is not yet the default for
the tarfile module shipped with Python 3.6, so backends using the
tarfile module need to explicitly pass format=tarfile.PAX_FORMAT.

Some backends may have extra requirements for creating sdists, such as
version control tools. However, some frontends may prefer to make
intermediate sdists when producing wheels, to ensure consistency. If the
backend cannot produce an sdist because a dependency is missing, or for
another well understood reason, it should raise an exception of a
specific type which it makes available as UnsupportedOperation on the
backend object. If the frontend gets this exception while building an
sdist as an intermediate for a wheel, it should fall back to building a
wheel directly. The backend does not need to define this exception type
if it would never raise it.

Optional hooks

get_requires_for_build_wheel

    def get_requires_for_build_wheel(config_settings=None):
        ...

This hook MUST return an additional list of strings containing PEP 508
dependency specifications, above and beyond those specified in the
pyproject.toml file, to be installed when calling the build_wheel or
prepare_metadata_for_build_wheel hooks.

Example:

    def get_requires_for_build_wheel(config_settings):
        return ["wheel >= 0.25", "setuptools"]

If not defined, the default implementation is equivalent to return [].

prepare_metadata_for_build_wheel

    def prepare_metadata_for_build_wheel(metadata_directory, config_settings=None):
        ...

Must create a .dist-info directory containing wheel metadata inside the
specified metadata_directory (i.e., creates a directory like
{metadata_directory}/{package}-{version}.dist-info/). This directory
MUST be a valid .dist-info directory as defined in the wheel
specification, except that it need not contain RECORD or signatures. The
hook MAY also create other files inside this directory, and a build
frontend MUST preserve, but otherwise ignore, such files; the intention
here is that in cases where the metadata depends on build-time
decisions, the build backend may need to record these decisions in some
convenient format for re-use by the actual wheel-building step.

This must return the basename (not the full path) of the .dist-info
directory it creates, as a unicode string.

If a build frontend needs this information and the method is not
defined, it should call build_wheel and look at the resulting metadata
directly.

get_requires_for_build_sdist

    def get_requires_for_build_sdist(config_settings=None):
        ...

This hook MUST return an additional list of strings containing PEP 508
dependency specifications, above and beyond those specified in the
pyproject.toml file. These dependencies will be installed when calling
the build_sdist hook.

If not defined, the default implementation is equivalent to return [].

Note

Editable installs

This PEP originally specified another hook, install_editable, to do an
editable install (as with pip install -e). It was removed due to the
complexity of the topic, but may be specified in a later PEP.

Briefly, the questions to be answered include: what reasonable ways
existing of implementing an 'editable install'? Should the backend or
the frontend pick how to make an editable install? And if the frontend
does, what does it need from the backend to do so.

Config settings

    config_settings

This argument, which is passed to all hooks, is an arbitrary dictionary
provided as an "escape hatch" for users to pass ad-hoc configuration
into individual package builds. Build backends MAY assign any semantics
they like to this dictionary. Build frontends SHOULD provide some
mechanism for users to specify arbitrary string-key/string-value pairs
to be placed in this dictionary. For example, they might support some
syntax like --package-config CC=gcc. In case a user provides duplicate
string-keys, build frontends SHOULD combine the corresponding
string-values into a list of strings. Build frontends MAY also provide
arbitrary other mechanisms for users to place entries in this
dictionary. For example, pip might choose to map a mix of modern and
legacy command line arguments like:

    pip install                                           \
      --package-config CC=gcc                             \
      --global-option="--some-global-option"              \
      --build-option="--build-option1"                    \
      --build-option="--build-option2"

into a config_settings dictionary like:

    {
     "CC": "gcc",
     "--global-option": ["--some-global-option"],
     "--build-option": ["--build-option1", "--build-option2"],
    }

Of course, it's up to users to make sure that they pass options which
make sense for the particular build backend and package that they are
building.

The hooks may be called with positional or keyword arguments, so
backends implementing them should be careful to make sure that their
signatures match both the order and the names of the arguments above.

All hooks are run with working directory set to the root of the source
tree, and MAY print arbitrary informational text on stdout and stderr.
They MUST NOT read from stdin, and the build frontend MAY close stdin
before invoking the hooks.

The build frontend may capture stdout and/or stderr from the backend. If
the backend detects that an output stream is not a terminal/console
(e.g. not sys.stdout.isatty()), it SHOULD ensure that any output it
writes to that stream is UTF-8 encoded. The build frontend MUST NOT fail
if captured output is not valid UTF-8, but it MAY not preserve all the
information in that case (e.g. it may decode using the replace error
handler in Python). If the output stream is a terminal, the build
backend is responsible for presenting its output accurately, as for any
program running in a terminal.

If a hook raises an exception, or causes the process to terminate, then
this indicates an error.

Build environment

One of the responsibilities of a build frontend is to set up the Python
environment in which the build backend will run.

We do not require that any particular "virtual environment" mechanism be
used; a build frontend might use virtualenv, or venv, or no special
mechanism at all. But whatever mechanism is used MUST meet the following
criteria:

-   All requirements specified by the project's build-requirements must
    be available for import from Python. In particular:

    -   The get_requires_for_build_wheel and
        get_requires_for_build_sdist hooks are executed in an
        environment which contains the bootstrap requirements specified
        in the pyproject.toml file.
    -   The prepare_metadata_for_build_wheel and build_wheel hooks are
        executed in an environment which contains the bootstrap
        requirements from pyproject.toml and those specified by the
        get_requires_for_build_wheel hook.
    -   The build_sdist hook is executed in an environment which
        contains the bootstrap requirements from pyproject.toml and
        those specified by the get_requires_for_build_sdist hook.

-   This must remain true even for new Python subprocesses spawned by
    the build environment, e.g. code like:

        import sys, subprocess
        subprocess.check_call([sys.executable, ...])

    must spawn a Python process which has access to all the project's
    build-requirements. This is necessary e.g. for build backends that
    want to run legacy setup.py scripts in a subprocess.

-   All command-line scripts provided by the build-required packages
    must be present in the build environment's PATH. For example, if a
    project declares a build-requirement on flit, then the following
    must work as a mechanism for running the flit command-line tool:

        import subprocess
        import shutil
        subprocess.check_call([shutil.which("flit"), ...])

A build backend MUST be prepared to function in any environment which
meets the above criteria. In particular, it MUST NOT assume that it has
access to any packages except those that are present in the stdlib, or
that are explicitly declared as build-requirements.

Frontends should call each hook in a fresh subprocess, so that backends
are free to change process global state (such as environment variables
or the working directory). A Python library will be provided which
frontends can use to easily call hooks this way.

Recommendations for build frontends (non-normative)

A build frontend MAY use any mechanism for setting up a build
environment that meets the above criteria. For example, simply
installing all build-requirements into the global environment would be
sufficient to build any compliant package -- but this would be
sub-optimal for a number of reasons. This section contains non-normative
advice to frontend implementors.

A build frontend SHOULD, by default, create an isolated environment for
each build, containing only the standard library and any explicitly
requested build-dependencies. This has two benefits:

-   It allows for a single installation run to build multiple packages
    that have contradictory build-requirements. E.g. if package1
    build-requires pbr==1.8.1, and package2 build-requires pbr==1.7.2,
    then these cannot both be installed simultaneously into the global
    environment -- which is a problem when the user requests
    pip install package1 package2. Or if the user already has pbr==1.8.1
    installed in their global environment, and a package build-requires
    pbr==1.7.2, then downgrading the user's version would be rather
    rude.
-   It acts as a kind of public health measure to maximize the number of
    packages that actually do declare accurate build-dependencies. We
    can write all the strongly worded admonitions to package authors we
    want, but if build frontends don't enforce isolation by default,
    then we'll inevitably end up with lots of packages on PyPI that
    build fine on the original author's machine and nowhere else, which
    is a headache that no-one needs.

However, there will also be situations where build-requirements are
problematic in various ways. For example, a package author might
accidentally leave off some crucial requirement despite our best
efforts; or, a package might declare a build-requirement on foo >= 1.0
which worked great when 1.0 was the latest version, but now 1.1 is out
and it has a showstopper bug; or, the user might decide to build a
package against numpy==1.7 -- overriding the package's preferred
numpy==1.8 -- to guarantee that the resulting build will be compatible
at the C ABI level with an older version of numpy (even if this means
the resulting build is unsupported upstream). Therefore, build frontends
SHOULD provide some mechanism for users to override the above defaults.
For example, a build frontend could have a
--build-with-system-site-packages option that causes the
--system-site-packages option to be passed to virtualenv-or-equivalent
when creating build environments, or a
--build-requirements-override=my-requirements.txt option that overrides
the project's normal build-requirements.

The general principle here is that we want to enforce hygiene on package
authors, while still allowing end-users to open up the hood and apply
duct tape when necessary.

In-tree build backends

In certain circumstances, projects may wish to include the source code
for the build backend directly in the source tree, rather than
referencing the backend via the requires key. Two specific situations
where this would be expected are:

-   Backends themselves, which want to use their own features for
    building themselves ("self-hosting backends")
-   Project-specific backends, typically consisting of a custom wrapper
    around a standard backend, where the wrapper is too project-specific
    to be worth distributing independently ("in-tree backends")

Projects can specify that their backend code is hosted in-tree by
including the backend-path key in pyproject.toml. This key contains a
list of directories, which the frontend will add to the start of
sys.path when loading the backend, and running the backend hooks.

There are two restrictions on the content of the backend-path key:

-   Directories in backend-path are interpreted as relative to the
    project root, and MUST refer to a location within the source tree
    (after relative paths and symbolic links have been resolved).
-   The backend code MUST be loaded from one of the directories
    specified in backend-path (i.e., it is not permitted to specify
    backend-path and not have in-tree backend code).

The first restriction is to ensure that source trees remain
self-contained, and cannot refer to locations outside of the source
tree. Frontends SHOULD check this condition (typically by resolving the
location to an absolute path and resolving symbolic links, and then
checking it against the project root), and fail with an error message if
it is violated.

The backend-path feature is intended to support the implementation of
in-tree backends, and not to allow configuration of existing backends.
The second restriction above is specifically to ensure that this is how
the feature is used. Front ends MAY enforce this check, but are not
required to. Doing so would typically involve checking the backend's
__file__ attribute against the locations in backend-path.

Source distributions

We continue with the legacy sdist format, adding some new restrictions.
This format is mostly undefined, but basically comes down to: a file
named {NAME}-{VERSION}.{EXT}, which unpacks into a buildable source tree
called {NAME}-{VERSION}/. Traditionally these have always contained
setup.py-style source trees; we now allow them to also contain
pyproject.toml-style source trees.

Integration frontends require that an sdist named {NAME}-{VERSION}.{EXT}
will generate a wheel named {NAME}-{VERSION}-{COMPAT-INFO}.whl.

The new restrictions for sdists built by PEP 517 backends are:

-   They will be gzipped tar archives, with the .tar.gz extension. Zip
    archives, or other compression formats for tarballs, are not allowed
    at present.
-   Tar archives must be created in the modern POSIX.1-2001 pax tar
    format, which uses UTF-8 for file names.
-   The source tree contained in an sdist is expected to include the
    pyproject.toml file.

Evolutionary notes

A goal here is to make it as simple as possible to convert old-style
sdists to new-style sdists. (E.g., this is one motivation for supporting
dynamic build requirements.) The ideal would be that there would be a
single static pyproject.toml that could be dropped into any "version 0"
VCS checkout to convert it to the new shiny. This is probably not 100%
possible, but we can get close, and it's important to keep track of how
close we are... hence this section.

A rough plan would be: Create a build system package
(setuptools_pypackage or whatever) that knows how to speak whatever hook
language we come up with, and convert them into calls to setup.py. This
will probably require some sort of hooking or monkeypatching to
setuptools to provide a way to extract the setup_requires= argument when
needed, and to provide a new version of the sdist command that generates
the new-style format. This all seems doable and sufficient for a large
proportion of packages (though obviously we'll want to prototype such a
system before we finalize anything here). (Alternatively, these changes
could be made to setuptools itself rather than going into a separate
package.)

But there remain two obstacles that mean we probably won't be able to
automatically upgrade packages to the new format:

1)  There currently exist packages which insist on particular packages
    being available in their environment before setup.py is executed.
    This means that if we decide to execute build scripts in an isolated
    virtualenv-like environment, then projects will need to check
    whether they do this, and if so then when upgrading to the new
    system they will have to start explicitly declaring these
    dependencies (either via setup_requires= or via static declaration
    in pyproject.toml).
2)  There currently exist packages which do not declare consistent
    metadata (e.g. egg_info and bdist_wheel might get different
    install_requires=). When upgrading to the new system, projects will
    have to evaluate whether this applies to them, and if so they will
    need to stop doing that.

Rejected options

-   We discussed making the wheel and sdist hooks build unpacked
    directories containing the same contents as their respective
    archives. In some cases this could avoid the need to pack and unpack
    an archive, but this seems like premature optimisation. It's
    advantageous for tools to work with archives as the canonical
    interchange formats (especially for wheels, where the archive format
    is already standardised). Close control of archive creation is
    important for reproducible builds. And it's not clear that tasks
    requiring an unpacked distribution will be more common than those
    requiring an archive.
-   We considered an extra hook to copy files to a build directory
    before invoking build_wheel. Looking at existing build systems, we
    found that passing a build directory into build_wheel made more
    sense for many tools than pre-emptively copying files into a build
    directory.
-   The idea of passing build_wheel a build directory was then also
    deemed an unnecessary complication. Build tools can use a temporary
    directory or a cache directory to store intermediate files while
    building. If there is a need, a frontend-controlled cache directory
    could be added in the future.
-   For build_sdist to signal a failure for an expected reason, various
    options were debated at great length, including raising
    NotImplementedError and returning either NotImplemented or None.
    Please do not attempt to reopen this discussion without an extremely
    good reason, because we are quite tired of it.
-   Allowing the backend to be imported from files in the source tree
    would be more consistent with the way Python imports often work.
    However, not allowing this prevents confusing errors from clashing
    module names. The initial version of this PEP did not provide a
    means to allow backends to be imported from files within the source
    tree, but the backend-path key was added in the next revision to
    allow projects to opt into this behaviour if needed.

Summary of changes to PEP 517

The following changes were made to this PEP after the initial reference
implementation was released in pip 19.0.

-   Cycles in build requirements were explicitly prohibited.
-   Support for in-tree backends and self-hosting of backends was added
    by the introduction of the backend-path key in the [build-system]
    table.
-   Clarified that the setuptools.build_meta:__legacy__ PEP 517 backend
    is an acceptable alternative to directly invoking setup.py for
    source trees that don't specify build-backend explicitly.

Appendix A: Comparison to PEP 516

PEP 516 is a competing proposal to specify a build system interface,
which has now been rejected in favour of this PEP. The primary
difference is that our build backend is defined via a Python hook-based
interface rather than a command-line based interface.

This appendix documents the arguments advanced for this PEP over PEP
516.

We do not expect that specifying Python hooks rather than command line
interfaces will, by itself, reduce the complexity of calling into the
backend, because build frontends will in any case want to run hooks
inside a child -- this is important to isolate the build frontend itself
from the backend code and to better control the build backends execution
environment. So under both proposals, there will need to be some code in
pip to spawn a subprocess and talk to some kind of command-line/IPC
interface, and there will need to be some code in the subprocess that
knows how to parse these command line arguments and call the actual
build backend implementation. So this diagram applies to all proposals
equally:

    +-----------+          +---------------+           +----------------+
    | frontend  | -spawn-> | child cmdline | -Python-> |    backend     |
    |   (pip)   |          |   interface   |           | implementation |
    +-----------+          +---------------+           +----------------+

The key difference between the two approaches is how these interface
boundaries map onto project structure:

    .-= This PEP =-.

    +-----------+          +---------------+    |      +----------------+
    | frontend  | -spawn-> | child cmdline | -Python-> |    backend     |
    |   (pip)   |          |   interface   |    |      | implementation |
    +-----------+          +---------------+    |      +----------------+
                                                |
    |______________________________________|    |
       Owned by pip, updated in lockstep        |
                                                |
                                                |
                                     PEP-defined interface boundary
                                   Changes here require distutils-sig

    .-= Alternative =-.

    +-----------+    |     +---------------+           +----------------+
    | frontend  | -spawn-> | child cmdline | -Python-> |    backend     |
    |   (pip)   |    |     |   interface   |           | implementation |
    +-----------+    |     +---------------+           +----------------+
                     |
                     |     |____________________________________________|
                     |      Owned by build backend, updated in lockstep
                     |
        PEP-defined interface boundary
      Changes here require distutils-sig

By moving the PEP-defined interface boundary into Python code, we gain
three key advantages.

First, because there will likely be only a small number of build
frontends (pip, and... maybe a few others?), while there will likely be
a long tail of custom build backends (since these are chosen separately
by each package to match their particular build requirements), the
actual diagrams probably look more like:

    .-= This PEP =-.

    +-----------+          +---------------+           +----------------+
    | frontend  | -spawn-> | child cmdline | -Python+> |    backend     |
    |   (pip)   |          |   interface   |        |  | implementation |
    +-----------+          +---------------+        |  +----------------+
                                                    |
                                                    |  +----------------+
                                                    +> |    backend     |
                                                    |  | implementation |
                                                    |  +----------------+
                                                    :
                                                    :

    .-= Alternative =-.

    +-----------+          +---------------+           +----------------+
    | frontend  | -spawn+> | child cmdline | -Python-> |    backend     |
    |   (pip)   |       |  |   interface   |           | implementation |
    +-----------+       |  +---------------+           +----------------+
                        |
                        |  +---------------+           +----------------+
                        +> | child cmdline | -Python-> |    backend     |
                        |  |   interface   |           | implementation |
                        |  +---------------+           +----------------+
                        :
                        :

That is, this PEP leads to less total code in the overall ecosystem. And
in particular, it reduces the barrier to entry of making a new build
system. For example, this is a complete, working build backend:

    # mypackage_custom_build_backend.py
    import os.path
    import pathlib
    import shutil
    import tarfile

    SDIST_NAME = "mypackage-0.1"
    SDIST_FILENAME = SDIST_NAME + ".tar.gz"
    WHEEL_FILENAME = "mypackage-0.1-py2.py3-none-any.whl"

    #################
    # sdist creation
    #################

    def _exclude_hidden_and_special_files(archive_entry):
        """Tarfile filter to exclude hidden and special files from the archive"""
        if archive_entry.isfile() or archive_entry.isdir():
            if not os.path.basename(archive_entry.name).startswith("."):
                return archive_entry

    def _make_sdist(sdist_dir):
        """Make an sdist and return both the Python object and its filename"""
        sdist_path = pathlib.Path(sdist_dir) / SDIST_FILENAME
        sdist = tarfile.open(sdist_path, "w:gz", format=tarfile.PAX_FORMAT)
        # Tar up the whole directory, minus hidden and special files
        sdist.add(os.getcwd(), arcname=SDIST_NAME,
                  filter=_exclude_hidden_and_special_files)
        return sdist, SDIST_FILENAME

    def build_sdist(sdist_dir, config_settings):
        """PEP 517 sdist creation hook"""
        sdist, sdist_filename = _make_sdist(sdist_dir)
        return sdist_filename

    #################
    # wheel creation
    #################

    def get_requires_for_build_wheel(config_settings):
        """PEP 517 wheel building dependency definition hook"""
        # As a simple static requirement, this could also just be
        # listed in the project's build system dependencies instead
        return ["wheel"]

    def build_wheel(wheel_directory,
                    metadata_directory=None, config_settings=None):
        """PEP 517 wheel creation hook"""
        from wheel.archive import archive_wheelfile
        path = os.path.join(wheel_directory, WHEEL_FILENAME)
        archive_wheelfile(path, "src/")
        return WHEEL_FILENAME

Of course, this is a terrible build backend: it requires the user to
have manually set up the wheel metadata in src/mypackage-0.1.dist-info/;
when the version number changes it must be manually updated in multiple
places... but it works, and more features could be added incrementally.
Much experience suggests that large successful projects often originate
as quick hacks (e.g., Linux -- "just a hobby, won't be big and
professional"; IPython/Jupyter -- a grad student's $PYTHONSTARTUP file),
so if our goal is to encourage the growth of a vibrant ecosystem of good
build tools, it's important to minimize the barrier to entry.

Second, because Python provides a simpler yet richer structure for
describing interfaces, we remove unnecessary complexity from the
specification -- and specifications are the worst place for complexity,
because changing specifications requires painful consensus-building
across many stakeholders. In the command-line interface approach, we
have to come up with ad hoc ways to map multiple different kinds of
inputs into a single linear command line (e.g. how do we avoid
collisions between user-specified configuration arguments and
PEP-defined arguments? how do we specify optional arguments? when
working with a Python interface these questions have simple, obvious
answers). When spawning and managing subprocesses, there are many fiddly
details that must be gotten right, subtle cross-platform differences,
and some of the most obvious approaches --e.g., using stdout to return
data for the build_requires operation -- can create unexpected pitfalls
(e.g., what happens when computing the build requirements requires
spawning some child processes, and these children occasionally print an
error message to stdout? obviously a careful build backend author can
avoid this problem, but the most obvious way of defining a Python
interface removes this possibility entirely, because the hook return
value is clearly demarcated).

In general, the need to isolate build backends into their own process
means that we can't remove IPC complexity entirely -- but by placing
both sides of the IPC channel under the control of a single project, we
make it much cheaper to fix bugs in the IPC interface than if fixing
bugs requires coordinated agreement and coordinated changes across the
ecosystem.

Third, and most crucially, the Python hook approach gives us much more
powerful options for evolving this specification in the future.

For concreteness, imagine that next year we add a new
build_sdist_from_vcs hook, which provides an alternative to the current
build_sdist hook where the frontend is responsible for passing version
control tracking metadata to backends (including indicating when all on
disk files are tracked), rather than individual backends having to query
that information themselves. In order to manage the transition, we'd
want it to be possible for build frontends to transparently use
build_sdist_from_vcs when available and fall back onto build_sdist
otherwise; and we'd want it to be possible for build backends to define
both methods, for compatibility with both old and new build frontends.

Furthermore, our mechanism should also fulfill two more goals: (a) If
new versions of e.g. pip and flit are both updated to support the new
interface, then this should be sufficient for it to be used; in
particular, it should not be necessary for every project that uses flit
to update its individual pyproject.toml file. (b) We do not want to have
to spawn extra processes just to perform this negotiation, because
process spawns can easily become a bottleneck when deploying large
multi-package stacks on some platforms (Windows).

In the interface described here, all of these goals are easy to achieve.
Because pip controls the code that runs inside the child process, it can
easily write it to do something like:

    command, backend, args = parse_command_line_args(...)
    if command == "build_sdist":
       if hasattr(backend, "build_sdist_from_vcs"):
           backend.build_sdist_from_vcs(...)
       elif hasattr(backend, "build_sdist"):
           backend.build_sdist(...)
       else:
           # error handling

In the alternative where the public interface boundary is placed at the
subprocess call, this is not possible -- either we need to spawn an
extra process just to query what interfaces are supported (as was
included in an earlier draft of PEP 516, an alternative to this), or
else we give up on autonegotiation entirely (as in the current version
of that PEP), meaning that any changes in the interface will require N
individual packages to update their pyproject.toml files before any
change can go live, and that any changes will necessarily be restricted
to new releases.

One specific consequence of this is that in this PEP, we're able to make
the prepare_metadata_for_build_wheel command optional. In our design,
this can be readily handled by build frontends, which can put code in
their subprocess runner like:

    def dump_wheel_metadata(backend, working_dir):
        """Dumps wheel metadata to working directory.

           Returns absolute path to resulting metadata directory
        """
        if hasattr(backend, "prepare_metadata_for_build_wheel"):
            subdir = backend.prepare_metadata_for_build_wheel(working_dir)
        else:
            wheel_fname = backend.build_wheel(working_dir)
            already_built = os.path.join(working_dir, "ALREADY_BUILT_WHEEL")
            with open(already_built, "w") as f:
                f.write(wheel_fname)
            subdir = unzip_metadata(os.path.join(working_dir, wheel_fname))
        return os.path.join(working_dir, subdir)

    def ensure_wheel_is_built(backend, output_dir, working_dir, metadata_dir):
        """Ensures built wheel is available in output directory

           Returns absolute path to resulting wheel file
        """
        already_built = os.path.join(working_dir, "ALREADY_BUILT_WHEEL")
        if os.path.exists(already_built):
            with open(already_built, "r") as f:
                wheel_fname = f.read().strip()
            working_path = os.path.join(working_dir, wheel_fname)
            final_path = os.path.join(output_dir, wheel_fname)
            os.rename(working_path, final_path)
            os.remove(already_built)
        else:
            wheel_fname = backend.build_wheel(output_dir, metadata_dir=metadata_dir)
        return os.path.join(output_dir, wheel_fname)

and thus expose a totally uniform interface to the rest of the frontend,
with no extra subprocess calls, no duplicated builds, etc. But obviously
this is the kind of code that you only want to write as part of a
private, within-project interface (e.g. the given example requires that
the working directory be shared between the two calls, but not with any
other wheel builds, and that the return value from the metadata helper
function will be passed back in to the wheel building one).

(And, of course, making the metadata command optional is one piece of
lowering the barrier to entry for developing new backends, as discussed
above.)

Other differences

Besides the key command line versus Python hook difference described
above, there are a few other differences in this proposal:

-   Metadata command is optional (as described above).
-   We return metadata as a directory, rather than a single METADATA
    file. This aligns better with the way that in practice wheel
    metadata is distributed across multiple files (e.g. entry points),
    and gives us more options in the future. (For example, instead of
    following the PEP 426 proposal of switching the format of METADATA
    to JSON, we might decide to keep the existing METADATA the way it is
    for backcompat, while adding new extensions as JSON "sidecar" files
    inside the same directory. Or maybe not; the point is it keeps our
    options more open.)
-   We provide a mechanism for passing information between the metadata
    step and the wheel building step. I guess everyone probably will
    agree this is a good idea?
-   We provide more detailed recommendations about the build
    environment, but these aren't normative anyway.

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:


text:
PEP: 582 Title: Python local packages directory Version: $Revision$
Last-Modified: $Date$ Author: Kushal Das<EMAIL_ADDRESS>Steve
Dower<EMAIL_ADDRESS>Donald Stufft<EMAIL_ADDRESS>Alyssa
Coghlan<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pep-582-python-local-packages-directory/963/
Status: Rejected Type: Standards Track Topic: Packaging Content-Type:
text/x-rst Created: 16-May-2018 Python-Version: 3.12 Post-History:
01-Mar-2019, Resolution:
https://discuss.python.org/t/pep-582-python-local-packages-directory/963/430

Abstract

This PEP proposes extending the existing mechanism for setting up
sys.path to include a new __pypackages__ directory, in addition to the
existing locations. The new directory will be added at the start of
sys.path, after the current working directory and just before the system
site-packages, to give packages installed there priority over other
locations.

This is similar to the existing mechanism of adding the current
directory (or the directory the script is located in), but by using a
subdirectory, additional libraries are kept separate from the user's
work.

Motivation

New Python programmers can benefit from being taught the value of
isolating an individual project's dependencies from their system
environment. However, the existing mechanism for doing this, virtual
environments, is known to be complex and error-prone for beginners to
understand. Explaining virtual environments is often a distraction when
trying to get a group of beginners set up - differences in platform and
shell environments require individual assistance, and the need for
activation in every new shell session makes it easy for students to make
mistakes when coming back to work after a break. This proposal offers a
lightweight solution that gives isolation without the user needing to
understand more advanced concepts.

Furthermore, standalone Python applications usually need 3rd party
libraries to function. Typically, they are either designed to be run
from a virtual environment, where the dependencies are installed into
the environment alongside the application, or they bundle their
dependencies in a subdirectory, and modify sys.path at application
startup. Virtual environments, while a common and effective solution
(used, for example, by the pipx tool), are somewhat awkward to set up
and manage, and are not relocatable. On the other hand, manual
manipulation of sys.path is boilerplate that developers need to get
right, and (being a runtime behaviour) it is not understood by tools
like linters and type checkers. The __pypackages__ proposal formalises
the idea of a "bundled dependencies" location, avoiding the boilerplate
and providing a standard location that development tools can be taught
to recognise.

It should be noted that in general, Python libraries cannot be simply
copied between machines, platforms, or even necessarily between Python
versions. This proposal does nothing to change that fact, and while it
is tempting to assume that bundling a script and its __pypackages__ is a
mechanism for distributing applications, this is explicitly not a goal
of this proposal. Developers remain responsible for the portability of
their code.

Rationale

While sys.path can be manipulated at runtime, the default value is
important, as it establishes a common baseline that users and tools can
agree on. The current default does not include a location that could be
viewed as "private to the current project", and yet that is a useful
concept.

This is similar to the npm node_modules directory, which is popular in
the Javascript community, and something that developers familiar with
that ecosystem often ask for from Python.

Specification

This PEP proposes to add a new step in the process of calculating
sys.path at startup.

When the interactive interpreter starts, if a __pypackages__ directory
is found in the current working directory, then it will be included in
sys.path after the entry for current working directory and just before
the system site-packages.

When the interpreter runs a script, Python will try to find
__pypackages__ in the same directory as the script. If found (along with
the current Python version directory inside), then it will be used,
otherwise Python will behave as it does currently.

The behaviour should work exactly the same as the way the existing
mechanism for adding the current working directory or script directory
to sys.path works. For example, __pypackages__ will be ignored if the -P
option or the PYTHONSAFEPATH environment variable is set.

In order to be recognised, the __pypackages__ directory must be laid out
according to a new localpackages scheme in the sysconfig module.
Specifically, both of the purelib and platlib directories must be
present, using the following code to determine the locations of those
directories:

    scheme = "localpackages"
    purelib = sysconfig.get_path("purelib", scheme, vars={"base": "__pypackages__", "platbase": "__pypackages__"})
    platlib = sysconfig.get_path("platlib", scheme, vars={"base": "__pypackages__", "platbase": "__pypackages__"})

These two locations will be added to sys.path, other directories or
files in the __pypackages__ directory will be silently ignored. The
paths will be based on Python versions.

Note

There is a possible option of having a separate new API, it is
documented at issue #3013.

Example

The following shows an example project directory structure, and
different ways the Python executable and any script will behave. The
example is for Unix-like systems - on Windows the subdirectories will be
different.

    foo
        __pypackages__
            lib
                python3.10
                           site-packages
                                         bottle
        myscript.py

    /> python foo/myscript.py
    sys.path[0] == 'foo'
    sys.path[1] == 'foo/__pypackages__/lib/python3.10/site-packages/'

    cd foo

    foo> /usr/bin/ansible
        #! /usr/bin/env python3
    foo> python /usr/bin/ansible

    foo> python myscript.py

    foo> python
    sys.path[0] == '.'
    sys.path[1] == './__pypackages__/lib/python3.10/site-packages'

    foo> python -m bottle

We have a project directory called foo and it has a __pypackages__
inside of it. We have bottle installed in that
__pypackages__/lib/python3.10/site-packages/, and have a myscript.py
file inside of the project directory. We have used whatever tool we
generally use to install bottle in that location.

For invoking a script, Python will try to find a __pypackages__ inside
of the directory that the script resides[1], /usr/bin. The same will
happen in case of the last example, where we are executing
/usr/bin/ansible from inside of the foo directory. In both cases, it
will not use the __pypackages__ in the current working directory.

Similarly, if we invoke myscript.py from the first example, it will use
the __pypackages__ directory that was in the foo directory.

If we go inside of the foo directory and start the Python executable
(the interpreter), it will find the __pypackages__ directory inside of
the current working directory and use it in the sys.path. The same
happens if we try to use the -m and use a module. In our example, bottle
module will be found inside of the __pypackages__ directory.

The above two examples are only cases where __pypackages__ from current
working directory is used.

In another example scenario, a trainer of a Python class can say "Today
we are going to learn how to use Twisted! To start, please checkout our
example project, go to that directory, and then run a given command to
install Twisted."

That will install Twisted into a directory separate from python3.
There's no need to discuss virtual environments, global versus user
installs, etc. as the install will be local by default. The trainer can
then just keep telling them to use python3 without any activation step,
etc.

Relationship to virtual environments

At its heart, this proposal is simply to modify the calculation of the
default value of sys.path, and does not relate at all to the virtual
environment mechanism. However, __pypackages__ can be viewed as
providing an isolation capability, and in that sense, it "competes" with
virtual environments.

However, there are significant differences:

  -   Virtual environments are isolated from the system environment,
      whereas __pypackages__ simply adds to the system environment.
  -   Virtual environments include a full "installation scheme", with
      directories for binaries, C header files, etc., whereas
      __pypackages__ is solely for Python library code.
  -   Virtual environments work most smoothly when "activated". This
      proposal needs no activation.

This proposal should be seen as independent of virtual environments, not
competing with them. At best, some use cases currently only served by
virtual environments can also be served (possibly better) by
__pypackages__.

It should be noted that libraries installed in __pypackages__ will be
visible in a virtual environment. This arguably breaks the isolation of
virtual environments, but it is no different in principle to the
presence of the current directory on sys.path (or mechanisms like the
PYTHONPATH environment variable). The only difference is in degree, as
the expectation is that people will more commonly install packages in
__pypackages__. The alternative would be to explicitly detect virtual
environments and disable __pypackages__ in that case - however that
would break scripts with bundled dependencies. The PEP authors believe
that developers using virtual environments should be experienced enough
to understand the issue and anticipate and avoid any problems.

Security Considerations

In theory, it is possible to add a library to the __pypackages__
directory that overrides a stdlib module or an installed 3rd party
library. For the __pypackages__ associated with a script, this is
assumed not to be a significant issue, as it is unlikely that anyone
would be able to write to __pypackages__ unless they also had the
ability to write to the script itself.

For a __pypackages__ directory in the current working directory, the
interactive interpreter could be affected. However, this is not
significantly different than the existing issue of someone having a
math.py module in their current directory, and while (just like that
case) it can cause user confusion, it does not introduce any new
security implications.

When running a script, any __pypackages__ directory in the current
working directory is ignored. This is the same approach Python uses for
adding the current working directory to sys.path and ensures that it is
not possible to change the behaviour of a script by modifying files in
the current directory.

Also, a __pypackages__ directory is only recognised in the current (or
script) directory. The interpreter will not scan for __pypackages__ in
parent directories. Doing so would open up the risk of security issues
if directory permissions on parents differ. In particular, scripts in
the bin directory or __pypackages__ (the scripts location in sysconfig
terms) have no special access to the libraries installed in
__pypackages__. Putting executable scripts in a bin directory is not
supported by this proposal.

How to Teach This

The original motivation for this proposal was to make it easier to teach
Python to beginners. To that end, it needs to be easy to explain, and
simple to use.

At the most basic level, this is similar to the existing mechanism where
the script directory is added to sys.path and can be taught in a similar
manner. However, for its intended use of "lightweight isolation", it
would likely be taught in terms of "things you put in a __pypackages__
directory are private to your script". The experience of the PEP authors
suggests that this would be significantly easier to teach than the
current alternative of introducing virtual environments.

Impact on Tools

As the intended use of the feature is to install 3rd party libraries in
the new directory, it is important that tools, particularly installers,
understand how to manage __pypackages__.

It is hoped that tools will introduce a dedicated "pypackages"
installation mode that is guaranteed to match the expected layout in all
cases. However, the question of how best to support the __pypackages__
layout is ultimately left to individual tool maintainers to consider and
decide on.

Tools that locate packages without actually running Python code (IDEs,
linters, type checkers, etc.) would need updating to recognise
__pypackages__. In the absence of such updates, the __pypackages__
directory would work similarly to directories currently added to
sys.path at runtime (i.e., the tool would probably ignore it).

Backwards Compatibility

The directory name __pypackages__ was chosen because it is unlikely to
be in common use. It is true that users who have chosen to use that name
for their own purposes will be impacted, but at the time this PEP was
written, this was viewed as a relatively low risk.

Unfortunately, in the time this PEP has been under discussion, a number
of tools have chosen to implement variations on what is being proposed
here, which are not all compatible with the final form of the PEP. As a
result, the risk of clashes is now higher than originally anticipated.

It would be possible to mitigate this by choosing a different name,
hopefully as uncommon as __pypackages__ originally was. But
realistically, any compatibility issues can be viewed as simply the
consequences of people trying to implement draft proposals, without
making the effort to track changes in the proposal. As such, it seems
reasonable to retain the __pypackages__ name, and put the burden of
addressing the compatibility issue on the tools that implemented the
draft version.

Impact on other Python implementations

Other Python implementations will need to replicate the new behavior of
the interpreter bootstrap, including locating the __pypackages__
directory and adding it the sys.path just before site packages, if it is
present. This is no different to any other Python change.

Reference Implementation

Here is a small script which will enable the implementation for Cpython
& in PyPy.

Rejected Ideas

-   Alternative names, such as __pylocal__ and python_modules.
    Ultimately, the name is arbitrary and the chosen name is good
    enough.
-   Additional features of virtual environments. This proposal is not a
    replacement for virtual environments, and such features are
    therefore out of scope.
-   We will not scan any parent directory to find __pypackages__. If we
    want to execute scripts inside of the ~/bin/ directory, then the
    __pypackages__ directory must be inside of the ~/bin/ directory.
    Doing any such scan for __pypackages__ (for the interpreter or a
    script) will have security implications and also increase startup
    time.
-   Raise an error if unexpected files or directories are present in
    __pypackages__. This is considered too strict, particularly as
    transitional approaches like pip install --prefix can create
    additional files in __pypackages__.
-   Using a different sysconfig scheme, or a dedicated pypackages
    scheme. While this is attractive in theory, it makes transition
    harder, as there will be no readily-available way of installing to
    __pypackages__ until tools implement explicit support. And while the
    PEP authors hope and assume that such support would be added, having
    the proposal dependent on such support in order to be usable seems
    like an unacceptable risk.

Copyright

This document has been placed in the public domain.

[1] In the case of symlinks, it is the directory where the actual script
resides, not the symlink pointing to the script

authors:
Kushal Das
Steve Dower