text:
PEP: 706 Title: Filter for tarfile.extractall Author: Petr Viktorin
<EMAIL_ADDRESS>Discussions-To: https://discuss.python.org/t/23903
Status: Final Type: Standards Track Content-Type: text/x-rst Created:
09-Feb-2023 Python-Version: 3.12 Post-History: 25-Jan-2023, 15-Feb-2023,
Resolution: https://discuss.python.org/t/23903/10

tarfile documentation <python:tarfile-extraction-filter>

Abstract

The extraction methods in tarfile gain a filter argument, which allows
rejecting files or modifying metadata as the archive is extracted. Three
built-in named filters are provided, aimed at limiting features that
might be surprising or dangerous. These can be used as-is, or serve as a
base for custom filters.

After a deprecation period, a strict (but safer) filter will become the
default.

Motivation

The tar format is used for several use cases, many of which have
different needs. For example:

-   A backup of a UNIX workstation should faithfully preserve all kinds
    of details like file permissions, symlinks to system configuration,
    and various kinds of special files.
-   When unpacking a data bundle, it’s much more important that the
    unpacking will not have unintended consequences – like exposing a
    password file by symlinking it to a public place.

To support all its use cases, the tar format has many features. In many
cases, it's best to ignore or disallow some of them when extracting an
archive.

Python allows extracting tar archives using tarfile.TarFile.extractall,
whose docs warn to never extract archives from untrusted sources without
prior inspection. However, it’s not clear what kind of inspection should
be done. Indeed, it’s quite tricky to do such an inspection correctly.
As a result, many people don’t bother, or do the check incorrectly,
resulting in security issues such as CVE-2007-4559.

Since tarfile was first written, it's become more accepted that warnings
in documentation are not enough. Whenever possible, an unsafe operation
should be explicitly requested; potentially dangerous operations should
look dangerous. However, TarFile.extractall looks benign in a code
review.

Tarfile extraction is also exposed via shutil.unpack_archive, which
allows the user to not care about the kind of archive they're dealing
with. The API is very inviting for extracting archives without prior
inspection, even though the docs again warn against it.

It has been argued that Python is not wrong -- it behaves exactly as
documented -- but that's beside the point. Let's improve the situation
rather than assign/avoid blame. Python and its docs are the best place
to improve things.

Rationale

How do we improve things? Unfortunately, we will need to change the
defaults, which implies breaking backwards compatibility.
TarFile.extractall <tarfile.TarFile.extractall> is what people reach for
when they need to extract a tarball. Its default behaviour needs to
change.

What would be the best behaviour? That depends on the use case. So,
we'll add several general “policies” to control extraction. They are
based on use cases, and ideally they should have straightforward
security implications:

-   Current behavior: trusting the archive. Suitable e.g. as a building
    block for libraries that do the check themselves, or extracting an
    archive you just made yourself.
-   Unpacking a UNIX archive: roughly following GNU tar, e.g. stripping
    leading / from filenames.
-   Unpacking a general data archive: the shutil.unpack_archive use
    case, where it's not important to preserve details specific to tar
    or Unix-like filesystems.

After a deprecation period, the last option -- the most limited but most
secure one -- will become the default.

Even with better general defaults, users should still verify the
archives they extract, and perhaps modify some of the metadata.
Superficially, the following looks like a reasonable way to do this
today:

-   Call TarFile.getmembers <tarfile.TarFile.getmembers>
-   Verify or modify each member's ~tarfile.TarInfo
-   Pass the result to extractall's members

However, there are some issues with this approach:

-   It's possible to modify TarInfo objects, but the changes to them
    affect all subsequent operations on the same TarFile object. This
    behavior is fine for most uses, but despite that, it would be very
    surprising if TarFile.extractall did this by default.
-   Calling getmembers can be expensive and it requires a seekable
    archive.
-   When verifying members in advance, it may be necessary to track how
    each member would have changed the filesystem, e.g. how symlinks are
    being set up. This is hard. We can't expect users to do it.

To solve these issues we'll:

-   Provide a supported way to “clone” and modify TarInfo objects. A
    replace method, similar to dataclasses.replace or
    namedtuple._replace <collections.somenamedtuple._replace> should do
    the trick.
-   Provide a “filter” hook in extractall's loop that can modify or
    discard members before they are processed.
-   Require that this hook is called just before extracting each member,
    so it can scan the current state of the disk. This will greatly
    simplify the implementation of policies (both in stdlib and user
    code), at the cost of not being able to do a precise “dry run”.

The hook API will be very similar to the existing filter argument for
TarFile.add <tarfile.TarFile.add>. We'll also name it filter. (In some
cases “policy” would be a more fitting name, but the API can be used for
more than security policies.)

The built-in policies/filters described above will be implemented using
the public filter API, so they can be used as building blocks or
examples.

Setting a precedent

If and when other libraries for archive extraction, such as zipfile,
gain similar functionality, they should mimic this API as much as it's
reasonable.

To enable this for simple cases, the built-in filters will have string
names; e.g. users can pass filter='data' instead of a specific function
that deals with ~tarfile.TarInfo objects.

The shutil.unpack_archive function will get a filter argument, which it
will pass to extractall.

Adding function-based API that would work across archive formats is out
of scope of this PEP.

Full disclosure & redistributor info

The PEP author works for Red Hat, a redistributor of Python with
different security needs and support periods than CPython in general.
Such redistributors may want to carry vendor patches to:

-   Allow configuring the defaults system-wide, and
-   Change the default as soon as possible, even in older Python
    versions.

The proposal makes this easy to do, and it allows users to query the
settings.

Specification

Modifying and forgetting member metadata

The ~tarfile.TarInfo class will gain a new method, replace(), which will
work similarly to dataclasses.replace. It will return a copy of the
TarInfo object with attributes replaced as specified by keyword-only
arguments:

-   name
-   mtime
-   mode
-   linkname
-   uid
-   gid
-   uname
-   gname

Any of these, except name and linkname, will be allowed to be set to
None. When extract or extractall encounters such a None, it will not set
that piece of metadata. (If uname or gname is None, it will fall back to
uid or gid as if the name wasn't found.) When addfile or tobuf
encounters such a None, it will raise a ValueError. When list encounters
such a None, it will print a placeholder string.

The documentation will mention why the method is there: TarInfo objects
retrieved from TarFile.getmembers <tarfile.TarFile.getmembers> are
“live”; modifying them directly will affect subsequent unrelated
operations.

Filters

TarFile.extract <tarfile.TarFile.extract> and
TarFile.extractall <tarfile.TarFile.extractall> methods will grow a
filter keyword-only parameter, which takes a callable that can be called
as:

    filter(/, member: TarInfo, path: str) -> TarInfo|None

where member is the member to be extracted, and path is the path to
where the archive is extracted (i.e., it'll be the same for every
member).

When used it will be called on each member as it is extracted, and
extraction will work with the result. If it returns None, the member
will be skipped.

The function can also raise an exception. This can, depending on
TarFile.errorlevel, abort the extraction or cause the member to be
skipped.

Note

If extraction is aborted, the archive may be left partially extracted.
It is the user’s responsibility to clean up.

We will also provide a set of defaults for common use cases. In addition
to a function, the filter argument can be one of the following strings:

-   'fully_trusted': Current behavior: honor the metadata as is. Should
    be used if the user trusts the archive completely, or implements
    their own complex verification.
-   'tar': Roughly follow defaults of the GNU tar command (when run as a
    normal user):
    -   Strip leading '/' and os.sep from filenames
    -   Refuse to extract files with absolute paths (after the /
        stripping above, e.g. C:/foo on Windows).
    -   Refuse to extract files whose absolute path (after following
        symlinks) would end up outside the destination. (Note that GNU
        tar instead delays creating some links.)
    -   Clear high mode bits (setuid, setgid, sticky) and group/other
        write bits (S_IWGRP|S_IWOTH <stat.S_IWGRP>). (This is an
        approximation of GNU tar's default, which limits the mode by the
        current umask setting.)
-   'data': Extract a "data" archive, disallowing common attack vectors
    but limiting functionality. In particular, many features specific to
    UNIX-style filesystems (or equivalently, to the tar archive format)
    are ignored, making this a good filter for cross-platform archives.
    In addition to tar:
    -   Refuse to extract links (hard or soft) that link to absolute
        paths.
    -   Refuse to extract links (hard or soft) which end up linking to a
        path outside of the destination. (On systems that don't support
        links, tarfile will, in most cases, fall back to creating
        regular files. This proposal doesn't change that behaviour.)
    -   Refuse to extract device files (including pipes).
    -   For regular files and hard links:
        -   Set the owner read and write permissions
            (S_IRUSR|S_IWUSR <stat.S_IRUSR>).
        -   Remove the group & other executable permission
            (S_IXGRP|S_IXOTH <stat.S_IXGRP>) if the owner doesn't have
            it (~stat.S_IXUSR).
    -   For other files (directories), ignore mode entirely (set it to
        None).
    -   Ignore user and group info (set uid, gid, uname, gname to None).

Any other string will cause a ValueError.

The corresponding filter functions will be available as
tarfile.fully_trusted_filter(), tarfile.tar_filter(), etc., so they can
be easily used in custom policies.

Note that these filters never return None. Skipping members this way is
a feature for user-defined filters.

Defaults and their configuration

~tarfile.TarFile will gain a new attribute, extraction_filter, to allow
configuring the default filter. By default it will be None, but users
can set it to a callable that will be used if the filter argument is
missing or None.

Note

String names won't be accepted here. That would encourage code like
my_tarfile.extraction_filter = 'data'. On Python versions without this
feature, this would do nothing, silently ignoring a security-related
request.

If both the argument and attribute are None:

-   In Python 3.12-3.13, a DeprecationWarning will be emitted and
    extraction will use the 'fully_trusted' filter.
-   In Python 3.14+, it will use the 'data' filter.

Applications and system integrators may wish to change extraction_filter
of the TarFile class itself to set a global default. When using a
function, they will generally want to wrap it in staticmethod() to
prevent injection of a self argument.

Subclasses of TarFile can also override extraction_filter.

FilterError

A new exception, FilterError, will be added to the tarfile module. It'll
have several new subclasses, one for each of the refusal reasons above.
FilterError's member attribute will contain the relevant TarInfo.

In the lists above, “refusing" to extract a file means that a
FilterError will be raised. As with other extraction errors, if the
TarFile.errorlevel is 1 or more, this will abort the extraction; with
errorlevel=0 the error will be logged and the member will be ignored,
but extraction will continue. Note that extractall() may leave the
archive partially extracted; it is the user's responsibility to clean
up.

Errorlevel, and fatal/non-fatal errors

Currently, ~tarfile.TarFile has an errorlevel argument/attribute, which
specifies how errors are handled:

-   With errorlevel=0, documentation says that “all errors are ignored
    when using ~tarfile.TarFile.extract and
    ~tarfile.TarFile.extractall”. The code only ignores non-fatal and
    fatal errors (see below), so, for example, you still get TypeError
    if you pass None as the destination path.

-   With errorlevel=1 (the default), all non-fatal errors are ignored.
    (They may be logged to sys.stderr by setting the debug
    argument/attribute.) Which errors are non-fatal is not defined in
    documentation, but code treats ExtractionError as such.
    Specifically, it's these issues:

    -   “unable to resolve link inside archive” (raised on systems that
        do not support symlinks)
    -   “fifo/special devices not supported by system” (not used for
        failures if the system supports these, e.g. for a
        PermissionError)
    -   “could not change owner/mode/modification time”

    Note that, for example, file name too long or out of disk space
    don't qualify. The non-fatal errors are not very likely to appear on
    a Unix-like system.

-   With errorlevel=2, all errors are raised, including fatal ones.
    Which errors are fatal is, again, not defined; in practice it's
    OSError.

A filter refusing to extract a member does not fit neatly into the
fatal/*non-fatal* categories.

-   This PEP does not change existing behavior. (Ideas for improvements
    are welcome in Discourse topic 25970.)
-   When a filter refuses to extract a member, the error should not pass
    silently by default.

To satisfy this, FilterError will be considered a fatal error, that is,
it'll be ignored only with errorlevel=0.

Users that want to ignore FilterError but not other fatal errors should
create a custom filter function, and call another filter in a try block.

Hints for further verification

Even with the proposed changes, tarfile will not be suited for
extracting untrusted files without prior inspection. Among other issues,
the proposed policies don't prevent denial-of-service attacks. Users
should do additional checks.

New docs will tell users to consider:

-   extracting to a new empty directory,
-   using external (e.g. OS-level) limits on disk, memory and CPU usage,
-   checking filenames against an allow-list of characters (to filter
    out control characters, confusables, etc.),
-   checking that filenames have expected extensions (discouraging files
    that execute when you “click on them”, or extension-less files like
    Windows special device names),
-   limiting the number of extracted files, total size of extracted
    data, and size of individual files,
-   checking for files that would be shadowed on case-insensitive
    filesystems.

Also, the docs will note that:

-   tar files commonly contain multiple versions of the same file: later
    ones are expected to overwrite earlier ones on extraction,
-   tarfile does not protect against issues with “live” data, e.g. an
    attacker tinkering with the destination directory while extracting
    (or adding) is going on (see the GNU tar manual for more info).

This list is not comprehensive, but the documentation is a good place to
collect such general tips. It can be moved into a separate document if
grows too long or if it needs to be consolidated with zipfile or shutil
(which is out of scope for this proposal).

TarInfo identity, and offset

With filters that use replace(), the TarInfo objects handled by the
extraction machinery will not necessarily be the same objects as those
present in members. This may affect TarInfo subclasses that override
methods like makelink and rely on object identity.

Such code can switch to comparing offset, the position of the member
header inside the file.

Note that both the overridable methods and offset are only documented in
source comments.

tarfile CLI

The CLI (python -m tarfile) will gain a --filter option that will take
the name of one of the provided default filters. It won't be possible to
specify a custom filter function.

If --filter is not given, the CLI will use the default filter
('fully_trusted' with a deprecation warning now, and 'data' from Python
3.14 on).

There will be no short option. (-f would be confusingly similar to the
filename option of GNU tar.)

Other archive libraries

If and when other archive libraries, such as zipfile, grow similar
functionality, their extraction functions should use a filter argument
that takes, at least, the strings 'fully_trusted' (which should disable
any security precautions) and 'data' (which should avoid features that
might surprise users).

Standardizing a function-based filter API is out of scope of this PEP.

Shutil

shutil.unpack_archive will gain a filter argument. If it's given, it
will be passed to the underlying extraction function. Passing it for a
zip archive will fail for now (until zipfile gains a filter argument, if
it ever does).

If filter is not specified (or left as None), it won't be passed on, so
extracting a tarball will use the default filter ('fully_trusted' with a
deprecation warning now, and 'data' from Python 3.14 on).

Complex filters

Note that some user-defined filters need, for example, to count
extracted members of do post-processing. This requires a more complex
API than a filter callable. However, that complex API need not be
exposed to tarfile. For example, with a hypothetical StatefulFilter
users would write:

    with StatefulFilter() as filter_func:
        my_tar.extract(path, filter=filter_func)

A simple StatefulFilter example will be added to the docs.

Note

The need for stateful filters is a reason against allowing registration
of custom filter names in addition to 'fully_trusted', 'tar' and 'data'.
With such a mechanism, API for (at least) set-up and tear-down would
need to be set in stone.

Backwards Compatibility

The default behavior of TarFile.extract <tarfile.TarFile.extract> and
TarFile.extractall <tarfile.TarFile.extractall> will change, after
raising DeprecationWarning for 2 releases (shortest deprecation period
allowed in Python's backwards compatibility policy <387>).

Additionally, code that relies on tarfile.TarInfo object identity may
break, see 706-offset.

Backporting & Forward Compatibility

This feature may be backported to older versions of Python.

In CPython, we don't add warnings to patch releases, so the default
filter should be changed to 'fully_trusted' in backports.

Other than that, all of the changes to tarfile should be backported, so
hasattr(tarfile, 'data_filter') becomes a reliable check for all of the
new functionality.

Note that CPython's usual policy is to avoid adding new APIs in security
backports. This feature does not make sense without a new API
(TarFile.extraction_filter and the filter argument), so we'll make an
exception. (See Discourse comment 23149/16 for details.)

Here are examples of code that takes into account that tarfile may or
may not have the proposed feature.

When copying these snippets, note that setting extraction_filter will
affect subsequent operations.

-   Fully trusted archive:

        my_tarfile.extraction_filter = (lambda member, path: member)
        my_tarfile.extractall()

-   Use the 'data' filter if available, but revert to Python 3.11
    behavior ('fully_trusted') if this feature is not available:

        my_tarfile.extraction_filter = getattr(tarfile, 'data_filter',
                                               (lambda member, path: member))
        my_tarfile.extractall()

    (This is an unsafe operation, so it should be spelled out
    explicitly, ideally with a comment.)

-   Use the 'data' filter; fail if it is not available:

        my_tarfile.extractall(filter=tarfile.data_filter)

    or:

        my_tarfile.extraction_filter = tarfile.data_filter
        my_tarfile.extractall()

-   Use the 'data' filter; warn if it is not available:

        if hasattr(tarfile, 'data_filter'):
            my_tarfile.extractall(filter='data')
        else:
            # remove this when no longer needed
            warn_the_user('Extracting may be unsafe; consider updating Python')
            my_tarfile.extractall()

Security Implications

This proposal improves security, at the expense of backwards
compatibility. In particular, it will help users avoid CVE-2007-4559.

How to Teach This

The API, usage notes and tips for further verification will be added to
the documentation. These should be usable for users who are familiar
with archives in general, but not with the specifics of UNIX filesystems
nor the related security issues.

Reference Implementation

See pull request #102953 on GitHub.

Rejected Ideas

SafeTarFile

An initial idea from Lars Gustäbel was to provide a separate class that
implements security checks (see gh-65308). There are two major issues
with this approach:

-   The name is misleading. General archive operations can never be made
    “safe” from all kinds of unwanted behavior, without impacting
    legitimate use cases.
-   It does not solve the problem of unsafe defaults.

However, many of the ideas behind SafeTarFile were reused in this PEP.

Add absolute_path option to tarfile

Issue gh-73974 asks for adding an absolute_path option to extraction
methods. This would be a minimal change to formally resolve
CVE-2007-4559. It doesn't go far enough to protect the unaware, nor to
empower the diligent and curious.

Other names for the 'tar' filter

The 'tar' filter exposes features specific to UNIX-like filesystems, so
it could be named 'unix'. Or 'unix-like', 'nix', '*nix', 'posix'?

Feature-wise, tar format and UNIX-like filesystem are essentially
equivalent, so tar is a good name.

Possible Further Work

Adding filters to zipfile and shutil.unpack_archive

For consistency, zipfile and shutil.unpack_archive could gain support
for a filter argument. However, this would require research that this
PEP's author can't promise for Python 3.12.

Filters for zipfile would probably not help security. Zip is used
primarily for cross-platform data bundles, and correspondingly,
ZipFile.extract <zipfile.ZipFile.extract>'s defaults are already similar
to what a 'data' filter would do. A 'fully_trusted' filter, which would
newly allow absolute paths and .. path components, might not be useful
for much except a unified unpack_archive API.

Filters should be useful for use cases other than security, but those
would usually need custom filter functions, and those would need API
that works with both ~tarfile.TarInfo and ~zipfile.ZipInfo. That is
definitely out of scope of this PEP.

If only this PEP is implemented and nothing changes for zipfile, the
effect for callers of unpack_archive is that the default for tar files
is changing from 'fully_trusted' to the more appropriate 'data'. In the
interim period, Python 3.12-3.13 will emit DeprecationWarning. That's
annoying, but there are several ways to handle it: e.g. add a filter
argument conditionally, set TarFile.extraction_filter globally, or
ignore/suppress the warning until Python 3.14.

Also, since many calls to unpack_archive are likely to be unsafe,
there's hope that the DeprecationWarning will often turn out to be a
helpful hint to review affected code.

Thanks

This proposal is based on prior work and discussions by many people, in
particular Lars Gustäbel, Gregory P. Smith, Larry Hastings, Joachim
Wagner, Jan Matejek, Jakub Wilk, Daniel Garcia, Lumír Balhar, Miro
Hrončok, and many others.

References

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 391 Title: Dictionary-Based Configuration For Logging Version:
$Revision$ Last-Modified: $Date$ Author: Vinay Sajip <vinay_sajip at
red-dove.com> Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 15-Oct-2009 Python-Version: 2.7, 3.2 Post-History:

Abstract

This PEP describes a new way of configuring logging using a dictionary
to hold configuration information.

Rationale

The present means for configuring Python's logging package is either by
using the logging API to configure logging programmatically, or else by
means of ConfigParser-based configuration files.

Programmatic configuration, while offering maximal control, fixes the
configuration in Python code. This does not facilitate changing it
easily at runtime, and, as a result, the ability to flexibly turn the
verbosity of logging up and down for different parts of a using
application is lost. This limits the usability of logging as an aid to
diagnosing problems - and sometimes, logging is the only diagnostic aid
available in production environments.

The ConfigParser-based configuration system is usable, but does not
allow its users to configure all aspects of the logging package. For
example, Filters cannot be configured using this system. Furthermore,
the ConfigParser format appears to engender dislike (sometimes strong
dislike) in some quarters. Though it was chosen because it was the only
configuration format supported in the Python standard at that time, many
people regard it (or perhaps just the particular schema chosen for
logging's configuration) as 'crufty' or 'ugly', in some cases apparently
on purely aesthetic grounds.

Recent versions of Python include JSON support in the standard library,
and this is also usable as a configuration format. In other
environments, such as Google App Engine, YAML is used to configure
applications, and usually the configuration of logging would be
considered an integral part of the application configuration. Although
the standard library does not contain YAML support at present, support
for both JSON and YAML can be provided in a common way because both of
these serialization formats allow deserialization to Python
dictionaries.

By providing a way to configure logging by passing the configuration in
a dictionary, logging will be easier to configure not only for users of
JSON and/or YAML, but also for users of custom configuration methods, by
providing a common format in which to describe the desired
configuration.

Another drawback of the current ConfigParser-based configuration system
is that it does not support incremental configuration: a new
configuration completely replaces the existing configuration. Although
full flexibility for incremental configuration is difficult to provide
in a multi-threaded environment, the new configuration mechanism will
allow the provision of limited support for incremental configuration.

Specification

The specification consists of two parts: the API and the format of the
dictionary used to convey configuration information (i.e. the schema to
which it must conform).

Naming

Historically, the logging package has not been PEP 8 conformant. At some
future time, this will be corrected by changing method and function
names in the package in order to conform with PEP 8. However, in the
interests of uniformity, the proposed additions to the API use a naming
scheme which is consistent with the present scheme used by logging.

API

The logging.config module will have the following addition:

-   A function, called dictConfig(), which takes a single argument
    - the dictionary holding the configuration. Exceptions will be
    raised if there are errors while processing the dictionary.

It will be possible to customize this API - see the section on API
Customization. Incremental configuration is covered in its own section.

Dictionary Schema - Overview

Before describing the schema in detail, it is worth saying a few words
about object connections, support for user-defined objects and access to
external and internal objects.

Object connections

The schema is intended to describe a set of logging objects - loggers,
handlers, formatters, filters - which are connected to each other in an
object graph. Thus, the schema needs to represent connections between
the objects. For example, say that, once configured, a particular logger
has attached to it a particular handler. For the purposes of this
discussion, we can say that the logger represents the source, and the
handler the destination, of a connection between the two. Of course in
the configured objects this is represented by the logger holding a
reference to the handler. In the configuration dict, this is done by
giving each destination object an id which identifies it unambiguously,
and then using the id in the source object's configuration to indicate
that a connection exists between the source and the destination object
with that id.

So, for example, consider the following YAML snippet:

    formatters:
      brief:
        # configuration for formatter with id 'brief' goes here
      precise:
        # configuration for formatter with id 'precise' goes here
    handlers:
      h1: #This is an id
       # configuration of handler with id 'h1' goes here
       formatter: brief
      h2: #This is another id
       # configuration of handler with id 'h2' goes here
       formatter: precise
    loggers:
      foo.bar.baz:
        # other configuration for logger 'foo.bar.baz'
        handlers: [h1, h2]

(Note: YAML will be used in this document as it is a little more
readable than the equivalent Python source form for the dictionary.)

The ids for loggers are the logger names which would be used
programmatically to obtain a reference to those loggers, e.g.
foo.bar.baz. The ids for Formatters and Filters can be any string value
(such as brief, precise above) and they are transient, in that they are
only meaningful for processing the configuration dictionary and used to
determine connections between objects, and are not persisted anywhere
when the configuration call is complete.

Handler ids are treated specially, see the section on Handler Ids,
below.

The above snippet indicates that logger named foo.bar.baz should have
two handlers attached to it, which are described by the handler ids h1
and h2. The formatter for h1 is that described by id brief, and the
formatter for h2 is that described by id precise.

User-defined objects

The schema should support user-defined objects for handlers, filters and
formatters. (Loggers do not need to have different types for different
instances, so there is no support - in the configuration -for
user-defined logger classes.)

Objects to be configured will typically be described by dictionaries
which detail their configuration. In some places, the logging system
will be able to infer from the context how an object is to be
instantiated, but when a user-defined object is to be instantiated, the
system will not know how to do this. In order to provide complete
flexibility for user-defined object instantiation, the user will need to
provide a 'factory' - a callable which is called with a configuration
dictionary and which returns the instantiated object. This will be
signalled by an absolute import path to the factory being made available
under the special key '()'. Here's a concrete example:

    formatters:
      brief:
        format: '%(message)s'
      default:
        format: '%(asctime)s %(levelname)-8s %(name)-15s %(message)s'
        datefmt: '%Y-%m-%d %H:%M:%S'
      custom:
          (): my.package.customFormatterFactory
          bar: baz
          spam: 99.9
          answer: 42

The above YAML snippet defines three formatters. The first, with id
brief, is a standard logging.Formatter instance with the specified
format string. The second, with id default, has a longer format and also
defines the time format explicitly, and will result in a
logging.Formatter initialized with those two format strings. Shown in
Python source form, the brief and default formatters have configuration
sub-dictionaries:

    {
      'format' : '%(message)s'
    }

and:

    {
      'format' : '%(asctime)s %(levelname)-8s %(name)-15s %(message)s',
      'datefmt' : '%Y-%m-%d %H:%M:%S'
    }

respectively, and as these dictionaries do not contain the special key
'()', the instantiation is inferred from the context: as a result,
standard logging.Formatter instances are created. The configuration
sub-dictionary for the third formatter, with id custom, is:

    {
      '()' : 'my.package.customFormatterFactory',
      'bar' : 'baz',
      'spam' : 99.9,
      'answer' : 42
    }

and this contains the special key '()', which means that user-defined
instantiation is wanted. In this case, the specified factory callable
will be used. If it is an actual callable it will be used directly -
otherwise, if you specify a string (as in the example) the actual
callable will be located using normal import mechanisms. The callable
will be called with the remaining items in the configuration
sub-dictionary as keyword arguments. In the above example, the formatter
with id custom will be assumed to be returned by the call:

    my.package.customFormatterFactory(bar='baz', spam=99.9, answer=42)

The key '()' has been used as the special key because it is not a valid
keyword parameter name, and so will not clash with the names of the
keyword arguments used in the call. The '()' also serves as a mnemonic
that the corresponding value is a callable.

Access to external objects

There are times where a configuration will need to refer to objects
external to the configuration, for example sys.stderr. If the
configuration dict is constructed using Python code then this is
straightforward, but a problem arises when the configuration is provided
via a text file (e.g. JSON, YAML). In a text file, there is no standard
way to distinguish sys.stderr from the literal string 'sys.stderr'. To
facilitate this distinction, the configuration system will look for
certain special prefixes in string values and treat them specially. For
example, if the literal string 'ext://sys.stderr' is provided as a value
in the configuration, then the ext:// will be stripped off and the
remainder of the value processed using normal import mechanisms.

The handling of such prefixes will be done in a way analogous to
protocol handling: there will be a generic mechanism to look for
prefixes which match the regular expression
^(?P<prefix>[a-z]+)://(?P<suffix>.*)$ whereby, if the prefix is
recognised, the suffix is processed in a prefix-dependent manner and the
result of the processing replaces the string value. If the prefix is not
recognised, then the string value will be left as-is.

The implementation will provide for a set of standard prefixes such as
ext:// but it will be possible to disable the mechanism completely or
provide additional or different prefixes for special handling.

Access to internal objects

As well as external objects, there is sometimes also a need to refer to
objects in the configuration. This will be done implicitly by the
configuration system for things that it knows about. For example, the
string value 'DEBUG' for a level in a logger or handler will
automatically be converted to the value logging.DEBUG, and the handlers,
filters and formatter entries will take an object id and resolve to the
appropriate destination object.

However, a more generic mechanism needs to be provided for the case of
user-defined objects which are not known to logging. For example, take
the instance of logging.handlers.MemoryHandler, which takes a target
which is another handler to delegate to. Since the system already knows
about this class, then in the configuration, the given target just needs
to be the object id of the relevant target handler, and the system will
resolve to the handler from the id. If, however, a user defines a
my.package.MyHandler which has a alternate handler, the configuration
system would not know that the alternate referred to a handler. To cater
for this, a generic resolution system will be provided which allows the
user to specify:

    handlers:
      file:
        # configuration of file handler goes here

      custom:
        (): my.package.MyHandler
        alternate: cfg://handlers.file

The literal string 'cfg://handlers.file' will be resolved in an
analogous way to the strings with the ext:// prefix, but looking in the
configuration itself rather than the import namespace. The mechanism
will allow access by dot or by index, in a similar way to that provided
by str.format. Thus, given the following snippet:

    handlers:
      email:
        class: logging.handlers.SMTPHandler
        mailhost: localhost
        fromaddr<EMAIL_ADDRESS>        toaddrs:
          -<EMAIL_ADDRESS>          -<EMAIL_ADDRESS>        subject: Houston, we have a problem.

in the configuration, the string 'cfg://handlers' would resolve to the
dict with key handlers, the string 'cfg://handlers.email would resolve
to the dict with key email in the handlers dict, and so on. The string
'cfg://handlers.email.toaddrs[1] would resolve to 'dev_team.domain.tld'
and the string 'cfg://handlers.email.toaddrs[0]' would resolve to the
value<EMAIL_ADDRESS>The subject value could be accessed
using either 'cfg://handlers.email.subject' or, equivalently,
'cfg://handlers.email[subject]'. The latter form only needs to be used
if the key contains spaces or non-alphanumeric characters. If an index
value consists only of decimal digits, access will be attempted using
the corresponding integer value, falling back to the string value if
needed.

Given a string cfg://handlers.myhandler.mykey.123, this will resolve to
config_dict['handlers']['myhandler']['mykey']['123']. If the string is
specified as cfg://handlers.myhandler.mykey[123], the system will
attempt to retrieve the value from
config_dict['handlers']['myhandler']['mykey'][123], and fall back to
config_dict['handlers']['myhandler']['mykey']['123'] if that fails.

Handler Ids

Some specific logging configurations require the use of handler levels
to achieve the desired effect. However, unlike loggers which can always
be identified by their names, handlers have no persistent handles
whereby levels can be changed via an incremental configuration call.

Therefore, this PEP proposes to add an optional name property to
handlers. If used, this will add an entry in a dictionary which maps the
name to the handler. (The entry will be removed when the handler is
closed.) When an incremental configuration call is made, handlers will
be looked up in this dictionary to set the handler level according to
the value in the configuration. See the section on incremental
configuration for more details.

In theory, such a "persistent name" facility could also be provided for
Filters and Formatters. However, there is not a strong case to be made
for being able to configure these incrementally. On the basis that
practicality beats purity, only Handlers will be given this new name
property. The id of a handler in the configuration will become its name.

The handler name lookup dictionary is for configuration use only and
will not become part of the public API for the package.

Dictionary Schema - Detail

The dictionary passed to dictConfig() must contain the following keys:

-   version - to be set to an integer value representing the schema
    version. The only valid value at present is 1, but having this key
    allows the schema to evolve while still preserving backwards
    compatibility.

All other keys are optional, but if present they will be interpreted as
described below. In all cases below where a 'configuring dict' is
mentioned, it will be checked for the special '()' key to see if a
custom instantiation is required. If so, the mechanism described above
is used to instantiate; otherwise, the context is used to determine how
to instantiate.

-   formatters - the corresponding value will be a dict in which each
    key is a formatter id and each value is a dict describing how to
    configure the corresponding Formatter instance.

    The configuring dict is searched for keys format and datefmt (with
    defaults of None) and these are used to construct a
    logging.Formatter instance.

-   filters - the corresponding value will be a dict in which each key
    is a filter id and each value is a dict describing how to configure
    the corresponding Filter instance.

    The configuring dict is searched for key name (defaulting to the
    empty string) and this is used to construct a logging.Filter
    instance.

-   handlers - the corresponding value will be a dict in which each key
    is a handler id and each value is a dict describing how to configure
    the corresponding Handler instance.

    The configuring dict is searched for the following keys:

    -   class (mandatory). This is the fully qualified name of the
        handler class.
    -   level (optional). The level of the handler.
    -   formatter (optional). The id of the formatter for this handler.
    -   filters (optional). A list of ids of the filters for this
        handler.

    All other keys are passed through as keyword arguments to the
    handler's constructor. For example, given the snippet:

        handlers:
          console:
            class : logging.StreamHandler
            formatter: brief
            level   : INFO
            filters: [allow_foo]
            stream  : ext://sys.stdout
          file:
            class : logging.handlers.RotatingFileHandler
            formatter: precise
            filename: logconfig.log
            maxBytes: 1024
            backupCount: 3

    the handler with id console is instantiated as a
    logging.StreamHandler, using sys.stdout as the underlying stream.
    The handler with id file is instantiated as a
    logging.handlers.RotatingFileHandler with the keyword arguments
    filename='logconfig.log', maxBytes=1024, backupCount=3.

-   loggers - the corresponding value will be a dict in which each key
    is a logger name and each value is a dict describing how to
    configure the corresponding Logger instance.

    The configuring dict is searched for the following keys:

    -   level (optional). The level of the logger.
    -   propagate (optional). The propagation setting of the logger.
    -   filters (optional). A list of ids of the filters for this
        logger.
    -   handlers (optional). A list of ids of the handlers for this
        logger.

    The specified loggers will be configured according to the level,
    propagation, filters and handlers specified.

-   root - this will be the configuration for the root logger.
    Processing of the configuration will be as for any logger, except
    that the propagate setting will not be applicable.

-   incremental - whether the configuration is to be interpreted as
    incremental to the existing configuration. This value defaults to
    False, which means that the specified configuration replaces the
    existing configuration with the same semantics as used by the
    existing fileConfig() API.

    If the specified value is True, the configuration is processed as
    described in the section on Incremental Configuration, below.

-   disable_existing_loggers - whether any existing loggers are to be
    disabled. This setting mirrors the parameter of the same name in
    fileConfig(). If absent, this parameter defaults to True. This value
    is ignored if incremental is True.

A Working Example

The following is an actual working configuration in YAML format (except
that the email addresses are bogus):

    formatters:
      brief:
        format: '%(levelname)-8s: %(name)-15s: %(message)s'
      precise:
        format: '%(asctime)s %(name)-15s %(levelname)-8s %(message)s'
    filters:
      allow_foo:
        name: foo
    handlers:
      console:
        class : logging.StreamHandler
        formatter: brief
        level   : INFO
        stream  : ext://sys.stdout
        filters: [allow_foo]
      file:
        class : logging.handlers.RotatingFileHandler
        formatter: precise
        filename: logconfig.log
        maxBytes: 1024
        backupCount: 3
      debugfile:
        class : logging.FileHandler
        formatter: precise
        filename: logconfig-detail.log
        mode: a
      email:
        class: logging.handlers.SMTPHandler
        mailhost: localhost
        fromaddr<EMAIL_ADDRESS>        toaddrs:
          -<EMAIL_ADDRESS>          -<EMAIL_ADDRESS>        subject: Houston, we have a problem.
    loggers:
      foo:
        level : ERROR
        handlers: [debugfile]
      spam:
        level : CRITICAL
        handlers: [debugfile]
        propagate: no
      bar.baz:
        level: WARNING
    root:
      level     : DEBUG
      handlers  : [console, file]

Incremental Configuration

It is difficult to provide complete flexibility for incremental
configuration. For example, because objects such as filters and
formatters are anonymous, once a configuration is set up, it is not
possible to refer to such anonymous objects when augmenting a
configuration.

Furthermore, there is not a compelling case for arbitrarily altering the
object graph of loggers, handlers, filters, formatters at run-time, once
a configuration is set up; the verbosity of loggers and handlers can be
controlled just by setting levels (and, in the case of loggers,
propagation flags). Changing the object graph arbitrarily in a safe way
is problematic in a multi-threaded environment; while not impossible,
the benefits are not worth the complexity it adds to the implementation.

Thus, when the incremental key of a configuration dict is present and is
True, the system will ignore any formatters and filters entries
completely, and process only the level settings in the handlers entries,
and the level and propagate settings in the loggers and root entries.

It's certainly possible to provide incremental configuration by other
means, for example making dictConfig() take an incremental keyword
argument which defaults to False. The reason for suggesting that a value
in the configuration dict be used is that it allows for configurations
to be sent over the wire as pickled dicts to a socket listener. Thus,
the logging verbosity of a long-running application can be altered over
time with no need to stop and restart the application.

Note: Feedback on incremental configuration needs based on your
practical experience will be particularly welcome.

API Customization

The bare-bones dictConfig() API will not be sufficient for all use
cases. Provision for customization of the API will be made by providing
the following:

-   A class, called DictConfigurator, whose constructor is passed the
    dictionary used for configuration, and which has a configure()
    method.
-   A callable, called dictConfigClass, which will (by default) be set
    to DictConfigurator. This is provided so that if desired,
    DictConfigurator can be replaced with a suitable user-defined
    implementation.

The dictConfig() function will call dictConfigClass passing the
specified dictionary, and then call the configure() method on the
returned object to actually put the configuration into effect:

    def dictConfig(config):
        dictConfigClass(config).configure()

This should cater to all customization needs. For example, a subclass of
DictConfigurator could call DictConfigurator.__init__() in its own
__init__(), then set up custom prefixes which would be usable in the
subsequent configure() call. The dictConfigClass would be bound to the
subclass, and then dictConfig() could be called exactly as in the
default, uncustomized state.

Change to Socket Listener Implementation

The existing socket listener implementation will be modified as follows:
when a configuration message is received, an attempt will be made to
deserialize to a dictionary using the json module. If this step fails,
the message will be assumed to be in the fileConfig format and processed
as before. If deserialization is successful, then dictConfig() will be
called to process the resulting dictionary.

Configuration Errors

If an error is encountered during configuration, the system will raise a
ValueError, TypeError, AttributeError or ImportError with a suitably
descriptive message. The following is a (possibly incomplete) list of
conditions which will raise an error:

-   A level which is not a string or which is a string not corresponding
    to an actual logging level
-   A propagate value which is not a boolean
-   An id which does not have a corresponding destination
-   A non-existent handler id found during an incremental call
-   An invalid logger name
-   Inability to resolve to an internal or external object

Discussion in the community

The PEP has been announced on python-dev and python-list. While there
hasn't been a huge amount of discussion, this is perhaps to be expected
for a niche topic.

Discussion threads on python-dev:

https://mail.python.org/pipermail/python-dev/2009-October/092695.html
https://mail.python.org/pipermail/python-dev/2009-October/092782.html
https://mail.python.org/pipermail/python-dev/2009-October/093062.html

And on python-list:

https://mail.python.org/pipermail/python-list/2009-October/1223658.html
https://mail.python.org/pipermail/python-list/2009-October/1224228.html

There have been some comments in favour of the proposal, no objections
to the proposal as a whole, and some questions and objections about
specific details. These are believed by the author to have been
addressed by making changes to the PEP.

Reference implementation

A reference implementation of the changes is available as a module
dictconfig.py with accompanying unit tests in test_dictconfig.py, at:

http://bitbucket.org/vinay.sajip/dictconfig

This incorporates all features other than the socket listener change.

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

license: Public Domain


text:
PEP: 3121 Title: Extension Module Initialization and Finalization
Version: $Revision$ Last-Modified: $Date$ Author: Martin von Löwis
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 27-Apr-2007 Python-Version: 3.0 Post-History:

PyInit_modulename and PyModuleDef

Abstract

Extension module initialization currently has a few deficiencies. There
is no cleanup for modules, the entry point name might give naming
conflicts, the entry functions don't follow the usual calling
convention, and multiple interpreters are not supported well. This PEP
addresses these issues.

Problems

Module Finalization

Currently, extension modules are initialized usually once and then
"live" forever. The only exception is when Py_Finalize() is called: then
the initialization routine is invoked a second time. This is bad from a
resource management point of view: memory and other resources might get
allocated each time initialization is called, but there is no way to
reclaim them. As a result, there is currently no way to completely
release all resources Python has allocated.

Entry point name conflicts

The entry point is currently called init<module>. This might conflict
with other symbols also called init<something>. In particular,
initsocket is known to have conflicted in the past (this specific
problem got resolved as a side effect of renaming the module to
_socket).

Entry point signature

The entry point is currently a procedure (returning void). This deviates
from the usual calling conventions; callers can find out whether there
was an error during initialization only by checking PyErr_Occurred. The
entry point should return a PyObject*, which will be the module created,
or NULL in case of an exception.

Multiple Interpreters

Currently, extension modules share their state across all interpreters.
This allows for undesirable information leakage across interpreters: one
script could permanently corrupt objects in an extension module,
possibly breaking all scripts in other interpreters.

Specification

The module initialization routines change their signature to:

    PyObject *PyInit_<modulename>()

The initialization routine will be invoked once per interpreter, when
the module is imported. It should return a new module object each time.

In order to store per-module state in C variables, each module object
will contain a block of memory that is interpreted only by the module.
The amount of memory used for the module is specified at the point of
creation of the module.

In addition to the initialization function, a module may implement a
number of additional callback functions, which are invoked when the
module's tp_traverse, tp_clear, and tp_free functions are invoked, and
when the module is reloaded.

The entire module definition is combined in a struct PyModuleDef:

    struct PyModuleDef{
      PyModuleDef_Base m_base;  /* To be filled out by the interpreter */
      Py_ssize_t m_size; /* Size of per-module data */
      PyMethodDef *m_methods;
      inquiry m_reload;
      traverseproc m_traverse;
      inquiry m_clear;
      freefunc m_free;
    };

Creation of a module is changed to expect an optional PyModuleDef*. The
module state will be null-initialized.

Each module method will be passed the module object as the first
parameter. To access the module data, a function:

    void* PyModule_GetState(PyObject*);

will be provided. In addition, to lookup a module more efficiently than
going through sys.modules, a function:

    PyObject* PyState_FindModule(struct PyModuleDef*);

will be provided. This lookup function will use an index located in the
m_base field, to find the module by index, not by name.

As all Python objects should be controlled through the Python memory
management, usage of "static" type objects is discouraged, unless the
type object itself has no memory-managed state. To simplify definition
of heap types, a new method:

    PyTypeObject* PyType_Copy(PyTypeObject*);

is added.

Example

xxmodule.c would be changed to remove the initxx function, and add the
following code instead:

    struct xxstate{
      PyObject *ErrorObject;
      PyObject *Xxo_Type;
    };

    #define xxstate(o) ((struct xxstate*)PyModule_GetState(o))

    static int xx_traverse(PyObject *m, visitproc v,
                           void *arg)
    {
      Py_VISIT(xxstate(m)->ErrorObject);
      Py_VISIT(xxstate(m)->Xxo_Type);
      return 0;
    }

    static int xx_clear(PyObject *m)
    {
      Py_CLEAR(xxstate(m)->ErrorObject);
      Py_CLEAR(xxstate(m)->Xxo_Type);
      return 0;
    }

    static struct PyModuleDef xxmodule = {
      {}, /* m_base */
      sizeof(struct xxstate),
      &xx_methods,
      0,  /* m_reload */
      xx_traverse,
      xx_clear,
      0,  /* m_free - not needed, since all is done in m_clear */
    }

    PyObject*
    PyInit_xx()
    {
      PyObject *res = PyModule_New("xx", &xxmodule);
      if (!res) return NULL;
      xxstate(res)->ErrorObject = PyErr_NewException("xx.error", NULL, NULL);
      if (!xxstate(res)->ErrorObject) {
        Py_DECREF(res);
        return NULL;
      }
      xxstate(res)->XxoType = PyType_Copy(&Xxo_Type);
      if (!xxstate(res)->Xxo_Type) {
        Py_DECREF(res);
        return NULL;
      }
      return res;
    }

Discussion

Tim Peters reports in[1] that PythonLabs considered such a feature at
one point, and lists the following additional hooks which aren't
currently supported in this PEP:

-   when the module object is deleted from sys.modules
-   when Py_Finalize is called
-   when Python exits
-   when the Python DLL is unloaded (Windows only)

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] Tim Peters, reporting earlier conversation about such a feature
https://mail.python.org/pipermail/python-3000/2006-April/000726.html


text:
PEP: 676 Title: PEP Infrastructure Process Author: Adam Turner
<EMAIL_ADDRESS>Sponsor: Mariatta<EMAIL_ADDRESS>PEP-Delegate: Barry Warsaw<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/10774 Status: Active Type: Process
Content-Type: text/x-rst Created: 01-Nov-2021 Post-History: 23-Sep-2021,
30-Nov-2021 Resolution: https://discuss.python.org/t/10774/99

Abstract

This PEP addresses the infrastructure around rendering PEP files from
reStructuredText files to HTML webpages. We aim to specify a
self-contained and maintainable solution for PEP readers, authors, and
editors.

Motivation

As of November 2021, Python Enhancement Proposals (PEPs) are rendered in
a multi-system, multi-stage process. A continuous integration (CI) task
runs a docutils script to render all PEP files individually. The CI task
then uploads a tar archive to a server, where it is retrieved and
rendered into the python.org website periodically.

This places a constraint on the python.org website to handle raw HTML
uploads and handle PEP rendering, and makes the appropriate place to
raise issues unclear in some cases[1].

This PEP provides a specification for self-contained rendering of PEPs.
This would:

-   reduce the amount of distributed configuration for supporting PEPs
-   enable quality-of-life improvements for those who read, write, and
    review PEPs
-   solve a number of outstanding issues, and lay the path for
    improvements
-   save volunteer maintainers' time

We propose that PEPs are accessed through peps.python.org at the
top-level (for example peps.python.org/pep-0008), and that all custom
tooling to support rendering PEPs is hosted in the python/peps
repository.

Rationale

Simplifying and Centralising Infrastructure

As of November 2021, to locally render a PEP file, a PEP author or
editor needs to create a full local instance of the python.org website
and run a number of disparate scripts, following documentation that
lives outside of the python/peps repository.

By contrast, the proposed implementation provides a single Makefile and
a Python script to render all PEP files, with options to target a
web-server or the local filesystem.

Using a single repository to host all tooling will clarify where to
raise issues, reducing volunteer time spent in triage.

Simplified and centralised tooling may also reduce the barrier to entry
to further improvements, as the scope of the PEP rendering
infrastructure is well defined.

Quality-of-Life Improvements and Resolving Issues

There are several requests for additional features in reading PEPs, such
as:

-   syntax highlighting[2]
-   use of .. code-block:: directives[3]
-   support for SVG images[4]
-   typographic quotation marks[5]
-   additional footer information[6]
-   intersphinx functionality[7]
-   dark mode theme[8]

These are "easy wins" from this proposal, and would serve to improve the
quality-of-life for consumers of PEPs (including reviewers and writers).

For example, the current (as of November 2021) system runs periodically
on a schedule. This means that updates to PEPs cannot be circulated
immediately, reducing productivity. The reference implementation renders
and publishes all PEPs on every commit to the repository, solving the
issue by design.

The reference implementation fixes several issues[9]. For example:

-   list styles are currently not respected by python.org's stylesheets
-   support for updating images in PEPs is challenging in python.org

Third-party providers such as Read the Docs or Netlify can enhance this
experience with features such as automatic rendering of pull requests.

Specification

The proposed specification for rendering the PEP files to HTML is as per
the reference implementation.

The rendered PEPs MUST be available at peps.python.org. These SHOULD be
hosted as static files, and MAY be behind a content delivery network
(CDN).

A service to render previews of pull requests SHOULD be provided. This
service MAY be integrated with the hosting and deployment solution.

The following redirect rules MUST be created for the python.org domain:

-   /peps/ -> https://peps.python.org/
-   /dev/peps/ -> https://peps.python.org/
-   /peps/(.*)\.html -> https://peps.python.org/$1
-   /dev/peps/(.*) -> https://peps.python.org/$1

The following nginx configuration would achieve this:

    location ~ ^/dev/peps/?(.*)$ {
        return 308 https://peps.python.org/$1/;
    }

    location ~ ^/peps/(.*)\.html$ {
        return 308 https://peps.python.org/$1/;
    }

    location ^/(dev/)?peps(/.*)?$ {
        return 308 https://peps.python.org/;
    }

Redirects MUST be implemented to preserve URL fragments for backward
compatibility purposes.

Backwards Compatibility

Due to server-side redirects to the new canonical URLs, links in
previously published materials referring to the old URL schemes will be
guaranteed to work. All PEPs will continue to render correctly, and a
custom stylesheet in the reference implementation improves presentation
for some elements (most notably code blocks and block quotes).
Therefore, this PEP presents no backwards compatibility issues.

Security Implications

The main python.org website will no longer process raw HTML uploads,
closing a potential threat vector. PEP rendering and deployment
processes will use modern, well-maintained code and secure automated
platforms, further reducing the potential attack surface. Therefore, we
see no negative security impact.

How to Teach This

The new canonical URLs will be publicised in the documentation. However,
this is mainly a backend infrastructure change, and there should be
minimal end-user impact. PEP 1 and PEP 12 will be updated as needed.

Reference Implementation

The proposed implementation has been merged into the python/peps
repository in a series of pull requests[10]. It uses the Sphinx
documentation system with a custom theme (supporting light and dark
colour schemes) and extensions.

This already automatically renders all PEPs on every commit, and
publishes them to python.github.io/peps. The high level documentation
for the system covers how to render PEPs locally and the implementation
of the system.

Rejected Ideas

It would likely be possible to amend the current (as of November 2021)
rendering process to include a subset of the quality-of-life
improvements and issue mitigations mentioned above. However, we do not
believe that this would solve the distributed tooling issue.

It would be possible to use the output from the proposed rendering
system and import it into python.org. We would argue that this would be
the worst of both worlds, as a great deal of complexity is added whilst
none is removed.

Acknowledgements

-   Hugo van Kemenade
-   Pablo Galindo Salgado
-   Éric Araujo
-   Mariatta
-   C.A.M. Gerlach

Footnotes

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] For example, pythondotorg#1024, pythondotorg#1038,
pythondotorg#1387, pythondotorg#1388, pythondotorg#1393,
pythondotorg#1564, pythondotorg#1913,

[2] Requested: pythondotorg#1063, pythondotorg#1206, pythondotorg#1638,
peps#159, comment in peps#1571, peps#1577,

[3] Requested: pythondotorg#1063, pythondotorg#1206, pythondotorg#1638,
peps#159, comment in peps#1571, peps#1577,

[4] Requested: peps#701

[5] Requested: peps#165

[6] Requested: pythondotorg#1564

[7] Requested: comment in peps#2

[8] Requested: in python-dev

[9] As of November 2021, see peps#1387, pythondotorg#824,
pythondotorg#1556,

[10] Implementation PRs: peps#1930, peps#1931, peps#1932, peps#1933,
peps#1934


text:
PEP: 634 Title: Structural Pattern Matching: Specification Author:
Brandt Bucher<EMAIL_ADDRESS>Guido van Rossum<EMAIL_ADDRESS>BDFL-Delegate: Discussions-To<EMAIL_ADDRESS>Status: Final Type:
Standards Track Created: 12-Sep-2020 Python-Version: 3.10 Post-History:
22-Oct-2020, 08-Feb-2021 Replaces: 622 Resolution:
<EMAIL_ADDRESS>
match

Abstract

This PEP provides the technical specification for the match statement.
It replaces PEP 622, which is hereby split in three parts:

-   PEP 634: Specification
-   PEP 635: Motivation and Rationale
-   PEP 636: Tutorial

This PEP is intentionally devoid of commentary; the motivation and all
explanations of our design choices are in PEP 635. First-time readers
are encouraged to start with PEP 636, which provides a gentler
introduction to the concepts, syntax and semantics of patterns.

Syntax and Semantics

See Appendix A for the complete grammar.

Overview and Terminology

The pattern matching process takes as input a pattern (following case)
and a subject value (following match). Phrases to describe the process
include "the pattern is matched with (or against) the subject value" and
"we match the pattern against (or with) the subject value".

The primary outcome of pattern matching is success or failure. In case
of success we may say "the pattern succeeds", "the match succeeds", or
"the pattern matches the subject value".

In many cases a pattern contains subpatterns, and success or failure is
determined by the success or failure of matching those subpatterns
against the value (e.g., for OR patterns) or against parts of the value
(e.g., for sequence patterns). This process typically processes the
subpatterns from left to right until the overall outcome is determined.
E.g., an OR pattern succeeds at the first succeeding subpattern, while a
sequence patterns fails at the first failing subpattern.

A secondary outcome of pattern matching may be one or more name
bindings. We may say "the pattern binds a value to a name". When
subpatterns tried until the first success, only the bindings due to the
successful subpattern are valid; when trying until the first failure,
the bindings are merged. Several more rules, explained below, apply to
these cases.

The Match Statement

Syntax:

    match_stmt: "match" subject_expr ':' NEWLINE INDENT case_block+ DEDENT
    subject_expr:
        | star_named_expression ',' star_named_expressions?
        | named_expression
    case_block: "case" patterns [guard] ':' block
    guard: 'if' named_expression

The rules star_named_expression, star_named_expressions,
named_expression and block are part of the standard Python grammar.

The rule patterns is specified below.

For context, match_stmt is a new alternative for compound_statement:

    compound_statement:
        | if_stmt
        ...
        | match_stmt

The match and case keywords are soft keywords, i.e. they are not
reserved words in other grammatical contexts (including at the start of
a line if there is no colon where expected). This implies that they are
recognized as keywords when part of a match statement or case block
only, and are allowed to be used in all other contexts as variable or
argument names.

Match Semantics

The match statement first evaluates the subject expression. If a comma
is present a tuple is constructed using the standard rules.

The resulting subject value is then used to select the first case block
whose patterns succeeds matching it and whose guard condition (if
present) is "truthy". If no case blocks qualify the match statement is
complete; otherwise, the block of the selected case block is executed.
The usual rules for executing a block nested inside a compound statement
apply (e.g. an if statement).

Name bindings made during a successful pattern match outlive the
executed block and can be used after the match statement.

During failed pattern matches, some subpatterns may succeed. For
example, while matching the pattern (0, x, 1) with the value [0, 1, 2],
the subpattern x may succeed if the list elements are matched from left
to right. The implementation may choose to either make persistent
bindings for those partial matches or not. User code including a match
statement should not rely on the bindings being made for a failed match,
but also shouldn't assume that variables are unchanged by a failed
match. This part of the behavior is left intentionally unspecified so
different implementations can add optimizations, and to prevent
introducing semantic restrictions that could limit the extensibility of
this feature.

The precise pattern binding rules vary per pattern type and are
specified below.

Guards

If a guard is present on a case block, once the pattern or patterns in
the case block succeed, the expression in the guard is evaluated. If
this raises an exception, the exception bubbles up. Otherwise, if the
condition is "truthy" the case block is selected; if it is "falsy" the
case block is not selected.

Since guards are expressions they are allowed to have side effects.
Guard evaluation must proceed from the first to the last case block, one
at a time, skipping case blocks whose pattern(s) don't all succeed.
(I.e., even if determining whether those patterns succeed may happen out
of order, guard evaluation must happen in order.) Guard evaluation must
stop once a case block is selected.

Irrefutable case blocks

A pattern is considered irrefutable if we can prove from its syntax
alone that it will always succeed. In particular, capture patterns and
wildcard patterns are irrefutable, and so are AS patterns whose
left-hand side is irrefutable, OR patterns containing at least one
irrefutable pattern, and parenthesized irrefutable patterns.

A case block is considered irrefutable if it has no guard and its
pattern is irrefutable.

A match statement may have at most one irrefutable case block, and it
must be last.

Patterns

The top-level syntax for patterns is as follows:

    patterns: open_sequence_pattern | pattern
    pattern: as_pattern | or_pattern
    as_pattern: or_pattern 'as' capture_pattern
    or_pattern: '|'.closed_pattern+
    closed_pattern:
        | literal_pattern
        | capture_pattern
        | wildcard_pattern
        | value_pattern
        | group_pattern
        | sequence_pattern
        | mapping_pattern
        | class_pattern

AS Patterns

Syntax:

    as_pattern: or_pattern 'as' capture_pattern

(Note: the name on the right may not be _.)

An AS pattern matches the OR pattern on the left of the as keyword
against the subject. If this fails, the AS pattern fails. Otherwise, the
AS pattern binds the subject to the name on the right of the as keyword
and succeeds.

OR Patterns

Syntax:

    or_pattern: '|'.closed_pattern+

When two or more patterns are separated by vertical bars (|), this is
called an OR pattern. (A single closed pattern is just that.)

Only the final subpattern may be irrefutable.

Each subpattern must bind the same set of names.

An OR pattern matches each of its subpatterns in turn to the subject,
until one succeeds. The OR pattern is then deemed to succeed. If none of
the subpatterns succeed the OR pattern fails.

Literal Patterns

Syntax:

    literal_pattern:
        | signed_number
        | signed_number '+' NUMBER
        | signed_number '-' NUMBER
        | strings
        | 'None'
        | 'True'
        | 'False'
    signed_number: NUMBER | '-' NUMBER

The rule strings and the token NUMBER are defined in the standard Python
grammar.

Triple-quoted strings are supported. Raw strings and byte strings are
supported. F-strings are not supported.

The forms signed_number '+' NUMBER and signed_number '-' NUMBER are only
permitted to express complex numbers; they require a real number on the
left and an imaginary number on the right.

A literal pattern succeeds if the subject value compares equal to the
value expressed by the literal, using the following comparisons rules:

-   Numbers and strings are compared using the == operator.
-   The singleton literals None, True and False are compared using the
    is operator.

Capture Patterns

Syntax:

    capture_pattern: !"_" NAME

The single underscore (_) is not a capture pattern (this is what !"_"
expresses). It is treated as a wildcard pattern.

A capture pattern always succeeds. It binds the subject value to the
name using the scoping rules for name binding established for the walrus
operator in PEP 572. (Summary: the name becomes a local variable in the
closest containing function scope unless there's an applicable nonlocal
or global statement.)

In a given pattern, a given name may be bound only once. This disallows
for example case x, x: ... but allows case [x] | x: ....

Wildcard Pattern

Syntax:

    wildcard_pattern: "_"

A wildcard pattern always succeeds. It binds no name.

Value Patterns

Syntax:

    value_pattern: attr
    attr: name_or_attr '.' NAME
    name_or_attr: attr | NAME

The dotted name in the pattern is looked up using the standard Python
name resolution rules. However, when the same value pattern occurs
multiple times in the same match statement, the interpreter may cache
the first value found and reuse it, rather than repeat the same lookup.
(To clarify, this cache is strictly tied to a given execution of a given
match statement.)

The pattern succeeds if the value found thus compares equal to the
subject value (using the == operator).

Group Patterns

Syntax:

    group_pattern: '(' pattern ')'

(For the syntax of pattern, see Patterns above. Note that it contains no
comma -- a parenthesized series of items with at least one comma is a
sequence pattern, as is ().)

A parenthesized pattern has no additional syntax. It allows users to add
parentheses around patterns to emphasize the intended grouping.

Sequence Patterns

Syntax:

    sequence_pattern:
      | '[' [maybe_sequence_pattern] ']'
      | '(' [open_sequence_pattern] ')'
    open_sequence_pattern: maybe_star_pattern ',' [maybe_sequence_pattern]
    maybe_sequence_pattern: ','.maybe_star_pattern+ ','?
    maybe_star_pattern: star_pattern | pattern
    star_pattern: '*' (capture_pattern | wildcard_pattern)

(Note that a single parenthesized pattern without a trailing comma is a
group pattern, not a sequence pattern. However a single pattern enclosed
in [...] is still a sequence pattern.)

There is no semantic difference between a sequence pattern using [...],
a sequence pattern using (...), and an open sequence pattern.

A sequence pattern may contain at most one star subpattern. The star
subpattern may occur in any position. If no star subpattern is present,
the sequence pattern is a fixed-length sequence pattern; otherwise it is
a variable-length sequence pattern.

For a sequence pattern to succeed the subject must be a sequence, where
being a sequence is defined as its class being one of the following:

-   a class that inherits from collections.abc.Sequence
-   a Python class that has been registered as a
    collections.abc.Sequence
-   a builtin class that has its Py_TPFLAGS_SEQUENCE bit set
-   a class that inherits from any of the above (including classes
    defined before a parent's Sequence registration)

The following standard library classes will have their
Py_TPFLAGS_SEQUENCE bit set:

-   array.array
-   collections.deque
-   list
-   memoryview
-   range
-   tuple

Note

Although str, bytes, and bytearray are usually considered sequences,
they are not included in the above list and do not match sequence
patterns.

A fixed-length sequence pattern fails if the length of the subject
sequence is not equal to the number of subpatterns.

A variable-length sequence pattern fails if the length of the subject
sequence is less than the number of non-star subpatterns.

The length of the subject sequence is obtained using the builtin len()
function (i.e., via the __len__ protocol). However, the interpreter may
cache this value in a similar manner as described for value patterns.

A fixed-length sequence pattern matches the subpatterns to corresponding
items of the subject sequence, from left to right. Matching stops (with
a failure) as soon as a subpattern fails. If all subpatterns succeed in
matching their corresponding item, the sequence pattern succeeds.

A variable-length sequence pattern first matches the leading non-star
subpatterns to the corresponding items of the subject sequence, as for a
fixed-length sequence. If this succeeds, the star subpattern matches a
list formed of the remaining subject items, with items removed from the
end corresponding to the non-star subpatterns following the star
subpattern. The remaining non-star subpatterns are then matched to the
corresponding subject items, as for a fixed-length sequence.

Mapping Patterns

Syntax:

    mapping_pattern: '{' [items_pattern] '}'
    items_pattern: ','.key_value_pattern+ ','?
    key_value_pattern:
        | (literal_pattern | value_pattern) ':' pattern
        | double_star_pattern
    double_star_pattern: '**' capture_pattern

(Note that **_ is disallowed by this syntax.)

A mapping pattern may contain at most one double star pattern, and it
must be last.

A mapping pattern may not contain duplicate key values. (If all key
patterns are literal patterns this is considered a syntax error;
otherwise this is a runtime error and will raise ValueError.)

For a mapping pattern to succeed the subject must be a mapping, where
being a mapping is defined as its class being one of the following:

-   a class that inherits from collections.abc.Mapping
-   a Python class that has been registered as a collections.abc.Mapping
-   a builtin class that has its Py_TPFLAGS_MAPPING bit set
-   a class that inherits from any of the above (including classes
    defined before a parent's Mapping registration)

The standard library classes dict and mappingproxy will have their
Py_TPFLAGS_MAPPING bit set.

A mapping pattern succeeds if every key given in the mapping pattern is
present in the subject mapping, and the pattern for each key matches the
corresponding item of the subject mapping. Keys are always compared with
the == operator. If a '**' NAME form is present, that name is bound to a
dict containing remaining key-value pairs from the subject mapping.

If duplicate keys are detected in the mapping pattern, the pattern is
considered invalid, and a ValueError is raised.

Key-value pairs are matched using the two-argument form of the subject's
get() method. As a consequence, matched key-value pairs must already be
present in the mapping, and not created on-the-fly by __missing__ or
__getitem__. For example, collections.defaultdict instances will only be
matched by patterns with keys that were already present when the match
statement was entered.

Class Patterns

Syntax:

    class_pattern:
        | name_or_attr '(' [pattern_arguments ','?] ')'
    pattern_arguments:
        | positional_patterns [',' keyword_patterns]
        | keyword_patterns
    positional_patterns: ','.pattern+
    keyword_patterns: ','.keyword_pattern+
    keyword_pattern: NAME '=' pattern

A class pattern may not repeat the same keyword multiple times.

If name_or_attr is not an instance of the builtin type, TypeError is
raised.

A class pattern fails if the subject is not an instance of name_or_attr.
This is tested using isinstance().

If no arguments are present, the pattern succeeds if the isinstance()
check succeeds. Otherwise:

-   If only keyword patterns are present, they are processed as follows,
    one by one:
    -   The keyword is looked up as an attribute on the subject.
        -   If this raises an exception other than AttributeError, the
            exception bubbles up.
        -   If this raises AttributeError the class pattern fails.
        -   Otherwise, the subpattern associated with the keyword is
            matched against the attribute value. If this fails, the
            class pattern fails. If it succeeds, the match proceeds to
            the next keyword.
    -   If all keyword patterns succeed, the class pattern as a whole
        succeeds.
-   If any positional patterns are present, they are converted to
    keyword patterns (see below) and treated as additional keyword
    patterns, preceding the syntactic keyword patterns (if any).

Positional patterns are converted to keyword patterns using the
__match_args__ attribute on the class designated by name_or_attr, as
follows:

-   For a number of built-in types (specified below), a single
    positional subpattern is accepted which will match the entire
    subject. (Keyword patterns work as for other types here.)
-   The equivalent of getattr(cls, "__match_args__", ())) is called.
-   If this raises an exception the exception bubbles up.
-   If the returned value is not a tuple, the conversion fails and
    TypeError is raised.
-   If there are more positional patterns than the length of
    __match_args__ (as obtained using len()), TypeError is raised.
-   Otherwise, positional pattern i is converted to a keyword pattern
    using __match_args__[i] as the keyword, provided it the latter is a
    string; if it is not, TypeError is raised.
-   For duplicate keywords, TypeError is raised.

Once the positional patterns have been converted to keyword patterns,
the match proceeds as if there were only keyword patterns.

As mentioned above, for the following built-in types the handling of
positional subpatterns is different: bool, bytearray, bytes, dict,
float, frozenset, int, list, set, str, and tuple.

This behavior is roughly equivalent to the following:

    class C:
        __match_args__ = ("__match_self_prop__",)
        @property
        def __match_self_prop__(self):
            return self

Side Effects and Undefined Behavior

The only side-effect produced explicitly by the matching process is the
binding of names. However, the process relies on attribute access,
instance checks, len(), equality and item access on the subject and some
of its components. It also evaluates value patterns and the class name
of class patterns. While none of those typically create any
side-effects, in theory they could. This proposal intentionally leaves
out any specification of what methods are called or how many times. This
behavior is therefore undefined and user code should not rely on it.

Another undefined behavior is the binding of variables by capture
patterns that are followed (in the same case block) by another pattern
that fails. These may happen earlier or later depending on the
implementation strategy, the only constraint being that capture
variables must be set before guards that use them explicitly are
evaluated. If a guard consists of an and clause, evaluation of the
operands may even be interspersed with pattern matching, as long as
left-to-right evaluation order is maintained.

The Standard Library

To facilitate the use of pattern matching, several changes will be made
to the standard library:

-   Namedtuples and dataclasses will have auto-generated __match_args__.
-   For dataclasses the order of attributes in the generated
    __match_args__ will be the same as the order of corresponding
    arguments in the generated __init__() method. This includes the
    situations where attributes are inherited from a superclass. Fields
    with init=False are excluded from __match_args__.

In addition, a systematic effort will be put into going through existing
standard library classes and adding __match_args__ where it looks
beneficial.

Appendix A -- Full Grammar

Here is the full grammar for match_stmt. This is an additional
alternative for compound_stmt. Remember that match and case are soft
keywords, i.e. they are not reserved words in other grammatical contexts
(including at the start of a line if there is no colon where expected).
By convention, hard keywords use single quotes while soft keywords use
double quotes.

Other notation used beyond standard EBNF:

-   SEP.RULE+ is shorthand for RULE (SEP RULE)*
-   !RULE is a negative lookahead assertion

    match_stmt: "match" subject_expr ':' NEWLINE INDENT case_block+ DEDENT
    subject_expr:
        | star_named_expression ',' [star_named_expressions]
        | named_expression
    case_block: "case" patterns [guard] ':' block
    guard: 'if' named_expression

    patterns: open_sequence_pattern | pattern
    pattern: as_pattern | or_pattern
    as_pattern: or_pattern 'as' capture_pattern
    or_pattern: '|'.closed_pattern+
    closed_pattern:
        | literal_pattern
        | capture_pattern
        | wildcard_pattern
        | value_pattern
        | group_pattern
        | sequence_pattern
        | mapping_pattern
        | class_pattern

    literal_pattern:
        | signed_number !('+' | '-')
        | signed_number '+' NUMBER
        | signed_number '-' NUMBER
        | strings
        | 'None'
        | 'True'
        | 'False'
    signed_number: NUMBER | '-' NUMBER

    capture_pattern: !"_" NAME !('.' | '(' | '=')

    wildcard_pattern: "_"

    value_pattern: attr !('.' | '(' | '=')
    attr: name_or_attr '.' NAME
    name_or_attr: attr | NAME

    group_pattern: '(' pattern ')'

    sequence_pattern:
      | '[' [maybe_sequence_pattern] ']'
      | '(' [open_sequence_pattern] ')'
    open_sequence_pattern: maybe_star_pattern ',' [maybe_sequence_pattern]
    maybe_sequence_pattern: ','.maybe_star_pattern+ ','?
    maybe_star_pattern: star_pattern | pattern
    star_pattern: '*' (capture_pattern | wildcard_pattern)

    mapping_pattern: '{' [items_pattern] '}'
    items_pattern: ','.key_value_pattern+ ','?
    key_value_pattern:
        | (literal_pattern | value_pattern) ':' pattern
        | double_star_pattern
    double_star_pattern: '**' capture_pattern

    class_pattern:
        | name_or_attr '(' [pattern_arguments ','?] ')'
    pattern_arguments:
        | positional_patterns [',' keyword_patterns]
        | keyword_patterns
    positional_patterns: ','.pattern+
    keyword_patterns: ','.keyword_pattern+
    keyword_pattern: NAME '=' pattern

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 535 Title: Rich comparison chaining Version: $Revision$
Last-Modified: $Date$ Author: Alyssa Coghlan<EMAIL_ADDRESS>Status: Deferred Type: Standards Track Content-Type: text/x-rst
Requires: 532 Created: 12-Nov-2016 Python-Version: 3.8

PEP Deferral

Further consideration of this PEP has been deferred until Python 3.8 at
the earliest.

Abstract

Inspired by PEP 335, and building on the circuit breaking protocol
described in PEP 532, this PEP proposes a change to the definition of
chained comparisons, where the comparison chaining will be updated to
use the left-associative circuit breaking operator (else) rather than
the logical disjunction operator (and) if the left hand comparison
returns a circuit breaker as its result.

While there are some practical complexities arising from the current
handling of single-valued arrays in NumPy, this change should be
sufficient to allow elementwise chained comparison operations for
matrices, where the result is a matrix of boolean values, rather than
raising ValueError or tautologically returning True (indicating a
non-empty matrix).

Relationship with other PEPs

This PEP has been extracted from earlier iterations of PEP 532, as a
follow-on use case for the circuit breaking protocol, rather than an
essential part of its introduction.

The specific proposal in this PEP to handle the element-wise comparison
use case by changing the semantic definition of comparison chaining is
drawn directly from Guido's rejection of PEP 335.

Specification

A chained comparison like 0 < x < 10 written as:

    LEFT_BOUND LEFT_OP EXPR RIGHT_OP RIGHT_BOUND

is currently roughly semantically equivalent to:

    _expr = EXPR
    _lhs_result = LEFT_BOUND LEFT_OP _expr
    _expr_result = _lhs_result and (_expr RIGHT_OP RIGHT_BOUND)

Using the circuit breaking concepts introduced in PEP 532, this PEP
proposes that comparison chaining be changed to explicitly check if the
left comparison returns a circuit breaker, and if so, use else rather
than and to implement the comparison chaining:

    _expr = EXPR
    _lhs_result = LEFT_BOUND LEFT_OP _expr
    if hasattr(type(_lhs_result), "__else__"):
        _expr_result = _lhs_result else (_expr RIGHT_OP RIGHT_BOUND)
    else:
        _expr_result = _lhs_result and (_expr RIGHT_OP RIGHT_BOUND)

This allows types like NumPy arrays to control the behaviour of chained
comparisons by returning suitably defined circuit breakers from
comparison operations.

The expansion of this logic to an arbitrary number of chained comparison
operations would be the same as the existing expansion for and.

Rationale

In ultimately rejecting PEP 335, Guido van Rossum noted[1]:

  The NumPy folks brought up a somewhat separate issue: for them, the
  most common use case is chained comparisons (e.g. A < B < C).

To understand this observation, we first need to look at how comparisons
work with NumPy arrays:

    >>> import numpy as np
    >>> increasing = np.arange(5)
    >>> increasing
    array([0, 1, 2, 3, 4])
    >>> decreasing = np.arange(4, -1, -1)
    >>> decreasing
    array([4, 3, 2, 1, 0])
    >>> increasing < decreasing
    array([ True,  True, False, False, False], dtype=bool)

Here we see that NumPy array comparisons are element-wise by default,
comparing each element in the left hand array to the corresponding
element in the right hand array, and producing a matrix of boolean
results.

If either side of the comparison is a scalar value, then it is broadcast
across the array and compared to each individual element:

    >>> 0 < increasing
    array([False,  True,  True,  True,  True], dtype=bool)
    >>> increasing < 4
    array([ True,  True,  True,  True, False], dtype=bool)

However, this broadcasting idiom breaks down if we attempt to use
chained comparisons:

    >>> 0 < increasing < 4
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

The problem is that internally, Python implicitly expands this chained
comparison into the form:

    >>> 0 < increasing and increasing < 4
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

And NumPy only permits implicit coercion to a boolean value for
single-element arrays where a.any() and a.all() can be assured of having
the same result:

    >>> np.array([False]) and np.array([False])
    array([False], dtype=bool)
    >>> np.array([False]) and np.array([True])
    array([False], dtype=bool)
    >>> np.array([True]) and np.array([False])
    array([False], dtype=bool)
    >>> np.array([True]) and np.array([True])
    array([ True], dtype=bool)

The proposal in this PEP would allow this situation to be changed by
updating the definition of element-wise comparison operations in NumPy
to return a dedicated subclass that implements the new circuit breaking
protocol and also changes the result array's interpretation in a boolean
context to always return False and hence never trigger the
short-circuiting behaviour:

    class ComparisonResultArray(np.ndarray):
        def __bool__(self):
            # Element-wise comparison chaining never short-circuits
            return False
        def _raise_NotImplementedError(self):
            msg = ("Comparison array truth values are ambiguous outside "
                   "chained comparisons. Use a.any() or a.all()")
            raise NotImplementedError(msg)
        def __not__(self):
            self._raise_NotImplementedError()
        def __then__(self, result):
            self._raise_NotImplementedError()
        def __else__(self, result):
            return np.logical_and(self, other.view(ComparisonResultArray))

With this change, the chained comparison example above would be able to
return:

    >>> 0 < increasing < 4
    ComparisonResultArray([ False,  True,  True,  True, False], dtype=bool)

Implementation

Actual implementation has been deferred pending in-principle interest in
the idea of making the changes proposed in PEP 532.

...TBD...

References

Copyright

This document has been placed in the public domain under the terms of
the CC0 1.0 license: https://creativecommons.org/publicdomain/zero/1.0/

[1] PEP 335 rejection notification
(https://mail.python.org/pipermail/python-dev/2012-March/117510.html)