text:
PEP: 282 Title: A Logging System Author: Vinay Sajip <vinay_sajip at
red-dove.com>, Trent Mick<EMAIL_ADDRESS>Status: Final Type:
Standards Track Content-Type: text/x-rst Created: 04-Feb-2002
Python-Version: 2.3 Post-History:

Abstract

This PEP describes a proposed logging package for Python's standard
library.

Basically the system involves the user creating one or more logger
objects on which methods are called to log debugging notes, general
information, warnings, errors etc. Different logging 'levels' can be
used to distinguish important messages from less important ones.

A registry of named singleton logger objects is maintained so that

1)  different logical logging streams (or 'channels') exist (say, one
    for 'zope.zodb' stuff and another for 'mywebsite'-specific stuff)
2)  one does not have to pass logger object references around.

The system is configurable at runtime. This configuration mechanism
allows one to tune the level and type of logging done while not touching
the application itself.

Motivation

If a single logging mechanism is enshrined in the standard library, 1)
logging is more likely to be done 'well', and 2) multiple libraries will
be able to be integrated into larger applications which can be logged
reasonably coherently.

Influences

This proposal was put together after having studied the following
logging packages:

-   java.util.logging in JDK 1.4 (a.k.a. JSR047)[1]
-   log4j[2]
-   the Syslog package from the Protomatter project[3]
-   MAL's mx.Log package[4]

Simple Example

This shows a very simple example of how the logging package can be used
to generate simple logging output on stderr.

    --------- mymodule.py -------------------------------
    import logging
    log = logging.getLogger("MyModule")

    def doIt():
            log.debug("Doin' stuff...")
            #do stuff...
            raise TypeError, "Bogus type error for testing"
    -----------------------------------------------------

    --------- myapp.py ----------------------------------
    import mymodule, logging

    logging.basicConfig()

    log = logging.getLogger("MyApp")

    log.info("Starting my app")
    try:
            mymodule.doIt()
    except Exception, e:
            log.exception("There was a problem.")
    log.info("Ending my app")
    -----------------------------------------------------

    $ python myapp.py

    INFO:MyApp: Starting my app
    DEBUG:MyModule: Doin' stuff...
    ERROR:MyApp: There was a problem.
    Traceback (most recent call last):
            File "myapp.py", line 9, in ?
                    mymodule.doIt()
            File "mymodule.py", line 7, in doIt
                    raise TypeError, "Bogus type error for testing"
    TypeError: Bogus type error for testing

    INFO:MyApp: Ending my app

The above example shows the default output format. All aspects of the
output format should be configurable, so that you could have output
formatted like this:

    2002-04-19 07:56:58,174 MyModule   DEBUG - Doin' stuff...

    or just

    Doin' stuff...

Control Flow

Applications make logging calls on Logger objects. Loggers are organized
in a hierarchical namespace and child Loggers inherit some logging
properties from their parents in the namespace.

Logger names fit into a "dotted name" namespace, with dots (periods)
indicating sub-namespaces. The namespace of logger objects therefore
corresponds to a single tree data structure.

-   "" is the root of the namespace
-   "Zope" would be a child node of the root
-   "Zope.ZODB" would be a child node of "Zope"

These Logger objects create LogRecord objects which are passed to
Handler objects for output. Both Loggers and Handlers may use logging
levels and (optionally) Filters to decide if they are interested in a
particular LogRecord. When it is necessary to output a LogRecord
externally, a Handler can (optionally) use a Formatter to localize and
format the message before sending it to an I/O stream.

Each Logger keeps track of a set of output Handlers. By default all
Loggers also send their output to all Handlers of their ancestor
Loggers. Loggers may, however, also be configured to ignore Handlers
higher up the tree.

The APIs are structured so that calls on the Logger APIs can be cheap
when logging is disabled. If logging is disabled for a given log level,
then the Logger can make a cheap comparison test and return. If logging
is enabled for a given log level, the Logger is still careful to
minimize costs before passing the LogRecord into the Handlers. In
particular, localization and formatting (which are relatively expensive)
are deferred until the Handler requests them.

The overall Logger hierarchy can also have a level associated with it,
which takes precedence over the levels of individual Loggers. This is
done through a module-level function:

    def disable(lvl):
        """
        Do not generate any LogRecords for requests with a severity less
        than 'lvl'.
        """
        ...

Levels

The logging levels, in increasing order of importance, are:

-   DEBUG
-   INFO
-   WARN
-   ERROR
-   CRITICAL

The term CRITICAL is used in preference to FATAL, which is used by
log4j. The levels are conceptually the same - that of a serious, or very
serious, error. However, FATAL implies death, which in Python implies a
raised and uncaught exception, traceback, and exit. Since the logging
module does not enforce such an outcome from a FATAL-level log entry, it
makes sense to use CRITICAL in preference to FATAL.

These are just integer constants, to allow simple comparison of
importance. Experience has shown that too many levels can be confusing,
as they lead to subjective interpretation of which level should be
applied to any particular log request.

Although the above levels are strongly recommended, the logging system
should not be prescriptive. Users may define their own levels, as well
as the textual representation of any levels. User defined levels must,
however, obey the constraints that they are all positive integers and
that they increase in order of increasing severity.

User-defined logging levels are supported through two module-level
functions:

    def getLevelName(lvl):
            """Return the text for level 'lvl'."""
            ...

    def addLevelName(lvl, lvlName):
            """
            Add the level 'lvl' with associated text 'levelName', or
            set the textual representation of existing level 'lvl' to be
            'lvlName'."""
            ...

Loggers

Each Logger object keeps track of a log level (or threshold) that it is
interested in, and discards log requests below that level.

A Manager class instance maintains the hierarchical namespace of named
Logger objects. Generations are denoted with dot-separated names: Logger
"foo" is the parent of Loggers "foo.bar" and "foo.baz".

The Manager class instance is a singleton and is not directly exposed to
users, who interact with it using various module-level functions.

The general logging method is:

    class Logger:
        def log(self, lvl, msg, *args, **kwargs):
            """Log 'str(msg) % args' at logging level 'lvl'."""
            ...

However, convenience functions are defined for each logging level:

    class Logger:
        def debug(self, msg, *args, **kwargs): ...
        def info(self, msg, *args, **kwargs): ...
        def warn(self, msg, *args, **kwargs): ...
        def error(self, msg, *args, **kwargs): ...
        def critical(self, msg, *args, **kwargs): ...

Only one keyword argument is recognized at present - "exc_info". If
true, the caller wants exception information to be provided in the
logging output. This mechanism is only needed if exception information
needs to be provided at any logging level. In the more common case,
where exception information needs to be added to the log only when
errors occur, i.e. at the ERROR level, then another convenience method
is provided:

    class Logger:
        def exception(self, msg, *args): ...

This should only be called in the context of an exception handler, and
is the preferred way of indicating a desire for exception information in
the log. The other convenience methods are intended to be called with
exc_info only in the unusual situation where you might want to provide
exception information in the context of an INFO message, for example.

The "msg" argument shown above will normally be a format string;
however, it can be any object x for which str(x) returns the format
string. This facilitates, for example, the use of an object which
fetches a locale- specific message for an internationalized/localized
application, perhaps using the standard gettext module. An outline
example:

    class Message:
        """Represents a message"""
        def __init__(self, id):
            """Initialize with the message ID"""

        def __str__(self):
            """Return an appropriate localized message text"""

    ...

    logger.info(Message("abc"), ...)

Gathering and formatting data for a log message may be expensive, and a
waste if the logger was going to discard the message anyway. To see if a
request will be honoured by the logger, the isEnabledFor() method can be
used:

    class Logger:
        def isEnabledFor(self, lvl):
            """
            Return true if requests at level 'lvl' will NOT be
            discarded.
            """
            ...

so instead of this expensive and possibly wasteful DOM to XML
conversion:

    ...
    hamletStr = hamletDom.toxml()
    log.info(hamletStr)
    ...

one can do this:

    if log.isEnabledFor(logging.INFO):
        hamletStr = hamletDom.toxml()
        log.info(hamletStr)

When new loggers are created, they are initialized with a level which
signifies "no level". A level can be set explicitly using the setLevel()
method:

    class Logger:
        def setLevel(self, lvl): ...

If a logger's level is not set, the system consults all its ancestors,
walking up the hierarchy until an explicitly set level is found. That is
regarded as the "effective level" of the logger, and can be queried via
the getEffectiveLevel() method:

    def getEffectiveLevel(self): ...

Loggers are never instantiated directly. Instead, a module-level
function is used:

    def getLogger(name=None): ...

If no name is specified, the root logger is returned. Otherwise, if a
logger with that name exists, it is returned. If not, a new logger is
initialized and returned. Here, "name" is synonymous with "channel
name".

Users can specify a custom subclass of Logger to be used by the system
when instantiating new loggers:

    def setLoggerClass(klass): ...

The passed class should be a subclass of Logger, and its __init__ method
should call Logger.__init__.

Handlers

Handlers are responsible for doing something useful with a given
LogRecord. The following core Handlers will be implemented:

-   StreamHandler: A handler for writing to a file-like object.
-   FileHandler: A handler for writing to a single file or set of
    rotating files.
-   SocketHandler: A handler for writing to remote TCP ports.
-   DatagramHandler: A handler for writing to UDP sockets, for low-cost
    logging. Jeff Bauer already had such a system[5].
-   MemoryHandler: A handler that buffers log records in memory until
    the buffer is full or a particular condition occurs [6].
-   SMTPHandler: A handler for sending to email addresses via SMTP.
-   SysLogHandler: A handler for writing to Unix syslog via UDP.
-   NTEventLogHandler: A handler for writing to event logs on Windows
    NT, 2000 and XP.
-   HTTPHandler: A handler for writing to a Web server with either GET
    or POST semantics.

Handlers can also have levels set for them using the setLevel() method:

    def setLevel(self, lvl): ...

The FileHandler can be set up to create a rotating set of log files. In
this case, the file name passed to the constructor is taken as a "base"
file name. Additional file names for the rotation are created by
appending .1, .2, etc. to the base file name, up to a maximum as
specified when rollover is requested. The setRollover method is used to
specify a maximum size for a log file and a maximum number of backup
files in the rotation.

    def setRollover(maxBytes, backupCount): ...

If maxBytes is specified as zero, no rollover ever occurs and the log
file grows indefinitely. If a non-zero size is specified, when that size
is about to be exceeded, rollover occurs. The rollover method ensures
that the base file name is always the most recent, .1 is the next most
recent, .2 the next most recent after that, and so on.

There are many additional handlers implemented in the test/example
scripts provided with[7] - for example, XMLHandler and SOAPHandler.

LogRecords

A LogRecord acts as a receptacle for information about a logging event.
It is little more than a dictionary, though it does define a getMessage
method which merges a message with optional runarguments.

Formatters

A Formatter is responsible for converting a LogRecord to a string
representation. A Handler may call its Formatter before writing a
record. The following core Formatters will be implemented:

-   Formatter: Provide printf-like formatting, using the % operator.
-   BufferingFormatter: Provide formatting for multiple messages, with
    header and trailer formatting support.

Formatters are associated with Handlers by calling setFormatter() on a
handler:

    def setFormatter(self, form): ...

Formatters use the % operator to format the logging message. The format
string should contain %(name)x and the attribute dictionary of the
LogRecord is used to obtain message-specific data. The following
attributes are provided:

  --------------------- -------------------------------------------------------------------------------------------------------------------------------------------------
  %(name)s              Name of the logger (logging channel)
  %(levelno)s           Numeric logging level for the message (DEBUG, INFO, WARN, ERROR, CRITICAL)
  %(levelname)s         Text logging level for the message ("DEBUG", "INFO", "WARN", "ERROR", "CRITICAL")
  %(pathname)s          Full pathname of the source file where the logging call was issued (if available)
  %(filename)s          Filename portion of pathname
  %(module)s            Module from which logging call was made
  %(lineno)d            Source line number where the logging call was issued (if available)
  %(created)f           Time when the LogRecord was created (time.time() return value)
  %(asctime)s           Textual time when the LogRecord was created
  %(msecs)d             Millisecond portion of the creation time
  %(relativeCreated)d   Time in milliseconds when the LogRecord was created, relative to the time the logging module was loaded (typically at application startup time)
  %(thread)d            Thread ID (if available)
  %(message)s           The result of record.getMessage(), computed just as the record is emitted
  --------------------- -------------------------------------------------------------------------------------------------------------------------------------------------

If a formatter sees that the format string includes "(asctime)s", the
creation time is formatted into the LogRecord's asctime attribute. To
allow flexibility in formatting dates, Formatters are initialized with a
format string for the message as a whole, and a separate format string
for date/time. The date/time format string should be in time.strftime
format. The default value for the message format is "%(message)s". The
default date/time format is ISO8601.

The formatter uses a class attribute, "converter", to indicate how to
convert a time from seconds to a tuple. By default, the value of
"converter" is "time.localtime". If needed, a different converter (e.g.
"time.gmtime") can be set on an individual formatter instance, or the
class attribute changed to affect all formatter instances.

Filters

When level-based filtering is insufficient, a Filter can be called by a
Logger or Handler to decide if a LogRecord should be output. Loggers and
Handlers can have multiple filters installed, and any one of them can
veto a LogRecord being output.

    class Filter:
        def filter(self, record):
            """
            Return a value indicating true if the record is to be
            processed.  Possibly modify the record, if deemed
            appropriate by the filter.
            """

The default behaviour allows a Filter to be initialized with a Logger
name. This will only allow through events which are generated using the
named logger or any of its children. For example, a filter initialized
with "A.B" will allow events logged by loggers "A.B", "A.B.C",
"A.B.C.D", "A.B.D" etc. but not "A.BB", "B.A.B" etc. If initialized with
the empty string, all events are passed by the Filter. This filter
behaviour is useful when it is desired to focus attention on one
particular area of an application; the focus can be changed simply by
changing a filter attached to the root logger.

There are many examples of Filters provided in[8].

Configuration

The main benefit of a logging system like this is that one can control
how much and what logging output one gets from an application without
changing that application's source code. Therefore, although
configuration can be performed through the logging API, it must also be
possible to change the logging configuration without changing an
application at all. For long-running programs like Zope, it should be
possible to change the logging configuration while the program is
running.

Configuration includes the following:

-   What logging level a logger or handler should be interested in.
-   What handlers should be attached to which loggers.
-   What filters should be attached to which handlers and loggers.
-   Specifying attributes specific to certain handlers and filters.

In general each application will have its own requirements for how a
user may configure logging output. However, each application will
specify the required configuration to the logging system through a
standard mechanism.

The most simple configuration is that of a single handler, writing to
stderr, attached to the root logger. This configuration is set up by
calling the basicConfig() function once the logging module has been
imported.

    def basicConfig(): ...

For more sophisticated configurations, this PEP makes no specific
proposals, for the following reasons:

-   A specific proposal may be seen as prescriptive.
-   Without the benefit of wide practical experience in the Python
    community, there is no way to know whether any given configuration
    approach is a good one. That practice can't really come until the
    logging module is used, and that means until after Python 2.3 has
    shipped.
-   There is a likelihood that different types of applications may
    require different configuration approaches, so that no "one size
    fits all".

The reference implementation[9] has a working configuration file format,
implemented for the purpose of proving the concept and suggesting one
possible alternative. It may be that separate extension modules, not
part of the core Python distribution, are created for logging
configuration and log viewing, supplemental handlers and other features
which are not of interest to the bulk of the community.

Thread Safety

The logging system should support thread-safe operation without any
special action needing to be taken by its users.

Module-Level Functions

To support use of the logging mechanism in short scripts and small
applications, module-level functions debug(), info(), warn(), error(),
critical() and exception() are provided. These work in the same way as
the correspondingly named methods of Logger - in fact they delegate to
the corresponding methods on the root logger. A further convenience
provided by these functions is that if no configuration has been done,
basicConfig() is automatically called.

At application exit, all handlers can be flushed by calling the
function:

    def shutdown(): ...

This will flush and close all handlers.

Implementation

The reference implementation is Vinay Sajip's logging module[10].

Packaging

The reference implementation is implemented as a single module. This
offers the simplest interface - all users have to do is "import logging"
and they are in a position to use all the functionality available.

References

Copyright

This document has been placed in the public domain.

[1] java.util.logging
http://java.sun.com/j2se/1.4/docs/guide/util/logging/

[2] log4j: a Java logging package https://logging.apache.org/log4j/

[3] Protomatter's Syslog
http://protomatter.sourceforge.net/1.1.6/index.html
http://protomatter.sourceforge.net/1.1.6/javadoc/com/protomatter/syslog/syslog-whitepaper.html

[4] MAL mentions his mx.Log logging module:
https://mail.python.org/pipermail/python-dev/2002-February/019767.html

[5] Jeff Bauer's Mr. Creosote
http://starship.python.net/crew/jbauer/creosote/

[6] java.util.logging
http://java.sun.com/j2se/1.4/docs/guide/util/logging/

[7] Vinay Sajip's logging module.
https://old.red-dove.com/python_logging.html

[8] Vinay Sajip's logging module.
https://old.red-dove.com/python_logging.html

[9] Vinay Sajip's logging module.
https://old.red-dove.com/python_logging.html

[10] Vinay Sajip's logging module.
https://old.red-dove.com/python_logging.html


text:
PEP: 606 Title: Python Compatibility Version Author: Victor Stinner
<EMAIL_ADDRESS>Status: Rejected Type: Standards Track
Content-Type: text/x-rst Created: 18-Oct-2019 Python-Version: 3.9

Abstract

Add sys.set_python_compat_version(version) to enable partial
compatibility with requested Python version. Add
sys.get_python_compat_version().

Modify a few functions in the standard library to implement partial
compatibility with Python 3.8.

Add sys.set_python_min_compat_version(version) to deny backward
compatibility with Python versions older than version.

Add -X compat_version=VERSION and -X min_compat_version=VERSION command
line options. Add PYTHONCOMPATVERSION and PYTHONCOMPATMINVERSION
environment variables.

Rationale

The need to evolve frequently

To remain relevant and useful, Python has to evolve frequently; some
enhancements require incompatible changes. Any incompatible change can
break an unknown number of Python projects. Developers can decide to not
implement a feature because of that.

Users want to get the latest Python version to obtain new features and
better performance. A few incompatible changes can prevent them from
using their applications on the latest Python version.

This PEP proposes to add a partial compatibility with old Python
versions as a tradeoff to fit both use cases.

The main issue with the migration from Python 2 to Python 3 is not that
Python 3 is backward incompatible, but how incompatible changes were
introduced.

Partial compatibility to minimize the Python maintenance burden

While technically it would be possible to provide full compatibility
with old Python versions, this PEP proposes to minimize the number of
functions handling backward compatibility to reduce the maintenance
burden of the Python project (CPython).

Each change introducing backport compatibility to a function should be
properly discussed to estimate the maintenance cost in the long-term.

Backward compatibility code will be dropped on each Python release, on a
case-by-case basis. Each compatibility function can be supported for a
different number of Python releases depending on its maintenance cost
and the estimated risk (number of broken projects) if it's removed.

The maintenance cost does not only come from the code implementing the
backward compatibility, but also comes from the additional tests.

Cases excluded from backward compatibility

The performance overhead of any compatibility code must be low when
sys.set_python_compat_version() is not called.

The C API is out of the scope of this PEP: Py_LIMITED_API macro and the
stable ABI are solving this problem differently, see the PEP 384:
Defining a Stable ABI <384>.

Security fixes which break backward compatibility on purpose will not
get a compatibility layer; security matters more than compatibility. For
example, http.client.HTTPSConnection was modified in Python 3.4.3 to
performs all the necessary certificate and hostname checks by default.
It was a deliberate change motivated by PEP 476: Enabling
certificate verification by default for stdlib http clients
<476> (bpo-22417).

The Python language does not provide backward compatibility.

Changes which are not clearly incompatible are not covered by this PEP.
For example, Python 3.9 changed the default protocol in the pickle
module to Protocol 4 which was first introduced in Python 3.4. This
change is backward compatible up to Python 3.4. There is no need to use
the Protocol 3 by default when compatibility with Python 3.8 is
requested.

The new DeprecationWarning and PendingDeprecatingWarning warnings in
Python 3.9 will not be disabled in Python 3.8 compatibility mode. If a
project runs its test suite using -Werror (treat any warning as an
error), these warnings must be fixed, or specific deprecation warnings
must be ignored on a case-by-case basis.

Upgrading a project to a newer Python

Without backward compatibility, all incompatible changes must be fixed
at once, which can be a blocker issue. It is even worse when a project
is upgraded to a newer Python which is separated by multiple releases
from the old Python.

Postponing an upgrade only makes things worse: each skipped release adds
more incompatible changes. The technical debt only steadily increases
over time.

With backward compatibility, it becomes possible to upgrade Python
incrementally in a project, without having to fix all of the issues at
once.

The "all-or-nothing" is a showstopper to port large Python 2 code bases
to Python 3. The list of incompatible changes between Python 2 and
Python 3 is long, and it's getting longer with each Python 3.x release.

Cleaning up Python and DeprecationWarning

One of the Zen of Python (PEP 20)
<20> motto is:

  There should be one-- and preferably only one --obvious way to do it.

When Python evolves, new ways inevitably emerge. DeprecationWarnings are
emitted to suggest using the new way, but many developers ignore these
warnings, which are silent by default (except in the __main__ module:
see the PEP 565). Some developers simply ignore all warnings when there
are too many warnings, thus only bother with exceptions when the
deprecated code is removed.

Sometimes, supporting both ways has a minor maintenance cost, but
developers prefer to drop the old way to clean up their code. These
kinds of changes are backward incompatible.

Some developers can take the end of the Python 2 support as an
opportunity to push even more incompatible changes than usual.

Adding an opt-in backward compatibility prevents the breaking of
applications and allows developers to continue doing these cleanups.

Redistribute the maintenance burden

The backward compatibility involves authors of incompatible changes more
in the upgrade path.

Examples of backward compatibility

collections ABC aliases

collections.abc aliases to ABC classes have been removed from the
collections module in Python 3.9, after being deprecated since Python
3.3. For example, collections.Mapping no longer exists.

In Python 3.6, aliases were created in collections/__init__.py by
from _collections_abc import *.

In Python 3.7, a __getattr__() has been added to the collections module
to emit a DeprecationWarning upon first access to an attribute:

    def __getattr__(name):
        # For backwards compatibility, continue to make the collections ABCs
        # through Python 3.6 available through the collections module.
        # Note: no new collections ABCs were added in Python 3.7
        if name in _collections_abc.__all__:
            obj = getattr(_collections_abc, name)
            import warnings
            warnings.warn("Using or importing the ABCs from 'collections' instead "
                          "of from 'collections.abc' is deprecated since Python 3.3, "
                          "and in 3.9 it will be removed.",
                          DeprecationWarning, stacklevel=2)
            globals()[name] = obj
            return obj
        raise AttributeError(f'module {__name__!r} has no attribute {name!r}')

Compatibility with Python 3.8 can be restored in Python 3.9 by adding
back the __getattr__() function, but only when backward compatibility is
requested:

    def __getattr__(name):
        if (sys.get_python_compat_version() < (3, 9)
           and name in _collections_abc.__all__):
            ...
        raise AttributeError(f'module {__name__!r} has no attribute {name!r}')

Deprecated open() "U" mode

The "U" mode of open() is deprecated since Python 3.4 and emits a
DeprecationWarning. bpo-37330 proposes to drop this mode:
open(filename, "rU") would raise an exception.

This change falls into the "cleanup" category: it is not required to
implement a feature.

A backward compatibility mode would be trivial to implement and would be
welcomed by users.

Specification

sys functions

Add 3 functions to the sys module:

-   sys.set_python_compat_version(version): set the Python compatibility
    version. If it has been called previously, use the minimum of
    requested versions. Raise an exception if
    sys.set_python_min_compat_version(min_version) has been called and
    version < min_version. version must be greater than or equal to
    (3, 0).
-   sys.set_python_min_compat_version(min_version): set the minimum
    compatibility version. Raise an exception if
    sys.set_python_compat_version(old_version) has been called
    previously and old_version < min_version. min_version must be
    greater than or equal to (3, 0).
-   sys.get_python_compat_version(): get the Python compatibility
    version. Return a tuple of 3 integers.

A version must a tuple of 2 or 3 integers. (major, minor) version is
equivalent to (major, minor, 0).

By default, sys.get_python_compat_version() returns the current Python
version.

For example, to request compatibility with Python 3.8.0:

    import collections

    sys.set_python_compat_version((3, 8))

    # collections.Mapping alias, removed from Python 3.9, is available
    # again, even if collections has been imported before calling
    # set_python_compat_version().
    parent = collections.Mapping

Obviously, calling sys.set_python_compat_version(version) has no effect
on code executed before the call. Use -X compat_version=VERSION command
line option or PYTHONCOMPATVERSIONVERSION=VERSION environment variable
to set the compatibility version at Python startup.

Command line

Add -X compat_version=VERSION and -X min_compat_version=VERSION command
line options: call respectively sys.set_python_compat_version() and
sys.set_python_min_compat_version(). VERSION is a version string with 2
or 3 numbers (major.minor.micro or major.minor). For example,
-X compat_version=3.8 calls sys.set_python_compat_version((3, 8)).

Add PYTHONCOMPATVERSIONVERSION=VERSION and
PYTHONCOMPATMINVERSION=VERSION=VERSION environment variables: call
respectively sys.set_python_compat_version() and
sys.set_python_min_compat_version(). VERSION is a version string with
the same format as the command line options.

Backwards Compatibility

Introducing the sys.set_python_compat_version() function means that an
application will behave differently depending on the compatibility
version. Moreover, since the version can be decreased multiple times,
the application can behave differently depending on the import order.

Python 3.9 with sys.set_python_compat_version((3, 8)) is not fully
compatible with Python 3.8: the compatibility is only partial.

Security Implications

sys.set_python_compat_version() must not disable security fixes.

Alternatives

Provide a workaround for each incompatible change

An application can work around most incompatible changes which impacts
it.

For example, collections aliases can be added back using:

    import collections.abc
    collections.Mapping = collections.abc.Mapping
    collections.Sequence = collections.abc.Sequence

Handle backward compatibility in the parser

The parser is modified to support multiple versions of the Python
language (grammar).

The current Python parser cannot be easily modified for that. AST and
grammar are hardcoded to a single Python version.

In Python 3.8, compile() has an undocumented _feature_version to not
consider async and await as keywords.

The latest major language backward incompatible change was Python 3.7
which made async and await real keywords. It seems like Twisted was the
only affected project, and Twisted had a single affected function (it
used a parameter called async).

Handling backward compatibility in the parser seems quite complex, not
only to modify the parser, but also for developers who have to check
which version of the Python language is used.

from __future__ import python38_syntax

Add pythonXY_syntax to the __future__ module. It would enable backward
compatibility with Python X.Y syntax, but only for the current file.

With this option, there is no need to change
sys.implementation.cache_tag to use a different .pyc filename, since the
parser will always produce the same output for the same input (except
for the optimization level).

For example:

    from __future__ import python35_syntax

    async = 1
    await = 2

Update cache_tag

Modify the parser to use sys.get_python_compat_version() to choose the
version of the Python language.

sys.set_python_compat_version() updates sys.implementation.cache_tag to
include the compatibility version without the micro version as a suffix.
For example, Python 3.9 uses 'cpython-39' by default, but
sys.set_python_compat_version((3, 7, 2)) sets cache_tag to
'cpython-39-37'. Changes to the Python language are now allowed in micro
releases.

One problem is that import asyncio is likely to fail if
sys.set_python_compat_version((3, 6)) has been called previously. The
code of the asyncio module requires async and await to be real keywords
(change done in Python 3.7).

Another problem is that regular users cannot write .pyc files into
system directories, and so cannot create them on demand. It means that
.pyc optimization cannot be used in the backward compatibility mode.

One solution for that is to modify the Python installer and Python
package installers to precompile .pyc files not only for the current
Python version, but also for multiple older Python versions (up to
Python 3.0?).

Each .py file would have 3n .pyc files (3 optimization levels), where n
is the number of supported Python versions. For example, it means 6 .pyc
files, instead of 3, to support Python 3.8 and Python 3.9.

Temporary moratorium on incompatible changes

In 2009, PEP 3003 "Python Language Moratorium" proposed a temporary
moratorium (suspension) of all changes to the Python language syntax,
semantics, and built-ins for Python 3.1 and Python 3.2.

In May 2018, during the PEP 572 discussions, it was also proposed to
slow down Python changes: see the python-dev thread Slow down...

Barry Warsaw's call on this:

  I don’t believe that the way for Python to remain relevant and useful
  for the next 10 years is to cease all language evolution. Who knows
  what the computing landscape will look like in 5 years, let alone 10?
  Something as arbitrary as a 10-year moratorium is (again, IMHO) a
  death sentence for the language.

PEP 387

PEP 387 -- Backwards Compatibility Policy
<387> proposes a process to make incompatible changes. The main point is
the 4th step of the process:

  See if there's any feedback. Users not involved in the original
  discussions may comment now after seeing the warning. Perhaps
  reconsider.

PEP 497

PEP 497 -- A standard mechanism for backward compatibility
<497> proposes different solutions to provide backward compatibility.

Except for the __past__ mechanism idea, PEP 497 does not propose
concrete solutions:

  When an incompatible change to core language syntax or semantics is
  being made, Python-dev's policy is to prefer and expect that, wherever
  possible, a mechanism for backward compatibility be considered and
  provided for future Python versions after the breaking change is
  adopted by default, in addition to any mechanisms proposed for forward
  compatibility such as new future_statements.

Examples of incompatible changes

Python 3.8

Examples of Python 3.8 incompatible changes:

-   (During beta phase) PyCode_New() required a new parameter: it broke
    all Cython extensions (all projects distributing precompiled Cython
    code). This change has been reverted during the 3.8 beta phase and a
    new PyCode_NewWithPosOnlyArgs() function was added instead.
-   types.CodeType requires an additional mandatory parameter. The
    CodeType.replace() function was added to help projects to no longer
    depend on the exact signature of the CodeType constructor.
-   C extensions are no longer linked to libpython.
-   sys.abiflags changed from 'm' to an empty string. For example,
    python3.8m program is gone.
-   The C structure PyInterpreterState was made opaque.
    -   Blender:
        -   https://bugzilla.redhat.com/show_bug.cgi?id=1734980#c6
        -   https://developer.blender.org/D6038
-   XML attribute order: bpo-34160. Broken projects:
    -   coverage
    -   docutils
    -   pcs
    -   python-glyphsLib

Backward compatibility cannot be added for all these changes. For
example, changes in the C API and in the build system are out of the
scope of this PEP.

See What’s New In Python 3.8: API and Feature Removals for all changes.

See also the Porting to Python 3.8 section of What’s New In Python 3.8.

Python 3.7

Examples of Python 3.7 incompatible changes:

-   async and await are now reserved keywords.
-   Several undocumented internal imports were removed. One example is
    that os.errno is no longer available; use import errno directly
    instead. Note that such undocumented internal imports may be removed
    any time without notice, even in micro version releases.
-   Unknown escapes consisting of '\' and an ASCII letter in replacement
    templates for re.sub() were deprecated in Python 3.5, and will now
    cause an error.
-   The asyncio.windows_utils.socketpair() function has been removed: it
    was an alias to socket.socketpair().
-   asyncio no longer exports the selectors and _overlapped modules as
    asyncio.selectors and asyncio._overlapped. Replace
    from asyncio import selectors with import selectors.
-   PEP 479 is enabled for all code in Python 3.7, meaning that
    StopIteration exceptions raised directly or indirectly in coroutines
    and generators are transformed into RuntimeError exceptions.
-   socketserver.ThreadingMixIn.server_close() now waits until all
    non-daemon threads complete. Set the new block_on_close class
    attribute to False to get the pre-3.7 behaviour.
-   The struct.Struct.format type is now str instead of bytes.
-   repr for datetime.timedelta has changed to include the keyword
    arguments in the output.
-   tracemalloc.Traceback frames are now sorted from oldest to most
    recent to be more consistent with traceback.

Adding backward compatibility for most of these changes would be easy.

See also the Porting to Python 3.7 section of What’s New In Python 3.7.

Micro releases

Sometimes, incompatible changes are introduced in micro releases (micro
in major.minor.micro) to fix bugs or security vulnerabilities. Examples
include:

-   Python 3.7.2, compileall and py_compile module: the
    invalidation_mode parameter's default value is updated to None; the
    SOURCE_DATE_EPOCH environment variable no longer overrides the value
    of the invalidation_mode argument, and determines its default value
    instead.
-   Python 3.7.1, xml modules: the SAX parser no longer processes
    general external entities by default to increase security by
    default.
-   Python 3.5.2, os.urandom(): on Linux, if the getrandom() syscall
    blocks (the urandom entropy pool is not initialized yet), fall back
    on reading /dev/urandom.
-   Python 3.5.1, sys.setrecursionlimit(): a RecursionError exception is
    now raised if the new limit is too low at the current recursion
    depth.
-   Python 3.4.4, ssl.create_default_context(): RC4 was dropped from the
    default cipher string.
-   Python 3.4.3, http.client: HTTPSConnection now performs all the
    necessary certificate and hostname checks by default.
-   Python 3.4.2, email.message: EmailMessage.is_attachment() is now a
    method instead of a property, for consistency with
    Message.is_multipart().
-   Python 3.4.1, os.makedirs(name, mode=0o777, exist_ok=False): Before
    Python 3.4.1, if exist_ok was True and the directory existed,
    makedirs() would still raise an error if mode did not match the mode
    of the existing directory. Since this behavior was impossible to
    implement safely, it was removed in Python 3.4.1 (bpo-21082).

Examples of changes made in micro releases which are not backward
incompatible:

-   ssl.OP_NO_TLSv1_3 constant was added to 2.7.15, 3.6.3 and 3.7.0 for
    backwards compatibility with OpenSSL 1.0.2.
-   typing.AsyncContextManager was added to Python 3.6.2.
-   The zipfile module accepts a path-like object since Python 3.6.2.
-   loop.create_future() was added to Python 3.5.2 in the asyncio
    module.

No backward compatibility code is needed for these kinds of changes.

References

Accepted PEPs:

-   PEP 5 -- Guidelines for Language Evolution
    <5>
-   PEP 236 -- Back to the __future__
    <236>
-   PEP 411 -- Provisional packages in the Python standard library
    <411>
-   PEP 3002 -- Procedure for Backwards-Incompatible Changes
    <3002>

Draft PEPs:

-   PEP 602 -- Annual Release Cycle for Python
    <602>
-   PEP 605 -- A rolling feature release stream for CPython
    <605>
-   See also withdrawn PEP 598 -- Introducing incremental feature
    releases <598>

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

license: Public Domain


text:
PEP: 204 Title: Range Literals Author: Thomas Wouters
<EMAIL_ADDRESS>Status: Rejected Type: Standards Track Created:
14-Jul-2000 Python-Version: 2.0 Post-History:

After careful consideration, and a period of meditation, this proposal
has been rejected. The open issues, as well as some confusion between
ranges and slice syntax, raised enough questions for Guido not to accept
it for Python 2.0, and later to reject the proposal altogether. The new
syntax and its intentions were deemed not obvious enough.

[ TBD: Guido, amend/confirm this, please. Preferably both; this is a
PEP, it should contain all the reasons for rejection and/or
reconsideration, for future reference. ]

Introduction

This PEP describes the "range literal" proposal for Python 2.0. This PEP
tracks the status and ownership of this feature, slated for introduction
in Python 2.0. It contains a description of the feature and outlines
changes necessary to support the feature. This PEP summarizes
discussions held in mailing list forums, and provides URLs for further
information, where appropriate. The CVS revision history of this file
contains the definitive historical record.

List ranges

Ranges are sequences of numbers of a fixed stepping, often used in
for-loops. The Python for-loop is designed to iterate over a sequence
directly:

    >>> l = ['a', 'b', 'c', 'd']
    >>> for item in l:
    ...     print item
    a
    b
    c
    d

However, this solution is not always prudent. Firstly, problems arise
when altering the sequence in the body of the for-loop, resulting in the
for-loop skipping items. Secondly, it is not possible to iterate over,
say, every second element of the sequence. And thirdly, it is sometimes
necessary to process an element based on its index, which is not readily
available in the above construct.

For these instances, and others where a range of numbers is desired,
Python provides the range builtin function, which creates a list of
numbers. The range function takes three arguments, start, end and step.
start and step are optional, and default to 0 and 1, respectively.

The range function creates a list of numbers, starting at start, with a
step of step, up to, but not including end, so that range(10) produces a
list that has exactly 10 items, the numbers 0 through 9.

Using the range function, the above example would look like this:

    >>> for i in range(len(l)):
    ...     print l[i]
    a
    b
    c
    d

Or, to start at the second element of l and processing only every second
element from then on:

    >>> for i in range(1, len(l), 2):
    ...     print l[i]
    b
    d

There are several disadvantages with this approach:

-   Clarity of purpose: Adding another function call, possibly with
    extra arithmetic to determine the desired length and step of the
    list, does not improve readability of the code. Also, it is possible
    to "shadow" the builtin range function by supplying a local or
    global variable with the same name, effectively replacing it. This
    may or may not be a desired effect.
-   Efficiency: because the range function can be overridden, the Python
    compiler cannot make assumptions about the for-loop, and has to
    maintain a separate loop counter.
-   Consistency: There already is a syntax that is used to denote
    ranges, as shown below. This syntax uses the exact same arguments,
    though all optional, in the exact same way. It seems logical to
    extend this syntax to ranges, to form "range literals".

Slice Indices

In Python, a sequence can be indexed in one of two ways: retrieving a
single item, or retrieving a range of items. Retrieving a range of items
results in a new object of the same type as the original sequence,
containing zero or more items from the original sequence. This is done
using a "range notation":

    >>> l[2:4]
    ['c', 'd']

This range notation consists of zero, one or two indices separated by a
colon. The first index is the start index, the second the end. When
either is left out, they default to respectively the start and the end
of the sequence.

There is also an extended range notation, which incorporates step as
well. Though this notation is not currently supported by most builtin
types, if it were, it would work as follows:

    >>> l[1:4:2]
    ['b', 'd']

The third "argument" to the slice syntax is exactly the same as the step
argument to range(). The underlying mechanisms of the standard, and
these extended slices, are sufficiently different and inconsistent that
many classes and extensions outside of mathematical packages do not
implement support for the extended variant. While this should be
resolved, it is beyond the scope of this PEP.

Extended slices do show, however, that there is already a perfectly
valid and applicable syntax to denote ranges in a way that solve all of
the earlier stated disadvantages of the use of the range() function:

-   It is clearer, more concise syntax, which has already proven to be
    both intuitive and easy to learn.
-   It is consistent with the other use of ranges in Python (e.g.
    slices).
-   Because it is built-in syntax, instead of a builtin function, it
    cannot be overridden. This means both that a viewer can be certain
    about what the code does, and that an optimizer will not have to
    worry about range() being "shadowed".

The Proposed Solution

The proposed implementation of range-literals combines the syntax for
list literals with the syntax for (extended) slices, to form range
literals:

    >>> [1:10]
    [1, 2, 3, 4, 5, 6, 7, 8, 9]
    >>> [:5]
    [0, 1, 2, 3, 4]
    >>> [5:1:-1]
    [5, 4, 3, 2]

There is one minor difference between range literals and the slice
syntax: though it is possible to omit all of start, end and step in
slices, it does not make sense to omit end in range literals. In slices,
end would default to the end of the list, but this has no meaning in
range literals.

Reference Implementation

The proposed implementation can be found on SourceForge[1]. It adds a
new bytecode, BUILD_RANGE, that takes three arguments from the stack and
builds a list on the bases of those. The list is pushed back on the
stack.

The use of a new bytecode is necessary to be able to build ranges based
on other calculations, whose outcome is not known at compile time.

The code introduces two new functions to listobject.c, which are
currently hovering between private functions and full-fledged API calls.

PyList_FromRange() builds a list from start, end and step, returning
NULL if an error occurs. Its prototype is:

    PyObject * PyList_FromRange(long start, long end, long step)

PyList_GetLenOfRange() is a helper function used to determine the length
of a range. Previously, it was a static function in bltinmodule.c, but
is now necessary in both listobject.c and bltinmodule.c (for xrange). It
is made non-static solely to avoid code duplication. Its prototype is:

    long PyList_GetLenOfRange(long start, long end, long step)

Open issues

-   One possible solution to the discrepancy of requiring the end
    argument in range literals is to allow the range syntax to create a
    "generator", rather than a list, such as the xrange builtin function
    does. However, a generator would not be a list, and it would be
    impossible, for instance, to assign to items in the generator, or
    append to it.

    The range syntax could conceivably be extended to include tuples
    (i.e. immutable lists), which could then be safely implemented as
    generators. This may be a desirable solution, especially for large
    number arrays: generators require very little in the way of storage
    and initialization, and there is only a small performance impact in
    calculating and creating the appropriate number on request. (TBD: is
    there any at all? Cursory testing suggests equal performance even in
    the case of ranges of length 1)

    However, even if idea was adopted, would it be wise to "special
    case" the second argument, making it optional in one instance of the
    syntax, and non-optional in other cases ?

-   Should it be possible to mix range syntax with normal list literals,
    creating a single list? E.g.:

        >>> [5, 6, 1:6, 7, 9]

    to create:

        [5, 6, 1, 2, 3, 4, 5, 7, 9]

-   How should range literals interact with another proposed new
    feature, "list comprehensions" <202>? Specifically, should it be
    possible to create lists in list comprehensions? E.g.:

        >>> [x:y for x in (1, 2) y in (3, 4)]

    Should this example return a single list with multiple ranges:

        [1, 2, 1, 2, 3, 2, 2, 3]

    Or a list of lists, like so:

        [[1, 2], [1, 2, 3], [2], [2, 3]]

    However, as the syntax and semantics of list comprehensions are
    still subject of hot debate, these issues are probably best
    addressed by the "list comprehensions" PEP.

-   Range literals accept objects other than integers: it performs
    PyInt_AsLong() on the objects passed in, so as long as the objects
    can be coerced into integers, they will be accepted. The resulting
    list, however, is always composed of standard integers.

    Should range literals create a list of the passed-in type? It might
    be desirable in the cases of other builtin types, such as longs and
    strings:

        >>> [ 1L : 2L<<64 : 2<<32L ]
        >>> ["a":"z":"b"]
        >>> ["a":"z":2]

    However, this might be too much "magic" to be obvious. It might also
    present problems with user-defined classes: even if the base class
    can be found and a new instance created, the instance may require
    additional arguments to __init__, causing the creation to fail.

-   The PyList_FromRange() and PyList_GetLenOfRange() functions need to
    be classified: are they part of the API, or should they be made
    private functions?

Copyright

This document has been placed in the Public Domain.

References

[1] http://sourceforge.net/patch/?func=detailpatch&patch_id=100902&group_id=5470


text:
PEP: 733 Title: An Evaluation of Python's Public C API Author: Erlend
Egeberg Aasland<EMAIL_ADDRESS>Domenico Andreoli
<EMAIL_ADDRESS>Stefan Behnel<EMAIL_ADDRESS>Carl
Friedrich Bolz-Tereick<EMAIL_ADDRESS>Simon Cross
<EMAIL_ADDRESS>Steve Dower<EMAIL_ADDRESS>Tim
Felgentreff<EMAIL_ADDRESS>David Hewitt
<EMAIL_ADDRESS>Shantanu Jain
<hauntsaninja at gmail.com>, Wenzel Jakob<EMAIL_ADDRESS>Irit
Katriel<EMAIL_ADDRESS>Marc-Andre Lemburg<EMAIL_ADDRESS>Donghee
Na<EMAIL_ADDRESS>Karl Nelson<EMAIL_ADDRESS>Ronald
Oussoren<EMAIL_ADDRESS>Antoine Pitrou<EMAIL_ADDRESS>Neil Schemenauer<EMAIL_ADDRESS>Mark Shannon<EMAIL_ADDRESS>Stepan Sindelar<EMAIL_ADDRESS>Gregory P. Smith
<EMAIL_ADDRESS>Eric Snow<EMAIL_ADDRESS>Victor
Stinner<EMAIL_ADDRESS>Guido van Rossum<EMAIL_ADDRESS>Petr
Viktorin<EMAIL_ADDRESS>Carol Willing<EMAIL_ADDRESS>William Woodruff<EMAIL_ADDRESS>David Woods
<EMAIL_ADDRESS>Jelle Zijlstra<EMAIL_ADDRESS>Status: Draft Type: Informational Created: 16-Oct-2023 Post-History:
01-Nov-2023

Abstract

This informational PEP describes our shared view of the public C API.
The document defines:

-   purposes of the C API
-   stakeholders and their particular use cases and requirements
-   strengths of the C API
-   problems of the C API categorized into nine areas of weakness

This document does not propose solutions to any of the identified
problems. By creating a shared list of C API issues, this document will
help to guide continuing discussion about change proposals and to
identify evaluation criteria.

Introduction

Python's C API was not designed for the different purposes it currently
fulfills. It evolved from what was initially the internal API between
the C code of the interpreter and the Python language and libraries. In
its first incarnation, it was exposed to make it possible to embed
Python into C/C++ applications and to write extension modules in C/C++.
These capabilities were instrumental to the growth of Python's
ecosystem. Over the decades, the C API grew to provide different tiers
of stability, conventions changed, and new usage patterns have emerged,
such as bindings to languages other than C/C++. In the next few years,
new developments are expected to further test the C API, such as the
removal of the GIL and the development of a JIT compiler. However, this
growth was not supported by clearly documented guidelines, resulting in
inconsistent approaches to API design in different subsystems of
CPython. In addition, CPython is no longer the only implementation of
Python, and some of the design decisions made when it was, are difficult
for alternative implementations to work with [Issue 64]. In the
meantime, lessons were learned and mistakes in both the design and the
implementation of the C API were identified.

Evolving the C API is hard due to the combination of backwards
compatibility constraints and its inherent complexity, both technical
and social. Different types of users bring different, sometimes
conflicting, requirements. The tradeoff between stability and progress
is an ongoing, highly contentious topic of discussion when suggestions
are made for incremental improvements. Several proposals have been put
forward for improvement, redesign or replacement of the C API, each
representing a deep analysis of the problems. At the 2023 Language
Summit, three back-to-back sessions were devoted to different aspects of
the C API. There is general agreement that a new design can remedy the
problems that the C API has accumulated over the last 30 years, while at
the same time updating it for use cases that it was not originally
designed for.

However, there was also a sense at the Language Summit that we are
trying to discuss solutions without a clear common understanding of the
problems that we are trying to solve. We decided that we need to agree
on the current problems with the C API, before we are able to evaluate
any of the proposed solutions. We therefore created the capi-workgroup
repository on GitHub in order to collect everyone's ideas on that
question.

Over 60 different issues were created on that repository, each
describing a problem with the C API. We categorized them and identified
a number of recurring themes. The sections below mostly correspond to
these themes, and each contains a combined description of the issues
raised in that category, along with links to the individual issues. In
addition, we included a section that aims to identify the different
stakeholders of the C API, and the particular requirements that each of
them has.

C API Stakeholders

As mentioned in the introduction, the C API was originally created as
the internal interface between CPython's interpreter and the Python
layer. It was later exposed as a way for third-party developers to
extend and embed Python programs. Over the years, new types of
stakeholders emerged, with different requirements and areas of focus.
This section describes this complex state of affairs in terms of the
actions that different stakeholders need to perform through the C API.

Common Actions for All Stakeholders

There are actions which are generic, and required by all types of API
users:

-   Define functions and call them
-   Define new types
-   Create instances of builtin and user-defined types
-   Perform operations on object instances
-   Introspect objects, including types, instances, and functions
-   Raise and handle exceptions
-   Import modules
-   Access to Python's OS interface

The following sections look at the unique requirements of various
stakeholders.

Extension Writers

Extension writers are the traditional users of the C API. Their
requirements are the common actions listed above. They also commonly
need to:

-   Create new modules
-   Efficiently interface between modules at the C level

Authors of Embedded Python Applications

Applications with an embedded Python interpreter. Examples are Blender
and OBS.

They need to be able to:

-   Configure the interpreter (import paths, inittab, sys.argv, memory
    allocator, etc.).
-   Interact with the execution model and program lifetime, including
    clean interpreter shutdown and restart.
-   Represent complex data models in a way Python can use without having
    to create deep copies.
-   Provide and import frozen modules.
-   Run and manage multiple independent interpreters (in particular,
    when embedded in a library that wants to avoid global effects).

Python Implementations

Python implementations such as CPython, PyPy, GraalPy, IronPython,
RustPython, MicroPython, and Jython), may take very different approaches
for the implementation of different subsystems. They need:

-   The API to be abstract and hide implementation details.
-   A specification of the API, ideally with a test suite that ensures
    compatibility.
-   It would be nice to have an ABI that can be shared across Python
    implementations.

Alternative APIs and Binding Generators

There are several projects that implement alternatives to the C API,
which offer extension users advantanges over programming directly with
the C API. These APIs are implemented with the C API, and in some cases
by using CPython internals.

There are also libraries that create bindings between Python and other
object models, paradigms or languages.

There is overlap between these categories: binding generators usually
provide alternative APIs, and vice versa.

Examples are Cython, cffi, pybind11 and nanobind for C++, PyO3 for Rust,
Shiboken used by PySide for Qt, PyGObject for GTK, Pygolo for Go, JPype
for Java, PyJNIus for Android, PyObjC for Objective-C, SWIG for C/C++,
Python.NET for .NET (C#), HPy, Mypyc, Pythran and pythoncapi-compat.
CPython's DSL for parsing function arguments, the Argument Clinic, can
also be seen as belonging to this category of stakeholders.

Alternative APIs need minimal building blocks for accessing CPython
efficiently. They don't necessarily need an ergonomic API, because they
typically generate code that is not intended to be read by humans. But
they do need it to be comprehensive enough so that they can avoid
accessing internals, without sacrificing performance.

Binding generators often need to:

-   Create custom objects (e.g. function/module objects and traceback
    entries) that match the behavior of equivalent Python code as
    closely as possible.
-   Dynamically create objects which are static in traditional C
    extensions (e.g. classes/modules), and need CPython to manage their
    state and lifetime.
-   Dynamically adapt foreign objects (strings, GC'd containers), with
    low overhead.
-   Adapt external mechanisms, execution models and guarantees to the
    Python way (stackful coroutines, continuations,
    one-writer-or-multiple-readers semantics, virtual multiple
    inheritance, 1-based indexing, super-long inheritance chains,
    goroutines, channels, etc.).

These tools might also benefit from a choice between a more stable and a
faster (possibly lower-level) API. Their users could then decide whether
they can afford to regenerate the code often or trade some performance
for more stability and less maintenance work.

Strengths of the C API

While the bulk of this document is devoted to problems with the C API
that we would like to see fixed in any new design, it is also important
to point out the strengths of the C API, and to make sure that they are
preserved.

As mentioned in the introduction, the C API enabled the development and
growth of the Python ecosystem over the last three decades, while
evolving to support use cases that it was not originally designed for.
This track record in itself is indication of how effective and valuable
it has been.

A number of specific strengths were mentioned in the capi-workgroup
discussions. Heap types were identified as much safer and easier to use
than static types [Issue 4].

API functions that take a C string literal for lookups based on a Python
string are very convenient [Issue 30].

The limited API demonstrates that an API which hides implementation
details makes it easier to evolve Python [Issue 30].

C API problems

The remainder of this document summarizes and categorizes the problems
that were reported on the capi-workgroup repository. The issues are
grouped into several categories.

API Evolution and Maintenance

The difficulty of making changes in the C API is central to this report.
It is implicit in many of the issues we discuss here, particularly when
we need to decide whether an incremental bugfix can resolve the issue,
or whether it can only be addressed as part of an API redesign [Issue
44]. The benefit of each incremental change is often viewed as too small
to justify the disruption. Over time, this implies that every mistake we
make in an API's design or implementation remains with us indefinitely.

We can take two views on this issue. One is that this is a problem and
the solution needs to be baked into any new C API we design, in the form
of a process for incremental API evolution, which includes deprecation
and removal of API elements. The other possible approach is that this is
not a problem to be solved, but rather a feature of any API. In this
view, API evolution should not be incremental, but rather through large
redesigns, each of which learns from the mistakes of the past and is not
shackled by backwards compatibility requirements (in the meantime, new
API elements may be added, but nothing can ever be removed). A
compromise approach is somewhere between these two extremes, fixing
issues which are easy or important enough to tackle incrementally, and
leaving others alone.

The problem we have in CPython is that we don't have an agreed, official
approach to API evolution. Different members of the core team are
pulling in different directions and this is an ongoing source of
disagreements. Any new C API needs to come with a clear decision about
the model that its maintenance will follow, as well as the technical and
organizational processes by which this will work.

If the model does include provisions for incremental evolution of the
API, it will include processes for managing the impact of the change on
users [Issue 60], perhaps through introducing an external backwards
compatibility module [Issue 62], or a new API tier of "blessed"
functions [Issue 55].

API Specification and Abstraction

The C API does not have a formal specification, it is currently defined
as whatever the reference implementation (CPython) contains in a
particular version. The documentation acts as an incomplete description,
which is not sufficient for verifying the correctness of either the full
API, the limited API, or the stable ABI. As a result, the C API may
change significantly between releases without needing a more visible
specification update, and this leads to a number of problems.

Bindings for languages other than C/C++ must parse C code [Issue 7].
Some C language features are hard to handle in this way, because they
produce compiler-dependent output (such as enums) or require a C
preprocessor/compiler rather than just a parser (such as macros) [Issue
35].

Furthermore, C header files tend to expose more than what is intended to
be part of the public API [Issue 34]. In particular, implementation
details such as the precise memory layouts of internal data structures
can be exposed [Issue 22 and PEP 620]. This can make API evolution very
difficult, in particular when it occurs in the stable ABI as in the case
of ob_refcnt and ob_type, which are accessed via the reference counting
macros [Issue 45].

We identified a deeper issue in relation to the way that reference
counting is exposed. The way that C extensions are required to manage
references with calls to Py_INCREF and Py_DECREF is specific to
CPython's memory model, and is hard for alternative Python
implementations to emulate. [Issue 12].

Another set of problems arises from the fact that a PyObject* is exposed
in the C API as an actual pointer rather than a handle. The address of
an object serves as its ID and is used for comparison, and this
complicates matters for alternative Python implementations that move
objects during GC [Issue 37].

A separate issue is that object references are opaque to the runtime,
discoverable only through calls to tp_traverse/tp_clear, which have
their own purposes. If there was a way for the runtime to know the
structure of the object graph, and keep up with changes in it, this
would make it possible for alternative implementations to implement
different memory management schemes [Issue 33].

Object Reference Management

There does not exist a consistent naming convention for functions which
makes their reference semantics obvious, and this leads to error prone C
API functions, where they do not follow the typical behaviour. When a C
API function returns a PyObject*, the caller typically gains ownership
of a reference to the object. However, there are exceptions where a
function returns a "borrowed" reference, which the caller can access but
does not own a reference to. Similarly, functions typically do not
change the ownership of references to their arguments, but there are
exceptions where a function "steals" a reference, i.e., the ownership of
the reference is permanently transferred from the caller to the callee
by the call [Issue 8 and Issue 52]. The terminology used to describe
these situations in the documentation can also be improved [Issue 11].

A more radical change is necessary in the case of functions that return
"borrowed" references (such as PyList_GetItem) [Issue 5 and Issue 21] or
pointers to parts of the internal structure of an object (such as
PyBytes_AsString) [Issue 57]. In both cases, the reference/pointer is
valid for as long as the owning object holds the reference, but this
time is hard to reason about. Such functions should not exist in the API
without a mechanism that can make them safe.

For containers, the API is currently missing bulk operations on the
references of contained objects. This is particularly important for a
stable ABI where INCREF and DECREF cannot be macros, making bulk
operations expensive when implemented as a sequence of function calls
[Issue 15].

Type Definition and Object Creation

The C API has functions that make it possible to create incomplete or
inconsistent Python objects, such as PyTuple_New and PyUnicode_New. This
causes problems when the object is tracked by GC or its
tp_traverse/tp_clear functions are called. A related issue is with
functions such as PyTuple_SetItem which is used to modify a partially
initialized tuple (tuples are immutable once fully initialized) [Issue
56].

We identified a few issues with type definition APIs. For legacy
reasons, there is often a significant amount of code duplication between
tp_new and tp_vectorcall [Issue 24]. The type slot function should be
called indirectly, so that their signatures can change to include
context information [Issue 13]. Several aspects of the type definition
and creation process are not well defined, such as which stage of the
process is responsible for initializing and clearing different fields of
the type object [Issue 49].

Error Handling

Error handling in the C API is based on the error indicator which is
stored on the thread state (in global scope). The design intention was
that each API function returns a value indicating whether an error has
occurred (by convention, -1 or NULL). When the program knows that an
error occurred, it can fetch the exception object which is stored in the
error indicator. We identified a number of problems which are related to
error handling, pointing at APIs which are too easy to use incorrectly.

There are functions that do not report all errors that occur while they
execute. For example, PyDict_GetItem clears any errors that occur when
it calls the key's hash function, or while performing a lookup in the
dictionary [Issue 51].

Python code never executes with an in-flight exception (by definition),
and typically code using native functions should also be interrupted by
an error being raised. This is not checked in most C API functions, and
there are places in the interpreter where error handling code calls a C
API function while an exception is set. For example, see the call to
PyUnicode_FromString in the error handler of _PyErr_WriteUnraisableMsg
[Issue 2].

There are functions that do not return a value, so a caller is forced to
query the error indicator in order to identify whether an error has
occurred. An example is PyBuffer_Release [Issue 20]. There are other
functions which do have a return value, but this return value does not
unambiguously indicate whether an error has occurred. For example,
PyLong_AsLong returns -1 in case of error, or when the value of the
argument is indeed -1 [Issue 1]. In both cases, the API is error prone
because it is possible that the error indicator was already set before
the function was called, and the error is incorrectly attributed. The
fact that the error was not detected before the call is a bug in the
calling code, but the behaviour of the program in this case doesn't make
it easy to identify and debug the problem.

There are functions that take a PyObject* argument, with special meaning
when it is NULL. For example, if PyObject_SetAttr receives NULL as the
value to set, this means that the attribute should be cleared. This is
error prone because it could be that NULL indicates an error in the
construction of the value, and the program failed to check for this
error. The program will misinterpret the NULL to mean something
different than error [Issue 47].

API Tiers and Stability Guarantees

The different API tiers provide different tradeoffs of stability vs API
evolution, and sometimes performance.

The stable ABI was identified as an area that needs to be looked into.
At the moment it is incomplete and not widely adopted. At the same time,
its existence is making it hard to make changes to some implementation
details, because it exposes struct fields such as ob_refcnt, ob_type and
ob_size. There was some discussion about whether the stable ABI is worth
keeping. Arguments on both sides can be found in [Issue 4] and [Issue
9].

Alternatively, it was suggested that in order to be able to evolve the
stable ABI, we need a mechanism to support multiple versions of it in
the same Python binary. It was pointed out that versioning individual
functions within a single ABI version is not enough because it may be
necessary to evolve, together, a group of functions that interoperate
with each other [Issue 39].

The limited API was introduced in 3.2 as a blessed subset of the C API
which is recommended for users who would like to restrict themselves to
high quality APIs which are not likely to change often. The
Py_LIMITED_API flag allows users to restrict their program to older
versions of the limited API, but we now need the opposite option, to
exclude older versions. This would make it possible to evolve the
limited API by replacing flawed elements in it [Issue 54]. More
generally, in a redesign we should revisit the way that API tiers are
specified and consider designing a method that will unify the way we
currently select between the different tiers [Issue 59].

API elements whose names begin with an underscore are considered
private, essentially an API tier with no stability guarantees. However,
this was only clarified recently, in PEP 689. It is not clear what the
change policy should be with respect to such API elements that predate
PEP 689 [Issue 58].

There are API functions which have an unsafe (but fast) version as well
as a safe version which performs error checking (for example,
PyTuple_GET_ITEM vs PyTuple_GetItem). It may help to be able to group
them into their own tiers - the "unsafe API" tier and the "safe API"
tier [Issue 61].

Use of the C Language

A number of issues were raised with respect to the way that CPython uses
the C language. First there is the issue of which C dialect we use, and
how we test our compatibility with it, as well as API header
compatibility with C++ dialects [Issue 42].

Usage of const in the API is currently sparse, but it is not clear
whether this is something that we should consider changing [Issue 38].

We currently use the C types long and int, where fixed-width integers
such as int32_t and int64_t may now be better choices [Issue 27].

We are using C language features which are hard for other languages to
interact with, such as macros, variadic arguments, enums, bitfields, and
non-function symbols [Issue 35].

There are API functions that take a PyObject* arg which must be of a
more specific type (such as PyTuple_Size, which fails if its arg is not
a PyTupleObject*). It is an open question whether this is a good pattern
to have, or whether the API should expect the more specific type [Issue
31].

There are functions in the API that take concrete types, such as
PyDict_GetItemString which performs a dictionary lookup for a key
specified as a C string rather than PyObject*. At the same time, for
PyDict_ContainsString it is not considered appropriate to add a concrete
type alternative. The principle around this should be documented in the
guidelines [Issue 23].

Implementation Flaws

Below is a list of localized implementation flaws. Most of these can
probably be fixed incrementally, if we choose to do so. They should, in
any case, be avoided in any new API design.

There are functions that don't follow the convention of returning 0 for
success and -1 for failure. For example, PyArg_ParseTuple returns 0 for
success and non-zero for failure [Issue 25].

The macros Py_CLEAR and Py_SETREF access their arg more than once, so if
the arg is an expression with side effects, they are duplicated [Issue
3].

The meaning of Py_SIZE depends on the type and is not always reliable
[Issue 10].

Some API function do not have the same behaviour as their Python
equivalents. The behaviour of PyIter_Next is different from tp_iternext.
[Issue 29]. The behaviour of PySet_Contains is different from
set.__contains__ [Issue 6].

The fact that PyArg_ParseTupleAndKeywords takes a non-const char* array
as argument makes it more difficult to use [Issue 28].

Python.h does not expose the whole API. Some headers (like marshal.h)
are not included from Python.h. [Issue 43].

Naming

PyLong and PyUnicode use names which no longer match the Python types
they represent (int/str). This could be fixed in a new API [Issue 14].

There are identifiers in the API which are lacking a Py/_Py prefix
[Issue 46].

Missing Functionality

This section consists of a list of feature requests, i.e., functionality
that was identified as missing in the current C API.

Debug Mode

A debug mode that can be activated without recompilation and which
activates various checks that can help detect various types of errors
[Issue 36].

Introspection

There aren't currently reliable introspection capabilities for objects
defined in C in the same way as there are for Python objects [Issue 32].

Efficient type checking for heap types [Issue 17].

Improved Interaction with Other Languages

Interfacing with other GC based languages, and integrating their GC with
Python's GC [Issue 19].

Inject foreign stack frames to the traceback [Issue 18].

Concrete strings that can be used in other languages [Issue 16].

References

1.  Python/C API Reference Manual
2.  2023 Language Summit Blog Post: Three Talks on the C API
3.  capi-workgroup on GitHub
4.  Irit's Core Sprint 2023 slides about C API workgroup
5.  Petr's Core Sprint 2023 slides
6.  HPy team's Core Sprint 2023 slides for Things to Learn from HPy
7.  Victor's slides of Core Sprint 2023 Python C API talk
8.  The Python's stability promise — Cristián Maureira-Fredes, PySide
    maintainer
9.  Report on the issues PySide had 5 years ago when switching to the
    stable ABI

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.