text:
PEP: 643 Title: Metadata for Package Source Distributions Author: Paul
Moore<EMAIL_ADDRESS>BDFL-Delegate: Paul Ganssle
<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pep-643-metadata-for-package-source-distributions/5577
Status: Final Type: Standards Track Topic: Packaging Content-Type:
text/x-rst Created: 24-Oct-2020 Post-History: 24-Oct-2020, 01-Nov-2020,
02-Nov-2020, 14-Nov-2020 Resolution:
https://discuss.python.org/t/pep-643-metadata-for-package-source-distributions/5577/53

packaging:core-metadata

Abstract

Python package metadata is stored in the distribution file in a standard
format, defined in the Core Metadata Specification. However, for source
distributions, while the format of the data is defined, there has
traditionally been a lot of inconsistency in what data is recorded in
the source distribution. See here for a discussion of this issue.

As a result, metadata consumers are unable to rely on the data available
from source distributions, and need to use the (costly) PEP 517 build
mechanisms to extract medatata.

This PEP defines a standard that allows build backends to reliably store
package metadata in the source distribution, while still retaining the
necessary flexibility to handle metadata fields that have to be
calculated at build time.

Motivation

There are a number of issues with the way that metadata is currently
stored in source distributions:

-   The details of how to store metadata, while standardised, are not
    easy to find.
-   The specification requires an old metadata version, and has not been
    updated in line with changes to the core metadata spec.
-   There is no way in the spec to distinguish between "this field has
    been omitted because its value will not be known until build time"
    and "this field does not have a value".
-   The core metadata specification allows most fields to be optional,
    meaning that the previous issue affects nearly every metadata field.

This PEP proposes an update to the metadata specification to allow
recording of fields which are expected to be "filled in later", and
updates the source distribution specification to clarify that backends
should record sdist metadata using that version of the spec (or later).

Rationale

This PEP allows projects to define source distribution metadata values
as being "dynamic". In this context, saying that a field is "dynamic"
means that the value has not been fixed at the time that the source
distribution was generated. Dynamic values will be supplied by the build
backend at the time when the wheel is generated, and could depend on
details of the build environment.

PEP 621 has a similar concept, of "dynamic" values that will be "filled
in later", and so we choose to use the same term here by analogy.

Specification

This PEP defines the relationship between metadata values specified in a
source distribution, and the corresponding values in wheels built from
it. It requires build backends to clearly mark any fields which will not
simply be copied unchanged from the sdist to the wheel.

In addition, this PEP makes the PyPA Specifications document the
canonical location for the specification of the source distribution
format (collecting the information in PEP 517 and in this PEP).

A new field, Dynamic, will be added to the Core Metadata Specification.
This field will be multiple use, and will be allowed to contain the name
of another core metadata field.

When found in the metadata of a source distribution, the following rules
apply:

1.  If a field is not marked as Dynamic, then the value of the field in
    any wheel built from the sdist MUST match the value in the sdist. If
    the field is not in the sdist, and not marked as Dynamic, then it
    MUST NOT be present in the wheel.
2.  If a field is marked as Dynamic, it may contain any valid value in a
    wheel built from the sdist (including not being present at all).
3.  Backends MUST NOT mark a field as Dynamic if they can determine that
    it was generated from data that will not change at build time.

Backends MAY record the value they calculated for a field they mark as
Dynamic in a source distribution. Consumers, however, MUST NOT treat
this value as canonical, but MAY use it as an hint about what the final
value in a wheel could be.

In any context other than a source distribution, if a field is marked as
Dynamic, that indicates that the value was generated at wheel build time
and may not match the value in the sdist (or in other builds of the
project). Backends are not required to record this information, though,
and consumers MUST NOT assume that the lack of a Dynamic marking has any
significance, except in a source distribution.

The fields Name and Version MUST NOT be marked as Dynamic.

As it adds a new metadata field, this PEP updates the core metadata
format to version 2.2.

Source distributions SHOULD use the latest version of the core metadata
specification that was available when they were created.

Build backends are strongly encouraged to only mark fields as Dynamic
when absolutely necessary, and to encourage projects to avoid backend
features that require the use of Dynamic. Projects should prefer to use
environment markers on static values to adapt to details of the install
location.

Backwards Compatibility

As this proposal increments the core metadata version, it is compatible
with existing source distributions, which will use an older metadata
version. Tools can determine whether a source distribution conforms to
this PEP by checking the metadata version.

Security Implications

As this specification is purely for the storage of data that is intended
to be publicly available, there are no security implications.

How to Teach This

This is a data storage format for project metadata, and so will not
typically be visible to end users. There is therefore no need to teach
users how to use the format. Developers wanting to reference the
metadata will be able to find the details in the PyPA Specifications.

Rejected Ideas

1.  Rather than marking fields as Dynamic, fields should be assumed to
    be dynamic unless explicitly marked as Static.

    This is logically equivalent to the current proposal, but it implies
    that fields being dynamic is the norm. Packaging tools can be much
    more efficient in the presence of metadata that is known to be
    static, so the PEP chooses to make dynamic fields the exception, and
    require backends to "opt in" to making a field dynamic.

    In addition, if dynamic is the default, then in future, as more and
    more metadata becomes static, metadata files will include an
    increasing number of Static declarations.

2.  Rather than having a Dynamic field, add a special value that
    indicates that a field is "not yet defined".

    Again, this is logically equivalent to the current proposal. It
    makes "being dynamic" an explicit choice, but requires a special
    value. As some fields can contain arbitrary text, choosing a such a
    value is somewhat awkward (although likely not a problem in
    practice). There does not seem to be enough benefit to this approach
    to make it worth using instead of the proposed mechanism.

3.  Special handling of Requires-Python.

    Early drafts of the PEP needed special discussion of
    Requires-Python, because the lack of environment markers for this
    field meant that it might be difficult to require it to be static.
    The final form of the PEP no longer needs this, as the idea of a
    whitelist of fields allowed to be dynamic was dropped.

4.  Restrict the use of Dynamic to a minimal "white list" of permitted
    fields.

    This approach was likely to prove extremely difficult for setuptools
    to implement in a backward compatible way, due to the dynamic nature
    of the setuptools interface. Instead, the proposal now allows most
    fields to be dynamic, but encourages backends to avoid dynamic
    values unless essential.

Open Issues

None

References

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 579 Title: Refactoring C functions and methods Author: Jeroen
Demeyer<EMAIL_ADDRESS>BDFL-Delegate: Petr Viktorin Status: Final
Type: Informational Content-Type: text/x-rst Created: 04-Jun-2018
Post-History: 20-Jun-2018

Approval Notice

This PEP describes design issues addressed in PEP 575, PEP 580, PEP 590
(and possibly later proposals).

As noted in PEP 1 <1#pep-types>:

  Informational PEPs do not necessarily represent a Python community
  consensus or recommendation, so users and implementers are free to
  ignore Informational PEPs or follow their advice.

While there is no consensus on whether the issues or the solutions in
this PEP are valid, the list is still useful to guide further design.

Abstract

This meta-PEP collects various issues with CPython's existing
implementation of built-in functions (functions implemented in C) and
methods.

Fixing all these issues is too much for one PEP, so that will be
delegated to other standards track PEPs. However, this PEP does give
some brief ideas of possible fixes. This is mainly meant to coordinate
an overall strategy. For example, a proposed solution may sound too
complicated for fixing any one single issue, but it may be the best
overall solution for multiple issues.

This PEP is purely informational: it does not imply that all issues will
eventually be fixed, nor that they will be fixed using the solution
proposed here.

It also serves as a check-list of possible requested features to verify
that a given fix does not make those other features harder to implement.

The major proposed change is replacing PyMethodDef by a new structure
PyCCallDef which collects everything needed for calling the
function/method. In the PyTypeObject structure, a new field
tp_ccalloffset is added giving an offset to a PyCCallDef * in the object
structure.

NOTE: This PEP deals only with CPython implementation details, it does
not affect the Python language or standard library.

Issues

This lists various issues with built-in functions and methods, together
with a plan for a solution and (if applicable) pointers to standards
track PEPs discussing the details.

1. Naming

The word "built-in" is overused in Python. From a quick skim of the
Python documentation, it mostly refers to things from the builtins
module. In other words: things which are available in the global
namespace without a need for importing them. This conflicts with the use
of the word "built-in" to mean "implemented in C".

Solution: since the C structure for built-in functions and methods is
already called PyCFunctionObject, let's use the name "cfunction" and
"cmethod" instead of "built-in function" and "built-in method".

2. Not extendable

The various classes involved (such as builtin_function_or_method) cannot
be subclassed:

    >>> from types import BuiltinFunctionType
    >>> class X(BuiltinFunctionType):
    ...     pass
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: type 'builtin_function_or_method' is not an acceptable base type

This is a problem because it makes it impossible to add features such as
introspection support to these classes.

If one wants to implement a function in C with additional functionality,
an entirely new class must be implemented from scratch. The problem with
this is that the existing classes like builtin_function_or_method are
special-cased in the Python interpreter to allow faster calling (for
example, by using METH_FASTCALL). It is currently impossible to have a
custom class with the same optimizations.

Solution: make the existing optimizations available to arbitrary
classes. This is done by adding a new PyTypeObject field tp_ccalloffset
(or can we re-use tp_print for that?) specifying the offset of a
PyCCallDef pointer. This is a new structure holding all information
needed to call a cfunction and it would be used instead of PyMethodDef.
This implements the new "C call" protocol.

For constructing cfunctions and cmethods, PyMethodDef arrays will still
be used (for example, in tp_methods) but that will be the only remaining
purpose of the PyMethodDef structure.

Additionally, we can also make some function classes subclassable.
However, this seems less important once we have tp_ccalloffset.

Reference: PEP 580

3. cfunctions do not become methods

A cfunction like repr does not implement __get__ to bind as a method:

    >>> class X:
    ...     meth = repr
    >>> x = X()
    >>> x.meth()
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: repr() takes exactly one argument (0 given)

In this example, one would have expected that x.meth() returns repr(x)
by applying the normal rules of methods.

This is surprising and a needless difference between cfunctions and
Python functions. For the standard built-in functions, this is not
really a problem since those are not meant to used as methods. But it
does become a problem when one wants to implement a new cfunction with
the goal of being usable as method.

Again, a solution could be to create a new class behaving just like
cfunctions but which bind as methods. However, that would lose some
existing optimizations for methods, such as the LOAD_METHOD/CALL_METHOD
opcodes.

Solution: the same as the previous issue. It just shows that handling
self and __get__ should be part of the new C call protocol.

For backwards compatibility, we would keep the existing non-binding
behavior of cfunctions. We would just allow it in custom classes.

Reference: PEP 580

4. Semantics of inspect.isfunction

Currently, inspect.isfunction returns True only for instances of
types.FunctionType. That is, true Python functions.

A common use case for inspect.isfunction is checking for introspection:
it guarantees for example that inspect.getfile() will work. Ideally, it
should be possible for other classes to be treated as functions too.

Solution: introduce a new InspectFunction abstract base class and use
that to implement inspect.isfunction. Alternatively, use duck typing for
inspect.isfunction (as proposed in[1]):

    def isfunction(obj):
        return hasattr(type(obj), "__code__")

5. C functions should have access to the function object

The underlying C function of a cfunction currently takes a self argument
(for bound methods) and then possibly a number of arguments. There is no
way for the C function to actually access the Python cfunction object
(the self in __call__ or tp_call). This would for example allow
implementing the C call protocol for Python functions
(types.FunctionType): the C function which implements calling Python
functions needs access to the __code__ attribute of the function.

This is also needed for PEP 573 where all cfunctions require access to
their "parent" (the module for functions of a module or the defining
class for methods).

Solution: add a new PyMethodDef flag to specify that the C function
takes an additional argument (as first argument), namely the function
object.

References: PEP 580, PEP 573

6. METH_FASTCALL is private and undocumented

The METH_FASTCALL mechanism allows calling cfunctions and cmethods using
a C array of Python objects instead of a tuple. This was introduced in
Python 3.6 for positional arguments only and extended in Python 3.7 with
support for keyword arguments.

However, given that it is undocumented, it is presumably only supposed
to be used by CPython itself.

Solution: since this is an important optimization, everybody should be
encouraged to use it. Now that the implementation of METH_FASTCALL is
stable, document it!

As part of the C call protocol, we should also add a C API function :

    PyObject *PyCCall_FastCall(PyObject *func, PyObject *const *args, Py_ssize_t nargs, PyObject *keywords)

Reference: PEP 580

7. Allowing native C arguments

A cfunction always takes its arguments as Python objects (say, an array
of PyObject pointers). In cases where the cfunction is really wrapping a
native C function (for example, coming from ctypes or some compiler like
Cython), this is inefficient: calls from C code to C code are forced to
use Python objects to pass arguments.

Analogous to the buffer protocol which allows access to C data, we
should also allow access to the underlying C callable.

Solution: when wrapping a C function with native arguments (for example,
a C long) inside a cfunction, we should also store a function pointer to
the underlying C function, together with its C signature.

Argument Clinic could automatically do this by storing a pointer to the
"impl" function.

8. Complexity

There are a huge number of classes involved to implement all variations
of methods. This is not a problem by itself, but a compounding issue.

For ordinary Python classes, the table below gives the classes for
various kinds of methods. The columns refer to the class in the class
__dict__, the class for unbound methods (bound to the class) and the
class for bound methods (bound to the instance):

  kind            __dict__       unbound    bound
  --------------- -------------- ---------- ----------
  Normal method   function       function   method
  Static method   staticmethod   function   function
  Class method    classmethod    method     method
  Slot method     function       function   method

This is the analogous table for extension types (C classes):

  kind            __dict__                 unbound                      bound
  --------------- ------------------------ ---------------------------- ----------------------------
  Normal method   method_descriptor        method_descriptor            builtin_function_or_method
  Static method   staticmethod             builtin_function_or_method   builtin_function_or_method
  Class method    classmethod_descriptor   builtin_function_or_method   builtin_function_or_method
  Slot method     wrapper_descriptor       wrapper_descriptor           method-wrapper

There are a lot of classes involved and these two tables look very
different. There is no good reason why Python methods should be treated
fundamentally different from C methods. Also the features are slightly
different: for example, method supports __func__ but
builtin_function_or_method does not.

Since CPython has optimizations for calls to most of these objects, the
code for dealing with them can also become complex. A good example of
this is the call_function function in Python/ceval.c.

Solution: all these classes should implement the C call protocol. Then
the complexity in the code can mostly be fixed by checking for the C
call protocol (tp_ccalloffset != 0) instead of doing type checks.

Furthermore, it should be investigated whether some of these classes can
be merged and whether method can be re-used also for bound methods of
extension types (see PEP 576 for the latter, keeping in mind that this
may have some minor backwards compatibility issues). This is not a goal
by itself but just something to keep in mind when working on these
classes.

9. PyMethodDef is too limited

The typical way to create a cfunction or cmethod in an extension module
is by using a PyMethodDef to define it. These are then stored in an
array PyModuleDef.m_methods (for cfunctions) or PyTypeObject.tp_methods
(for cmethods). However, because of the stable ABI (PEP 384), we cannot
change the PyMethodDef structure.

So, this means that we cannot add new fields for creating
cfunctions/cmethods this way. This is probably the reason for the hack
that __doc__ and __text_signature__ are stored in the same C string
(with the __doc__ and __text_signature__ descriptors extracting the
relevant part).

Solution: stop assuming that a single PyMethodDef entry is sufficient to
describe a cfunction/cmethod. Instead, we could add some flag which
means that one of the PyMethodDef fields is instead a pointer to an
additional structure. Or, we could add a flag to use two or more
consecutive PyMethodDef entries in the array to store more data. Then
the PyMethodDef array would be used only to construct
cfunctions/cmethods but it would no longer be used after that.

10. Slot wrappers have no custom documentation

Right now, slot wrappers like __init__ or __lt__ only have very generic
documentation, not at all specific to the class:

    >>> list.__init__.__doc__
    'Initialize self.  See help(type(self)) for accurate signature.'
    >>> list.__lt__.__doc__
    'Return self<value.'

The same happens for the signature:

    >>> list.__init__.__text_signature__
    '($self, /, *args, **kwargs)'

As you can see, slot wrappers do support __doc__ and __text_signature__.
The problem is that these are stored in struct wrapperbase, which is
common for all wrappers of a specific slot (for example, the same
wrapperbase is used for str.__eq__ and int.__eq__).

Solution: rethink the slot wrapper class to allow docstrings (and text
signatures) for each instance separately.

This still leaves the question of how extension modules should specify
the documentation. The PyTypeObject entries like tp_init are just
function pointers, we cannot do anything with those. One solution would
be to add entries to the tp_methods array just for adding docstrings.
Such an entry could look like :

    {"__init__", NULL, METH_SLOTDOC, "pointer to __init__ doc goes here"}

11. Static methods and class methods should be callable

Instances of staticmethod and classmethod should be callable.
Admittedly, there is no strong use case for this, but it has
occasionally been requested (see for example[2]).

Making static/class methods callable would increase consistency. First
of all, function decorators typically add functionality or modify a
function, but the result remains callable. This is not true for
@staticmethod and @classmethod.

Second, class methods of extension types are already callable:

    >>> fromhex = float.__dict__["fromhex"]
    >>> type(fromhex)
    <class 'classmethod_descriptor'>
    >>> fromhex(float, "0xff")
    255.0

Third, one can see function, staticmethod and classmethod as different
kinds of unbound methods: they all become method when bound, but the
implementation of __get__ is slightly different. From this point of
view, it looks strange that function is callable but the others are not.

Solution: when changing the implementation of staticmethod, classmethod,
we should consider making instances callable. Even if this is not a goal
by itself, it may happen naturally because of the implementation.

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] Duck-typing inspect.isfunction()
(https://bugs.python.org/issue30071)

[2] Not all method descriptors are callable
(https://bugs.python.org/issue20309)

license: Public Domain


text:
PEP: 743 Title: Add Py_COMPAT_API_VERSION to the Python C API Author:
Victor Stinner<EMAIL_ADDRESS>Petr Viktorin<EMAIL_ADDRESS>PEP-Delegate: C API Working Group Status: Draft Type: Standards Track
Created: 11-Mar-2024 Python-Version: 3.14

Abstract

Add Py_COMPAT_API_VERSION C macro that hides some deprecated and
soft-deprecated symbols, allowing users to opt out of using API with
known issues that other API solves. The macro is versioned, allowing
users to update (or not) on their own pace.

Also, add namespaced alternatives for API without the Py_ prefix, and
soft-deprecate the original names.

Motivation

Some of Python's C API has flaws that are only obvious in hindsight.

If an API prevents adding features or optimizations, or presents a
serious security risk or maintenance burden, we can deprecate and remove
it as described in PEP 387.

However, this leaves us with some API that has “sharp edges” -- it works
fine for its current users, but should be avoided in new code. For
example:

-   API that cannot signal an exception, so failures are either ignored
    or exit the process with a fatal error. For example
    PyObject_HasAttr.
-   API that is not thread-safe, for example by borrowing references
    from mutable objects, or exposing unfinished mutable objects. For
    example PyDict_GetItemWithError.
-   API with names that don't use the Py/_Py prefix, and so can clash
    with other code. For example: setter.

It is important to note that despite such flaws, it's usually possible
to use the API correctly. For example, in a single-threaded environment,
thread safety is not an issue. We do not want to break working code,
even if it uses API that would be wrong in some -- or even most -- other
contexts.

On the other hand, we want to steer users away from such “undesirable”
API in new code, especially if a safer alternative exists.

Adding the Py prefix

Some names defined in CPython headers is not namespaced: it that lacks
the Py prefix (or a variant: _Py, and alternative capitalizations). For
example, we declare a function type named simply setter.

While such names are not exported in the ABI (as checked by
make smelly), they can clash with user code and, more importantly, with
libraries linked to third-party extensions.

While it would be possible to provide namespaced aliases and
(soft-)deprecate these names, the only way to make them not clash with
third-party code is to not define them in Python headers at all.

Rationale

We want to allow an easy way for users to avoid “undesirable” API if
they choose to do so.

It might be be sufficient to leave this to third-party linters. For that
we'd need a good way to expose a list of (soft-)deprecated API to such
linters. While adding that, we can -- rather easily -- do the linter's
job directly in CPython headers, avoiding the neel for an extra tool.
Unlike Python, C makes it rather easy to limit available API -- for a
whole project or for each individual source file -- by having users
define an “opt-in” macro.

We already do something similar with Py_LIMITED_API, which limits the
available API to a subset that compiles to stable ABI. (In hindsight, we
should have used a different macro name for that particular kind of
limiting, but it's too late to change that now.)

To prevent working code from breaking as we identify more “undesirable”
API and add safer alternatives to it, the opt-in macro should be
versioned. Users can choose a version they need based on their
compatibility requirements, and update it at their own pace.

To be clear, this mechanism is not a replacement for deprecation.
Deprecation is for API that prevents new features or optimizations, or
presents a security risk or maintenance burden. This mechanism, on the
other hand, is meant for cases where “we found a slightly better way of
doing things” -- perhaps one that's harder to misuse, or just has a less
misleading name. (On a lighter note: many people configure a code
quality checker to shout at them about the number of blank lines between
functions. Let's help them identify more substantial “code smells”!)

The proposed macro does not change any API definitions; it only hides
them. So, if code compiles with the macro, it'll also compile without
it, with identical behaviour. This has implications for core devs: to
deal with undesirable behaviour, we'll need to introduce new, better
API, and then discourage the old one. In turn, this implies that we
should look at an individual API and fix all its known issues at once,
rather than do codebase-wide sweeps for a single kind of issue, so that
we avoid multiple renames of the same function.

Adding the Py prefix

An opt-in macro allows us to omit definitions that could clash with
third-party libraries.

Specification

We introduce a Py_COMPAT_API_VERSION macro. If this macro is defined
before #include <Python.h>, some API definitions -- as described below
-- will be omitted from the Python header files.

The macro only omits complete top-level definitions exposed from
<Python.h>. Other things (the ABI, structure definitions, macro
expansions, static inline function bodies, etc.) are not affected.

The C API working group (PEP 731) has authority over the set of omitted
definitions.

The set of omitted definitions will be tied to a particular feature
release of CPython, and is finalized in each 3.x.0 Beta 1 release. In
rare cases, entries can be removed (i.e. made available for use) at any
time.

The macro should be defined to a version in the format used by
PY_VERSION_HEX, with the “micro”, “release” and “serial” fields set to
zero. For example, to omit API deemed undesirable in 3.14.0b1, users
should define Py_COMPAT_API_VERSION to 0x030e0000.

Requirements for omitted API

An API that is omitted with Py_COMPAT_API_VERSION must:

-   be soft-deprecated (see PEP 387);
-   for all known use cases of the API, have a documented alternative or
    workaround;
-   have tests to ensure it keeps working (except for 1:1 renames using
    #define or typedef);
-   be documented (except if it was never mentioned in previous versions
    of the documentation); and
-   be approved by the C API working group. (The WG may give blanket
    approvals for groups of related API; see Initial set below for
    examples.)

Note that Py_COMPAT_API_VERSION is meant for API that can be trivially
replaced by a better alternative. API without a replacement should
generally be deprecated instead.

Location

All API definitions omitted by Py_COMPAT_API_VERSION will be moved to a
new header, Include/legacy.h.

This is meant to help linter authors compile lists, so they can flag the
API with warnings rather than errors.

Note that for simple renaming of source-only constructs (macros, types),
we expect names to be omitted in the same version -- or the same PR --
that adds a replacement. This means that the original definition will be
renamed, and a typedef or #define for the old name added to
Include/legacy.h.

Documentation

Documentation for omitted API should generally:

-   appear after the recommended replacement,
-   reference the replacement (e.g. “Similar to X, but…”), and
-   focus on differences from the replacement and migration advice.

Exceptions are possible if there is a good reason for them.

Initial set

The following API will be omitted with Py_COMPAT_API_VERSION set to
0x030e0000 (3.14) or greater:

-   Omit API returning borrowed references:

      Omitted API              Replacement
      ------------------------ ---------------------------
      PyDict_GetItem()         PyDict_GetItemRef()
      PyDict_GetItemString()   PyDict_GetItemStringRef()
      PyImport_AddModule()     PyImport_AddModuleRef()
      PyList_GetItem()         PyList_GetItemRef()

-   Omit deprecated APIs:

      Omitted Deprecated API             Replacement
      ---------------------------------- -----------------------------------------
      PY_FORMAT_SIZE_T                   "z"
      PY_UNICODE_TYPE                    wchar_t
      PyCode_GetFirstFree()              PyUnstable_Code_GetFirstFree()
      PyCode_New()                       PyUnstable_Code_New()
      PyCode_NewWithPosOnlyArgs()        PyUnstable_Code_NewWithPosOnlyArgs()
      PyImport_ImportModuleNoBlock()     PyImport_ImportModule()
      PyMem_DEL()                        PyMem_Free()
      PyMem_Del()                        PyMem_Free()
      PyMem_FREE()                       PyMem_Free()
      PyMem_MALLOC()                     PyMem_Malloc()
      PyMem_NEW()                        PyMem_New()
      PyMem_REALLOC()                    PyMem_Realloc()
      PyMem_RESIZE()                     PyMem_Resize()
      PyModule_GetFilename()             PyModule_GetFilenameObject()
      PyOS_AfterFork()                   PyOS_AfterFork_Child()
      PyObject_DEL()                     PyObject_Free()
      PyObject_Del()                     PyObject_Free()
      PyObject_FREE()                    PyObject_Free()
      PyObject_MALLOC()                  PyObject_Malloc()
      PyObject_REALLOC()                 PyObject_Realloc()
      PySlice_GetIndicesEx()             (two calls; see current docs)
      PyThread_ReInitTLS()               (no longer needed)
      PyThread_create_key()              PyThread_tss_alloc()
      PyThread_delete_key()              PyThread_tss_free()
      PyThread_delete_key_value()        PyThread_tss_delete()
      PyThread_get_key_value()           PyThread_tss_get()
      PyThread_set_key_value()           PyThread_tss_set()
      PyUnicode_AsDecodedObject()        PyUnicode_Decode()
      PyUnicode_AsDecodedUnicode()       PyUnicode_Decode()
      PyUnicode_AsEncodedObject()        PyUnicode_AsEncodedString()
      PyUnicode_AsEncodedUnicode()       PyUnicode_AsEncodedString()
      PyUnicode_IS_READY()               (no longer needed)
      PyUnicode_READY()                  (no longer needed)
      PyWeakref_GET_OBJECT()             PyWeakref_GetRef()
      PyWeakref_GetObject()              PyWeakref_GetRef()
      Py_UNICODE                         wchar_t
      _PyCode_GetExtra()                 PyUnstable_Code_GetExtra()
      _PyCode_SetExtra()                 PyUnstable_Code_SetExtra()
      _PyDict_GetItemStringWithError()   PyDict_GetItemStringRef()
      _PyEval_RequestCodeExtraIndex()    PyUnstable_Eval_RequestCodeExtraIndex()
      _PyHASH_BITS                       PyHASH_BITS
      _PyHASH_IMAG                       PyHASH_IMAG
      _PyHASH_INF                        PyHASH_INF
      _PyHASH_MODULUS                    PyHASH_MODULUS
      _PyHASH_MULTIPLIER                 PyHASH_MULTIPLIER
      _PyObject_EXTRA_INIT               (no longer needed)
      _PyThreadState_UncheckedGet()      PyThreadState_GetUnchecked()
      _PyUnicode_AsString()              PyUnicode_AsUTF8()
      _Py_HashPointer()                  Py_HashPointer()
      _Py_T_OBJECT                       Py_T_OBJECT_EX
      _Py_WRITE_RESTRICTED               (no longer needed)

-   Soft-deprecate and omit APIs:

      Omitted Deprecated API      Replacement
      --------------------------- -----------------------------------
      PyDict_GetItemWithError()   PyDict_GetItemRef()
      PyDict_SetDefault()         PyDict_SetDefaultRef()
      PyMapping_HasKey()          PyMapping_HasKeyWithError()
      PyMapping_HasKeyString()    PyMapping_HasKeyStringWithError()
      PyObject_HasAttr()          PyObject_HasAttrWithError()
      PyObject_HasAttrString()    PyObject_HasAttrStringWithError()

-   Omit <structmember.h> legacy API:

    The header file structmember.h, which is not included from
    <Python.h> and must be included separately, will #error if
    Py_COMPAT_API_VERSION is defined. This affects the following API:

      Omitted Deprecated API   Replacement
      ------------------------ ---------------------------------
      T_SHORT                  Py_T_SHORT
      T_INT                    Py_T_INT
      T_LONG                   Py_T_LONG
      T_FLOAT                  Py_T_FLOAT
      T_DOUBLE                 Py_T_DOUBLE
      T_STRING                 Py_T_STRING
      T_OBJECT                 (tp_getset; docs to be written)
      T_CHAR                   Py_T_CHAR
      T_BYTE                   Py_T_BYTE
      T_UBYTE                  Py_T_UBYTE
      T_USHORT                 Py_T_USHORT
      T_UINT                   Py_T_UINT
      T_ULONG                  Py_T_ULONG
      T_STRING_INPLACE         Py_T_STRING_INPLACE
      T_BOOL                   Py_T_BOOL
      T_OBJECT_EX              Py_T_OBJECT_EX
      T_LONGLONG               Py_T_LONGLONG
      T_ULONGLONG              Py_T_ULONGLONG
      T_PYSSIZET               Py_T_PYSSIZET
      T_NONE                   (tp_getset; docs to be written)
      READONLY                 Py_READONLY
      PY_AUDIT_READ            Py_AUDIT_READ
      READ_RESTRICTED          Py_AUDIT_READ
      PY_WRITE_RESTRICTED      (no longer needed)
      RESTRICTED               Py_AUDIT_READ

-   Omit soft deprecated macros:

      Omitted Macros     Replacement
      ------------------ -----------------------------
      Py_IS_NAN()        isnan() (C99+ <math.h>)
      Py_IS_INFINITY()   isinf(X) (C99+ <math.h>)
      Py_IS_FINITE()     isfinite(X) (C99+ <math.h>)
      Py_MEMCPY()        memcpy() (C <string.h>)

-   Soft-deprecate and omit typedefs without the Py/_Py prefix (getter,
    setter, allocfunc, …), in favour of new ones that add the prefix
    (Py_getter , etc.)

-   Soft-deprecate and omit macros without the Py/_Py prefix (METH_O,
    CO_COROUTINE, FUTURE_ANNOTATIONS, WAIT_LOCK, …), favour of new ones
    that add the prefix (Py_METH_O , etc.).

-   Any others approved by the C API workgroup

If any of these proposed replacements, or associated documentation, are
not added in time for 3.14.0b1, they'll be omitted with later versions
of Py_COMPAT_API_VERSION. (We expect this for macros generated by
configure: HAVE_*, WITH_*, ALIGNOF_*, SIZEOF_*, and several without a
common prefix.)

Implementation

TBD

Open issues

The name Py_COMPAT_API_VERSION was taken from the earlier PEP; it
doesn't fit this version.

Backwards Compatibility

The macro is backwards compatible. Developers can introduce and update
the macro on their own pace, potentially for one source file at a time.

Discussions

-   C API Evolutions: Macro to hide deprecated functions (October 2023)
-   C API Problems: Opt-in macro for a new clean API? Subset of
    functions with no known issues (June 2023)
-   Finishing the Great Renaming (May 2024)

Prior Art

-   Py_LIMITED_API macro of PEP 384 "Defining a Stable ABI".
-   Rejected PEP 606 "Python Compatibility Version" which has a global
    scope.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 617 Title: New PEG parser for CPython Author: Guido van Rossum
<EMAIL_ADDRESS>Pablo Galindo<EMAIL_ADDRESS>Lysandros
Nikolaou<EMAIL_ADDRESS>Discussions-To<EMAIL_ADDRESS>Status: Final Type: Standards Track Created: 24-Mar-2020 Python-Version:
3.9 Post-History: 02-Apr-2020

python:full-grammar-specification

Overview

This PEP proposes replacing the current LL(1)-based parser of CPython
with a new PEG-based parser. This new parser would allow the elimination
of multiple "hacks" that exist in the current grammar to circumvent the
LL(1)-limitation. It would substantially reduce the maintenance costs in
some areas related to the compiling pipeline such as the grammar, the
parser and the AST generation. The new PEG parser will also lift the
LL(1) restriction on the current Python grammar.

Background on LL(1) parsers

The current Python grammar is an LL(1)-based grammar. A grammar can be
said to be LL(1) if it can be parsed by an LL(1) parser, which in turn
is defined as a top-down parser that parses the input from left to
right, performing leftmost derivation of the sentence, with just one
token of lookahead. The traditional approach to constructing or
generating an LL(1) parser is to produce a parse table which encodes the
possible transitions between all possible states of the parser. These
tables are normally constructed from the first sets and the follow sets
of the grammar:

-   Given a rule, the first set is the collection of all terminals that
    can occur first in a full derivation of that rule. Intuitively, this
    helps the parser decide among the alternatives in a rule. For
    instance, given the rule:

        rule: A | B

    if only A can start with the terminal a and only B can start with
    the terminal b and the parser sees the token b when parsing this
    rule, it knows that it needs to follow the non-terminal B.

-   An extension to this simple idea is needed when a rule may expand to
    the empty string. Given a rule, the follow set is the collection of
    terminals that can appear immediately to the right of that rule in a
    partial derivation. Intuitively, this solves the problem of the
    empty alternative. For instance, given this rule:

        rule: A 'b'

    if the parser has the token b and the non-terminal A can only start
    with the token a, then the parser can tell that this is an invalid
    program. But if A could expand to the empty string (called an
    ε-production), then the parser would recognise a valid empty A,
    since the next token b is in the follow set of A.

The current Python grammar does not contain ε-productions, so the follow
sets are not needed when creating the parse tables. Currently, in
CPython, a parser generator program reads the grammar and produces a
parsing table representing a set of deterministic finite automata (DFA)
that can be included in a C program, the parser. The parser is a
pushdown automaton that uses this data to produce a Concrete Syntax Tree
(CST) sometimes known directly as a "parse tree". In this process, the
first sets are used indirectly when generating the DFAs.

LL(1) parsers and grammars are usually efficient and simple to implement
and generate. However, it is not possible, under the LL(1) restriction,
to express certain common constructs in a way natural to the language
designer and the reader. This includes some in the Python language.

As LL(1) parsers can only look one token ahead to distinguish
possibilities, some rules in the grammar may be ambiguous. For instance
the rule:

    rule: A | B

is ambiguous if the first sets of both A and B have some elements in
common. When the parser sees a token in the input program that both A
and B can start with, it is impossible for it to deduce which option to
expand, as no further token of the program can be examined to
disambiguate. The rule may be transformed to equivalent LL(1) rules, but
then it may be harder for a human reader to grasp its meaning. Examples
later in this document show that the current LL(1)-based grammar suffers
a lot from this scenario.

Another broad class of rules precluded by LL(1) is left-recursive rules.
A rule is left-recursive if it can derive to a sentential form with
itself as the leftmost symbol. For instance this rule:

    rule: rule 'a'

is left-recursive because the rule can be expanded to an expression that
starts with itself. As will be described later, left-recursion is the
natural way to express certain desired language properties directly in
the grammar.

Background on PEG parsers

A PEG (Parsing Expression Grammar) grammar differs from a context-free
grammar (like the current one) in the fact that the way it is written
more closely reflects how the parser will operate when parsing it. The
fundamental technical difference is that the choice operator is ordered.
This means that when writing:

    rule: A | B | C

a context-free-grammar parser (like an LL(1) parser) will generate
constructions that given an input string will deduce which alternative
(A, B or C) must be expanded, while a PEG parser will check if the first
alternative succeeds and only if it fails, will it continue with the
second or the third one in the order in which they are written. This
makes the choice operator not commutative.

Unlike LL(1) parsers, PEG-based parsers cannot be ambiguous: if a string
parses, it has exactly one valid parse tree. This means that a PEG-based
parser cannot suffer from the ambiguity problems described in the
previous section.

PEG parsers are usually constructed as a recursive descent parser in
which every rule in the grammar corresponds to a function in the program
implementing the parser and the parsing expression (the "expansion" or
"definition" of the rule) represents the "code" in said function. Each
parsing function conceptually takes an input string as its argument, and
yields one of the following results:

-   A "success" result. This result indicates that the expression can be
    parsed by that rule and the function may optionally move forward or
    consume one or more characters of the input string supplied to it.
-   A "failure" result, in which case no input is consumed.

Notice that "failure" results do not imply that the program is incorrect
or a parsing failure because as the choice operator is ordered, a
"failure" result merely indicates "try the following option". A direct
implementation of a PEG parser as a recursive descent parser will
present exponential time performance in the worst case as compared with
LL(1) parsers, because PEG parsers have infinite lookahead (this means
that they can consider an arbitrary number of tokens before deciding for
a rule). Usually, PEG parsers avoid this exponential time complexity
with a technique called "packrat parsing"[1] which not only loads the
entire program in memory before parsing it but also allows the parser to
backtrack arbitrarily. This is made efficient by memoizing the rules
already matched for each position. The cost of the memoization cache is
that the parser will naturally use more memory than a simple LL(1)
parser, which normally are table-based. We will explain later in this
document why we consider this cost acceptable.

Rationale

In this section, we describe a list of problems that are present in the
current parser machinery in CPython that motivates the need for a new
parser.

Some rules are not actually LL(1)

Although the Python grammar is technically an LL(1) grammar (because it
is parsed by an LL(1) parser) several rules are not LL(1) and several
workarounds are implemented in the grammar and in other parts of CPython
to deal with this. For example, consider the rule for assignment
expressions:

    namedexpr_test: [NAME ':='] test

This simple rule is not compatible with the Python grammar as NAME is
among the elements of the first set of the rule test. To work around
this limitation the actual rule that appears in the current grammar is:

    namedexpr_test: test [':=' test]

Which is a much broader rule than the previous one allowing constructs
like [x for x in y] := [1,2,3]. The way the rule is limited to its
desired form is by disallowing these unwanted constructions when
transforming the parse tree to the abstract syntax tree. This is not
only inelegant but a considerable maintenance burden as it forces the
AST creation routines and the compiler into a situation in which they
need to know how to separate valid programs from invalid programs, which
should be a responsibility solely of the parser. This also leads to the
actual grammar file not reflecting correctly what the actual grammar is
(that is, the collection of all valid Python programs).

Similar workarounds appear in multiple other rules of the current
grammar. Sometimes this problem is unsolvable. For instance, bpo-12782:
Multiple context expressions do not support parentheses for continuation
across lines shows how making an LL(1) rule that supports writing:

    with (
        open("a_really_long_foo") as foo,
        open("a_really_long_baz") as baz,
        open("a_really_long_bar") as bar
    ):
      ...

is not possible since the first sets of the grammar items that can
appear as context managers include the open parenthesis, making the rule
ambiguous. This rule is not only consistent with other parts of the
language (like the rule for multiple imports), but is also very useful
to auto-formatting tools, as parenthesized groups are normally used to
group elements to be formatted together (in the same way the tools
operate on the contents of lists, sets...).

Complicated AST parsing

Another problem of the current parser is that there is a huge coupling
between the AST generation routines and the particular shape of the
produced parse trees. This makes the code for generating the AST
especially complicated as many actions and choices are implicit. For
instance, the AST generation code knows what alternatives of a certain
rule are produced based on the number of child nodes present in a given
parse node. This makes the code difficult to follow as this property is
not directly related to the grammar file and is influenced by
implementation details. As a result of this, a considerable amount of
the AST generation code needs to deal with inspecting and reasoning
about the particular shape of the parse trees that it receives.

Lack of left recursion

As described previously, a limitation of LL(1) grammars is that they
cannot allow left-recursion. This makes writing some rules very
unnatural and far from how programmers normally think about the program.
For instance this construct (a simpler variation of several rules
present in the current grammar):

    expr: expr '+' term | term

cannot be parsed by an LL(1) parser. The traditional remedy is to
rewrite the grammar to circumvent the problem:

    expr: term ('+' term)*

The problem that appears with this form is that the parse tree is forced
to have a very unnatural shape. This is because with this rule, for the
input program a + b + c the parse tree will be flattened
(['a', '+', 'b', '+', 'c']) and must be post-processed to construct a
left-recursive parse tree ([['a', '+', 'b'], '+', 'c']). Being forced to
write the second rule not only leads to the parse tree not correctly
reflecting the desired associativity, but also imposes further pressure
on later compilation stages to detect and post-process these cases.

Intermediate parse tree

The last problem present in the current parser is the intermediate
creation of a parse tree or Concrete Syntax Tree that is later
transformed to an Abstract Syntax Tree. Although the construction of a
CST is very common in parser and compiler pipelines, in CPython this
intermediate CST is not used by anything else (it is only indirectly
exposed by the parser module and a surprisingly small part of the code
in the CST production is reused in the module). Which is worse: the
whole tree is kept in memory, keeping many branches that consist of
chains of nodes with a single child. This has been shown to consume a
considerable amount of memory (for instance in bpo-26415: Excessive peak
memory consumption by the Python parser).

Having to produce an intermediate result between the grammar and the AST
is not only undesirable but also makes the AST generation step much more
complicated, raising considerably the maintenance burden.

The new proposed PEG parser

The new proposed PEG parser contains the following pieces:

-   A parser generator that can read a grammar file and produce a PEG
    parser written in Python or C that can parse said grammar.
-   A PEG meta-grammar that automatically generates a Python parser that
    is used for the parser generator itself (this means that there are
    no manually-written parsers).
-   A generated parser (using the parser generator) that can directly
    produce C and Python AST objects.

Left recursion

PEG parsers normally do not support left recursion but we have
implemented a technique similar to the one described in Medeiros et
al.[2] but using the memoization cache instead of static variables. This
approach is closer to the one described in Warth et al.[3]. This allows
us to write not only simple left-recursive rules but also more
complicated rules that involve indirect left-recursion like:

    rule1: rule2 | 'a'
    rule2: rule3 | 'b'
    rule3: rule1 | 'c'

and "hidden left-recursion" like:

    rule: 'optional'? rule '@' some_other_rule

Syntax

The grammar consists of a sequence of rules of the form:

    rule_name: expression

Optionally, a type can be included right after the rule name, which
specifies the return type of the C or Python function corresponding to
the rule:

    rule_name[return_type]: expression

If the return type is omitted, then a void * is returned in C and an Any
in Python.

Grammar Expressions

# comment

Python-style comments.

e1 e2

Match e1, then match e2.

    rule_name: first_rule second_rule

e1 | e2

Match e1 or e2.

The first alternative can also appear on the line after the rule name
for formatting purposes. In that case, a | must be used before the first
alternative, like so:

    rule_name[return_type]:
        | first_alt
        | second_alt

( e )

Match e.

    rule_name: (e)

A slightly more complex and useful example includes using the grouping
operator together with the repeat operators:

    rule_name: (e1 e2)*

[ e ] or e?

Optionally match e.

    rule_name: [e]

A more useful example includes defining that a trailing comma is
optional:

    rule_name: e (',' e)* [',']

e*

Match zero or more occurrences of e.

    rule_name: (e1 e2)*

e+

Match one or more occurrences of e.

    rule_name: (e1 e2)+

s.e+

Match one or more occurrences of e, separated by s. The generated parse
tree does not include the separator. This is otherwise identical to
(e (s e)*).

    rule_name: ','.e+

&e

Succeed if e can be parsed, without consuming any input.

!e

Fail if e can be parsed, without consuming any input.

An example taken from the proposed Python grammar specifies that a
primary consists of an atom, which is not followed by a . or a ( or a [:

    primary: atom !'.' !'(' !'['

~

Commit to the current alternative, even if it fails to parse.

    rule_name: '(' ~ some_rule ')' | some_alt

In this example, if a left parenthesis is parsed, then the other
alternative won’t be considered, even if some_rule or ‘)’ fail to be
parsed.

Variables in the Grammar

A subexpression can be named by preceding it with an identifier and an =
sign. The name can then be used in the action (see below), like this:

    rule_name[return_type]: '(' a=some_other_rule ')' { a }

Grammar actions

To avoid the intermediate steps that obscure the relationship between
the grammar and the AST generation the proposed PEG parser allows
directly generating AST nodes for a rule via grammar actions. Grammar
actions are language-specific expressions that are evaluated when a
grammar rule is successfully parsed. These expressions can be written in
Python or C depending on the desired output of the parser generator.
This means that if one would want to generate a parser in Python and
another in C, two grammar files should be written, each one with a
different set of actions, keeping everything else apart from said
actions identical in both files. As an example of a grammar with Python
actions, the piece of the parser generator that parses grammar files is
bootstrapped from a meta-grammar file with Python actions that generate
the grammar tree as a result of the parsing.

In the specific case of the new proposed PEG grammar for Python, having
actions allows directly describing how the AST is composed in the
grammar itself, making it more clear and maintainable. This AST
generation process is supported by the use of some helper functions that
factor out common AST object manipulations and some other required
operations that are not directly related to the grammar.

To indicate these actions each alternative can be followed by the action
code inside curly-braces, which specifies the return value of the
alternative:

    rule_name[return_type]:
        | first_alt1 first_alt2 { first_alt1 }
        | second_alt1 second_alt2 { second_alt1 }

If the action is omitted and C code is being generated, then there are
two different possibilities:

1.  If there’s a single name in the alternative, this gets returned.
2.  If not, a dummy name object gets returned (this case should be
    avoided).

If the action is omitted and Python code is being generated, then a list
with all the parsed expressions gets returned (this is meant for
debugging).

The full meta-grammar for the grammars supported by the PEG generator
is:

    start[Grammar]: grammar ENDMARKER { grammar }

    grammar[Grammar]:
        | metas rules { Grammar(rules, metas) }
        | rules { Grammar(rules, []) }

    metas[MetaList]:
        | meta metas { [meta] + metas }
        | meta { [meta] }

    meta[MetaTuple]:
        | "@" NAME NEWLINE { (name.string, None) }
        | "@" a=NAME b=NAME NEWLINE { (a.string, b.string) }
        | "@" NAME STRING NEWLINE { (name.string, literal_eval(string.string)) }

    rules[RuleList]:
        | rule rules { [rule] + rules }
        | rule { [rule] }

    rule[Rule]:
        | rulename ":" alts NEWLINE INDENT more_alts DEDENT {
              Rule(rulename[0], rulename[1], Rhs(alts.alts + more_alts.alts)) }
        | rulename ":" NEWLINE INDENT more_alts DEDENT { Rule(rulename[0], rulename[1], more_alts) }
        | rulename ":" alts NEWLINE { Rule(rulename[0], rulename[1], alts) }

    rulename[RuleName]:
        | NAME '[' type=NAME '*' ']' {(name.string, type.string+"*")}
        | NAME '[' type=NAME ']' {(name.string, type.string)}
        | NAME {(name.string, None)}

    alts[Rhs]:
        | alt "|" alts { Rhs([alt] + alts.alts)}
        | alt { Rhs([alt]) }

    more_alts[Rhs]:
        | "|" alts NEWLINE more_alts { Rhs(alts.alts + more_alts.alts) }
        | "|" alts NEWLINE { Rhs(alts.alts) }

    alt[Alt]:
        | items '$' action { Alt(items + [NamedItem(None, NameLeaf('ENDMARKER'))], action=action) }
        | items '$' { Alt(items + [NamedItem(None, NameLeaf('ENDMARKER'))], action=None) }
        | items action { Alt(items, action=action) }
        | items { Alt(items, action=None) }

    items[NamedItemList]:
        | named_item items { [named_item] + items }
        | named_item { [named_item] }

    named_item[NamedItem]:
        | NAME '=' ~ item {NamedItem(name.string, item)}
        | item {NamedItem(None, item)}
        | it=lookahead {NamedItem(None, it)}

    lookahead[LookaheadOrCut]:
        | '&' ~ atom {PositiveLookahead(atom)}
        | '!' ~ atom {NegativeLookahead(atom)}
        | '~' {Cut()}

    item[Item]:
        | '[' ~ alts ']' {Opt(alts)}
        |  atom '?' {Opt(atom)}
        |  atom '*' {Repeat0(atom)}
        |  atom '+' {Repeat1(atom)}
        |  sep=atom '.' node=atom '+' {Gather(sep, node)}
        |  atom {atom}

    atom[Plain]:
        | '(' ~ alts ')' {Group(alts)}
        | NAME {NameLeaf(name.string) }
        | STRING {StringLeaf(string.string)}

    # Mini-grammar for the actions

    action[str]: "{" ~ target_atoms "}" { target_atoms }

    target_atoms[str]:
        | target_atom target_atoms { target_atom + " " + target_atoms }
        | target_atom { target_atom }

    target_atom[str]:
        | "{" ~ target_atoms "}" { "{" + target_atoms + "}" }
        | NAME { name.string }
        | NUMBER { number.string }
        | STRING { string.string }
        | "?" { "?" }
        | ":" { ":" }

As an illustrative example this simple grammar file allows directly
generating a full parser that can parse simple arithmetic expressions
and that returns a valid C-based Python AST:

    start[mod_ty]: a=expr_stmt* $ { Module(a, NULL, p->arena) }
    expr_stmt[stmt_ty]: a=expr NEWLINE { _Py_Expr(a, EXTRA) }
    expr[expr_ty]:
        | l=expr '+' r=term { _Py_BinOp(l, Add, r, EXTRA) }
        | l=expr '-' r=term { _Py_BinOp(l, Sub, r, EXTRA) }
        | t=term { t }

    term[expr_ty]:
        | l=term '*' r=factor { _Py_BinOp(l, Mult, r, EXTRA) }
        | l=term '/' r=factor { _Py_BinOp(l, Div, r, EXTRA) }
        | f=factor { f }

    factor[expr_ty]:
        | '(' e=expr ')' { e }
        | a=atom { a }

    atom[expr_ty]:
        | n=NAME { n }
        | n=NUMBER { n }
        | s=STRING { s }

Here EXTRA is a macro that expands to
start_lineno, start_col_offset, end_lineno, end_col_offset, p->arena,
those being variables automatically injected by the parser; p points to
an object that holds on to all state for the parser.

A similar grammar written to target Python AST objects:

    start: expr NEWLINE? ENDMARKER { ast.Expression(expr) }
    expr:
        | expr '+' term { ast.BinOp(expr, ast.Add(), term) }
        | expr '-' term { ast.BinOp(expr, ast.Sub(), term) }
        | term { term }

    term:
        | l=term '*' r=factor { ast.BinOp(l, ast.Mult(), r) }
        | term '/' factor { ast.BinOp(term, ast.Div(), factor) }
        | factor { factor }

    factor:
        | '(' expr ')' { expr }
        | atom { atom }

    atom:
        | NAME { ast.Name(id=name.string, ctx=ast.Load()) }
        | NUMBER { ast.Constant(value=ast.literal_eval(number.string)) }

Migration plan

This section describes the migration plan when porting to the new
PEG-based parser if this PEP is accepted. The migration will be executed
in a series of steps that allow initially to fallback to the previous
parser if needed:

1.  Starting with Python 3.9 alpha 6, include the new PEG-based parser
    machinery in CPython with a command-line flag and environment
    variable that allows switching between the new and the old parsers
    together with explicit APIs that allow invoking the new and the old
    parsers independently. At this step, all Python APIs like ast.parse
    and compile will use the parser set by the flags or the environment
    variable and the default parser will be the new PEG-based parser.
2.  Between Python 3.9 and Python 3.10, the old parser and related code
    (like the "parser" module) will be kept until a new Python release
    happens (Python 3.10). In the meanwhile and until the old parser is
    removed, no new Python Grammar addition will be added that requires
    the PEG parser. This means that the grammar will be kept LL(1) until
    the old parser is removed.
3.  In Python 3.10, remove the old parser, the command-line flag, the
    environment variable and the "parser" module and related code.

Performance and validation

We have done extensive timing and validation of the new parser, and this
gives us confidence that the new parser is of high enough quality to
replace the current parser.

Validation

To start with validation, we regularly compile the entire Python 3.8
stdlib and compare every aspect of the resulting AST with that produced
by the standard compiler. (In the process we found a few bugs in the
standard parser's treatment of line and column numbers, which we have
all fixed upstream via a series of issues and PRs.)

We have also occasionally compiled a much larger codebase (the approx.
3800 most popular packages on PyPI) and this has helped us find a (very)
few additional bugs in the new parser.

(One area we have not explored extensively is rejection of all wrong
programs. We have unit tests that check for a certain number of explicit
rejections, but more work could be done, e.g. by using a fuzzer that
inserts random subtle bugs into existing code. We're open to help in
this area.)

Performance

We have tuned the performance of the new parser to come within 10% of
the current parser both in speed and memory consumption. While the
PEG/packrat parsing algorithm inherently consumes more memory than the
current LL(1) parser, we have an advantage because we don't construct an
intermediate CST.

Below are some benchmarks. These are focused on compiling source code to
bytecode, because this is the most realistic situation. Returning an AST
to Python code is not as representative, because the process to convert
the internal AST (only accessible to C code) to an external AST (an
instance of ast.AST) takes more time than the parser itself.

All measurements reported here are done on a recent MacBook Pro, taking
the median of three runs. No particular care was taken to stop other
applications running on the same machine.

The first timings are for our canonical test file, which has 100,000
lines endlessly repeating the following three lines:

    1 + 2 + 4 + 5 + 6 + 7 + 8 + 9 + 10 + ((((((11 * 12 * 13 * 14 * 15 + 16 * 17 + 18 * 19 * 20))))))
    2*3 + 4*5*6
    12 + (2 * 3 * 4 * 5 + 6 + 7 * 8)

-   Just parsing and throwing away the internal AST takes 1.16 seconds
    with a max RSS of 681 MiB.
-   Parsing and converting to ast.AST takes 6.34 seconds, max RSS 1029
    MiB.
-   Parsing and compiling to bytecode takes 1.28 seconds, max RSS 681
    MiB.
-   With the current parser, parsing and compiling takes 1.44 seconds,
    max RSS 836 MiB.

For this particular test file, the new parser is faster and uses less
memory than the current parser (compare the last two bullets).

We also did timings with a more realistic payload, the entire Python 3.8
stdlib. This payload consists of 1,641 files, 749,570 lines, 27,622,497
bytes. (Though 11 files can't be compiled by any Python 3 parser due to
encoding issues, sometimes intentional.)

-   Compiling and throwing away the internal AST took 2.141 seconds.
    That's 350,040 lines/sec, or 12,899,367 bytes/sec. The max RSS was
    74 MiB (the largest file in the stdlib is much smaller than our
    canonical test file).
-   Compiling to bytecode took 3.290 seconds. That's 227,861 lines/sec,
    or 8,396,942 bytes/sec. Max RSS 77 MiB.
-   Compiling to bytecode using the current parser took 3.367 seconds.
    That's 222,620 lines/sec, or 8,203,780 bytes/sec. Max RSS 70 MiB.

Comparing the last two bullets we find that the new parser is slightly
faster but uses slightly (about 10%) more memory. We believe this is
acceptable. (Also, there are probably some more tweaks we can make to
reduce memory usage.)

Rejected Alternatives

We did not seriously consider alternative ways to implement the new
parser, but here's a brief discussion of LALR(1).

Thirty years ago the first author decided to go his own way with
Python's parser rather than using LALR(1), which was the industry
standard at the time (e.g. Bison and Yacc). The reasons were primarily
emotional (gut feelings, intuition), based on past experience using Yacc
in other projects, where grammar development took more effort than
anticipated (in part due to shift-reduce conflicts). A specific
criticism of Bison and Yacc that still holds is that their meta-grammar
(the notation used to feed the grammar into the parser generator) does
not support EBNF conveniences like [optional_clause] or
(repeated_clause)*. Using a custom parser generator, a syntax tree
matching the structure of the grammar could be generated automatically,
and with EBNF that tree could match the "human-friendly" structure of
the grammar.

Other variants of LR were not considered, nor was LL (e.g. ANTLR). PEG
was selected because it was easy to understand given a basic
understanding of recursive-descent parsing.

References

[4] Guido's series on PEG parsing
https://medium.com/@gvanrossum_83706/peg-parsing-series-de5d41b2ed60

Copyright

This document has been placed in the public domain.

[1] Ford, Bryan https://pdos.csail.mit.edu/~baford/packrat/thesis/

[2] Medeiros et al. https://arxiv.org/pdf/1207.0443.pdf

[3] Warth et al. https://web.cs.ucla.edu/~todd/research/pepm08.pdf


text:
PEP: 427 Title: The Wheel Binary Package Format 1.0 Version: $Revision$
Last-Modified: $Date$ Author: Daniel Holth<EMAIL_ADDRESS>BDFL-Delegate: Alyssa Coghlan<EMAIL_ADDRESS>Discussions-To:
<EMAIL_ADDRESS>Status: Final Type: Standards Track Topic:
Packaging Content-Type: text/x-rst Created: 20-Sep-2012 Post-History:
18-Oct-2012, 15-Feb-2013 Resolution:
https://mail.python.org/pipermail/python-dev/2013-February/124103.html

packaging:binary-distribution-format

Abstract

This PEP describes a built-package format for Python called "wheel".

A wheel is a ZIP-format archive with a specially formatted file name and
the .whl extension. It contains a single distribution nearly as it would
be installed according to PEP 376 with a particular installation scheme.
Although a specialized installer is recommended, a wheel file may be
installed by simply unpacking into site-packages with the standard
'unzip' tool while preserving enough information to spread its contents
out onto their final paths at any later time.

PEP Acceptance

This PEP was accepted, and the defined wheel version updated to 1.0, by
Alyssa Coghlan on 16th February, 2013[1]

Rationale

Python needs a package format that is easier to install than sdist.
Python's sdist packages are defined by and require the distutils and
setuptools build systems, running arbitrary code to build-and-install,
and re-compile, code just so it can be installed into a new virtualenv.
This system of conflating build-install is slow, hard to maintain, and
hinders innovation in both build systems and installers.

Wheel attempts to remedy these problems by providing a simpler interface
between the build system and the installer. The wheel binary package
format frees installers from having to know about the build system,
saves time by amortizing compile time over many installations, and
removes the need to install a build system in the target environment.

Details

Installing a wheel 'distribution-1.0-py32-none-any.whl'

Wheel installation notionally consists of two phases:

-   Unpack.
    a.  Parse distribution-1.0.dist-info/WHEEL.
    b.  Check that installer is compatible with Wheel-Version. Warn if
        minor version is greater, abort if major version is greater.
    c.  If Root-Is-Purelib == 'true', unpack archive into purelib
        (site-packages).
    d.  Else unpack archive into platlib (site-packages).
-   Spread.
    a.  Unpacked archive includes distribution-1.0.dist-info/ and (if
        there is data) distribution-1.0.data/.
    b.  Move each subtree of distribution-1.0.data/ onto its destination
        path. Each subdirectory of distribution-1.0.data/ is a key into
        a dict of destination directories, such as
        distribution-1.0.data/(purelib|platlib|headers|scripts|data).
        The initially supported paths are taken from
        distutils.command.install.
    c.  If applicable, update scripts starting with #!python to point to
        the correct interpreter.
    d.  Update distribution-1.0.dist-info/RECORD with the installed
        paths.
    e.  Remove empty distribution-1.0.data directory.
    f.  Compile any installed .py to .pyc. (Uninstallers should be smart
        enough to remove .pyc even if it is not mentioned in RECORD.)

Recommended installer features

Rewrite #!python.

    In wheel, scripts are packaged in
    {distribution}-{version}.data/scripts/. If the first line of a file
    in scripts/ starts with exactly b'#!python', rewrite to point to the
    correct interpreter. Unix installers may need to add the +x bit to
    these files if the archive was created on Windows.

    The b'#!pythonw' convention is allowed. b'#!pythonw' indicates a GUI
    script instead of a console script.

Generate script wrappers.

    In wheel, scripts packaged on Unix systems will certainly not have
    accompanying .exe wrappers. Windows installers may want to add them
    during install.

Recommended archiver features

Place .dist-info at the end of the archive.

    Archivers are encouraged to place the .dist-info files physically at
    the end of the archive. This enables some potentially interesting
    ZIP tricks including the ability to amend the metadata without
    rewriting the entire archive.

File Format

File name convention

The wheel filename is
{distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whl.

distribution

    Distribution name, e.g. 'django', 'pyramid'.

version

    Distribution version, e.g. 1.0.

build tag

    Optional build number. Must start with a digit. Acts as a
    tie-breaker if two wheel file names are the same in all other
    respects (i.e. name, version, and other tags). Sort as an empty
    tuple if unspecified, else sort as a two-item tuple with the first
    item being the initial digits as an int, and the second item being
    the remainder of the tag as a str.

language implementation and version tag

    E.g. 'py27', 'py2', 'py3'.

abi tag

    E.g. 'cp33m', 'abi3', 'none'.

platform tag

    E.g. 'linux_x86_64', 'any'.

For example, distribution-1.0-1-py27-none-any.whl is the first build of
a package called 'distribution', and is compatible with Python 2.7 (any
Python 2.7 implementation), with no ABI (pure Python), on any CPU
architecture.

The last three components of the filename before the extension are
called "compatibility tags." The compatibility tags express the
package's basic interpreter requirements and are detailed in PEP 425.

Escaping and Unicode

Each component of the filename is escaped by replacing runs of
non-alphanumeric characters with an underscore _:

    re.sub("[^\w\d.]+", "_", distribution, re.UNICODE)

The archive filename is Unicode. It will be some time before the tools
are updated to support non-ASCII filenames, but they are supported in
this specification.

The filenames inside the archive are encoded as UTF-8. Although some ZIP
clients in common use do not properly display UTF-8 filenames, the
encoding is supported by both the ZIP specification and Python's
zipfile.

File contents

The contents of a wheel file, where {distribution} is replaced with the
name of the package, e.g. beaglevote and {version} is replaced with its
version, e.g. 1.0.0, consist of:

1.  /, the root of the archive, contains all files to be installed in
    purelib or platlib as specified in WHEEL. purelib and platlib are
    usually both site-packages.

2.  {distribution}-{version}.dist-info/ contains metadata.

3.  {distribution}-{version}.data/ contains one subdirectory for each
    non-empty install scheme key not already covered, where the
    subdirectory name is an index into a dictionary of install paths
    (e.g. data, scripts, headers, purelib, platlib).

4.  Python scripts must appear in scripts and begin with exactly
    b'#!python' in order to enjoy script wrapper generation and #!python
    rewriting at install time. They may have any or no extension.

5.  {distribution}-{version}.dist-info/METADATA is Metadata version 1.1
    or greater format metadata.

6.  {distribution}-{version}.dist-info/WHEEL is metadata about the
    archive itself in the same basic key: value format:

        Wheel-Version: 1.0
        Generator: bdist_wheel 1.0
        Root-Is-Purelib: true
        Tag: py2-none-any
        Tag: py3-none-any
        Build: 1

7.  Wheel-Version is the version number of the Wheel specification.

8.  Generator is the name and optionally the version of the software
    that produced the archive.

9.  Root-Is-Purelib is true if the top level directory of the archive
    should be installed into purelib; otherwise the root should be
    installed into platlib.

10. Tag is the wheel's expanded compatibility tags; in the example the
    filename would contain py2.py3-none-any.

11. Build is the build number and is omitted if there is no build
    number.

12. A wheel installer should warn if Wheel-Version is greater than the
    version it supports, and must fail if Wheel-Version has a greater
    major version than the version it supports.

13. Wheel, being an installation format that is intended to work across
    multiple versions of Python, does not generally include .pyc files.

14. Wheel does not contain setup.py or setup.cfg.

This version of the wheel specification is based on the distutils
install schemes and does not define how to install files to other
locations. The layout offers a superset of the functionality provided by
the existing wininst and egg binary formats.

The .dist-info directory

1.  Wheel .dist-info directories include at a minimum METADATA, WHEEL,
    and RECORD.
2.  METADATA is the package metadata, the same format as PKG-INFO as
    found at the root of sdists.
3.  WHEEL is the wheel metadata specific to a build of the package.
4.  RECORD is a list of (almost) all the files in the wheel and their
    secure hashes. Unlike PEP 376, every file except RECORD, which
    cannot contain a hash of itself, must include its hash. The hash
    algorithm must be sha256 or better; specifically, md5 and sha1 are
    not permitted, as signed wheel files rely on the strong hashes in
    RECORD to validate the integrity of the archive.
5.  PEP 376's INSTALLER and REQUESTED are not included in the archive.
6.  RECORD.jws is used for digital signatures. It is not mentioned in
    RECORD.
7.  RECORD.p7s is allowed as a courtesy to anyone who would prefer to
    use S/MIME signatures to secure their wheel files. It is not
    mentioned in RECORD.
8.  During extraction, wheel installers verify all the hashes in RECORD
    against the file contents. Apart from RECORD and its signatures,
    installation will fail if any file in the archive is not both
    mentioned and correctly hashed in RECORD.

The .data directory

Any file that is not normally installed inside site-packages goes into
the .data directory, named as the .dist-info directory but with the
.data/ extension:

    distribution-1.0.dist-info/

    distribution-1.0.data/

The .data directory contains subdirectories with the scripts, headers,
documentation and so forth from the distribution. During installation
the contents of these subdirectories are moved onto their destination
paths.

Signed wheel files

Wheel files include an extended RECORD that enables digital signatures.
PEP 376's RECORD is altered to include a secure hash
digestname=urlsafe_b64encode_nopad(digest) (urlsafe base64 encoding with
no trailing = characters) as the second column instead of an md5sum. All
possible entries are hashed, including any generated files such as .pyc
files, but not RECORD which cannot contain its own hash. For example:

    file.py,sha256=AVTFPZpEKzuHr7OvQZmhaU3LvwKz06AJw8mT\_pNh2yI,3144
    distribution-1.0.dist-info/RECORD,,

The signature file(s) RECORD.jws and RECORD.p7s are not mentioned in
RECORD at all since they can only be added after RECORD is generated.
Every other file in the archive must have a correct hash in RECORD or
the installation will fail.

If JSON web signatures are used, one or more JSON Web Signature JSON
Serialization (JWS-JS) signatures is stored in a file RECORD.jws
adjacent to RECORD. JWS is used to sign RECORD by including the SHA-256
hash of RECORD as the signature's JSON payload:

    { "hash": "sha256=ADD-r2urObZHcxBW3Cr-vDCu5RJwT4CaRTHiFmbcIYY" }

(The hash value is the same format used in RECORD.)

If RECORD.p7s is used, it must contain a detached S/MIME format
signature of RECORD.

A wheel installer is not required to understand digital signatures but
MUST verify the hashes in RECORD against the extracted file contents.
When the installer checks file hashes against RECORD, a separate
signature checker only needs to establish that RECORD matches the
signature.

See

-   7515
-   https://datatracker.ietf.org/doc/html/draft-jones-jose-jws-json-serialization.html
-   7517
-   https://datatracker.ietf.org/doc/html/draft-jones-jose-json-private-key.html

Comparison to .egg

1.  Wheel is an installation format; egg is importable. Wheel archives
    do not need to include .pyc and are less tied to a specific Python
    version or implementation. Wheel can install (pure Python) packages
    built with previous versions of Python so you don't always have to
    wait for the packager to catch up.
2.  Wheel uses .dist-info directories; egg uses .egg-info. Wheel is
    compatible with the new world of Python packaging and the new
    concepts it brings.
3.  Wheel has a richer file naming convention for today's
    multi-implementation world. A single wheel archive can indicate its
    compatibility with a number of Python language versions and
    implementations, ABIs, and system architectures. Historically the
    ABI has been specific to a CPython release, wheel is ready for the
    stable ABI.
4.  Wheel is lossless. The first wheel implementation bdist_wheel always
    generates egg-info, and then converts it to a .whl. It is also
    possible to convert existing eggs and bdist_wininst distributions.
5.  Wheel is versioned. Every wheel file contains the version of the
    wheel specification and the implementation that packaged it.
    Hopefully the next migration can simply be to Wheel 2.0.
6.  Wheel is a reference to the other Python.

FAQ

Wheel defines a .data directory. Should I put all my data there?

  This specification does not have an opinion on how you should organize
  your code. The .data directory is just a place for any files that are
  not normally installed inside site-packages or on the PYTHONPATH. In
  other words, you may continue to use
  pkgutil.get_data(package, resource) even though those files will
  usually not be distributed in wheel's .data directory.

Why does wheel include attached signatures?

  Attached signatures are more convenient than detached signatures
  because they travel with the archive. Since only the individual files
  are signed, the archive can be recompressed without invalidating the
  signature or individual files can be verified without having to
  download the whole archive.

Why does wheel allow JWS signatures?

  The JOSE specifications of which JWS is a part are designed to be easy
  to implement, a feature that is also one of wheel's primary design
  goals. JWS yields a useful, concise pure-Python implementation.

Why does wheel also allow S/MIME signatures?

  S/MIME signatures are allowed for users who need or want to use
  existing public key infrastructure with wheel.

  Signed packages are only a basic building block in a secure package
  update system. Wheel only provides the building block.

What's the deal with "purelib" vs. "platlib"?

  Wheel preserves the "purelib" vs. "platlib" distinction, which is
  significant on some platforms. For example, Fedora installs pure
  Python packages to '/usr/lib/pythonX.Y/site-packages' and platform
  dependent packages to '/usr/lib64/pythonX.Y/site-packages'.

  A wheel with "Root-Is-Purelib: false" with all its files in
  {name}-{version}.data/purelib is equivalent to a wheel with
  "Root-Is-Purelib: true" with those same files in the root, and it is
  legal to have files in both the "purelib" and "platlib" categories.

  In practice a wheel should have only one of "purelib" or "platlib"
  depending on whether it is pure Python or not and those files should
  be at the root with the appropriate setting given for
  "Root-is-purelib".

Is it possible to import Python code directly from a wheel file?

  Technically, due to the combination of supporting installation via
  simple extraction and using an archive format that is compatible with
  zipimport, a subset of wheel files do support being placed directly on
  sys.path. However, while this behaviour is a natural consequence of
  the format design, actually relying on it is generally discouraged.

  Firstly, wheel is designed primarily as a distribution format, so
  skipping the installation step also means deliberately avoiding any
  reliance on features that assume full installation (such as being able
  to use standard tools like pip and virtualenv to capture and manage
  dependencies in a way that can be properly tracked for auditing and
  security update purposes, or integrating fully with the standard build
  machinery for C extensions by publishing header files in the
  appropriate place).

  Secondly, while some Python software is written to support running
  directly from a zip archive, it is still common for code to be written
  assuming it has been fully installed. When that assumption is broken
  by trying to run the software from a zip archive, the failures can
  often be obscure and hard to diagnose (especially when they occur in
  third party libraries). The two most common sources of problems with
  this are the fact that importing C extensions from a zip archive is
  not supported by CPython (since doing so is not supported directly by
  the dynamic loading machinery on any platform) and that when running
  from a zip archive the __file__ attribute no longer refers to an
  ordinary filesystem path, but to a combination path that includes both
  the location of the zip archive on the filesystem and the relative
  path to the module inside the archive. Even when software correctly
  uses the abstract resource APIs internally, interfacing with external
  components may still require the availability of an actual on-disk
  file.

  Like metaclasses, monkeypatching and metapath importers, if you're not
  already sure you need to take advantage of this feature, you almost
  certainly don't need it. If you do decide to use it anyway, be aware
  that many projects will require a failure to be reproduced with a
  fully installed package before accepting it as a genuine bug.

References

Appendix

Example urlsafe-base64-nopad implementation:

    # urlsafe-base64-nopad for Python 3
    import base64

    def urlsafe_b64encode_nopad(data):
        return base64.urlsafe_b64encode(data).rstrip(b'=')

    def urlsafe_b64decode_nopad(data):
        pad = b'=' * (4 - (len(data) & 3))
        return base64.urlsafe_b64decode(data + pad)

Copyright

This document has been placed into the public domain.

[1] PEP acceptance
(https://mail.python.org/pipermail/python-dev/2013-February/124103.html)