text:
PEP: 761 Title: Deprecating PGP signatures for CPython artifacts Author:
Seth Michael Larson<EMAIL_ADDRESS>Sponsor: Hugo van Kemenade
Discussions-To:
https://discuss.python.org/t/pep-761-deprecating-pgp-signatures-for-cpython-artifacts/67180
Status: Draft Type: Process Created: 08-Oct-2024 Python-Version: 3.14
Post-History: 25-Sep-2024, 09-Oct-2024

Abstract

Since Python 3.11.0, CPython has provided two verifiable digital
signatures for all CPython artifacts: PGP and Sigstore.

PGP's design requires the maintenance and protection of long-lived
private keys by trusted parties. PGP's security and ergonomics have been
criticized by security practitioners for many years now, with the
biggest issue being that there were few alternatives for "artifact
signing" being proposed or adopted.

Sigstore's design philosophy has focused on the ergonomics of signing
and verifying and uses short-lived keys with strongly-bound
human-readable identities via OpenID Connect. Sigstore has both
development and adoption momentum, seeing adoption by PyPI, NPM,
Homebrew, and GitHub, among other ecosystems.

This PEP proposes to move CPython to using Sigstore exclusively for
signing artifacts through a deprecation and eventual discontinuance of
providing PGP signatures with new release managers.

Motivation

CPython's releases are release-manager-centric, where a single person
maintains multiple CPython releases from pre-release to end-of-life over
the course of many years.

Requiring release managers to maintain and protect PGP private keys for
seven or more years is an unnecessary burden in the new age of ergonomic
and ephemeral signing keys. Comparatively, Sigstore only requires
release managers to click a button during the release process to OAuth
sign-on to their identity provider. Maintaining the integrity of
accounts on identity providers like GitHub is already an expectation of
being a Python release manager or core team member, such as through
multi-factor authentication and strong unique passwords.

Rationale

Preserve expectations across a Python release

To avoid breaking downstream verifiers, the expectations for
verification materials availability SHOULD NOT be changed during a
feature release's lifecycle.

Release managers, not releases

The discontinuation of PGP signatures doesn't necessarily have to happen
on a "release manager boundary"; a new Python release could be a
potential boundary.

Because the primary motivation for deprecating PGP is ergonomics,
deciding to drop PGP for one release while a release manager still has
obligations to provide PGP signatures for other releases for multiple
years isn't much savings of effort.

A new release manager also represents a new PGP public key that
downstream verifiers need to adopt. By choosing to make the change
during this period, this minimizes the breakage to a place in downstream
maintenance where a change will already be necessary.

Gordian knot of signing methods and verifiers

CPython providing both PGP and Sigstore signatures concurrently creates
a "Gordian knot" where verifiers are disincentivized to migrate to a new
signature method due to the continued and expected availability of an
existing signature method, thus propagating the apparent demand for
maintaining the existing signature method.

This situation slows down the adoption of new signature methods like
Sigstore for both signature-producing projects and signature-verifying
ecosystems by not creating a "need" to automate and integrate the
signature method into verifier tooling.

By changing the expectation of what future signature methods will be
available, the incentive-knot can be broken by spurring the adoption of
the new signature method in downstream tooling. This change to verifier
tooling also makes other upstream projects able to migrate to publishing
only Sigstore signatures, resulting in a positive feedback loop of
adoption.

Specification

Because PGP keys are tied to a release manager identity, the change to
availability of PGP signatures will be tied to release managers instead
of individual releases (3.13, 3.14, etc). This PEP both deprecates and
proposes a discontinuation timeline for PGP signatures.

Deprecation and discontinuation of PGP signatures

This PEP deprecates PGP signatures for future CPython releases and
recommends verifiers to adopt Sigstore to verify CPython artifacts as an
alternative to PGP.

This PEP also removes the expectation that PGP signatures be published
by future release managers that don't already maintain a stable Python
release. At the time of writing this would be Hugo van Kemenade, as 3.14
is the next Python version without a stable release.

Releases which already have a stable release (3.13, 3.12, 3.11, etc) are
not affected and will continue to provide PGP signatures for artifacts
until they are end-of-life. All existing PGP signatures will continue to
work as expected.

Delaying discontinuation of PGP signatures

This PEP provides a mechanism to delay the discontinuation of PGP
signatures from active and upcoming CPython releases in case of
extraordinary circumstances. Deprecation of PGP signatures can't be
changed without a superseding PEP.

The Steering Council MAY at a future date after this PEP's acceptance
decide to delay the discontinuation of PGP signatures to a future
CPython release. If the Steering Council decides to delay the
discontinuation of PGP signatures then all active release managers MUST
provide PGP signatures for their covered CPython artifacts for the
remainder of their tenure as a release manager. This includes all steps
required to do so, such as generating a new PGP key and publishing their
identity to python.org.

The discontinuation of PGP signatures then is automatically scheduled
for the next release manager without a stable release, to be highlighted
in the Steering Council decision.

Backwards Compatibility

This proposal would remove the ability to verify future CPython
artifacts using PGP. Any downstream verifiers using PGP for CPython
artifacts would need to either start using Sigstore, verify their source
code of CPython through other means, or stop verification altogether for
future CPython releases.

Security Implications

PGP and Sigstore have different security models, so by removing PGP
signatures this means that all users only have the option to rely on the
security model provided by Sigstore.

In general, the security model required for artifact signatures is being
able to detect whether a given artifact is from the expected source and
hasn't been modified, regardless of the security or integrity of the
hosting service (in CPython's case: python.org/downloads).

Sigstore's security model depends more on centralized infrastructure
compared to PGP, such as the "public good" signature transparency log
(Rekor), certificate authority and transparency log (Fulcio), and the
security of OpenID Connect identity providers like Google and GitHub.

CPython's development already depends on the security of some of these
services and the others are better resourced than any individual release
manager to provide long-term public key management.

How to Teach This

CPython already documents how to verify artifacts using Sigstore based
on the pre-published identities of release managers. Documentation will
be updated to indicate the deprecation and future expectations of PGP
signatures.

Verifying signatures of CPython artifacts isn't something we should
expect from new Python users. Instead, Sigstore is more likely to be a
part of a downstream integrator's build pipeline such as a Linux distro,
Homebrew, pyenv, or others that programmatically fetch and build CPython
from source.

Rejected Ideas

Continue publishing PGP signatures indefinitely

Being a release manager is already a difficult, time-consuming, and
long-term commitment that is typically done on a volunteer basis. Thus
we see removal of PGP key management duties as a step towards reducing
burnout and stress of future release managers and improving the
sustainability of CPython.

Appendix

Support for offline verification

During the pre-PEP discussion, there was a question of whether offline
verification was supported by Sigstore. Using a Sigstore bundle
(.sigstore) file, Sigstore clients support verifying the artifact
completely offline.

Using offline verification with Sigstore requires disabling root of
trust updates and "pinning" a root of trust in a file to use during
verification.

Pinning a root of trust means signatures made after a new root of trust
is established would no longer be able to verify using a "pinned"
previous root of trust. New roots of trust are expected to be rare
events, such as when the root of trust is compromised, and in this case
verifiers would want signatures to fail to verify.

Offline verification also makes revocation checks impossible, but this
is similar to PGP's model where revocation of keys requires an online
lookup.

Barring rare events like root of trust compromise, using offline
verification with Sigstore doesn't impose additional operations
requirements to verifiers.

Support for a pre-compiled executable for verification

During discussion there were requests for a pre-compiled executable that
could be used for verifying Sigstore bundles without needing to either
install a Go build toolchain to build sigstore-go from source or already
have a working Python installation for sigstore-python.

Cosign is another Sigstore project that provides pre-compiled standalone
binaries and supports verifying bundles offline:

    # Download Cosign
    wget https://github.com/sigstore/cosign/releases/download/v2.4.1/cosign-linux-amd64

    # For offline verification, also need the Root of Trust. Can be grabbed
    # from GitHub at: https://github.com/sigstore/root-signing/blob/main/targets/trusted_root.json
    wget https://raw.githubusercontent.com/sigstore/root-signing/refs/heads/main/targets/trusted_root.json

    # Download CPython artifacts
    wget https://www.python.org/ftp/python/3.13.0/Python-3.13.0.tgz
    wget https://www.python.org/ftp/python/3.13.0/Python-3.13.0.tgz.sigstore

    ./cosign-linux-amd64 verify-blob \
      --new-bundle-format \
      --certificate-oidc-issuer 'https://accounts.google.com' \
      --certificate-identity<EMAIL_ADDRESS>\
      --bundle ./Python-3.13.0.tgz.sigstore \
      # --offline and --trust-root optional for offline verification
      --offline \
      --trust-root ./trusted_root.json \
      ./Python-3.13.0.tgz

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 575 Title: Unifying function/method classes Author: Jeroen Demeyer
<EMAIL_ADDRESS>Status: Withdrawn Type: Standards Track
Content-Type: text/x-rst Created: 27-Mar-2018 Python-Version: 3.8
Post-History: 31-Mar-2018, 12-Apr-2018, 27-Apr-2018, 05-May-2018

Withdrawal notice

See PEP 580 for a better solution to allowing fast calling of custom
classes.

See PEP 579 for a broader discussion of some of the other issues from
this PEP.

Abstract

Reorganize the class hierarchy for functions and methods with the goal
of reducing the difference between built-in functions (implemented in C)
and Python functions. Mainly, make built-in functions behave more like
Python functions without sacrificing performance.

A new base class base_function is introduced and the various function
classes, as well as method (renamed to bound_method), inherit from it.

We also allow subclassing the Python function class.

Motivation

Currently, CPython has two different function classes: the first is
Python functions, which is what you get when defining a function with
def or lambda. The second is built-in functions such as len, isinstance
or numpy.dot. These are implemented in C.

These two classes are implemented completely independently and have
different functionality. In particular, it is currently not possible to
implement a function efficiently in C (only built-in functions can do
that) while still allowing introspection like inspect.signature or
inspect.getsourcefile (only Python functions can do that). This is a
problem for projects like Cython[1] that want to do exactly that.

In Cython, this was worked around by inventing a new function class
called cyfunction. Unfortunately, a new function class creates problems:
the inspect module does not recognize such functions as being
functions[2] and the performance is worse (CPython has specific
optimizations for calling built-in functions).

A second motivation is more generally making built-in functions and
methods behave more like Python functions and methods. For example,
Python unbound methods are just functions but unbound methods of
extension types (e.g. dict.get) are a distinct class. Bound methods of
Python classes have a __func__ attribute, bound methods of extension
types do not.

Third, this PEP allows great customization of functions. The function
class becomes subclassable and custom function subclasses are also
allowed for functions implemented in C. In the latter case, this can be
done with the same performance as true built-in functions. All functions
can access the function object (the self in __call__), paving the way
for PEP 573.

New classes

This is the new class hierarchy for functions and methods:

    object
       |
       |
    base_function
    /       |     \
    /        |      \
    /         |   defined_function
    /          |        \
    cfunction (*)    |         \
       |       function
       |
    bound_method (*)

The two classes marked with () donot* allow subclassing; the others do.

There is no difference between functions and unbound methods, while
bound methods are instances of bound_method.

base_function

The class base_function becomes a new base class for all function types.
It is based on the existing builtin_function_or_method class, but with
the following differences and new features:

1.  It acts as a descriptor implementing __get__ to turn a function into
    a method if m_self is NULL. If m_self is not NULL, then this is a
    no-op: the existing function is returned instead.
2.  A new read-only attribute __parent__, represented in the C structure
    as m_parent. If this attribute exists, it represents the defining
    object. For methods of extension types, this is the defining class
    (__class__ in plain Python) and for functions of a module, this is
    the defining module. In general, it can be any Python object. If
    __parent__ is a class, it carries special semantics: in that case,
    the function must be called with self being an instance of that
    class. Finally, __qualname__ and __reduce__ will use __parent__ as
    namespace (instead of __self__ before).
3.  A new attribute __objclass__ which equals __parent__ if __parent__
    is a class. Otherwise, accessing __objclass__ raises AttributeError.
    This is meant to be backwards compatible with method_descriptor.
4.  The field ml_doc and the attributes __doc__ and __text_signature__
    (see Argument Clinic <436>) are not supported.
5.  A new flag METH_PASS_FUNCTION for ml_flags. If this flag is set, the
    C function stored in ml_meth is called with an additional first
    argument equal to the function object.
6.  A new flag METH_BINDING for ml_flags which only applies to functions
    of a module (not methods of a class). If this flag is set, then
    m_self is set to NULL instead of the module. This allows the
    function to behave more like a Python function as it enables
    __get__.
7.  A new flag METH_CALL_UNBOUND to disable self slicing.
8.  A new flag METH_PYTHON for ml_flags. This flag indicates that this
    function should be treated as Python function. Ideally, use of this
    flag should be avoided because it goes against the duck typing
    philosophy. It is still needed in a few places though, for example
    profiling.

The goal of base_function is that it supports all different ways of
calling functions and methods in just one structure. For example, the
new flag METH_PASS_FUNCTION will be used by the implementation of
methods.

It is not possible to directly create instances of base_function (tp_new
is NULL). However, it is legal for C code to manually create instances.

These are the relevant C structures:

    PyTypeObject PyBaseFunction_Type;

    typedef struct {
        PyObject_HEAD
        PyCFunctionDef *m_ml;     /* Description of the C function to call */
        PyObject *m_self;         /* __self__: anything, can be NULL; readonly */
        PyObject *m_module;       /* __module__: anything (typically str) */
        PyObject *m_parent;       /* __parent__: anything, can be NULL; readonly */
        PyObject *m_weakreflist;  /* List of weak references */
    } PyBaseFunctionObject;

    typedef struct {
        const char *ml_name;   /* The name of the built-in function/method */
        PyCFunction ml_meth;   /* The C function that implements it */
        int ml_flags;          /* Combination of METH_xxx flags, which mostly
                                  describe the args expected by the C func */
    } PyCFunctionDef;

Subclasses may extend PyCFunctionDef with extra fields.

The Python attribute __self__ returns m_self, except if METH_STATIC is
set. In that case or if m_self is NULL, then there is no __self__
attribute at all. For that reason, we write either m_self or __self__ in
this PEP with slightly different meanings.

cfunction

This is the new version of the old builtin_function_or_method class. The
name cfunction was chosen to avoid confusion with "built-in" in the
sense of "something in the builtins module". It also fits better with
the C API which use the PyCFunction prefix.

The class cfunction is a copy of base_function, with the following
differences:

1.  m_ml points to a PyMethodDef structure, extending PyCFunctionDef
    with an additional ml_doc field to implement __doc__ and
    __text_signature__ as read-only attributes:

        typedef struct {
            const char *ml_name;
            PyCFunction ml_meth;
            int ml_flags;
            const char *ml_doc;
        } PyMethodDef;

    Note that PyMethodDef is part of the Python Stable ABI <384> and it
    is used by practically all extension modules, so we absolutely
    cannot change this structure.

2.  Argument Clinic <436> is supported.

3.  __self__ always exists. In the cases where base_function.__self__
    would raise AttributeError, instead None is returned.

The type object is PyTypeObject PyCFunction_Type and we define
PyCFunctionObject as alias of PyBaseFunctionObject (except for the type
of m_ml).

defined_function

The class defined_function is an abstract base class meant to indicate
that the function has introspection support. Instances of
defined_function are required to support all attributes that Python
functions have, namely __code__, __globals__, __doc__, __defaults__,
__kwdefaults__, __closure__ and __annotations__. There is also a
__dict__ to support attributes added by the user.

None of these is required to be meaningful. In particular, __code__ may
not be a working code object, possibly only a few fields may be filled
in. This PEP does not dictate how the various attributes are
implemented. They may be simple struct members or more complicated
descriptors. Only read-only support is required, none of the attributes
is required to be writable.

The class defined_function is mainly meant for auto-generated C code,
for example produced by Cython[3]. There is no API to create instances
of it.

The C structure is the following:

    PyTypeObject PyDefinedFunction_Type;

    typedef struct {
        PyBaseFunctionObject base;
        PyObject *func_dict;        /* __dict__: dict or NULL */
    } PyDefinedFunctionObject;

TODO: maybe find a better name for defined_function. Other proposals:
inspect_function (anything that satisfies inspect.isfunction),
builtout_function (a function that is better built out; pun on builtin),
generic_function (original proposal but conflicts with
functools.singledispatch generic functions), user_function (defined by
the user as opposed to CPython).

function

This is the class meant for functions implemented in Python. Unlike the
other function types, instances of function can be created from Python
code. This is not changed, so we do not describe the details in this
PEP.

The layout of the C structure is the following:

    PyTypeObject PyFunction_Type;

    typedef struct {
        PyBaseFunctionObject base;
        PyObject *func_dict;        /* __dict__: dict or NULL */
        PyObject *func_code;        /* __code__: code */
        PyObject *func_globals;     /* __globals__: dict; readonly */
        PyObject *func_name;        /* __name__: string */
        PyObject *func_qualname;    /* __qualname__: string */
        PyObject *func_doc;         /* __doc__: can be anything or NULL */
        PyObject *func_defaults;    /* __defaults__: tuple or NULL */
        PyObject *func_kwdefaults;  /* __kwdefaults__: dict or NULL */
        PyObject *func_closure;     /* __closure__: tuple of cell objects or NULL; readonly */
        PyObject *func_annotations; /* __annotations__: dict or NULL */
        PyCFunctionDef _ml;         /* Storage for base.m_ml */
    } PyFunctionObject;

The descriptor __name__ returns func_name. When setting __name__, also
base.m_ml->ml_name is updated with the UTF-8 encoded name.

The _ml field reserves space to be used by base.m_ml.

A base_function instance must have the flag METH_PYTHON set if and only
if it is an instance of function.

When constructing an instance of function from code and globals, an
instance is created with base.m_ml = &_ml, base.m_self = NULL.

To make subclassing easier, we also add a copy constructor: if f is an
instance of function, then types.FunctionType(f) copies f. This
conveniently allows using a custom function type as decorator:

    >>> from types import FunctionType
    >>> class CustomFunction(FunctionType):
    ...     pass
    >>> @CustomFunction
    ... def f(x):
    ...     return x
    >>> type(f)
    <class '__main__.CustomFunction'>

This also removes many use cases of functools.wraps: wrappers can be
replaced by subclasses of function.

bound_method

The class bound_method is used for all bound methods, regardless of the
class of the underlying function. It adds one new attribute on top of
base_function: __func__ points to that function.

bound_method replaces the old method class which was used only for
Python functions bound as method.

There is a complication because we want to allow constructing a method
from an arbitrary callable. This may be an already-bound method or
simply not an instance of base_function. Therefore, in practice there
are two kinds of methods:

-   For arbitrary callables, we use a single fixed PyCFunctionDef
    structure with the METH_PASS_FUNCTION flag set.
-   For methods which bind instances of base_function (more precisely,
    which have the Py_TPFLAGS_BASEFUNCTION flag set) that have self
    slicing, we instead use the PyCFunctionDef from the original
    function. This way, we don't lose any performance when calling bound
    methods. In this case, the __func__ attribute is only used to
    implement various attributes but not for calling the method.

When constructing a new method from a base_function, we check that the
self object is an instance of __objclass__ (if a class was specified as
parent) and raise a TypeError otherwise.

The C structure is:

    PyTypeObject PyMethod_Type;

    typedef struct {
        PyBaseFunctionObject base;
        PyObject *im_func;  /* __func__: function implementing the method; readonly */
    } PyMethodObject;

Calling base_function instances

We specify the implementation of __call__ for instances of
base_function.

Checking __objclass__

First of all, a type check is done if the __parent__ of the function is
a class (recall that __objclass__ then becomes an alias of __parent__):
if m_self is NULL (this is the case for unbound methods of extension
types), then the function must be called with at least one positional
argument and the first (typically called self) must be an instance of
__objclass__. If not, a TypeError is raised.

Note that bound methods have m_self != NULL, so the __objclass__ is not
checked. Instead, the __objclass__ check is done when constructing the
method.

Flags

For convenience, we define a new constant: METH_CALLFLAGS combines all
flags from PyCFunctionDef.ml_flags which specify the signature of the C
function to be called. It is equal to :

    METH_VARARGS | METH_FASTCALL | METH_NOARGS | METH_O | METH_KEYWORDS | METH_PASS_FUNCTION

Exactly one of the first four flags above must be set and only
METH_VARARGS and METH_FASTCALL may be combined with METH_KEYWORDS.
Violating these rules is undefined behaviour.

There are one new flags which affects calling functions, namely
METH_PASS_FUNCTION and METH_CALL_UNBOUND. Some flags are already
documented in[4]. We explain the others below.

Self slicing

If the function has m_self == NULL and the flag METH_CALL_UNBOUND is not
set, then the first positional argument (if any) is removed from *args
and instead passed as first argument to the C function. Effectively, the
first positional argument is treated as __self__. This is meant to
support unbound methods such that the C function does not see the
difference between bound and unbound method calls. This does not affect
keyword arguments in any way.

This process is called self slicing and a function is said to have self
slicing if m_self == NULL and METH_CALL_UNBOUND is not set.

Note that a METH_NOARGS function which has self slicing effectively has
one argument, namely self. Analogously, a METH_O function with self
slicing has two arguments.

METH_PASS_FUNCTION

If this flag is set, then the C function is called with an additional
first argument, namely the function itself (the base_function instance).
As special case, if the function is a bound_method, then the underlying
function of the method is passed (but not recursively: if a bound_method
wraps a bound_method, then __func__ is only applied once).

For example, an ordinary METH_VARARGS function has signature
(PyObject *self, PyObject *args). With
METH_VARARGS | METH_PASS_FUNCTION, this becomes
(PyObject *func, PyObject *self, PyObject *args).

METH_FASTCALL

This is an existing but undocumented flag. We suggest to officially
support and document it.

If the flag METH_FASTCALL is set without METH_KEYWORDS, then the ml_meth
field is of type PyCFunctionFast which takes the arguments
(PyObject *self, PyObject *const *args, Py_ssize_t nargs). Such a
function takes only positional arguments and they are passed as plain C
array args of length nargs.

If the flags METH_FASTCALL | METH_KEYWORDS are set, then the ml_meth
field is of type PyCFunctionFastKeywords which takes the arguments
(PyObject *self, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames).
The positional arguments are passed as C array args of length nargs. The
values of the keyword arguments follow in that array, starting at
position nargs. The keys (names) of the keyword arguments are passed as
a tuple in kwnames. As an example, assume that 3 positional and 2
keyword arguments are given. Then args is an array of length 3 + 2 = 5,
nargs equals 3 and kwnames is a 2-tuple.

Automatic creation of built-in functions

Python automatically generates instances of cfunction for extension
types (using the PyTypeObject.tp_methods field) and modules (using the
PyModuleDef.m_methods field). The arrays PyTypeObject.tp_methods and
PyModuleDef.m_methods must be arrays of PyMethodDef structures.

Unbound methods of extension types

The type of unbound methods changes from method_descriptor to cfunction.
The object which appears as unbound method is the same object which
appears in the class __dict__. Python automatically sets the __parent__
attribute to the defining class.

Built-in functions of a module

For the case of functions of a module, __parent__ will be set to the
module. Unless the flag METH_BINDING is given, also __self__ will be set
to the module (for backwards compatibility).

An important consequence is that such functions by default do not become
methods when used as attribute (base_function.__get__ only does that if
m_self was NULL). One could consider this a bug, but this was done for
backwards compatibility reasons: in an initial post on python-ideas[5]
the consensus was to keep this misfeature of built-in functions.

However, to allow this anyway for specific or newly implemented built-in
functions, the METH_BINDING flag prevents setting __self__.

Further changes

New type flag

A new PyTypeObject flag (for tp_flags) is added: Py_TPFLAGS_BASEFUNCTION
to indicate that instances of this type are functions which can be
called and bound as method like a base_function.

This is different from flags like Py_TPFLAGS_LIST_SUBCLASS because it
indicates more than just a subclass: it also indicates a default
implementation of __call__ and __get__. In particular, such subclasses
of base_function must follow the implementation from the section Calling
base_function instances.

This flag is automatically set for extension types which inherit the
tp_call and tp_descr_get implementation from base_function. Extension
types can explicitly specify it if they override __call__ or __get__ in
a compatible way. The flag Py_TPFLAGS_BASEFUNCTION must never be set for
a heap type because that would not be safe (heap types can be changed
dynamically).

C API functions

We list some relevant Python/C API macros and functions. Some of these
are existing (possibly changed) functions, some are new:

-   int PyBaseFunction_CheckFast(PyObject *op): return true if op is an
    instance of a class with the Py_TPFLAGS_BASEFUNCTION set. This is
    the function that you need to use to determine whether it is
    meaningful to access the base_function internals.
-   int PyBaseFunction_Check(PyObject *op): return true if op is an
    instance of base_function.
-   PyObject *PyBaseFunction_New(PyTypeObject *cls, PyCFunctionDef *ml, PyObject *self, PyObject *module, PyObject *parent):
    create a new instance of cls (which must be a subclass of
    base_function) from the given data.
-   int PyCFunction_Check(PyObject *op): return true if op is an
    instance of cfunction.
-   int PyCFunction_NewEx(PyMethodDef* ml, PyObject *self, PyObject* module):
    create a new instance of cfunction. As special case, if self is
    NULL, then set self = Py_None instead (for backwards compatibility).
    If self is a module, then __parent__ is set to self. Otherwise,
    __parent__ is NULL.
-   For many existing PyCFunction_... and PyMethod_ functions, we define
    a new function PyBaseFunction_... acting on base_function instances.
    The old functions are kept as aliases of the new functions.
-   int PyFunction_Check(PyObject *op): return true if op is an instance
    of base_function with the METH_PYTHON flag set (this is equivalent
    to checking whether op is an instance of function).
-   int PyFunction_CheckFast(PyObject *op): equivalent to
    PyFunction_Check(op) && PyBaseFunction_CheckFast(op).
-   int PyFunction_CheckExact(PyObject *op): return true if the type of
    op is function.
-   PyObject *PyFunction_NewPython(PyTypeObject *cls, PyObject *code, PyObject *globals, PyObject *name, PyObject *qualname):
    create a new instance of cls (which must be a subclass of function)
    from the given data.
-   PyObject *PyFunction_New(PyObject *code, PyObject *globals): create
    a new instance of function.
-   PyObject *PyFunction_NewWithQualName(PyObject *code, PyObject *globals, PyObject *qualname):
    create a new instance of function.
-   PyObject *PyFunction_Copy(PyTypeObject *cls, PyObject *func): create
    a new instance of cls (which must be a subclass of function) by
    copying a given function.

Changes to the types module

Two types are added: types.BaseFunctionType corresponding to
base_function and types.DefinedFunctionType corresponding to
defined_function.

Apart from that, no changes to the types module are made. In particular,
types.FunctionType refers to function. However, the actual types will
change: in particular, types.BuiltinFunctionType will no longer be the
same as types.BuiltinMethodType.

Changes to the inspect module

The new function inspect.isbasefunction checks for an instance of
base_function.

inspect.isfunction checks for an instance of defined_function.

inspect.isbuiltin checks for an instance of cfunction.

inspect.isroutine checks isbasefunction or ismethoddescriptor.

NOTE: bpo-33261[6] should be fixed first.

Profiling

Currently, sys.setprofile supports c_call, c_return and c_exception
events for built-in functions. These events are generated when calling
or returning from a built-in function. By contrast, the call and return
events are generated by the function itself. So nothing needs to change
for the call and return events.

Since we no longer make a difference between C functions and Python
functions, we need to prevent the c_* events for Python functions. This
is done by not generating those events if the METH_PYTHON flag in
ml_flags is set.

Non-CPython implementations

Most of this PEP is only relevant to CPython. For other implementations
of Python, the two changes that are required are the base_function base
class and the fact that function can be subclassed. The classes
cfunction and defined_function are not required.

We require base_function for consistency but we put no requirements on
it: it is acceptable if this is just a copy of object. Support for the
new __parent__ (and __objclass__) attribute is not required. If there is
no defined_function class, then types.DefinedFunctionType should be an
alias of types.FunctionType.

Rationale

Why not simply change existing classes?

One could try to solve the problem by keeping the existing classes
without introducing a new base_function class.

That might look like a simpler solution but it is not: it would require
introspection support for 3 distinct classes: function,
builtin_function_or_method and method_descriptor. For the latter two
classes, "introspection support" would mean at a minimum allowing
subclassing. But we don't want to lose performance, so we want fast
subclass checks. This would require two new flags in tp_flags. And we
want subclasses to allow __get__ for built-in functions, so we should
implement the LOAD_METHOD opcode for built-in functions too. More
generally, a lot of functionality would need to be duplicated and the
end result would be far more complex code.

It is also not clear how the introspection of built-in function
subclasses would interact with __text_signature__. Having two
independent kinds of inspect.signature support on the same class sounds
like asking for problems.

And this would not fix some of the other differences between built-in
functions and Python functions that were mentioned in the motivation.

Why __text_signature__ is not a solution

Built-in functions have an attribute __text_signature__, which gives the
signature of the function as plain text. The default values are
evaluated by ast.literal_eval. Because of this, it supports only a small
number of standard Python classes and not arbitrary Python objects.

And even if __text_signature__ would allow arbitrary signatures somehow,
that is only one piece of introspection: it does not help with
inspect.getsourcefile for example.

defined_function versus function

In many places, a decision needs to be made whether the old function
class should be replaced by defined_function or the new function class.
This is done by thinking of the most likely use case:

1.  types.FunctionType refers to function because that type might be
    used to construct instances using types.FunctionType(...).
2.  inspect.isfunction() refers to defined_function because this is the
    class where introspection is supported.
3.  The C API functions must refer to function because we do not specify
    how the various attributes of defined_function are implemented. We
    expect that this is not a problem since there is typically no reason
    for introspection to be done by C extensions.

Scope of this PEP: which classes are involved?

The main motivation of this PEP is fixing function classes, so we
certainly want to unify the existing classes builtin_function_or_method
and function.

Since built-in functions and methods have the same class, it seems
natural to include bound methods too. And since there are no "unbound
methods" for Python functions, it makes sense to get rid of unbound
methods for extension types.

For now, no changes are made to the classes staticmethod, classmethod
and classmethod_descriptor. It would certainly make sense to put these
in the base_function class hierarchy and unify classmethod and
classmethod_descriptor. However, this PEP is already big enough and this
is left as a possible future improvement.

Slot wrappers for extension types like __init__ or __eq__ are quite
different from normal methods. They are also typically not called
directly because you would normally write foo[i] instead of
foo.__getitem__(i). So these are left outside the scope of this PEP.

Python also has an instancemethod class, which seems to be a relic from
Python 2, where it was used for bound and unbound methods. It is not
clear whether there is still a use case for it. In any case, there is no
reason to deal with it in this PEP.

TODO: should instancemethod be deprecated? It doesn't seem used at all
within CPython 3.7, but maybe external packages use it?

Not treating METH_STATIC and METH_CLASS

Almost nothing in this PEP refers to the flags METH_STATIC and
METH_CLASS. These flags are checked only by the automatic creation of
built-in functions. When a staticmethod, classmethod or
classmethod_descriptor is bound (i.e. __get__ is called), a
base_function instance is created with m_self != NULL. For a
classmethod, this is obvious since m_self is the class that the method
is bound to. For a staticmethod, one can take an arbitrary Python object
for m_self. For backwards compatibility, we choose m_self = __parent__
for static methods of extension types.

------------------------------------------------------------------------

It may look strange at first sight to add the __self__ slot in
base_function as opposed to bound_method. We took this idea from the
existing builtin_function_or_method class. It allows us to have a single
general implementation of __call__ and __get__ for the various function
classes discussed in this PEP.

It also makes it easy to support existing built-in functions which set
__self__ to the module (for example, sys.exit.__self__ is sys).

Two implementations of __doc__

base_function does not support function docstrings. Instead, the classes
cfunction and function each have their own way of dealing with
docstrings (and bound_method just takes the __doc__ from the wrapped
function).

For cfunction, the docstring is stored (together with the text
signature) as C string in the read-only ml_doc field of a PyMethodDef.
For function, the docstring is stored as a writable Python object and it
does not actually need to be a string. It looks hard to unify these two
very different ways of dealing with __doc__. For backwards
compatibility, we keep the existing implementations.

For defined_function, we require __doc__ to be implemented but we do not
specify how. A subclass can implement __doc__ the same way as cfunction
or using a struct member or some other way.

Subclassing

We disallow subclassing of cfunction and bound_method to enable fast
type checks for PyCFunction_Check and PyMethod_Check.

We allow subclassing of the other classes because there is no reason to
disallow it. For Python modules, the only relevant class to subclass is
function because the others cannot be instantiated anyway.

Replacing tp_call: METH_PASS_FUNCTION and METH_CALL_UNBOUND

The new flags METH_PASS_FUNCTION and METH_CALL_UNBOUND are meant to
support cases where formerly a custom tp_call was used. It reduces the
number of special fast paths in Python/ceval.c for calling objects:
instead of treating Python functions, built-in functions and method
descriptors separately, there would only be a single check.

The signature of tp_call is essentially the signature of
PyBaseFunctionObject.m_ml.ml_meth with flags
METH_VARARGS | METH_KEYWORDS | METH_PASS_FUNCTION | METH_CALL_UNBOUND
(the only difference is an added self argument). Therefore, it should be
easy to change existing tp_call slots to use the base_function
implementation instead.

It also makes sense to use METH_PASS_FUNCTION without METH_CALL_UNBOUND
in cases where the C function simply needs access to additional metadata
from the function, such as the __parent__. This is for example needed to
support PEP 573. Converting existing methods to use METH_PASS_FUNCTION
is trivial: it only requires adding an extra argument to the C function.

Backwards compatibility

While designing this PEP, great care was taken to not break backwards
compatibility too much. Most of the potentially incompatible changes are
changes to CPython implementation details which are different anyway in
other Python interpreters. In particular, Python code which correctly
runs on PyPy will very likely continue to work with this PEP.

The standard classes and functions like staticmethod, functools.partial
or operator.methodcaller do not need to change at all.

Changes to types and inspect

The proposed changes to types and inspect are meant to minimize changes
in behaviour. However, it is unavoidable that some things change and
this can cause code which uses types or inspect to break. In the Python
standard library for example, changes are needed in the doctest module
because of this.

Also, tools which take various kinds of functions as input will need to
deal with the new function hierarchy and the possibility of custom
function classes.

Python functions

For Python functions, essentially nothing changes. The attributes that
existed before still exist and Python functions can be initialized,
called and turned into methods as before.

The name function is kept for backwards compatibility. While it might
make sense to change the name to something more specific like
python_function, that would require a lot of annoying changes in
documentation and testsuites.

Built-in functions of a module

Also for built-in functions, nothing changes. We keep the old behaviour
that such functions do not bind as methods. This is a consequence of the
fact that __self__ is set to the module.

Built-in bound and unbound methods

The types of built-in bound and unbound methods will change. However,
this does not affect calling such methods because the protocol in
base_function.__call__ (in particular the handling of __objclass__ and
self slicing) was specifically designed to be backwards compatible. All
attributes which existed before (like __objclass__ and __self__) still
exist.

New attributes

Some objects get new special double-underscore attributes. For example,
the new attribute __parent__ appears on all built-in functions and all
methods get a __func__ attribute. The fact that __self__ is now a
special read-only attribute for Python functions caused trouble in[7].
Generally, we expect that not much will break though.

method_descriptor and PyDescr_NewMethod

The class method_descriptor and the constructor PyDescr_NewMethod should
be deprecated. They are no longer used by CPython itself but are still
supported.

Two-phase Implementation

TODO: this section is optional. If this PEP is accepted, it should be
decided whether to apply this two-phase implementation or not.

As mentioned above, the changes to types and inspect can break some
existing code. In order to further minimize breakage, this PEP could be
implemented in two phases.

Phase one: keep existing classes but add base classes

Initially, implement the base_function class and use it as common base
class but otherwise keep the existing classes (but not their
implementation).

In this proposal, the class hierarchy would become:

    object
       |
       |
    base_function
    /       |     \
    /        |      \
    /         |       \
    cfunction          |     defined_function
    |     |           |         \
    |     |      bound_method    \
    |     |                       \
    |  method_descriptor       function
    |
    builtin_function_or_method

The leaf classes builtin_function_or_method, method_descriptor,
bound_method and function correspond to the existing classes (with
method renamed to bound_method).

Automatically created functions created in modules become instances of
builtin_function_or_method. Unbound methods of extension types become
instances of method_descriptor.

The class method_descriptor is a copy of cfunction except that __get__
returns a builtin_function_or_method instead of a bound_method.

The class builtin_function_or_method has the same C structure as a
bound_method, but it inherits from cfunction. The __func__ attribute is
not mandatory: it is only defined when binding a method_descriptor.

We keep the implementation of the inspect functions as they are. Because
of this and because the existing classes are kept, backwards
compatibility is ensured for code doing type checks.

Since showing an actual DeprecationWarning would affect a lot of
correctly-functioning code, any deprecations would only appear in the
documentation. Another reason is that it is hard to show warnings for
calling isinstance(x, t) (but it could be done using __instancecheck__
hacking) and impossible for type(x) is t.

Phase two

Phase two is what is actually described in the rest of this PEP. In
terms of implementation, it would be a relatively small change compared
to phase one.

Reference Implementation

Most of this PEP has been implemented for CPython at
https://github.com/jdemeyer/cpython/tree/pep575

There are four steps, corresponding to the commits on that branch. After
each step, CPython is in a mostly working state.

1.  Add the base_function class and make it a subclass for cfunction.
    This is by far the biggest step as the complete __call__ protocol is
    implemented in this step.
2.  Rename method to bound_method and make it a subclass of
    base_function. Change unbound methods of extension types to be
    instances of cfunction such that bound methods of extension types
    are also instances of bound_method.
3.  Implement defined_function and function.
4.  Changes to other parts of Python, such as the standard library and
    testsuite.

Appendix: current situation

NOTE: This section is more useful during the draft period of the PEP, so
feel free to remove this once the PEP has been accepted.

For reference, we describe in detail the relevant existing classes in
CPython 3.7.

Each of the classes involved is an "orphan" class (no non-trivial
subclasses nor superclasses).

builtin_function_or_method: built-in functions and bound methods

These are of type PyCFunction_Type with structure PyCFunctionObject:

    typedef struct {
        PyObject_HEAD
        PyMethodDef *m_ml; /* Description of the C function to call */
        PyObject    *m_self; /* Passed as 'self' arg to the C func, can be NULL */
        PyObject    *m_module; /* The __module__ attribute, can be anything */
        PyObject    *m_weakreflist; /* List of weak references */
    } PyCFunctionObject;

    struct PyMethodDef {
        const char  *ml_name;   /* The name of the built-in function/method */
        PyCFunction ml_meth;    /* The C function that implements it */
        int         ml_flags;   /* Combination of METH_xxx flags, which mostly
                                   describe the args expected by the C func */
        const char  *ml_doc;    /* The __doc__ attribute, or NULL */
    };

where PyCFunction is a C function pointer (there are various forms of
this, the most basic takes two arguments for self and *args).

This class is used both for functions and bound methods: for a method,
the m_self slot points to the object:

    >>> dict(foo=42).get
    <built-in method get of dict object at 0x...>
    >>> dict(foo=42).get.__self__
    {'foo': 42}

In some cases, a function is considered a "method" of the module
defining it:

    >>> import os
    >>> os.kill
    <built-in function kill>
    >>> os.kill.__self__
    <module 'posix' (built-in)>

method_descriptor: built-in unbound methods

These are of type PyMethodDescr_Type with structure PyMethodDescrObject:

    typedef struct {
        PyDescrObject d_common;
        PyMethodDef *d_method;
    } PyMethodDescrObject;

    typedef struct {
        PyObject_HEAD
        PyTypeObject *d_type;
        PyObject *d_name;
        PyObject *d_qualname;
    } PyDescrObject;

function: Python functions

These are of type PyFunction_Type with structure PyFunctionObject:

    typedef struct {
        PyObject_HEAD
        PyObject *func_code;        /* A code object, the __code__ attribute */
        PyObject *func_globals;     /* A dictionary (other mappings won't do) */
        PyObject *func_defaults;    /* NULL or a tuple */
        PyObject *func_kwdefaults;  /* NULL or a dict */
        PyObject *func_closure;     /* NULL or a tuple of cell objects */
        PyObject *func_doc;         /* The __doc__ attribute, can be anything */
        PyObject *func_name;        /* The __name__ attribute, a string object */
        PyObject *func_dict;        /* The __dict__ attribute, a dict or NULL */
        PyObject *func_weakreflist; /* List of weak references */
        PyObject *func_module;      /* The __module__ attribute, can be anything */
        PyObject *func_annotations; /* Annotations, a dict or NULL */
        PyObject *func_qualname;    /* The qualified name */

        /* Invariant:
         *     func_closure contains the bindings for func_code->co_freevars, so
         *     PyTuple_Size(func_closure) == PyCode_GetNumFree(func_code)
         *     (func_closure may be NULL if PyCode_GetNumFree(func_code) == 0).
         */
    } PyFunctionObject;

In Python 3, there is no "unbound method" class: an unbound method is
just a plain function.

method: Python bound methods

These are of type PyMethod_Type with structure PyMethodObject:

    typedef struct {
        PyObject_HEAD
        PyObject *im_func;   /* The callable object implementing the method */
        PyObject *im_self;   /* The instance it is bound to */
        PyObject *im_weakreflist; /* List of weak references */
    } PyMethodObject;

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] Cython (http://cython.org/)

[2] Python bug 30071, Duck-typing inspect.isfunction()
(https://bugs.python.org/issue30071)

[3] Cython (http://cython.org/)

[4] PyMethodDef documentation
(https://docs.python.org/3.7/c-api/structures.html#c.PyMethodDef)

[5] PEP proposal: unifying function/method classes
(https://mail.python.org/pipermail/python-ideas/2018-March/049398.html)

[6] Python bug 33261, inspect.isgeneratorfunction fails on hand-created
methods (https://bugs.python.org/issue33261 and
https://github.com/python/cpython/pull/6448)

[7] Python bug 33265, contextlib.ExitStack abuses __self__
(https://bugs.python.org/issue33265 and
https://github.com/python/cpython/pull/6456)

license: Public Domain


text:
PEP: 657 Title: Include Fine Grained Error Locations in Tracebacks
Version: $Revision$ Last-Modified: $Date$ Author: Pablo Galindo
<EMAIL_ADDRESS>Batuhan Taskaya<EMAIL_ADDRESS>Ammar
Askar<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pep-657-include-fine-grained-error-locations-in-tracebacks/8629
Status: Final Type: Standards Track Content-Type: text/x-rst Created:
08-May-2021 Python-Version: 3.11 Post-History:

Abstract

This PEP proposes adding a mapping from each bytecode instruction to the
start and end column offsets of the line that generated them as well as
the end line number. This data will be used to improve tracebacks
displayed by the CPython interpreter in order to improve the debugging
experience. The PEP also proposes adding APIs that allow other tools
(such as coverage analysis tools, profilers, tracers, debuggers) to
consume this information from code objects.

Motivation

The primary motivation for this PEP is to improve the feedback presented
about the location of errors to aid with debugging.

Python currently keeps a mapping of bytecode to line numbers from
compilation. The interpreter uses this mapping to point to the source
line associated with an error. While this line-level granularity for
instructions is useful, a single line of Python code can compile into
dozens of bytecode operations making it hard to track which part of the
line caused the error.

Consider the following line of Python code:

    x['a']['b']['c']['d'] = 1

If any of the values in the dictionaries are None, the error shown is:

    Traceback (most recent call last):
      File "test.py", line 2, in <module>
        x['a']['b']['c']['d'] = 1
    TypeError: 'NoneType' object is not subscriptable

From the traceback, it is impossible to determine which one of the
dictionaries had the None element that caused the error. Users often
have to attach a debugger or split up their expression to track down the
problem.

However, if the interpreter had a mapping of bytecode to column offsets
as well as line numbers, it could helpfully display:

    Traceback (most recent call last):
      File "test.py", line 2, in <module>
        x['a']['b']['c']['d'] = 1
        ~~~~~~~~~~~^^^^^
    TypeError: 'NoneType' object is not subscriptable

indicating to the user that the object x['a']['b'] must have been None.
This highlighting will occur for every frame in the traceback. For
instance, if a similar error is part of a complex function call chain,
the traceback would display the code associated to the current
instruction in every frame:

    Traceback (most recent call last):
      File "test.py", line 14, in <module>
        lel3(x)
        ^^^^^^^
      File "test.py", line 12, in lel3
        return lel2(x) / 23
               ^^^^^^^
      File "test.py", line 9, in lel2
        return 25 + lel(x) + lel(x)
                    ^^^^^^
      File "test.py", line 6, in lel
        return 1 + foo(a,b,c=x['z']['x']['y']['z']['y'], d=e)
                             ~~~~~~~~~~~~~~~~^^^^^
    TypeError: 'NoneType' object is not subscriptable

This problem presents itself in the following situations.

-   When passing down multiple objects to function calls while accessing
    the same attribute in them. For instance, this error:

        Traceback (most recent call last):
          File "test.py", line 19, in <module>
            foo(a.name, b.name, c.name)
        AttributeError: 'NoneType' object has no attribute 'name'

    With the improvements in this PEP this would show:

        Traceback (most recent call last):
          File "test.py", line 17, in <module>
            foo(a.name, b.name, c.name)
                        ^^^^^^
        AttributeError: 'NoneType' object has no attribute 'name'

-   When dealing with lines with complex mathematical expressions,
    especially with libraries such as numpy where arithmetic operations
    can fail based on the arguments. For example: :

        Traceback (most recent call last):
          File "test.py", line 1, in <module>
            x = (a + b) @ (c + d)
        ValueError: operands could not be broadcast together with shapes (1,2) (2,3)

    There is no clear indication as to which operation failed, was it
    the addition on the left, the right or the matrix multiplication in
    the middle? With this PEP the new error message would look like:

        Traceback (most recent call last):
          File "test.py", line 1, in <module>
            x = (a + b) @ (c + d)
                           ~~^~~
        ValueError: operands could not be broadcast together with shapes (1,2) (2,3)

    Giving a much clearer and easier to debug error message.

Debugging aside, this extra information would also be useful for code
coverage tools, enabling them to measure expression-level coverage
instead of just line-level coverage. For instance, given the following
line: :

    x = foo() if bar() else baz()

coverage, profile or state analysis tools will highlight the full line
in both branches, making it impossible to differentiate what branch was
taken. This is a known problem in pycoverage.

Similar efforts to this PEP have taken place in other languages such as
Java in the form of JEP358. NullPointerExceptions in Java were similarly
nebulous when it came to lines with complicated expressions. A
NullPointerException would provide very little aid in finding the root
cause of an error. The implementation for JEP358 is fairly complex,
requiring walking back through the bytecode by using a control flow
graph analyzer and decompilation techniques to recover the source code
that led to the null pointer. Although the complexity of this solution
is high and requires maintenance for the decompiler every time Java
bytecode is changed, this improvement was deemed to be worth it for the
extra information provided for just one exception type.

Rationale

In order to identify the range of source code being executed when
exceptions are raised, this proposal requires adding new data for every
bytecode instruction. This will have an impact on the size of pyc files
on disk and the size of code objects in memory. The authors of this
proposal have chosen the data types in a way that tries to minimize this
impact. The proposed overhead is storing two uint8_t (one for the start
offset and one for the end offset) and the end line information for
every bytecode instruction (in the same encoded fashion as the start
line is stored currently).

As an illustrative example to gauge the impact of this change, we have
calculated that including the start and end offsets will increase the
size of the standard library’s pyc files by 22% (6MB) from 28.4MB to
34.7MB. The overhead in memory usage will be the same (assuming the full
standard library is loaded into the same program). We believe that this
is a very acceptable number since the order of magnitude of the overhead
is very small, especially considering the storage size and memory
capabilities of modern computers. Additionally, in general the memory
size of a Python program is not dominated by code objects. To check this
assumption we have executed the test suite of several popular PyPI
projects (including NumPy, pytest, Django and Cython) as well as several
applications (Black, pylint, mypy executed over either mypy or the
standard library) and we found that code objects represent normally 3-6%
of the average memory size of the program.

We understand that the extra cost of this information may not be
acceptable for some users, so we propose an opt-out mechanism which will
cause generated code objects to not have the extra information while
also allowing pyc files to not include the extra information.

Specification

In order to have enough information to correctly resolve the location
within a given line where an error was raised, a map linking bytecode
instructions to column offsets (start and end offset) and end line
numbers is needed. This is similar in fashion to how line numbers are
currently linked to bytecode instructions.

The following changes will be performed as part of the implementation of
this PEP:

-   The offset information will be exposed to Python via a new attribute
    in the code object class called co_positions that will return a
    sequence of four-element tuples containing the full location of
    every instruction (including start line, end line, start column
    offset and end column offset) or None if the code object was created
    without the offset information.

-   One new C-API function: :

        int PyCode_Addr2Location(
            PyCodeObject *co, int addrq,
            int *start_line, int *start_column,
            int *end_line, int *end_column)

    will be added so the end line, the start column offsets and the end
    column offset can be obtained given the index of a bytecode
    instruction. This function will set the values to 0 if the
    information is not available.

The internal storage, compression and encoding of the information is
left as an implementation detail and can be changed at any point as long
as the public API remains unchanged.

Offset semantics

These offsets are propagated by the compiler from the ones stored
currently in all AST nodes. The output of the public APIs (co_positions
and PyCode_Addr2Location) that deal with these attributes use 0-indexed
offsets (just like the AST nodes), but the underlying implementation is
free to represent the actual data in whatever form they choose to be
most efficient. The error code regarding information not available is
None for the co_positions() API, and -1 for the PyCode_Addr2Location
API. The availability of the information highly depends on whether the
offsets fall under the range, as well as the runtime flags for the
interpreter configuration.

The AST nodes use int types to store these values. The current
implementation, however, utilizes uint8_t types as an implementation
detail to minimize storage impact. This decision allows offsets to go
from 0 to 255, while offsets bigger than these values will be treated as
missing (returning -1 on the PyCode_Addr2Location and None API in the
co_positions() API).

As specified previously, the underlying storage of the offsets should be
considered an implementation detail, as the public APIs to obtain this
values will return either C int types or Python int objects, which
allows to implement better compression/encoding in the future if bigger
ranges would need to be supported. This PEP proposes to start with this
simpler version and defer improvements to future work.

Displaying tracebacks

When displaying tracebacks, the default exception hook will be modified
to query this information from the code objects and use it to display a
sequence of carets for every displayed line in the traceback if the
information is available. For instance:

    File "test.py", line 6, in lel
      return 1 + foo(a,b,c=x['z']['x']['y']['z']['y'], d=e)
                           ~~~~~~~~~~~~~~~~^^^^^
    TypeError: 'NoneType' object is not subscriptable

When displaying tracebacks, instruction offsets will be taken from the
traceback objects. This makes highlighting exceptions that are re-raised
work naturally without the need to store the new information in the
stack. For example, for this code:

    def foo(x):
        1 + 1/0 + 2

    def bar(x):
        try:
            1 + foo(x) + foo(x)
        except Exception as e:
            raise ValueError("oh no!") from e

    bar(bar(bar(2)))

The printed traceback would look like this:

    Traceback (most recent call last):
      File "test.py", line 6, in bar
        1 + foo(x) + foo(x)
            ^^^^^^
      File "test.py", line 2, in foo
        1 + 1/0 + 2
            ~^~
    ZeroDivisionError: division by zero

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last):
      File "test.py", line 10, in <module>
        bar(bar(bar(2)))
                ^^^^^^
      File "test.py", line 8, in bar
        raise ValueError("oh no!") from e
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ValueError: oh no

While this code:

    def foo(x):
        1 + 1/0 + 2
    def bar(x):
        try:
            1 + foo(x) + foo(x)
        except Exception:
            raise
    bar(bar(bar(2)))

Will be displayed as:

    Traceback (most recent call last):
      File "test.py", line 10, in <module>
        bar(bar(bar(2)))
                ^^^^^^
      File "test.py", line 6, in bar
        1 + foo(x) + foo(x)
            ^^^^^^
      File "test.py", line 2, in foo
        1 + 1/0 + 2
            ~^~
    ZeroDivisionError: division by zero

Maintaining the current behavior, only a single line will be displayed
in tracebacks. For instructions that span multiple lines (the end offset
and the start offset belong to different lines), the end line number
must be inspected to know if the end offset applies to the same line as
the starting offset.

Opt-out mechanism

To offer an opt-out mechanism for those users that care about the
storage and memory overhead and to allow third party tools and other
programs that are currently parsing tracebacks to catch up the following
methods will be provided to deactivate this feature:

-   A new environment variable: PYTHONNODEBUGRANGES.
-   A new command line option for the dev mode:
    python -Xno_debug_ranges.

If any of these methods are used, the Python compiler will not populate
code objects with the new information (None will be used instead) and
any unmarshalled code objects that contain the extra information will
have it stripped away and replaced with None). Additionally, the
traceback machinery will not show the extended location information even
if the information was present. This method allows users to:

-   Create smaller pyc files by using one of the two methods when said
    files are created.
-   Don't load the extra information from pyc files if those were
    created with the extra information in the first place.
-   Deactivate the extra information when displaying tracebacks (the
    caret characters indicating the location of the error).

Doing this has a very small performance hit as the interpreter state
needs to be fetched when code objects are created to look up the
configuration. Creating code objects is not a performance sensitive
operation so this should not be a concern.

Backwards Compatibility

The change is fully backwards compatible.

Reference Implementation

A reference implementation can be found in the implementation fork.

Rejected Ideas

Use a single caret instead of a range

It has been proposed to use a single caret instead of highlighting the
full range when reporting errors as a way to simplify the feature. We
have decided to not go this route for the following reasons:

-   Deriving the location of the caret is not straightforward using the
    current layout of the AST. This is because the AST nodes only record
    the start and end line numbers as well as the start and end column
    offsets. As the AST nodes do not preserve the original tokens (by
    design) deriving the exact location of some tokens is not possible
    without extra re-parsing. For instance, currently binary operators
    have nodes for the operands but the type of the operator is stored
    in an enumeration so its location cannot be derived from the node
    (this is just an example of how this problem manifest, and not the
    only one).

-   Deriving the ranges from AST nodes greatly simplifies the
    implementation and reduces a lot the maintenance cost and the
    possibilities of errors. This is because using the ranges is always
    possible to do generically for any AST node, while any other custom
    information would need to be extracted differently from different
    types of nodes. Given how error-prone getting the locations manually
    was when this used to be a manual process when generating the AST,
    we believe that a generic solution is a very important property to
    pursue.

-   Storing the information to highlight a single caret will be very
    limiting for tools such as coverage tools and profilers as well as
    for tools like IPython and IDEs that want to make use of this new
    feature. As this message from the author of "friendly-traceback"
    mentions, the reason is that without the full range (including end
    lines) these tools will find very difficult to highlight correctly
    the relevant source code. For instance, for this code:

        something = foo(a,b,c) if bar(a,b,c) else other(b,c,d)

    tools (such as coverage reporters) want to be able to highlight the
    totality of the call that is covered by the executed bytecode (let's
    say foo(a,b,c)) and not just a single character. Even if is
    technically possible to re-parse and re-tokenize the source code to
    re-construct the information, it is not possible to do this reliably
    and would result in a much worse user experience.

-   Many users have reported that a single caret is much harder to read
    than a full range, and this motivated using ranges to highlight
    syntax errors, which was very well received. Additionally, it has
    been noted that users with vision problems can identify the ranges
    much easily than a single caret character, which we believe is a
    great advantage of using ranges.

Have a configure flag to opt out

Having a configure flag to opt out of the overhead even when executing
Python in non-optimized mode may sound desirable, but it may cause
problems when reading pyc files that were created with a version of the
interpreter that was not compiled with the flag activated. This can lead
to crashes that would be very difficult to debug for regular users and
will make different pyc files incompatible between each other. As this
pyc could be shipped as part of libraries or applications without the
original source, it is also not always possible to force recompilation
of said pyc files. For these reasons we have decided to use the -O flag
to opt-out of this behaviour.

Lazy loading of column information

One potential solution to reduce the memory usage of this feature is to
not load the column information from the pyc file when code is imported.
Only if an uncaught exception bubbles up or if a call to the C-API
functions is made will the column information be loaded from the pyc
file. This is similar to how we only read source lines to display them
in the traceback when an exception bubbles up. While this would indeed
lower memory usage, it also results in a far more complex implementation
requiring changes to the importing machinery to selectively ignore a
part of the code object. We consider this an interesting avenue to
explore but ultimately we think is out of the scope for this particular
PEP. It also means that column information will not be available if the
user is not using pyc files or for code objects created dynamically at
runtime.

Implement compression

Although it would be possible to implement some form of compression over
the pyc files and the new data in code objects, we believe that this is
out of the scope of this proposal due to its larger impact (in the case
of pyc files) and the fact that we expect column offsets to not compress
well due to the lack of patterns in them (in case of the new data in
code objects).

Acknowledgments

Thanks to Carl Friedrich Bolz-Tereick for showing an initial prototype
of this idea for the Pypy interpreter and for the helpful discussion.

References

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.