text:
PEP: 435 Title: Adding an Enum type to the Python standard library
Version: $Revision$ Last-Modified: $Date$ Author: Barry Warsaw
<EMAIL_ADDRESS>Eli Bendersky<EMAIL_ADDRESS>Ethan Furman
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 23-Feb-2013 Python-Version: 3.4 Post-History:
23-Feb-2013, 02-May-2013 Replaces: 354 Resolution:
https://mail.python.org/pipermail/python-dev/2013-May/126112.html

Abstract

This PEP proposes adding an enumeration type to the Python standard
library.

An enumeration is a set of symbolic names bound to unique, constant
values. Within an enumeration, the values can be compared by identity,
and the enumeration itself can be iterated over.

Status of discussions

The idea of adding an enum type to Python is not new - PEP 354 is a
previous attempt that was rejected in 2005. Recently a new set of
discussions was initiated[1] on the python-ideas mailing list. Many new
ideas were proposed in several threads; after a lengthy discussion Guido
proposed adding flufl.enum to the standard library[2]. During the PyCon
2013 language summit the issue was discussed further. It became clear
that many developers want to see an enum that subclasses int, which can
allow us to replace many integer constants in the standard library by
enums with friendly string representations, without ceding backwards
compatibility. An additional discussion among several interested core
developers led to the proposal of having IntEnum as a special case of
Enum.

The key dividing issue between Enum and IntEnum is whether comparing to
integers is semantically meaningful. For most uses of enumerations, it's
a feature to reject comparison to integers; enums that compare to
integers lead, through transitivity, to comparisons between enums of
unrelated types, which isn't desirable in most cases. For some uses,
however, greater interoperability with integers is desired. For
instance, this is the case for replacing existing standard library
constants (such as socket.AF_INET) with enumerations.

Further discussion in late April 2013 led to the conclusion that
enumeration members should belong to the type of their enum:
type(Color.red) == Color. Guido has pronounced a decision on this
issue[3], as well as related issues of not allowing to subclass
enums[4], unless they define no enumeration members[5].

The PEP was accepted by Guido on May 10th, 2013[6].

Motivation

[Based partly on the Motivation stated in PEP 354*]*

The properties of an enumeration are useful for defining an immutable,
related set of constant values that may or may not have a semantic
meaning. Classic examples are days of the week (Sunday through Saturday)
and school assessment grades ('A' through 'D', and 'F'). Other examples
include error status values and states within a defined process.

It is possible to simply define a sequence of values of some other basic
type, such as int or str, to represent discrete arbitrary values.
However, an enumeration ensures that such values are distinct from any
others including, importantly, values within other enumerations, and
that operations without meaning ("Wednesday times two") are not defined
for these values. It also provides a convenient printable representation
of enum values without requiring tedious repetition while defining them
(i.e. no GREEN = 'green').

Module and type name

We propose to add a module named enum to the standard library. The main
type exposed by this module is Enum. Hence, to import the Enum type user
code will run:

    >>> from enum import Enum

Proposed semantics for the new enumeration type

Creating an Enum

Enumerations are created using the class syntax, which makes them easy
to read and write. An alternative creation method is described in
Functional API. To define an enumeration, subclass Enum as follows:

    >>> from enum import Enum
    >>> class Color(Enum):
    ...     red = 1
    ...     green = 2
    ...     blue = 3

A note on nomenclature: we call Color an enumeration (or enum) and
Color.red, Color.green are enumeration members (or enum members).
Enumeration members also have values (the value of Color.red is 1, etc.)

Enumeration members have human readable string representations:

    >>> print(Color.red)
    Color.red

...while their repr has more information:

    >>> print(repr(Color.red))
    <Color.red: 1>

The type of an enumeration member is the enumeration it belongs to:

    >>> type(Color.red)
    <Enum 'Color'>
    >>> isinstance(Color.green, Color)
    True
    >>>

Enums also have a property that contains just their item name:

    >>> print(Color.red.name)
    red

Enumerations support iteration, in definition order:

    >>> class Shake(Enum):
    ...   vanilla = 7
    ...   chocolate = 4
    ...   cookies = 9
    ...   mint = 3
    ...
    >>> for shake in Shake:
    ...   print(shake)
    ...
    Shake.vanilla
    Shake.chocolate
    Shake.cookies
    Shake.mint

Enumeration members are hashable, so they can be used in dictionaries
and sets:

    >>> apples = {}
    >>> apples[Color.red] = 'red delicious'
    >>> apples[Color.green] = 'granny smith'
    >>> apples
    {<Color.red: 1>: 'red delicious', <Color.green: 2>: 'granny smith'}

Programmatic access to enumeration members

Sometimes it's useful to access members in enumerations programmatically
(i.e. situations where Color.red won't do because the exact color is not
known at program-writing time). Enum allows such access:

    >>> Color(1)
    <Color.red: 1>
    >>> Color(3)
    <Color.blue: 3>

If you want to access enum members by name, use item access:

    >>> Color['red']
    <Color.red: 1>
    >>> Color['green']
    <Color.green: 2>

Duplicating enum members and values

Having two enum members with the same name is invalid:

    >>> class Shape(Enum):
    ...   square = 2
    ...   square = 3
    ...
    Traceback (most recent call last):
    ...
    TypeError: Attempted to reuse key: square

However, two enum members are allowed to have the same value. Given two
members A and B with the same value (and A defined first), B is an alias
to A. By-value lookup of the value of A and B will return A. By-name
lookup of B will also return A:

    >>> class Shape(Enum):
    ...   square = 2
    ...   diamond = 1
    ...   circle = 3
    ...   alias_for_square = 2
    ...
    >>> Shape.square
    <Shape.square: 2>
    >>> Shape.alias_for_square
    <Shape.square: 2>
    >>> Shape(2)
    <Shape.square: 2>

Iterating over the members of an enum does not provide the aliases:

    >>> list(Shape)
    [<Shape.square: 2>, <Shape.diamond: 1>, <Shape.circle: 3>]

The special attribute __members__ is an ordered dictionary mapping names
to members. It includes all names defined in the enumeration, including
the aliases:

    >>> for name, member in Shape.__members__.items():
    ...   name, member
    ...
    ('square', <Shape.square: 2>)
    ('diamond', <Shape.diamond: 1>)
    ('circle', <Shape.circle: 3>)
    ('alias_for_square', <Shape.square: 2>)

The __members__ attribute can be used for detailed programmatic access
to the enumeration members. For example, finding all the aliases:

    >>> [name for name, member in Shape.__members__.items() if member.name != name]
    ['alias_for_square']

Comparisons

Enumeration members are compared by identity:

    >>> Color.red is Color.red
    True
    >>> Color.red is Color.blue
    False
    >>> Color.red is not Color.blue
    True

Ordered comparisons between enumeration values are not supported. Enums
are not integers (but see IntEnum below):

    >>> Color.red < Color.blue
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: unorderable types: Color() < Color()

Equality comparisons are defined though:

    >>> Color.blue == Color.red
    False
    >>> Color.blue != Color.red
    True
    >>> Color.blue == Color.blue
    True

Comparisons against non-enumeration values will always compare not equal
(again, IntEnum was explicitly designed to behave differently, see
below):

    >>> Color.blue == 2
    False

Allowed members and attributes of enumerations

The examples above use integers for enumeration values. Using integers
is short and handy (and provided by default by the Functional API), but
not strictly enforced. In the vast majority of use-cases, one doesn't
care what the actual value of an enumeration is. But if the value is
important, enumerations can have arbitrary values.

Enumerations are Python classes, and can have methods and special
methods as usual. If we have this enumeration:

    class Mood(Enum):
      funky = 1
      happy = 3

      def describe(self):
        # self is the member here
        return self.name, self.value

      def __str__(self):
        return 'my custom str! {0}'.format(self.value)

      @classmethod
      def favorite_mood(cls):
        # cls here is the enumeration
        return cls.happy

Then:

    >>> Mood.favorite_mood()
    <Mood.happy: 3>
    >>> Mood.happy.describe()
    ('happy', 3)
    >>> str(Mood.funky)
    'my custom str! 1'

The rules for what is allowed are as follows: all attributes defined
within an enumeration will become members of this enumeration, with the
exception of __dunder__ names and descriptors[7]; methods are
descriptors too.

Restricted subclassing of enumerations

Subclassing an enumeration is allowed only if the enumeration does not
define any members. So this is forbidden:

    >>> class MoreColor(Color):
    ...   pink = 17
    ...
    TypeError: Cannot extend enumerations

But this is allowed:

    >>> class Foo(Enum):
    ...   def some_behavior(self):
    ...     pass
    ...
    >>> class Bar(Foo):
    ...   happy = 1
    ...   sad = 2
    ...

The rationale for this decision was given by Guido in[8]. Allowing to
subclass enums that define members would lead to a violation of some
important invariants of types and instances. On the other hand, it makes
sense to allow sharing some common behavior between a group of
enumerations, and subclassing empty enumerations is also used to
implement IntEnum.

IntEnum

A variation of Enum is proposed which is also a subclass of int. Members
of an IntEnum can be compared to integers; by extension, integer
enumerations of different types can also be compared to each other:

    >>> from enum import IntEnum
    >>> class Shape(IntEnum):
    ...   circle = 1
    ...   square = 2
    ...
    >>> class Request(IntEnum):
    ...   post = 1
    ...   get = 2
    ...
    >>> Shape == 1
    False
    >>> Shape.circle == 1
    True
    >>> Shape.circle == Request.post
    True

However they still can't be compared to Enum:

    >>> class Shape(IntEnum):
    ...   circle = 1
    ...   square = 2
    ...
    >>> class Color(Enum):
    ...   red = 1
    ...   green = 2
    ...
    >>> Shape.circle == Color.red
    False

IntEnum values behave like integers in other ways you'd expect:

    >>> int(Shape.circle)
    1
    >>> ['a', 'b', 'c'][Shape.circle]
    'b'
    >>> [i for i in range(Shape.square)]
    [0, 1]

For the vast majority of code, Enum is strongly recommended, since
IntEnum breaks some semantic promises of an enumeration (by being
comparable to integers, and thus by transitivity to other unrelated
enumerations). It should be used only in special cases where there's no
other choice; for example, when integer constants are replaced with
enumerations and backwards compatibility is required with code that
still expects integers.

Other derived enumerations

IntEnum will be part of the enum module. However, it would be very
simple to implement independently:

    class IntEnum(int, Enum):
        pass

This demonstrates how similar derived enumerations can be defined, for
example a StrEnum that mixes in str instead of int.

Some rules:

1.  When subclassing Enum, mix-in types must appear before Enum itself
    in the sequence of bases, as in the IntEnum example above.
2.  While Enum can have members of any type, once you mix in an
    additional type, all the members must have values of that type, e.g.
    int above. This restriction does not apply to mix-ins which only add
    methods and don't specify another data type such as int or str.

Pickling

Enumerations can be pickled and unpickled:

    >>> from enum.tests.fruit import Fruit
    >>> from pickle import dumps, loads
    >>> Fruit.tomato is loads(dumps(Fruit.tomato))
    True

The usual restrictions for pickling apply: picklable enums must be
defined in the top level of a module, since unpickling requires them to
be importable from that module.

Functional API

The Enum class is callable, providing the following functional API:

    >>> Animal = Enum('Animal', 'ant bee cat dog')
    >>> Animal
    <Enum 'Animal'>
    >>> Animal.ant
    <Animal.ant: 1>
    >>> Animal.ant.value
    1
    >>> list(Animal)
    [<Animal.ant: 1>, <Animal.bee: 2>, <Animal.cat: 3>, <Animal.dog: 4>]

The semantics of this API resemble namedtuple. The first argument of the
call to Enum is the name of the enumeration. Pickling enums created with
the functional API will work on CPython and PyPy, but for IronPython and
Jython you may need to specify the module name explicitly as follows:

    >>> Animals = Enum('Animals', 'ant bee cat dog', module=__name__)

The second argument is the source of enumeration member names. It can be
a whitespace-separated string of names, a sequence of names, a sequence
of 2-tuples with key/value pairs, or a mapping (e.g. dictionary) of
names to values. The last two options enable assigning arbitrary values
to enumerations; the others auto-assign increasing integers starting
with 1. A new class derived from Enum is returned. In other words, the
above assignment to Animal is equivalent to:

    >>> class Animals(Enum):
    ...   ant = 1
    ...   bee = 2
    ...   cat = 3
    ...   dog = 4

The reason for defaulting to 1 as the starting number and not 0 is that
0 is False in a boolean sense, but enum members all evaluate to True.

Proposed variations

Some variations were proposed during the discussions in the mailing
list. Here's some of the more popular ones.

flufl.enum

flufl.enum was the reference implementation upon which this PEP was
originally based. Eventually, it was decided against the inclusion of
flufl.enum because its design separated enumeration members from
enumerations, so the former are not instances of the latter. Its design
also explicitly permits subclassing enumerations for extending them with
more members (due to the member/enum separation, the type invariants are
not violated in flufl.enum with such a scheme).

Not having to specify values for enums

Michael Foord proposed (and Tim Delaney provided a proof-of-concept
implementation) to use metaclass magic that makes this possible:

    class Color(Enum):
        red, green, blue

The values get actually assigned only when first looked up.

Pros: cleaner syntax that requires less typing for a very common task
(just listing enumeration names without caring about the values).

Cons: involves much magic in the implementation, which makes even the
definition of such enums baffling when first seen. Besides, explicit is
better than implicit.

Using special names or forms to auto-assign enum values

A different approach to avoid specifying enum values is to use a special
name or form to auto assign them. For example:

    class Color(Enum):
        red = None          # auto-assigned to 0
        green = None        # auto-assigned to 1
        blue = None         # auto-assigned to 2

More flexibly:

    class Color(Enum):
        red = 7
        green = None        # auto-assigned to 8
        blue = 19
        purple = None       # auto-assigned to 20

Some variations on this theme:

1.  A special name auto imported from the enum package.
2.  Georg Brandl proposed ellipsis (...) instead of None to achieve the
    same effect.

Pros: no need to manually enter values. Makes it easier to change the
enum and extend it, especially for large enumerations.

Cons: actually longer to type in many simple cases. The argument of
explicit vs. implicit applies here as well.

Use-cases in the standard library

The Python standard library has many places where the usage of enums
would be beneficial to replace other idioms currently used to represent
them. Such usages can be divided to two categories: user-code facing
constants, and internal constants.

User-code facing constants like os.SEEK_*, socket module constants,
decimal rounding modes and HTML error codes could require backwards
compatibility since user code may expect integers. IntEnum as described
above provides the required semantics; being a subclass of int, it does
not affect user code that expects integers, while on the other hand
allowing printable representations for enumeration values:

    >>> import socket
    >>> family = socket.AF_INET
    >>> family == 2
    True
    >>> print(family)
    SocketFamily.AF_INET

Internal constants are not seen by user code but are employed internally
by stdlib modules. These can be implemented with Enum. Some examples
uncovered by a very partial skim through the stdlib: binhex, imaplib,
http/client, urllib/robotparser, idlelib, concurrent.futures,
turtledemo.

In addition, looking at the code of the Twisted library, there are many
use cases for replacing internal state constants with enums. The same
can be said about a lot of networking code (especially implementation of
protocols) and can be seen in test protocols written with the Tulip
library as well.

Acknowledgments

This PEP was initially proposing including the flufl.enum package[9] by
Barry Warsaw into the stdlib, and is inspired in large parts by it. Ben
Finney is the author of the earlier enumeration PEP 354.

References

Copyright

This document has been placed in the public domain.

[1] https://mail.python.org/pipermail/python-ideas/2013-January/019003.html

[2] https://mail.python.org/pipermail/python-ideas/2013-February/019373.html

[3] To make enums behave similarly to Python classes like bool, and
behave in a more intuitive way. It would be surprising if the type of
Color.red would not be Color. (Discussion in
https://mail.python.org/pipermail/python-dev/2013-April/125687.html)

[4] Subclassing enums and adding new members creates an unresolvable
situation; on one hand MoreColor.red and Color.red should not be the
same object, and on the other isinstance checks become confusing if they
are not. The discussion also links to Stack Overflow discussions that
make additional arguments.
(https://mail.python.org/pipermail/python-dev/2013-April/125716.html)

[5] It may be useful to have a class defining some behavior (methods,
with no actual enumeration members) mixed into an enum, and this would
not create the problem discussed in. (Discussion in
https://mail.python.org/pipermail/python-dev/2013-May/125859.html)

[6] https://mail.python.org/pipermail/python-dev/2013-May/126112.html

[7] http://docs.python.org/3/howto/descriptor.html

[8] Subclassing enums and adding new members creates an unresolvable
situation; on one hand MoreColor.red and Color.red should not be the
same object, and on the other isinstance checks become confusing if they
are not. The discussion also links to Stack Overflow discussions that
make additional arguments.
(https://mail.python.org/pipermail/python-dev/2013-April/125716.html)

[9] http://pythonhosted.org/flufl.enum/


text:
PEP: 445 Title: Add new APIs to customize Python memory allocators
Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner
<EMAIL_ADDRESS>BDFL-Delegate: Antoine Pitrou
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 15-Jun-2013 Python-Version: 3.4 Resolution:
https://mail.python.org/pipermail/python-dev/2013-July/127222.html

Abstract

This PEP proposes new Application Programming Interfaces (API) to
customize Python memory allocators. The only implementation required to
conform to this PEP is CPython, but other implementations may choose to
be compatible, or to re-use a similar scheme.

Rationale

Use cases:

-   Applications embedding Python which want to isolate Python memory
    from the memory of the application, or want to use a different
    memory allocator optimized for its Python usage
-   Python running on embedded devices with low memory and slow CPU. A
    custom memory allocator can be used for efficiency and/or to get
    access all the memory of the device.
-   Debug tools for memory allocators:
    -   track the memory usage (find memory leaks)
    -   get the location of a memory allocation: Python filename and
        line number, and the size of a memory block
    -   detect buffer underflow, buffer overflow and misuse of Python
        allocator APIs (see Redesign Debug Checks on Memory Block
        Allocators as Hooks)
    -   force memory allocations to fail to test handling of the
        MemoryError exception

Proposal

New Functions and Structures

-   Add a new GIL-free (no need to hold the GIL) memory allocator:

    -   void* PyMem_RawMalloc(size_t size)
    -   void* PyMem_RawRealloc(void *ptr, size_t new_size)
    -   void PyMem_RawFree(void *ptr)
    -   The newly allocated memory will not have been initialized in any
        way.
    -   Requesting zero bytes returns a distinct non-NULL pointer if
        possible, as if PyMem_Malloc(1) had been called instead.

-   Add a new PyMemAllocator structure:

        typedef struct {
            /* user context passed as the first argument to the 3 functions */
            void *ctx;

            /* allocate a memory block */
            void* (*malloc) (void *ctx, size_t size);

            /* allocate or resize a memory block */
            void* (*realloc) (void *ctx, void *ptr, size_t new_size);

            /* release a memory block */
            void (*free) (void *ctx, void *ptr);
        } PyMemAllocator;

-   Add a new PyMemAllocatorDomain enum to choose the Python allocator
    domain. Domains:

    -   PYMEM_DOMAIN_RAW: PyMem_RawMalloc(), PyMem_RawRealloc() and
        PyMem_RawFree()
    -   PYMEM_DOMAIN_MEM: PyMem_Malloc(), PyMem_Realloc() and
        PyMem_Free()
    -   PYMEM_DOMAIN_OBJ: PyObject_Malloc(), PyObject_Realloc() and
        PyObject_Free()

-   Add new functions to get and set memory block allocators:

    -   void PyMem_GetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator)
    -   void PyMem_SetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator)
    -   The new allocator must return a distinct non-NULL pointer when
        requesting zero bytes
    -   For the PYMEM_DOMAIN_RAW domain, the allocator must be
        thread-safe: the GIL is not held when the allocator is called.

-   Add a new PyObjectArenaAllocator structure:

        typedef struct {
            /* user context passed as the first argument to the 2 functions */
            void *ctx;

            /* allocate an arena */
            void* (*alloc) (void *ctx, size_t size);

            /* release an arena */
            void (*free) (void *ctx, void *ptr, size_t size);
        } PyObjectArenaAllocator;

-   Add new functions to get and set the arena allocator used by
    pymalloc:

    -   void PyObject_GetArenaAllocator(PyObjectArenaAllocator *allocator)
    -   void PyObject_SetArenaAllocator(PyObjectArenaAllocator *allocator)

-   Add a new function to reinstall the debug checks on memory
    allocators when a memory allocator is replaced with
    PyMem_SetAllocator():

    -   void PyMem_SetupDebugHooks(void)
    -   Install the debug hooks on all memory block allocators. The
        function can be called more than once, hooks are only installed
        once.
    -   The function does nothing is Python is not compiled in debug
        mode.

-   Memory block allocators always return NULL if size is greater than
    PY_SSIZE_T_MAX. The check is done before calling the inner function.

Note

The pymalloc allocator is optimized for objects smaller than 512 bytes
with a short lifetime. It uses memory mappings with a fixed size of 256
KB called "arenas".

Here is how the allocators are set up by default:

-   PYMEM_DOMAIN_RAW, PYMEM_DOMAIN_MEM: malloc(), realloc() and free();
    call malloc(1) when requesting zero bytes
-   PYMEM_DOMAIN_OBJ: pymalloc allocator which falls back on
    PyMem_Malloc() for allocations larger than 512 bytes
-   pymalloc arena allocator: VirtualAlloc() and VirtualFree() on
    Windows, mmap() and munmap() when available, or malloc() and free()

Redesign Debug Checks on Memory Block Allocators as Hooks

Since Python 2.3, Python implements different checks on memory
allocators in debug mode:

-   Newly allocated memory is filled with the byte 0xCB, freed memory is
    filled with the byte 0xDB.
-   Detect API violations, ex: PyObject_Free() called on a memory block
    allocated by PyMem_Malloc()
-   Detect write before the start of the buffer (buffer underflow)
-   Detect write after the end of the buffer (buffer overflow)

In Python 3.3, the checks are installed by replacing PyMem_Malloc(),
PyMem_Realloc(), PyMem_Free(), PyObject_Malloc(), PyObject_Realloc() and
PyObject_Free() using macros. The new allocator allocates a larger
buffer and writes a pattern to detect buffer underflow, buffer overflow
and use after free (by filling the buffer with the byte 0xDB). It uses
the original PyObject_Malloc() function to allocate memory. So
PyMem_Malloc() and PyMem_Realloc() indirectly call PyObject_Malloc() and
PyObject_Realloc().

This PEP redesigns the debug checks as hooks on the existing allocators
in debug mode. Examples of call traces without the hooks:

-   PyMem_RawMalloc() => _PyMem_RawMalloc() => malloc()
-   PyMem_Realloc() => _PyMem_RawRealloc() => realloc()
-   PyObject_Free() => _PyObject_Free()

Call traces when the hooks are installed (debug mode):

-   PyMem_RawMalloc() => _PyMem_DebugMalloc() => _PyMem_RawMalloc() =>
    malloc()
-   PyMem_Realloc() => _PyMem_DebugRealloc() => _PyMem_RawRealloc() =>
    realloc()
-   PyObject_Free() => _PyMem_DebugFree() => _PyObject_Free()

As a result, PyMem_Malloc() and PyMem_Realloc() now call malloc() and
realloc() in both release mode and debug mode, instead of calling
PyObject_Malloc() and PyObject_Realloc() in debug mode.

When at least one memory allocator is replaced with
PyMem_SetAllocator(), the PyMem_SetupDebugHooks() function must be
called to reinstall the debug hooks on top on the new allocator.

Don't call malloc() directly anymore

PyObject_Malloc() falls back on PyMem_Malloc() instead of malloc() if
size is greater or equal than 512 bytes, and PyObject_Realloc() falls
back on PyMem_Realloc() instead of realloc()

Direct calls to malloc() are replaced with PyMem_Malloc(), or
PyMem_RawMalloc() if the GIL is not held.

External libraries like zlib or OpenSSL can be configured to allocate
memory using PyMem_Malloc() or PyMem_RawMalloc(). If the allocator of a
library can only be replaced globally (rather than on an
object-by-object basis), it shouldn't be replaced when Python is
embedded in an application.

For the "track memory usage" use case, it is important to track memory
allocated in external libraries to have accurate reports, because these
allocations can be large (e.g. they can raise a MemoryError exception)
and would otherwise be missed in memory usage reports.

Examples

Use case 1: Replace Memory Allocators, keep pymalloc

Dummy example wasting 2 bytes per memory block, and 10 bytes per
pymalloc arena:

    #include <stdlib.h>

    size_t alloc_padding = 2;
    size_t arena_padding = 10;

    void* my_malloc(void *ctx, size_t size)
    {
        int padding = *(int *)ctx;
        return malloc(size + padding);
    }

    void* my_realloc(void *ctx, void *ptr, size_t new_size)
    {
        int padding = *(int *)ctx;
        return realloc(ptr, new_size + padding);
    }

    void my_free(void *ctx, void *ptr)
    {
        free(ptr);
    }

    void* my_alloc_arena(void *ctx, size_t size)
    {
        int padding = *(int *)ctx;
        return malloc(size + padding);
    }

    void my_free_arena(void *ctx, void *ptr, size_t size)
    {
        free(ptr);
    }

    void setup_custom_allocator(void)
    {
        PyMemAllocator alloc;
        PyObjectArenaAllocator arena;

        alloc.ctx = &alloc_padding;
        alloc.malloc = my_malloc;
        alloc.realloc = my_realloc;
        alloc.free = my_free;

        PyMem_SetAllocator(PYMEM_DOMAIN_RAW, &alloc);
        PyMem_SetAllocator(PYMEM_DOMAIN_MEM, &alloc);
        /* leave PYMEM_DOMAIN_OBJ unchanged, use pymalloc */

        arena.ctx = &arena_padding;
        arena.alloc = my_alloc_arena;
        arena.free = my_free_arena;
        PyObject_SetArenaAllocator(&arena);

        PyMem_SetupDebugHooks();
    }

Use case 2: Replace Memory Allocators, override pymalloc

If you have a dedicated allocator optimized for allocations of objects
smaller than 512 bytes with a short lifetime, pymalloc can be overridden
(replace PyObject_Malloc()).

Dummy example wasting 2 bytes per memory block:

    #include <stdlib.h>

    size_t padding = 2;

    void* my_malloc(void *ctx, size_t size)
    {
        int padding = *(int *)ctx;
        return malloc(size + padding);
    }

    void* my_realloc(void *ctx, void *ptr, size_t new_size)
    {
        int padding = *(int *)ctx;
        return realloc(ptr, new_size + padding);
    }

    void my_free(void *ctx, void *ptr)
    {
        free(ptr);
    }

    void setup_custom_allocator(void)
    {
        PyMemAllocator alloc;
        alloc.ctx = &padding;
        alloc.malloc = my_malloc;
        alloc.realloc = my_realloc;
        alloc.free = my_free;

        PyMem_SetAllocator(PYMEM_DOMAIN_RAW, &alloc);
        PyMem_SetAllocator(PYMEM_DOMAIN_MEM, &alloc);
        PyMem_SetAllocator(PYMEM_DOMAIN_OBJ, &alloc);

        PyMem_SetupDebugHooks();
    }

The pymalloc arena does not need to be replaced, because it is no more
used by the new allocator.

Use case 3: Setup Hooks On Memory Block Allocators

Example to setup hooks on all memory block allocators:

    struct {
        PyMemAllocator raw;
        PyMemAllocator mem;
        PyMemAllocator obj;
        /* ... */
    } hook;

    static void* hook_malloc(void *ctx, size_t size)
    {
        PyMemAllocator *alloc = (PyMemAllocator *)ctx;
        void *ptr;
        /* ... */
        ptr = alloc->malloc(alloc->ctx, size);
        /* ... */
        return ptr;
    }

    static void* hook_realloc(void *ctx, void *ptr, size_t new_size)
    {
        PyMemAllocator *alloc = (PyMemAllocator *)ctx;
        void *ptr2;
        /* ... */
        ptr2 = alloc->realloc(alloc->ctx, ptr, new_size);
        /* ... */
        return ptr2;
    }

    static void hook_free(void *ctx, void *ptr)
    {
        PyMemAllocator *alloc = (PyMemAllocator *)ctx;
        /* ... */
        alloc->free(alloc->ctx, ptr);
        /* ... */
    }

    void setup_hooks(void)
    {
        PyMemAllocator alloc;
        static int installed = 0;

        if (installed)
            return;
        installed = 1;

        alloc.malloc = hook_malloc;
        alloc.realloc = hook_realloc;
        alloc.free = hook_free;
        PyMem_GetAllocator(PYMEM_DOMAIN_RAW, &hook.raw);
        PyMem_GetAllocator(PYMEM_DOMAIN_MEM, &hook.mem);
        PyMem_GetAllocator(PYMEM_DOMAIN_OBJ, &hook.obj);

        alloc.ctx = &hook.raw;
        PyMem_SetAllocator(PYMEM_DOMAIN_RAW, &alloc);

        alloc.ctx = &hook.mem;
        PyMem_SetAllocator(PYMEM_DOMAIN_MEM, &alloc);

        alloc.ctx = &hook.obj;
        PyMem_SetAllocator(PYMEM_DOMAIN_OBJ, &alloc);
    }

Note

PyMem_SetupDebugHooks() does not need to be called because memory
allocator are not replaced: the debug checks on memory block allocators
are installed automatically at startup.

Performances

The implementation of this PEP (issue #3329) has no visible overhead on
the Python benchmark suite.

Results of the Python benchmarks suite (-b 2n3): some tests are 1.04x
faster, some tests are 1.04 slower. Results of pybench microbenchmark:
"+0.1%" slower globally (diff between -4.9% and +5.6%).

The full output of benchmarks is attached to the issue #3329.

Rejected Alternatives

More specific functions to get/set memory allocators

It was originally proposed a larger set of C API functions, with one
pair of functions for each allocator domain:

-   void PyMem_GetRawAllocator(PyMemAllocator *allocator)
-   void PyMem_GetAllocator(PyMemAllocator *allocator)
-   void PyObject_GetAllocator(PyMemAllocator *allocator)
-   void PyMem_SetRawAllocator(PyMemAllocator *allocator)
-   void PyMem_SetAllocator(PyMemAllocator *allocator)
-   void PyObject_SetAllocator(PyMemAllocator *allocator)

This alternative was rejected because it is not possible to write
generic code with more specific functions: code must be duplicated for
each memory allocator domain.

Make PyMem_Malloc() reuse PyMem_RawMalloc() by default

If PyMem_Malloc() called PyMem_RawMalloc() by default, calling
PyMem_SetAllocator(PYMEM_DOMAIN_RAW, alloc) would also patch
PyMem_Malloc() indirectly.

This alternative was rejected because PyMem_SetAllocator() would have a
different behaviour depending on the domain. Always having the same
behaviour is less error-prone.

Add a new PYDEBUGMALLOC environment variable

It was proposed to add a new PYDEBUGMALLOC environment variable to
enable debug checks on memory block allocators. It would have had the
same effect as calling the PyMem_SetupDebugHooks(), without the need to
write any C code. Another advantage is to allow to enable debug checks
even in release mode: debug checks would always be compiled in, but only
enabled when the environment variable is present and non-empty.

This alternative was rejected because a new environment variable would
make Python initialization even more complex. PEP 432 tries to simplify
the CPython startup sequence.

Use macros to get customizable allocators

To have no overhead in the default configuration, customizable
allocators would be an optional feature enabled by a configuration
option or by macros.

This alternative was rejected because the use of macros implies having
to recompile extensions modules to use the new allocator and allocator
hooks. Not having to recompile Python nor extension modules makes debug
hooks easier to use in practice.

Pass the C filename and line number

Define allocator functions as macros using __FILE__ and __LINE__ to get
the C filename and line number of a memory allocation.

Example of PyMem_Malloc macro with the modified PyMemAllocator
structure:

    typedef struct {
        /* user context passed as the first argument
           to the 3 functions */
        void *ctx;

        /* allocate a memory block */
        void* (*malloc) (void *ctx, const char *filename, int lineno,
                         size_t size);

        /* allocate or resize a memory block */
        void* (*realloc) (void *ctx, const char *filename, int lineno,
                          void *ptr, size_t new_size);

        /* release a memory block */
        void (*free) (void *ctx, const char *filename, int lineno,
                      void *ptr);
    } PyMemAllocator;

    void* _PyMem_MallocTrace(const char *filename, int lineno,
                             size_t size);

    /* the function is still needed for the Python stable ABI */
    void* PyMem_Malloc(size_t size);

    #define PyMem_Malloc(size) \
            _PyMem_MallocTrace(__FILE__, __LINE__, size)

The GC allocator functions would also have to be patched. For example,
_PyObject_GC_Malloc() is used in many C functions and so objects of
different types would have the same allocation location.

This alternative was rejected because passing a filename and a line
number to each allocator makes the API more complex: pass 3 new
arguments (ctx, filename, lineno) to each allocator function, instead of
just a context argument (ctx). Having to also modify GC allocator
functions adds too much complexity for a little gain.

GIL-free PyMem_Malloc()

In Python 3.3, when Python is compiled in debug mode, PyMem_Malloc()
indirectly calls PyObject_Malloc() which requires the GIL to be held (it
isn't thread-safe). That's why PyMem_Malloc() must be called with the
GIL held.

This PEP changes PyMem_Malloc(): it now always calls malloc() rather
than PyObject_Malloc(). The "GIL must be held" restriction could
therefore be removed from PyMem_Malloc().

This alternative was rejected because allowing to call PyMem_Malloc()
without holding the GIL can break applications which setup their own
allocators or allocator hooks. Holding the GIL is convenient to develop
a custom allocator: no need to care about other threads. It is also
convenient for a debug allocator hook: Python objects can be safely
inspected, and the C API may be used for reporting.

Moreover, calling PyGILState_Ensure() in a memory allocator has
unexpected behaviour, especially at Python startup and when creating of
a new Python thread state. It is better to free custom allocators of the
responsibility of acquiring the GIL.

Don't add PyMem_RawMalloc()

Replace malloc() with PyMem_Malloc(), but only if the GIL is held.
Otherwise, keep malloc() unchanged.

The PyMem_Malloc() is used without the GIL held in some Python
functions. For example, the main() and Py_Main() functions of Python
call PyMem_Malloc() whereas the GIL do not exist yet. In this case,
PyMem_Malloc() would be replaced with malloc() (or PyMem_RawMalloc()).

This alternative was rejected because PyMem_RawMalloc() is required for
accurate reports of the memory usage. When a debug hook is used to track
the memory usage, the memory allocated by direct calls to malloc()
cannot be tracked. PyMem_RawMalloc() can be hooked and so all the memory
allocated by Python can be tracked, including memory allocated without
holding the GIL.

Use existing debug tools to analyze memory use

There are many existing debug tools to analyze memory use. Some
examples: Valgrind, Purify, Clang AddressSanitizer, failmalloc, etc.

The problem is to retrieve the Python object related to a memory pointer
to read its type and/or its content. Another issue is to retrieve the
source of the memory allocation: the C backtrace is usually useless
(same reasoning than macros using __FILE__ and __LINE__, see Pass the C
filename and line number), the Python filename and line number (or even
the Python traceback) is more useful.

This alternative was rejected because classic tools are unable to
introspect Python internals to collect such information. Being able to
setup a hook on allocators called with the GIL held allows to collect a
lot of useful data from Python internals.

Add a msize() function

Add another function to PyMemAllocator and PyObjectArenaAllocator
structures:

    size_t msize(void *ptr);

This function returns the size of a memory block or a memory mapping.
Return (size_t)-1 if the function is not implemented or if the pointer
is unknown (ex: NULL pointer).

On Windows, this function can be implemented using _msize() and
VirtualQuery().

The function can be used to implement a hook tracking the memory usage.
The free() method of an allocator only gets the address of a memory
block, whereas the size of the memory block is required to update the
memory usage.

The additional msize() function was rejected because only few platforms
implement it. For example, Linux with the GNU libc does not provide a
function to get the size of a memory block. msize() is not currently
used in the Python source code. The function would only be used to track
memory use, and make the API more complex. A debug hook can implement
the function internally, there is no need to add it to PyMemAllocator
and PyObjectArenaAllocator structures.

No context argument

Simplify the signature of allocator functions, remove the context
argument:

-   void* malloc(size_t size)
-   void* realloc(void *ptr, size_t new_size)
-   void free(void *ptr)

It is likely for an allocator hook to be reused for PyMem_SetAllocator()
and PyObject_SetAllocator(), or even PyMem_SetRawAllocator(), but the
hook must call a different function depending on the allocator. The
context is a convenient way to reuse the same custom allocator or hook
for different Python allocators.

In C++, the context can be used to pass this.

External Libraries

Examples of API used to customize memory allocators.

Libraries used by Python:

-   OpenSSL: CRYPTO_set_mem_functions() to set memory management
    functions globally
-   expat: parserCreate() has a per-instance memory handler
-   zlib: zlib 1.2.8 Manual, pass an opaque pointer
-   bz2: bzip2 and libbzip2, version 1.0.5, pass an opaque pointer
-   lzma: LZMA SDK - How to Use, pass an opaque pointer
-   lipmpdec: no opaque pointer (classic malloc API)

Other libraries:

-   glib: g_mem_set_vtable()
-   libxml2: xmlGcMemSetup(), global
-   Oracle's OCI: Oracle Call Interface Programmer's Guide, Release 2
    (9.2), pass an opaque pointer

The new ctx parameter of this PEP was inspired by the API of zlib and
Oracle's OCI libraries.

See also the GNU libc: Memory Allocation Hooks which uses a different
approach to hook memory allocators.

Memory Allocators

The C standard library provides the well known malloc() function. Its
implementation depends on the platform and of the C library. The GNU C
library uses a modified ptmalloc2, based on "Doug Lea's Malloc"
(dlmalloc). FreeBSD uses jemalloc. Google provides tcmalloc which is
part of gperftools.

malloc() uses two kinds of memory: heap and memory mappings. Memory
mappings are usually used for large allocations (ex: larger than 256
KB), whereas the heap is used for small allocations.

On UNIX, the heap is handled by brk() and sbrk() system calls, and it is
contiguous. On Windows, the heap is handled by HeapAlloc() and can be
discontiguous. Memory mappings are handled by mmap() on UNIX and
VirtualAlloc() on Windows, they can be discontiguous.

Releasing a memory mapping gives back immediately the memory to the
system. On UNIX, the heap memory is only given back to the system if the
released block is located at the end of the heap. Otherwise, the memory
will only be given back to the system when all the memory located after
the released memory is also released.

To allocate memory on the heap, an allocator tries to reuse free space.
If there is no contiguous space big enough, the heap must be enlarged,
even if there is more free space than required size. This issue is
called the "memory fragmentation": the memory usage seen by the system
is higher than real usage. On Windows, HeapAlloc() creates a new memory
mapping with VirtualAlloc() if there is not enough free contiguous
memory.

CPython has a pymalloc allocator for allocations smaller than 512 bytes.
This allocator is optimized for small objects with a short lifetime. It
uses memory mappings called "arenas" with a fixed size of 256 KB.

Other allocators:

-   Windows provides a Low-fragmentation Heap.
-   The Linux kernel uses slab allocation.
-   The glib library has a Memory Slice API: efficient way to allocate
    groups of equal-sized chunks of memory

This PEP allows to choose exactly which memory allocator is used for
your application depending on its usage of the memory (number of
allocations, size of allocations, lifetime of objects, etc.).

Links

CPython issues related to memory allocation:

-   Issue #3329: Add new APIs to customize memory allocators
-   Issue #13483: Use VirtualAlloc to allocate memory arenas
-   Issue #16742: PyOS_Readline drops GIL and calls PyOS_StdioReadline,
    which isn't thread safe
-   Issue #18203: Replace calls to malloc() with PyMem_Malloc() or
    PyMem_RawMalloc()
-   Issue #18227: Use Python memory allocators in external libraries
    like zlib or OpenSSL

Projects analyzing the memory usage of Python applications:

-   pytracemalloc
-   Meliae: Python Memory Usage Analyzer
-   Guppy-PE: umbrella package combining Heapy and GSL
-   PySizer (developed for Python 2.4)

Copyright

This document has been placed into the public domain.

license: Public Domain


text:
PEP: 8102 Title: 2021 Term Steering Council election Author: Ewa
Jodlowska<EMAIL_ADDRESS>Ee Durbin<EMAIL_ADDRESS>Joe Carey
<EMAIL_ADDRESS>Sponsor: Brett Cannon<EMAIL_ADDRESS>Status: Final
Type: Informational Topic: Governance Content-Type: text/x-rst Created:
29-Oct-2020

Abstract

This document describes the schedule and other details of the December
2020 election for the Python steering council, as specified in PEP 13.
This is the steering council election for the 2021 term.

Election Administration

The steering council appointed the Python Software Foundation Director
of Infrastructure, Ee Durbin, and Accounting Manager, Joe Carey, to
coadminister the election.

Python Software Foundation Executive Director, Ewa Jodlowska, will
communicate announcements regarding the election.

Schedule

There will be a two-week nomination period, followed by a two-week vote.

The nomination period was: November 1, 2020 through November 16, 2020
12:00 UTC (The end of November 15, 2020 Anywhere on Earth). As announced
on python-committers, and took place on discuss.python.org.

The voting period is: December 1, 2020 12:00 UTC through December 16,
2020 12:00 UTC (The end of December 15, 2020 Anywhere on Earth).

Candidates

Candidates must be nominated by a core team member. If the candidate is
a core team member, they may nominate themselves.

Nominees (in alphabetical order):

-   Brett Cannon
-   Matt Harrison
-   Christian Heimes
-   Raymond Hettinger
-   David Mertz
-   Pablo Galindo Salgado
-   Barry Warsaw
-   Carol Willing
-   Thomas Wouters
-   Mariatta

Withdrawn nominations:

-   None

Voter Roll

All active Python core team members are eligible to vote. Active status
is determined as described in PEP 13 <13#membership> and implemented via
the software at python/voters[1].

Ballots will be distributed based on the The Python Voter Roll for this
election [2].

While this file is not public as it contains private email addresses,
the Complete Voter Roll by name will be made available when the roll is
created.

Election Implementation

The election will be conducted using the Helios Voting Service.

Configuration

Short name: 2021-python-steering-council

Name: 2021 Python Steering Council Election

Description:
Election for the Python steering council, as specified in PEP 13. This is steering council election for the 2021 term.

type: Election

Use voter aliases: [X]

Randomize answer order: [X]

Private: [X]

Help Email Address<EMAIL_ADDRESS>
Voting starts at: December 1, 2020 00:00 UTC

Voting ends at: December 16, 2020 12:00 UTC

This will create an election in which:

-   Voting is not open to the public, only those on the Voter Roll may
    participate. Ballots will be emailed when voting starts.
-   Candidates are presented in random order, to help avoid bias.
-   Voter identities and ballots are protected against cryptographic
    advances.

Questions

Question 1

Select between 0 and - (approval) answers. Result Type: absolute

Question: Select candidates for the Python Steering Council

Answer #1 - #N: Candidates from Candidates_ Section

Results

Of 91 eligible voters, 74 cast ballots.

The top five vote-getters are:

-   Barry Warsaw
-   Brett Cannon
-   Carol Willing
-   Pablo Galindo Salgado
-   Thomas Wouters

No conflict of interest as defined in PEP 13 were observed.

The full vote counts are as follows:

  ----------------------------------------
  Candidate               Votes Received
  ----------------------- ----------------
  Pablo Galindo Salgado   59

  Carol Willing           57

  Barry Warsaw            56

  Brett Cannon            54

  Thomas Wouters          52

  Christian Heimes        43

  Mariatta                42

  Raymond Hettinger       31

  David Mertz             28

  Matt Harrison           15
  ----------------------------------------

Copyright

This document has been placed in the public domain.

Complete Voter Roll

Active Python core developers

    Abhilash Raj
    Alex Gaynor
    Alex Martelli
    Alexander Belopolsky
    Andrew Kuchling
    Andrew Svetlov
    Antoine Pitrou
    Barry Warsaw
    Batuhan Taskaya
    Benjamin Peterson
    Berker Peksağ
    Brandt Bucher
    Brett Cannon
    Brian Curtin
    Brian Quinlan
    Carol Willing
    Cheryl Sabella
    Chris Jerdonek
    Chris Withers
    Christian Heimes
    Christian Tismer
    Davin Potts
    Dino Viehland
    Donald Stufft
    Dong-hee Na
    Emily Morehouse
    Éric Araujo
    Eric Snow
    Eric V. Smith
    Ethan Furman
    Ezio Melotti
    Facundo Batista
    Fred Drake
    Georg Brandl
    Giampaolo Rodolà
    Gregory P. Smith
    Guido van Rossum
    Hynek Schlawack
    Inada Naoki
    Ivan Levkivskyi
    Jack Jansen
    Jason R. Coombs
    Jeremy Kloth
    Jesús Cea
    Joannah Nanjekye
    Julien Palard
    Karthikeyan Singaravelan
    Kurt B. Kaiser
    Kushal Das
    Kyle Stanley
    Larry Hastings
    Lisa Roach
    Łukasz Langa
    Lysandros Nikolaou
    Marc-André Lemburg
    Mariatta
    Mark Dickinson
    Mark Hammond
    Mark Shannon
    Matthias Klose
    Michael Foord
    Nathaniel J. Smith
    Ned Deily
    Neil Schemenauer
    Alyssa Coghlan
    Pablo Galindo
    Paul Ganssle
    Paul Moore
    Petr Viktorin
    R. David Murray
    Raymond Hettinger
    Robert Collins
    Ronald Oussoren
    Sandro Tosi
    Senthil Kumaran
    Serhiy Storchaka
    Stefan Behnel
    Steve Dower
    Steven D'Aprano
    Stéphane Wirtel
    Tal Einat
    Terry Jan Reedy
    Thomas Wouters
    Tim Golden
    Tim Peters
    Victor Stinner
    Vinay Sajip
    Walter Dörwald
    Xiang Zhang
    Yury Selivanov
    Zachary Ware

[1] This repository is private and accessible only to Python Core
Developers, administrators, and Python Software Foundation Staff as it
contains personal email addresses.

[2] This repository is private and accessible only to Python Core
Developers, administrators, and Python Software Foundation Staff as it
contains personal email addresses.

authors:
Ee Durbin
Ewa Jodlowska
Joe Carey


text:
PEP: 8105 Title: 2024 Term Steering Council election Author: Ee Durbin
<EMAIL_ADDRESS>Sponsor: Thomas Wouters<EMAIL_ADDRESS>Status:
Final Type: Informational Topic: Governance Created: 23-Oct-2023

Abstract

This document describes the schedule and other details of the 2023
election for the Python steering council, as specified in PEP 13. This
is the steering council election for the 2024 term (i.e. Python 3.13).

Election Administration

The steering council appointed the Python Software Foundation Director
of Infrastructure, Ee Durbin, to administer the election.

Schedule

There was a two-week nomination period, followed by a two-week vote.

The nomination period was: November 9, 2023 through November 22, 2023
AoE[1].

The voting period was: November 27, 2023 through December 11, 2023
AoE[2].

Candidates

Candidates must be nominated by a core team member. If the candidate is
a core team member, they may nominate themselves.

Nominees (in alphabetical order by first name):

-   Barry Warsaw
-   Donghee Na
-   Emily Morehouse
-   Gregory P. Smith
-   Mariatta
-   Pablo Galindo Salgado
-   Thomas Wouters
-   Victor Stinner

Withdrawn nominations:

-   None

Voter Roll

All active Python core team members are eligible to vote. Active status
is determined as described in PEP 13 <13#membership> and implemented via
the software at python/voters [3].

Ballots were be distributed based on the the Python Voter Roll[4] for
this election.

While this file is not public as it contains private email addresses,
the Complete Voter Roll is available with a list of all eligible voters
by name.

Election Implementation

The election was be conducted using the Helios Voting Service.

Configuration

Short name: 2024-python-steering-council

Name: 2024 Python Steering Council Election

Description:
Election for the Python steering council, as specified in PEP 13. This is steering council election for the 2024 term.

type: Election

Use voter aliases: [X]

Randomize answer order: [X]

Private: [X]

Help Email Address<EMAIL_ADDRESS>
Voting starts at: November 27, 2023 12:00 UTC

Voting ends at: December 11, 2023 12:00 UTC

This will create an election in which:

-   Voting is not open to the public, only those on the Voter Roll may
    participate. Ballots will be emailed when voting starts.
-   Candidates are presented in random order, to help avoid bias.
-   Voter identities and ballots are protected against cryptographic
    advances.

Questions

Question 1

Select between 0 and - (approval) answers. Result Type: absolute

Question: Select candidates for the Python Steering Council

Answer #1 - #N: Candidates from Candidates_ Section

Extended Election Implementation

Due to a mismatch between announced timeline and the initial election
implementation an extended election is held to provide the additional
day for voters who were disenfranchised by the initial election.

Results from the initial election were combined with the results from
the extended election. Only voters who DID NOT cast ballots in the
initial election were able to vote.

The election was be conducted using the Helios Voting Service.

Configuration

Short name: 2024-python-steering-council-extension

Name: 2024 Python Steering Council Election Extension

Description:
Election for the Python steering council, as specified in PEP 13. This is steering council election for the 2024 term.

type: Election

Use voter aliases: [X]

Randomize answer order: [X]

Private: [X]

Help Email Address<EMAIL_ADDRESS>
Voting starts at: December 11, 2023 12:00 UTC

Voting ends at: December 12, 2023 12:00 UTC

This will create an election in which:

-   Voting is not open to the public, only those on the Voter Roll may
    participate. Ballots will be emailed when voting starts.
-   Candidates are presented in random order, to help avoid bias.
-   Voter identities and ballots are protected against cryptographic
    advances.

Questions

Question 1

Select between 0 and - (approval) answers. Result Type: absolute

Question: Select candidates for the Python Steering Council

Answer #1 - #N: Candidates from Candidates_ Section

Results

Of 87 eligible voters, 68 cast ballots.

The top five vote-getters are:

-   Pablo Galindo Salgado
-   Gregory P. Smith
-   Barry Warsaw
-   Emily Morehouse
-   Thomas Wouters

No conflict of interest as defined in PEP 13 were observed.

The full vote counts are as follows:

Initial Election

56 of 87 eligible voters cast ballots.

  ----------------------------------------
  Candidate               Votes Received
  ----------------------- ----------------
  Barry Warsaw            35

  Donghee Na              27

  Emily Morehouse         38

  Gregory P. Smith        42

  Mariatta                25

  Pablo Galindo Salgado   48

  Thomas Wouters          30

  Victor Stinner          27
  ----------------------------------------

Election Extension

12 of 31 eligible voters disenfranchised in the initial election cast
ballots.

  ----------------------------------------
  Candidate               Votes Received
  ----------------------- ----------------
  Barry Warsaw            8

  Donghee Na              5

  Emily Morehouse         8

  Gregory P. Smith        10

  Mariatta                5

  Pablo Galindo Salgado   12

  Thomas Wouters          10

  Victor Stinner          8
  ----------------------------------------

Combined

68 of 87 eligible voters cast ballots

  ----------------------------------------
  Candidate               Votes Received
  ----------------------- ----------------
  Barry Warsaw            43

  Donghee Na              32

  Emily Morehouse         46

  Gregory P. Smith        52

  Mariatta                30

  Pablo Galindo Salgado   60

  Thomas Wouters          40

  Victor Stinner          35
  ----------------------------------------

Copyright

This document has been placed in the public domain.

Complete Voter Roll

Active Python core developers

    Adam Turner
    Alex Gaynor
    Alex Waygood
    Alexander Belopolsky
    Alyssa Coghlan
    Ammar Askar
    Andrew Svetlov
    Antoine Pitrou
    Barney Gale
    Barry Warsaw
    Batuhan Taskaya
    Benjamin Peterson
    Berker Peksağ
    Brandt Bucher
    Brett Cannon
    Brian Curtin
    C.A.M. Gerlach
    Carl Friedrich Bolz-Tereick
    Carl Meyer
    Carol Willing
    Cheryl Sabella
    Chris Withers
    Christian Heimes
    Dennis Sweeney
    Dino Viehland
    Donghee Na
    Emily Morehouse
    Éric Araujo
    Eric Snow
    Eric V. Smith
    Erlend Egeberg Aasland
    Ethan Furman
    Ezio Melotti
    Facundo Batista
    Filipe Laíns
    Georg Brandl
    Giampaolo Rodolà
    Gregory P. Smith
    Guido van Rossum
    Hugo van Kemenade
    Hynek Schlawack
    Inada Naoki
    Irit Katriel
    Ivan Levkivskyi
    Jason R. Coombs
    Jelle Zijlstra
    Jeremy Kloth
    Joannah Nanjekye
    Julien Palard
    Karthikeyan Singaravelan
    Ken Jin
    Kumar Aditya
    Kurt B. Kaiser
    Kushal Das
    Larry Hastings
    Łukasz Langa
    Lysandros Nikolaou
    Marc-André Lemburg
    Mariatta
    Mark Dickinson
    Mark Shannon
    Nathaniel J. Smith
    Ned Deily
    Neil Schemenauer
    Pablo Galindo
    Paul Ganssle
    Paul Moore
    Petr Viktorin
    Pradyun Gedam
    R. David Murray
    Raymond Hettinger
    Ronald Oussoren
    Senthil Kumaran
    Serhiy Storchaka
    Shantanu Jain
    Stefan Behnel
    Stéphane Wirtel
    Steve Dower
    Tal Einat
    Terry Jan Reedy
    Thomas Wouters
    Tim Golden
    Tim Peters
    Victor Stinner
    Vinay Sajip
    Yury Selivanov
    Zachary Ware

[1] AoE: Anywhere on Earth.

[2] AoE: Anywhere on Earth.

[3] This repository is private and accessible only to Python Core
Developers, administrators, and Python Software Foundation Staff as it
contains personal email addresses.

[4] This repository is private and accessible only to Python Core
Developers, administrators, and Python Software Foundation Staff as it
contains personal email addresses.


text:
PEP: 665 Title: A file format to list Python dependencies for
reproducibility of an application Author: Brett Cannon
<EMAIL_ADDRESS>Pradyun Gedam<EMAIL_ADDRESS>Tzu-ping Chung
<EMAIL_ADDRESS>PEP-Delegate: Paul Moore<EMAIL_ADDRESS>Discussions-To: https://discuss.python.org/t/9911 Status: Rejected Type:
Standards Track Topic: Packaging Created: 29-Jul-2021 Post-History:
29-Jul-2021, 03-Nov-2021, 25-Nov-2021 Superseded-By: 751 Resolution:
https://discuss.python.org/t/pep-665-take-2-a-file-format-to-list-python-dependencies-for-reproducibility-of-an-application/11736/140

Note

This PEP was rejected due to lukewarm reception from the community from
the lack of source distribution support.

Abstract

This PEP specifies a file format to specify the list of Python package
installation requirements for an application, and the relation between
the specified requirements. The list of requirements is considered
exhaustive for the installation target, and thus not requiring any
information beyond the platform being installed for, and the file
itself. The file format is flexible enough to allow installing the
requirements across different platforms, which allows for
reproducibility on multiple platforms from the same file.

Terminology

There are several terms whose definition must be agreed upon in order to
facilitate a discussion on the topic of this PEP.

A package is something you install as a dependency and use via the
import system. The packages on PyPI are an example of this.

An application or app is an end product that other external code does
not directly rely on via the import system (i.e. they are standalone).
Desktop applications, command-line tools, etc. are examples of
applications.

A lock file records the packages that are to be installed for an app.
Traditionally, the exact version of the package to be installed is
specified by a lock file, but specified packages are not always
installed on a given platform (according a filtering logic described in
a later section), which enables the lock file to describe
reproducibility across multiple platforms. Examples of this are
package-lock.json from npm, Poetry.lock from Poetry, etc.

Locking is the act of taking the input of the packages an app depends on
and producing a lock file from that.

A locker is a tool which produces a lock file.

An installer consumes a lock file to install what the lock file
specifies.

Motivation

Applications want reproducible installs for a few reasons (we are not
worrying about package development, integration into larger systems that
would handle locking dependencies external to the Python application, or
other situations where flexible installation requirements are desired
over strict, reproducible installations).

One, reproducibility eases development. When you and your fellow
developers all end up with the same files on a specific platform, you
make sure you are all developing towards the same experience for the
application. You also want your users to install the same files as you
expect to guarantee the experience is the same as you developed for
them.

Two, you want to be able to reproduce what gets installed across
multiple platforms. Thanks to Python's portability across operating
systems, CPUs, etc., it is very easy and often desirable to create
applications that are not restricted to a single platform. Thus, you
want to be flexible enough to allow for differences in your package
dependencies between platforms, while still having consistency and
reproducibility on any one specific platform.

Three, reproducibility is more secure. When you control exactly what
files are installed, you can make sure no malicious actor is attempting
to slip nefarious code into your application (i.e. some supply chain
attacks). By using a lock file which always leads to reproducible
installs, we can avoid certain risks entirely.

Four, relying on the wheel file format provides reproducibility without
requiring build tools to support reproducibility themselves. Thanks to
wheels being static and not executing code as part of installation,
wheels always lead to a reproducible result. Compare this to source
distributions (aka sdists) or source trees which only lead to a
reproducible install if their build tool supports reproducibility due to
inherent code execution. Unfortunately the vast majority of build tools
do not support reproducible builds, so this PEP helps alleviate that
issue by only supporting wheels as a package format.

This PEP proposes a standard for a lock file, as the current solutions
don't meet the outlined goals. Today, the closest we come to a lock file
standard is the requirements file format from pip. Unfortunately, that
format does not lead to inherently reproducible installs (it requires
optional features both in the requirements file and the installer
itself, to be discussed later).

The community itself has also shown a need for lock files based on the
fact that multiple tools have independently created their own lock file
formats:

1.  PDM
2.  pip-tools
3.  Pipenv
4.  Poetry
5.  Pyflow

Unfortunately, those tools all use differing lock file formats. This
means tooling around these tools must be unique. This impacts tooling
such as code editors and hosting providers, which want to be as flexible
as possible when it comes to accepting a user's application code, but
also have a limit as to how much development resources they can spend to
add support for yet another lock file format. A standardized format
would allow tools to focus their work on a single target, and make sure
that workflow decisions made by developers outside of the lock file
format are of no concern to e.g. hosting providers.

Other programming language communities have also shown the usefulness of
lock files by developing their own solution to this problem. Some of
those communities include:

1.  Dart
2.  npm/Node
3.  Go
4.  Rust

The trend in programming languages in the past decade seems to have been
toward providing a lock file solution.

Rationale

File Format

We wanted the file format to be easy to read as a diff when auditing a
change to the lock file. As such, and thanks to PEP 518 and
pyproject.toml, we decided to go with the TOML file format.

Secure by Design

Viewing the requirements file format as the closest we have to a lock
file standard, there are a few issues with the file format when it comes
to security. First is that the file format simply does not require you
to specify the exact version of a package. This is why tools like
pip-tools exist to help manage that users of requirements files.

Second, you must opt into specifying what files are acceptable to be
installed by using the --hash argument for a specific dependency. This
is also optional with pip-tools as it requires specifying the
--generate-hashes CLI argument. This requires --require-hashes for pip
to make sure no dependencies lack a hash to check.

Third, even when you control what files may be installed, it does not
prevent other packages from being installed. If a dependency is not
listed in the requirements file, pip will happily go searching for a
file to meet that need. You must specify --no-deps as an argument to pip
to prevent unintended dependency resolution outside of the requirements
file.

Fourth, the format allows for installing a source distribution file (aka
"sdist"). By its very nature, installing an sdist requires executing
arbitrary Python code, meaning that there is no control over what files
may be installed. Only by specifying --only-binary :all: can you
guarantee pip to only use a wheel file for each package.

To recap, in order for a requirements file to be as secure as what is
being proposed, a user should always do the following steps:

1.  Use pip-tools and its command pip-compile --generate-hashes
2.  Install the requirements file using
    pip install --require-hashes --no-deps --only-binary :all:

Critically, all of those flags, and both the specificity and exhaustion
of what to install that pip-tools provides, are optional for
requirements files.

As such, the proposal raised in this PEP is secure by design which
combats some supply chain attacks. Hashes for files which would be used
to install from are required. You can only install from wheels to
unambiguously define what files will be placed in the file system.
Installers must lead to an deterministic installation from a lock file
for a given platform. All of this leads to a reproducible installation
which you can deem trustworthy (when you have audited the lock file and
what it lists).

Cross-Platform

Various projects which already have a lock file, like PDM and Poetry,
provide a lock file which is cross-platform. This allows for a single
lock file to work on multiple platforms while still leading to the exact
same top-level requirements to be installed everywhere with the
installation being consistent/unambiguous on each platform.

As to why this is useful, let's use an example involving PyWeek (a
week-long game development competition). Assume you are developing on
Linux, while someone you choose to partner with is using macOS. Now
assume the judges are using Windows. How do you make sure everyone is
using the same top-level dependencies, while allowing for any
platform-specific requirements (e.g. a package requires a helper package
under Windows)?

With a cross-platform lock file, you can make sure that the key
requirements are met consistently across all platforms. You can then
also make sure that all users on the same platform get the same
reproducible installation.

Simple Installer

The separation of concerns between a locker and an installer allows for
an installer to have a much simpler operation to perform. As such, it
not only allows for installers to be easier to write, but facilitates in
making sure installers create unambiguous, reproducible installations
correctly.

The installer can also expend less computation/energy in creating the
installation. This is beneficial not only for faster installs, but also
from an energy consumption perspective, as installers are expected to be
run more often than lockers.

This has led to a design where the locker must do more work upfront to
the benefit installers. It also means the complexity of package
dependencies is simpler and easier to comprehend in a lock files to
avoid ambiguity.

Specification

Details

Lock files MUST use the TOML file format. This not only prevents the
need to have another file format in the Python packaging ecosystem
thanks to its adoption by PEP 518 for pyproject.toml, but also assists
in making lock files more human-readable.

Lock files MUST end their file names with .pylock.toml. The .toml part
unambiguously distinguishes the format of the file, and helps tools like
code editors support the file appropriately. The .pylock part
distinguishes the file from other TOML files the user has, to make the
logic easier for tools to create functionality specific to Python lock
files, instead of TOML files in general.

The following sections are the top-level keys of the TOML file data
format. Any field not listed as required is considered optional.

version

This field is required.

The version of the lock file being used. The key MUST be a string
consisting of a number that follows the same formatting as the
Metadata-Version key in the core metadata spec.

The value MUST be set to "1.0" until a future PEP allows for a different
value. The introduction of a new optional key to the file format SHOULD
increase the minor version. The introduction of a new required key or
changing the format MUST increase the major version. How to handle other
scenarios is left as a per-PEP decision.

Installers MUST warn the user if the lock file specifies a version whose
major version is supported but whose minor version is
unsupported/unrecognized (e.g. the installer supports "1.0", but the
lock file specifies "1.1").

Installers MUST raise an error if the lock file specifies a major
version which is unsupported (e.g. the installer supports "1.9" but the
lock file specifies "2.0").

created-at

This field is required.

The timestamp for when the lock file was generated (using TOML's native
timestamp type). It MUST be recorded using the UTC time zone to avoid
ambiguity.

If the SOURCE_DATE_EPOCH environment variable is set, it MUST be used as
the timestamp by the locker. This facilitates reproducibility of the
lock file itself.

[tool]

Tools may create their own sub-tables under the tool table. The rules
for this table match those for pyproject.toml and its [tool] table from
the build system declaration spec.

[metadata]

This table is required.

A table containing data applying to the overall lock file.

metadata.marker

A key storing a string containing an environment marker as specified in
the dependency specifier spec.

The locker MAY specify an environment marker which specifies any
restrictions the lock file was generated under.

If the installer is installing for an environment which does not satisfy
the specified environment marker, the installer MUST raise an error as
the lock file does not support the target installation environment.

metadata.tag

A key storing a string specifying platform compatibility tags (i.e.
wheel tags). The tag MAY be a compressed tag set.

If the installer is installing for an environment which does not satisfy
the specified tag (set), the installer MUST raise an error as the lock
file does not support the targeted installation environment.

metadata.requires

This field is required.

An array of strings following the dependency specifier spec. This array
represents the top-level package dependencies of the lock file and thus
the root of the dependency graph.

metadata.requires-python

A string specifying the supported version(s) of Python for this lock
file. It follows the same format as that specified for the
Requires-Python field in the core metadata spec.

[[package._name_._version_]]

This array is required.

An array per package and version containing entries for the potential
(wheel) files to install (as represented by _name_ and _version_,
respectively).

Lockers MUST normalize a project's name according to the simple
repository API. If extras are specified as part of the project to
install, the extras are to be included in the key name and are to be
sorted in lexicographic order.

Within the file, the tables for the projects SHOULD be sorted by:

1.  Project/key name in lexicographic order
2.  Package version, newest/highest to older/lowest according to the
    version specifiers spec
3.  Optional dependencies (extras) via lexicographic order
4.  File name based on the filename field (discussed below)

These recommendations are to help minimize diff changes between tool
executions.

package._name_._version_.filename

This field is required.

A string representing the base name of the file as represented by an
entry in the array (i.e. what os.path.basename()/pathlib.PurePath.name
represents). This field is required to simplify installers as the file
name is required to resolve wheel tags derived from the file name. It
also guarantees that the association of the array entry to the file it
is meant for is always clear.

[package._name_._version_.hashes]

This table is required.

A table with keys specifying a hash algorithm and values as the hash for
the file represented by this entry in the package._name_._version_
table.

Lockers SHOULD list hashes in lexicographic order. This is to help
minimize diff sizes and the potential to overlook hash value changes.

An installer MUST only install a file which matches one of the specified
hashes.

package._name_._version_.url

A string representing a URL where to get the file.

The installer MAY support any schemes it wants for URLs. A URL with no
scheme MUST be assumed to be a local file path (both relative paths to
the lock file and absolute paths). Installers MUST support, at minimum,
HTTPS URLs as well as local file paths.

An installer MAY choose to not use the URL to retrieve a file if a file
matching the specified hash can be found using alternative means (e.g.
on the file system in a cache directory).

package._name_._version_.direct

A boolean representing whether an installer should consider the project
installed "directly" as specified by the direct URL origin of installed
distributions spec.

If the key is true, then the installer MUST follow the direct URL origin
of installed distributions spec for recording the installation as
"direct".

package._name_._version_.requires-python

A string specifying the support version(s) of Python for this file. It
follows the same format as that specified for the Requires-Python field
in the core metadata spec.

package._name_._version_.requires

An array of strings following the dependency specifier spec which
represent the dependencies of this file.

Example

    version = "1.0"
    created-at = 2021-10-19T22:33:45.520739+00:00

    [tool]
    # Tool-specific table.

    [metadata]
    requires = ["mousebender", "coveragepy[toml]"]
    marker = "sys_platform == 'linux'"  # As an example for coverage.
    requires-python = ">=3.7"

    [[package.attrs."21.2.0"]]
    filename = "attrs-21.2.0-py2.py3-none-any.whl"
    hashes.sha256 = "149e90d6d8ac20db7a955ad60cf0e6881a3f20d37096140088356da6c716b0b1"
    url = "https://files.pythonhosted.org/packages/20/a9/ba6f1cd1a1517ff022b35acd6a7e4246371dfab08b8e42b829b6d07913cc/attrs-21.2.0-py2.py3-none-any.whl"
    requires-python = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*"

    [[package.attrs."21.2.0"]]
    # If attrs had another wheel file (e.g. that was platform-specific),
    # it could be listed here.

    [[package."coveragepy[toml]"."6.2.0"]]
    filename = "coverage-6.2-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl"
    hashes.sha256 = "c7912d1526299cb04c88288e148c6c87c0df600eca76efd99d84396cfe00ef1d"
    url = "https://files.pythonhosted.org/packages/da/64/468ca923e837285bd0b0a60bd9a287945d6b68e325705b66b368c07518b1/coverage-6.2-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl"
    requires-python = ">=3.6"
    requires = ["tomli"]

    [[package."coveragepy[toml]"."6.2.0"]]
    filename = "coverage-6.2-cp310-cp310-musllinux_1_1_x86_64.whl "
    hashes.sha256 = "276651978c94a8c5672ea60a2656e95a3cce2a3f31e9fb2d5ebd4c215d095840"
    url = "https://files.pythonhosted.org/packages/17/d6/a29f2cccacf2315150c31d8685b4842a6e7609279939a478725219794355/coverage-6.2-cp310-cp310-musllinux_1_1_x86_64.whl"
    requires-python = ">=3.6"
    requires = ["tomli"]

    # More wheel files for `coverage` could be listed for more
    # extensive support (i.e. all Linux-based wheels).

    [[package.mousebender."2.0.0"]]
    filename = "mousebender-2.0.0-py3-none-any.whl"
    hashes.sha256 = "a6f9adfbd17bfb0e6bb5de9a27083e01dfb86ed9c3861e04143d9fd6db373f7c"
    url = "https://files.pythonhosted.org/packages/f4/b3/f6fdbff6395e9b77b5619160180489410fb2f42f41272994353e7ecf5bdf/mousebender-2.0.0-py3-none-any.whl"
    requires-python = ">=3.6"
    requires = ["attrs", "packaging"]

    [[package.packaging."20.9"]]
    filename = "packaging-20.9-py2.py3-none-any.whl"
    hashes.blake-256 = "3e897ea760b4daa42653ece2380531c90f64788d979110a2ab51049d92f408af"
    hashes.sha256 = "67714da7f7bc052e064859c05c595155bd1ee9f69f76557e21f051443c20947a"
    url = "https://files.pythonhosted.org/packages/3e/89/7ea760b4daa42653ece2380531c90f64788d979110a2ab51049d92f408af/packaging-20.9-py2.py3-none-any.whl"
    requires-python = ">=3.6"
    requires = ["pyparsing"]

    [[package.pyparsing."2.4.7"]]
    filename = "pyparsing-2.4.7-py2.py3-none-any.whl"
    hashes.sha256 = "ef9d7589ef3c200abe66653d3f1ab1033c3c419ae9b9bdb1240a85b024efc88b"
    url = "https://files.pythonhosted.org/packages/8a/bb/488841f56197b13700afd5658fc279a2025a39e22449b7cf29864669b15d/pyparsing-2.4.7-py2.py3-none-any.whl"
    direct = true  # For demonstration purposes.
    requires-python = ">=2.6, !=3.0.*, !=3.1.*, !=3.2.*"

    [[package.tomli."2.0.0"]]
    filename = "tomli-2.0.0-py3-none-any.whl"
    hashes.sha256 = "b5bde28da1fed24b9bd1d4d2b8cba62300bfb4ec9a6187a957e8ddb9434c5224"
    url = "https://files.pythonhosted.org/packages/e2/9f/5e1557a57a7282f066351086e78f87289a3446c47b2cb5b8b2f614d8fe99/tomli-2.0.0-py3-none-any.whl"
    requires-python = ">=3.7"

Expectations for Lockers

Lockers MUST create lock files for which a topological sort of the
packages which qualify for installation on the specified platform
results in a graph for which only a single version of any package
qualifies for installation and there is at least one compatible file to
install for each package. This leads to a lock file for any supported
platform where the only decision an installer can make is what the
"best-fitting" wheel is to install (which is discussed below).

Lockers are expected to utilize metadata.marker, metadata.tag, and
metadata.requires-python as appropriate as well as environment markers
specified via requires and Python version requirements via
requires-python to enforce this result for installers. Put another way,
the information used in the lock file is not expected to be pristine/raw
from the locker's input and instead is to be changed as necessary to the
benefit of the locker's goals.

Expectations for Installers

The expected algorithm for resolving what to install is:

1.  Construct a dependency graph based on the data in the lock file with
    metadata.requires as the starting/root point.
2.  Eliminate all files that are unsupported by the specified platform.
3.  Eliminate all irrelevant edges between packages based on marker
    evaluation for requires.
4.  Raise an error if a package version is still reachable from the root
    of the dependency graph but lacks any compatible file.
5.  Verify that all packages left only have one version to install,
    raising an error otherwise.
6.  Install the best-fitting wheel file for each package which remains.

Installers MUST follow a deterministic algorithm determine what the
"best-fitting wheel file" is. A simple solution for this is to rely upon
the packaging project and its packaging.tags module to determine wheel
file precedence.

Installers MUST support installing into an empty environment. Installers
MAY support installing into an environment that already contains
installed packages (and whatever that would entail to be supported).

(Potential) Tool Support

The pip team has said they are interested in supporting this PEP if
accepted. The current proposal for pip may even supplant the need for
pip-tools.

PDM has also said they would support the PEP if accepted.

Pyflow has said they "like the idea" of the PEP.

Poetry has said they would not support the PEP as-is because "Poetry
supports sdists files, directory and VCS dependencies which are not
supported". Recording requirements at the file level, which is on
purpose to better reflect what can occur when it comes to dependencies,
"is contradictory to the design of Poetry". This also excludes export
support to a this PEP's lock file as "Poetry exports the information
present in the poetry.lock file into another format" and sdists and
source trees are included in Poetry.lock files. Thus it is not a clean
translation from Poetry's lock file to this PEP's lock file format.

Backwards Compatibility

As there is no pre-existing specification regarding lock files, there
are no explicit backwards compatibility concerns.

As for pre-existing tools that have their own lock file, some updating
will be required. Most document the lock file name, but not its
contents. For projects which do not commit their lock file to version
control, they will need to update the equivalent of their .gitignore
file. For projects that do commit their lock file to version control,
what file(s) get committed will need an update.

For projects which do document their lock file format like pipenv, they
will very likely need a major version release which changes the lock
file format.

Transition Plan

In general, this PEP could be considered successful if:

1.  Two pre-existing tools became lockers (e.g. pip-tools, PDM, pip via
    pip freeze).
2.  Pip became an installer.
3.  One major, non-Python-specific platform supported the file format
    (e.g. a cloud provider).

This would show interoperability, usability, and programming
community/business acceptance.

In terms of a transition plan, there are potentially multiple steps that
could lead to this desired outcome. Below is a somewhat idealized plan
that would see this PEP being broadly used.

Usability

First, a pip freeze equivalent tool could be developed which creates a
lock file. While installed packages do not by themselves provide enough
information to statically create a lock file, a user could provide local
directories and index URLs to construct one. This would then lead to
lock files that are stricter than a requirements file by limiting the
lock file to the current platform. This would also allow people to see
whether their environment would be reproducible.

Second, a stand-alone installer should be developed. As the requirements
on an installer are much simpler than what pip provides, it should be
reasonable to have an installer that is independently developed.

Third, a tool to convert a pinned requirements file as emitted by
pip-tools could be developed. Much like the pip freeze equivalent
outlined above, some input from the user may be needed. But this tool
could act as a transitioning step for anyone who has an appropriate
requirements file. This could also act as a test before potentially
having pip-tools grow some --lockfile flag to use this PEP.

All of this could be required before the PEP transitions from
conditional acceptance to full acceptance (and give the community a
chance to test if this PEP is potentially useful).

Interoperability

At this point, the goal would be to increase interoperability between
tools.

First, pip would become an installer. By having the most widely used
installer support the format, people can innovate on the locker side
while knowing people will have the tools necessary to actually consume a
lock file.

Second, pip becomes a locker. Once again, pip's reach would make the
format accessible for the vast majority of Python users very quickly.

Third, a project with a pre-existing lock file format supports at least
exporting to the lock file format (e.g. PDM or Pyflow). This would show
that the format meets the needs of other projects.

Acceptance

With the tooling available throughout the community, acceptance would be
shown via those not exclusively tied to the Python community supporting
the file format based on what they believe their users want.

First, tools that operate on requirements files like code editors having
equivalent support for lock files.

Second, consumers of requirements files like cloud providers would also
accept lock files.

At this point the PEP would have permeated out far enough to be on par
with requirements files in terms of general acceptance and potentially
more if projects had dropped their own lock files for this PEP.

Security Implications

A lock file should not introduce security issues but instead help solve
them. By requiring the recording of hashes for files, a lock file is
able to help prevent tampering with code since the hash details were
recorded. Relying on only wheel files means what files will be installed
can be known ahead of time and is reproducible. A lock file also helps
prevent unexpected package updates being installed which may in turn be
malicious.

How to Teach This

Teaching of this PEP will very much be dependent on the lockers and
installers being used for day-to-day use. Conceptually, though, users
could be taught that a lock file specifies what should be installed for
a project to work. The benefits of consistency and security should be
emphasized to help users realize why they should care about lock files.

Reference Implementation

A proof-of-concept locker can be found at
https://github.com/frostming/pep665_poc . No installer has been
implemented yet, but the design of this PEP suggests the locker is the
more difficult aspect to implement.

Rejected Ideas

File Formats Other Than TOML

JSON was briefly considered, but due to:

1.  TOML already being used for pyproject.toml
2.  TOML being more human-readable
3.  TOML leading to better diffs

the decision was made to go with TOML. There was some concern over
Python's standard library lacking a TOML parser, but most packaging
tools already use a TOML parser thanks to pyproject.toml so this issue
did not seem to be a showstopper. Some have also argued against this
concern in the past by the fact that if packaging tools abhor installing
dependencies and feel they can't vendor a package then the packaging
ecosystem has much bigger issues to rectify than the need to depend on a
third-party TOML parser.

Alternative Naming Schemes

Specifying a directory to install file to was considered, but ultimately
rejected due to people's distaste for the idea.

It was also suggested to not have a special file name suffix, but it was
decided that hurt discoverability by tools too much.

Supporting a Single Lock File

At one point the idea of only supporting single lock file which
contained all possible lock information was considered. But it quickly
became apparent that trying to devise a data format which could
encompass both a lock file format which could support multiple
environments as well as strict lock outcomes for reproducible builds
would become quite complex and cumbersome.

The idea of supporting a directory of lock files as well as a single
lock file named pyproject-lock.toml was also considered. But any
possible simplicity from skipping the directory in the case of a single
lock file seemed unnecessary. Trying to define appropriate logic for
what should be the pyproject-lock.toml file and what should go into
pyproject-lock.d seemed unnecessarily complicated.

Using a Flat List Instead of a Dependency Graph

The first version of this PEP proposed that the lock file have no
concept of a dependency graph. Instead, the lock file would list exactly
what should be installed for a specific platform such that installers
did not have to make any decisions about what to install, only
validating that the lock file would work for the target platform.

This idea was eventually rejected due to the number of combinations of
potential PEP 508 environment markers. The decision was made that trying
to have lockers generate all possible combinations as individual lock
files when a project wants to be cross-platform would be too much.

Use Wheel Tags in the File Name

Instead of having the metadata.tag field there was a suggestion of
encoding the tags into the file name. But due to the addition of the
metadata.marker field and what to do when no tags were needed, the idea
was dropped.

Alternative Names for requires

Some other names for what became requires were installs, needs, and
dependencies. Initially this PEP chose needs after asking a Python
beginner which term they preferred. But based on feedback on an earlier
draft of this PEP, requires was chosen as the term.

Accepting PEP 650

PEP 650 was an earlier attempt at trying to tackle this problem by
specifying an API for installers instead of standardizing on a lock file
format (ala PEP 517). The initial response to PEP 650 could be
considered mild/lukewarm. People seemed to be consistently confused over
which tools should provide what functionality to implement the PEP. It
also potentially incurred more overhead as it would require executing
Python APIs to perform any actions involving packaging.

This PEP chooses to standardize around an artifact instead of an API
(ala PEP 621). This would allow for more tool integrations as it removes
the need to specifically use Python to do things such as create a lock
file, update it, or even install packages listed in a lock file. It also
allows for easier introspection by forcing dependency graph details to
be written in a human-readable format. It also allows for easier sharing
of knowledge by standardizing what people need to know more (e.g.
tutorials become more portable between tools when it comes to
understanding the artifact they produce). It's also simply the approach
other language communities have taken and seem to be happy with.

Acceptance of this PEP would mean PEP 650 gets rejected.

Specifying Requirements per Package Instead of per File

An earlier draft of this PEP specified dependencies at the package level
instead of per file. While this has traditionally been how packaging
systems work, it actually did not reflect accurately how things are
specified. As such, this PEP was subsequently updated to reflect the
granularity that dependencies can truly be specified at.

Specify Where Lockers Gather Input

This PEP does not specify how a locker gets its input. An initial
suggestion was to partially reuse PEP 621, but due to disagreements on
how flexible the potential input should be in terms of specifying things
such as indexes, etc., it was decided this would best be left to a
separate PEP.

Allowing Source Distributions and Source Trees to be an Opt-In, Supported File Format

After extensive discussion, it was decided that this PEP would not
support source distributions (aka sdists) or source trees as an
acceptable format for code. Introducing sdists and source trees to this
PEP would immediately undo the reproducibility and security goals due to
needing to execute code to build the sdist or source tree. It would also
greatly increase the complexity for (at least) installers as the dynamic
build nature of sdists and source trees means the installer would need
to handle fully resolving whatever requirements the sdists produced
dynamically, both from a building and installation perspective.

Due to all of this, it was decided it was best to have a separate
discussion about what supporting sdists and source trees after this PEP
is accepted/rejected. As the proposed file format is versioned,
introducing sdists and source tree support in a later PEP is doable.

It should be noted, though, that this PEP is not stop an out-of-band
solution from being developed to be used in conjunction with this PEP.
Building wheel files from sdists and shipping them with code upon
deployment so they can be included in the lock file is one option.
Another is to use a requirements file just for sdists and source trees,
then relying on a lock file for all wheels.

Open Issues

None.

Acknowledgments

Thanks to Frost Ming of PDM and Sébastien Eustace of Poetry for
providing input around dynamic install-time resolution of PEP 508
requirements.

Thanks to Kushal Das for making sure reproducible builds stayed a
concern for this PEP.

Thanks to Andrea McInnes for initially settling the bikeshedding and
choosing the paint colour of needs (at which point people ralled around
the requires colour instead).

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.