text:
PEP: 532 Title: A circuit breaking protocol and binary operators
Version: $Revision$ Last-Modified: $Date$ Author: Alyssa Coghlan
<EMAIL_ADDRESS>Mark E. Haase<EMAIL_ADDRESS>Status: Deferred
Type: Standards Track Content-Type: text/x-rst Created: 30-Oct-2016
Python-Version: 3.8 Post-History: 05-Nov-2016

PEP Deferral

Further consideration of this PEP has been deferred until Python 3.8 at
the earliest.

Abstract

Inspired by PEP 335, PEP 505, PEP 531, and the related discussions, this
PEP proposes the definition of a new circuit breaking protocol (using
the method names __then__ and __else__) that provides a common
underlying semantic foundation for:

-   conditional expressions: LHS if COND else RHS
-   logical conjunction: LHS and RHS
-   logical disjunction: LHS or RHS
-   the None-aware operators proposed in PEP 505
-   the rich comparison chaining model proposed in PEP 535

Taking advantage of the new protocol, it further proposes that the
definition of conditional expressions be revised to also permit the use
of if and else respectively as right-associative and left-associative
general purpose short-circuiting operators:

-   Right-associative short-circuiting: LHS if RHS
-   Left-associative short-circuiting: LHS else RHS

In order to make logical inversion (not EXPR) consistent with the above
changes, it also proposes the introduction of a new logical inversion
protocol (using the method name __not__).

To force short-circuiting of a circuit breaker without having to
evaluate the expression creating it twice, a new
operator.short_circuit(obj) helper function will be added to the
operator module.

Finally, a new standard types.CircuitBreaker type is proposed to
decouple an object's truth value (as used to determine control flow)
from the value it returns from short-circuited circuit breaking
expressions, with the following factory functions added to the operator
module to represent particularly common switching idioms:

-   switching on bool(obj): operator.true(obj)
-   switching on not bool(obj): operator.false(obj)
-   switching on obj is value: operator.is_sentinel(obj, value)
-   switching on obj is not value: operator.is_not_sentinel(obj, value)

Relationship with other PEPs

This PEP builds on an extended history of work in other proposals. Some
of the key proposals are discussed below.

PEP 531: Existence checking protocol

This PEP is a direct successor to PEP 531, replacing the existence
checking protocol and the new ?then and ?else syntactic operators
defined there with the new circuit breaking protocol and adjustments to
conditional expressions and the not operator.

PEP 505: None-aware operators

This PEP complements the None-aware operator proposals in PEP 505, by
offering an underlying protocol-driven semantic framework that explains
their short-circuiting behaviour as highly optimised syntactic sugar for
particular uses of conditional expressions.

Given the changes proposed by this PEP:

-   LHS ?? RHS would roughly be is_not_sentinel(LHS, None) else RHS
-   EXPR?.attr would roughly be EXPR.attr if is_not_sentinel(EXPR, None)
-   EXPR?[key] would roughly be EXPR[key] if is_not_sentinel(EXPR, None)

In all three cases, the dedicated syntactic form would be optimised to
avoid actually creating the circuit breaker instance and instead
implement the underlying control flow directly. In the latter two cases,
the syntactic form would also avoid evaluating EXPR twice.

This means that while the None-aware operators would remain highly
specialised and specific to None, other sentinel values would still be
usable through the more general protocol-driven proposal in this PEP.

PEP 335: Overloadable Boolean operators

PEP 335 proposed the ability to overload the short-circuiting and and or
operators directly, with the ability to overload the semantics of
comparison chaining being one of the consequences of that change. The
proposal in an earlier version of this PEP to instead handle the
element-wise comparison use case by changing the semantic definition of
comparison chaining is drawn directly from Guido's rejection of PEP
335[1].

However, initial feedback on this PEP indicated that the number of
different proposals that it covered made it difficult to read, so that
part of the proposal has been separated out as PEP 535.

PEP 535: Rich comparison chaining

As noted above, PEP 535 is a proposal to build on the circuit breaking
protocol defined in this PEP in order to expand the rich comparison
support introduced in PEP 207 to also handle comparison chaining
operations like LEFT_BOUND < VALUE < RIGHT_BOUND.

Specification

The circuit breaking protocol (if-else)

Conditional expressions (LHS if COND else RHS) are currently interpreted
as an expression level equivalent to:

    if COND:
        _expr_result = LHS
    else:
        _expr_result = RHS

This PEP proposes changing that expansion to allow the checked condition
to implement a new "circuit breaking" protocol that allows it to see,
and potentially alter, the result of either or both branches of the
expression:

    _cb = COND
    _type_cb = type(cb)
    if _cb:
        _expr_result = LHS
        if hasattr(_type_cb, "__then__"):
            _expr_result = _type_cb.__then__(_cb, _expr_result)
    else:
        _expr_result = RHS
        if hasattr(_type_cb, "__else__"):
            _expr_result = _type_cb.__else__(_cb, _expr_result)

As shown, interpreter implementations would be required to access only
the protocol method needed for the branch of the conditional expression
that is actually executed. Consistent with other protocol methods, the
special methods would be looked up via the circuit breaker's type,
rather than directly on the instance.

Circuit breaking operators (binary if and binary else)

The proposed name of the protocol doesn't come from the proposed changes
to the semantics of conditional expressions. Rather, it comes from the
proposed addition of if and else as general purpose protocol driven
short-circuiting operators to complement the existing True and False
based short-circuiting operators (or and and, respectively) as well as
the None based short-circuiting operator proposed in PEP 505 (??).

Together, these two operators would be known as the circuit breaking
operators.

In order to support this usage, the definition of conditional
expressions in the language grammar would be updated to make both the if
clause and the else clause optional:

    test: else_test ['if' or_test ['else' test]] | lambdef
    else_test: or_test ['else' test]

Note that we would need to avoid the apparent simplification to
else_test ('if' else_test)* in order to make it easier for compiler
implementations to correctly preserve the semantics of normal
conditional expressions.

The definition of the test_nocond node in the grammar (which
deliberately excludes conditional expressions) would remain unchanged,
so the circuit breaking operators would require parentheses when used in
the if clause of comprehensions and generator expressions just as
conditional expressions themselves do.

This grammar definition means precedence/associativity in the otherwise
ambiguous case of expr1 if cond else expr2 else expr3 resolves as
(expr1 if cond else expr2) else epxr3. However, a guideline will also be
added to PEP 8 to say "don't do that", as such a construct will be
inherently confusing for readers, regardless of how the interpreter
executes it.

The right-associative circuit breaking operator (LHS if RHS) would then
be expanded as follows:

    _cb = RHS
    _expr_result = LHS if _cb else _cb

While the left-associative circuit breaking operator (LHS else RHS)
would be expanded as:

    _cb = LHS
    _expr_result = _cb if _cb else RHS

The key point to note in both cases is that when the circuit breaking
expression short-circuits, the condition expression is used as the
result of the expression unless the condition is a circuit breaker. In
the latter case, the appropriate circuit breaker protocol method is
called as usual, but the circuit breaker itself is supplied as the
method argument.

This allows circuit breakers to reliably detect short-circuiting by
checking for cases when the argument passed in as the candidate
expression result is self.

Overloading logical inversion (not)

Any circuit breaker definition will have a logical inverse that is still
a circuit breaker, but inverts the answer as to when to short circuit
the expression evaluation. For example, the operator.true and
operator.false circuit breakers proposed in this PEP are each other's
logical inverse.

A new protocol method, __not__(self), will be introduced to permit
circuit breakers and other types to override not expressions to return
their logical inverse rather than a coerced boolean result.

To preserve the semantics of existing language optimisations (such as
eliminating double negations directly in a boolean context as
redundant), __not__ implementations will be required to respect the
following invariant:

    assert not bool(obj) == bool(not obj)

However, symmetric circuit breakers (those that implement all of
__bool__, __not__, __then__ and __else__) would only be expected to
respect the full semantics of boolean logic when all circuit breakers
involved in the expression are using a consistent definition of "truth".
This is covered further in Respecting De Morgan's Laws.

Forcing short-circuiting behaviour

Invocation of a circuit breaker's short-circuiting behaviour can be
forced by using it as all three operands in a conditional expression:

    obj if obj else obj

Or, equivalently, as both operands in a circuit breaking expression:

    obj if obj
    obj else obj

Rather than requiring the using of any of these patterns, this PEP
proposes to add a dedicated function to the operator to explicitly
short-circuit a circuit breaker, while passing other objects through
unmodified:

    def short_circuit(obj)
        """Replace circuit breakers with their short-circuited result

        Passes other input values through unmodified.
        """
        return obj if obj else obj

Circuit breaking identity comparisons (is and is not)

In the absence of any standard circuit breakers, the proposed if and
else operators would largely just be unusual spellings of the existing
and and or logical operators.

However, this PEP further proposes to provide a new general purpose
types.CircuitBreaker type that implements the appropriate short
circuiting logic, as well as factory functions in the operator module
that correspond to the is and is not operators.

These would be defined in such a way that the following expressions
produce VALUE rather than False when the conditional check fails:

    EXPR if is_sentinel(VALUE, SENTINEL)
    EXPR if is_not_sentinel(VALUE, SENTINEL)

And similarly, these would produce VALUE rather than True when the
conditional check succeeds:

    is_sentinel(VALUE, SENTINEL) else EXPR
    is_not_sentinel(VALUE, SENTINEL) else EXPR

In effect, these comparisons would be defined such that the leading
VALUE if and trailing else VALUE clauses can be omitted as implied in
expressions of the following forms:

    # To handle "if" expressions, " else VALUE" is implied when omitted
    EXPR if is_sentinel(VALUE, SENTINEL) else VALUE
    EXPR if is_not_sentinel(VALUE, SENTINEL) else VALUE
    # To handle "else" expressions, "VALUE if " is implied when omitted
    VALUE if is_sentinel(VALUE, SENTINEL) else EXPR
    VALUE if is_not_sentinel(VALUE, SENTINEL) else EXPR

The proposed types.CircuitBreaker type would represent this behaviour
programmatically as follows:

    class CircuitBreaker:
        """Simple circuit breaker type"""
        def __init__(self, value, bool_value):
            self.value = value
            self.bool_value = bool(bool_value)
        def __bool__(self):
            return self.bool_value
        def __not__(self):
            return CircuitBreaker(self.value, not self.bool_value)
        def __then__(self, result):
            if result is self:
                return self.value
            return result
        def __else__(self, result):
            if result is self:
                return self.value
            return result

The key characteristic of these circuit breakers is that they are
ephemeral: when they are told that short circuiting has taken place (by
receiving a reference to themselves as the candidate expression result),
they return the original value, rather than the circuit breaking
wrapper.

The short-circuiting detection is defined such that the wrapper will
always be removed if you explicitly pass the same circuit breaker
instance to both sides of a circuit breaking operator or use one as all
three operands in a conditional expression:

    breaker = types.CircuitBreaker(foo, foo is None)
    assert operator.short_circuit(breaker) is foo
    assert (breaker if breaker) is foo
    assert (breaker else breaker) is foo
    assert (breaker if breaker else breaker) is foo
    breaker = types.CircuitBreaker(foo, foo is not None)
    assert operator.short_circuit(breaker) is foo
    assert (breaker if breaker) is foo
    assert (breaker else breaker) is foo
    assert (breaker if breaker else breaker) is foo

The factory functions in the operator module would then make it
straightforward to create circuit breakers that correspond to identity
checks using the is and is not operators:

    def is_sentinel(value, sentinel):
        """Returns a circuit breaker switching on 'value is sentinel'"""
        return types.CircuitBreaker(value, value is sentinel)

    def is_not_sentinel(value, sentinel):
        """Returns a circuit breaker switching on 'value is not sentinel'"""
        return types.CircuitBreaker(value, value is not sentinel)

Truth checking comparisons

Due to their short-circuiting nature, the runtime logic underlying the
and and or operators has never previously been accessible through the
operator or types modules.

The introduction of circuit breaking operators and circuit breakers
allows that logic to be captured in the operator module as follows:

    def true(value):
        """Returns a circuit breaker switching on 'bool(value)'"""
        return types.CircuitBreaker(value, bool(value))

    def false(value):
        """Returns a circuit breaker switching on 'not bool(value)'"""
        return types.CircuitBreaker(value, not bool(value))

-   LHS or RHS would be effectively true(LHS) else RHS
-   LHS and RHS would be effectively false(LHS) else RHS

No actual change would take place in these operator definitions, the new
circuit breaking protocol and operators would just provide a way to make
the control flow logic programmable, rather than hardcoding the sense of
the check at development time.

Respecting the rules of boolean logic, these expressions could also be
expanded in their inverted form by using the right-associative circuit
breaking operator instead:

-   LHS or RHS would be effectively RHS if false(LHS)
-   LHS and RHS would be effectively RHS if true(LHS)

None-aware operators

If both this PEP and PEP 505's None-aware operators were accepted, then
the proposed is_sentinel and is_not_sentinel circuit breaker factories
would be used to encapsulate the notion of "None checking": seeing if a
value is None and either falling back to an alternative value (an
operation known as "None-coalescing") or passing it through as the
result of the overall expression (an operation known as "None-severing"
or "None-propagating").

Given these circuit breakers, LHS ?? RHS would be roughly equivalent to
both of the following:

-   is_not_sentinel(LHS, None) else RHS
-   RHS if is_sentinel(LHS, None)

Due to the way they inject control flow into attribute lookup and
subscripting operations, None-aware attribute access and None-aware
subscripting can't be expressed directly in terms of the circuit
breaking operators, but they can still be defined in terms of the
underlying circuit breaking protocol.

In those terms, EXPR?.ATTR[KEY].SUBATTR() would be semantically
equivalent to:

    _lookup_base = EXPR
    _circuit_breaker = is_not_sentinel(_lookup_base, None)
    _expr_result = _lookup_base.ATTR[KEY].SUBATTR() if _circuit_breaker

Similarly, EXPR?[KEY].ATTR.SUBATTR() would be semantically equivalent
to:

    _lookup_base = EXPR
    _circuit_breaker = is_not_sentinel(_lookup_base, None)
    _expr_result = _lookup_base[KEY].ATTR.SUBATTR() if _circuit_breaker

The actual implementations of the None-aware operators would presumably
be optimised to skip actually creating the circuit breaker instance, but
the above expansions would still provide an accurate description of the
observable behaviour of the operators at runtime.

Rich chained comparisons

Refer to PEP 535 for a detailed discussion of this possible use case.

Other conditional constructs

No changes are proposed to if statements, while statements,
comprehensions, or generator expressions, as the boolean clauses they
contain are used entirely for control flow purposes and never return a
result as such.

However, it's worth noting that while such proposals are outside the
scope of this PEP, the circuit breaking protocol defined here would
already be sufficient to support constructs like:

    def is_not_none(obj):
        return is_sentinel(obj, None)

    while is_not_none(dynamic_query()) as result:
        ... # Code using result

and:

    if is_not_none(re.search(pattern, text)) as match:
        ... # Code using match

This could be done by assigning the result of
operator.short_circuit(CONDITION) to the name given in the as clause,
rather than assigning CONDITION to the given name directly.

Style guide recommendations

The following additions to PEP 8 are proposed in relation to the new
features introduced by this PEP:

-   Avoid combining conditional expressions (if-else) and the standalone
    circuit breaking operators (if and else) in a single expression -use
    one or the other depending on the situation, but not both.
-   Avoid using conditional expressions (if-else) and the standalone
    circuit breaking operators (if and else) as part of if conditions in
    if statements and the filter clauses of comprehensions and generator
    expressions.

Rationale

Adding new operators

Similar to PEP 335, early drafts of this PEP focused on making the
existing and and or operators less rigid in their interpretation, rather
than proposing new operators. However, this proved to be problematic for
a few key reasons:

-   the and and or operators have a long established and stable meaning,
    so readers would inevitably be surprised if their meaning now became
    dependent on the type of the left operand. Even new users would be
    confused by this change due to 25+ years of teaching material that
    assumes the current well-known semantics for these operators
-   Python interpreter implementations, including CPython, have taken
    advantage of the existing semantics of and and or when defining
    runtime and compile time optimisations, which would all need to be
    reviewed and potentially discarded if the semantics of those
    operations changed
-   it isn't clear what names would be appropriate for the new methods
    needed to define the protocol

Proposing short-circuiting binary variants of the existing if-else
ternary operator instead resolves all of those issues:

-   the runtime semantics of and and or remain entirely unchanged
-   while the semantics of the unary not operator do change, the
    invariant required of __not__ implementations means that existing
    expression optimisations in boolean contexts will remain valid.
-   __else__ is the short-circuiting outcome for if expressions due to
    the absence of a trailing else clause
-   __then__ is the short-circuiting outcome for else expressions due to
    the absence of a leading if clause (this connection would be even
    clearer if the method name was __if__, but that would be ambiguous
    given the other uses of the if keyword that won't invoke the circuit
    breaking protocol)

Naming the operator and protocol

The names "circuit breaking operator", "circuit breaking protocol" and
"circuit breaker" are all inspired by the phrase "short circuiting
operator": the general language design term for operators that only
conditionally evaluate their right operand.

The electrical analogy is that circuit breakers in Python detect and
handle short circuits in expressions before they trigger any exceptions
similar to the way that circuit breakers detect and handle short
circuits in electrical systems before they damage any equipment or harm
any humans.

The Python level analogy is that just as a break statement lets you
terminate a loop before it reaches its natural conclusion, a circuit
breaking expression lets you terminate evaluation of the expression and
produce a result immediately.

Using existing keywords

Using existing keywords has the benefit of allowing the new operators to
be introduced without a __future__ statement.

if and else are semantically appropriate for the proposed new protocol,
and the only additional syntactic ambiguity introduced arises when the
new operators are combined with the explicit if-else conditional
expression syntax.

The PEP handles that ambiguity by explicitly specifying how it should be
handled by interpreter implementers, but proposing to point out in PEP 8
that even though interpreters will understand it, human readers probably
won't, and hence it won't be a good idea to use both conditional
expressions and the circuit breaking operators in a single expression.

Naming the protocol methods

Naming the __else__ method was straightforward, as reusing the operator
keyword name results in a special method name that is both obvious and
unambiguous.

Naming the __then__ method was less straightforward, as there was
another possible option in using the keyword-based name __if__.

The problem with __if__ is that there would continue to be many cases
where the if keyword appeared, with an expression to its immediate
right, but the __if__ special method would not be invoked. Instead, the
bool() builtin and its underlying special methods (__bool__, __len__)
would be invoked, while __if__ had no effect.

With the boolean protocol already playing a part in conditional
expressions and the new circuit breaking protocol, the less ambiguous
name __then__ was chosen based on the terminology commonly used in
computer science and programming language design to describe the first
clause of an if statement.

Making binary if right-associative

The precedent set by conditional expressions means that a binary
short-circuiting if expression must necessarily have the condition on
the right as a matter of consistency.

With the right operand always being evaluated first, and the left
operand not being evaluated at all if the right operand is true in a
boolean context, the natural outcome is a right-associative operator.

Naming the standard circuit breakers

When used solely with the left-associative circuit breaking operator,
explicit circuit breaker names for unary checks read well if they start
with the preposition if_:

    operator.if_true(LHS) else RHS
    operator.if_false(LHS) else RHS

However, incorporating the if_ doesn't read as well when performing
logical inversion:

    not operator.if_true(LHS) else RHS
    not operator.if_false(LHS) else RHS

Or when using the right-associative circuit breaking operator:

    LHS if operator.if_true(RHS)
    LHS if operator.if_false(RHS)

Or when naming a binary comparison operation:

    operator.if_is_sentinel(VALUE, SENTINEL) else EXPR
    operator.if_is_not_sentinel(VALUE, SENTINEL) else EXPR

By contrast, omitting the preposition from the circuit breaker name
gives a result that reads reasonably well in all forms for unary checks:

    operator.true(LHS) else RHS       # Preceding "LHS if " implied
    operator.false(LHS) else RHS      # Preceding "LHS if " implied
    not operator.true(LHS) else RHS   # Preceding "LHS if " implied
    not operator.false(LHS) else RHS  # Preceding "LHS if " implied
    LHS if operator.true(RHS)         # Trailing " else RHS" implied
    LHS if operator.false(RHS)        # Trailing " else RHS" implied
    LHS if not operator.true(RHS)     # Trailing " else RHS" implied
    LHS if not operator.false(RHS)    # Trailing " else RHS" implied

And also reads well for binary checks:

    operator.is_sentinel(VALUE, SENTINEL) else EXPR
    operator.is_not_sentinel(VALUE, SENTINEL) else EXPR
    EXPR if operator.is_sentinel(VALUE, SENTINEL)
    EXPR if operator.is_not_sentinel(VALUE, SENTINEL)

Risks and concerns

This PEP has been designed specifically to address the risks and
concerns raised when discussing PEPs 335, 505 and 531.

-   it defines new operators and adjusts the definition of chained
    comparison (in a separate PEP) rather than impacting the existing
    and and or operators
-   the proposed new operators are general purpose short-circuiting
    binary operators that can even be used to express the existing
    semantics of and and or rather than focusing solely and inflexibly
    on identity checking against None
-   the changes to the not unary operator and the is and is not binary
    comparison operators are defined in such a way that control flow
    optimisations based on the existing semantics remain valid

One consequence of this approach is that this PEP on its own doesn't
produce much in the way of direct benefits to end users aside from
making it possible to omit some common None if prefixes and else None
suffixes from particular forms of conditional expression.

Instead, what it mainly provides is a common foundation that would allow
the None-aware operator proposals in PEP 505 and the rich comparison
chaining proposal in PEP 535 to be pursued atop a common underlying
semantic framework that would also be shared with conditional
expressions and the existing and and or operators.

Design Discussion

Protocol walk-through

The following diagram illustrates the core concepts behind the circuit
breaking protocol (although it glosses over the technical detail of
looking up the special methods via the type rather than the instance):

[diagram of circuit breaking protocol applied to ternary expression]

We will work through the following expression:

    >>> def is_not_none(obj):
    ...     return operator.is_not_sentinel(obj, None)
    >>> x if is_not_none(data.get("key")) else y

is_not_none is a helper function that invokes the proposed
operator.is_not_sentinel types.CircuitBreaker factory with None as the
sentinel value. data is a container (such as a builtin dict instance)
that returns None when the get() method is called with an unknown key.

We can rewrite the example to give a name to the circuit breaker
instance:

    >>> maybe_value = is_not_none(data.get("key"))
    >>> x if maybe_value else y

Here the maybe_value circuit breaker instance corresponds to breaker in
the diagram.

The ternary condition is evaluated by calling bool(maybe_value), which
is the same as Python's existing behavior. The change in behavior is
that instead of directly returning one of the operands x or y, the
circuit breaking protocol passes the relevant operand to the circuit
breaker used in the condition.

If bool(maybe_value) evaluates to True (i.e. the requested key exists
and its value is not None) then the interpreter calls
type(maybe_value).__then__(maybe_value, x). Otherwise, it calls
type(maybe_value).__else__(maybe_value, y).

The protocol also applies to the new if and else binary operators, but
in these cases, the interpreter needs a way to indicate the missing
third operand. It does this by re-using the circuit breaker itself in
that role.

Consider these two expressions:

    >>> x if data.get("key") is None
    >>> x if operator.is_sentinel(data.get("key"), None)

The first form of this expression returns x if data.get("key") is None,
but otherwise returns False, which almost certainly isn't what we want.

By contrast, the second form of this expression still returns x if
data.get("key") is None, but otherwise returns data.get("key"), which is
significantly more useful behaviour.

We can understand this behavior by rewriting it as a ternary expression
with an explicitly named circuit breaker instance:

    >>> maybe_value = operator.is_sentinel(data.get("key"), None)
    >>> x if maybe_value else maybe_value

If bool(maybe_value) is True (i.e. data.get("key") is None), then the
interpreter calls type(maybe_value).__then__(maybe_value, x). The
implementation of types.CircuitBreaker.__then__ doesn't see anything
that indicates short-circuiting has taken place, and hence returns x.

By contrast, if bool(maybe_value) is False (i.e. data.get("key") is not
None), the interpreter calls
type(maybe_value).__else__(maybe_value, maybe_value). The implementation
of types.CircuitBreaker.__else__ detects that the instance method has
received itself as its argument and returns the wrapped value (i.e.
data.get("key")) rather than the circuit breaker.

The same logic applies to else, only reversed:

    >>> is_not_none(data.get("key")) else y

This expression returns data.get("key") if it is not None, otherwise it
evaluates and returns y. To understand the mechanics, we rewrite the
expression as follows:

    >>> maybe_value = is_not_none(data.get("key"))
    >>> maybe_value if maybe_value else y

If bool(maybe_value) is True, then the expression short-circuits and the
interpreter calls type(maybe_value).__else__(maybe_value, maybe_value).
The implementation of types.CircuitBreaker.__then__ detects that the
instance method has received itself as its argument and returns the
wrapped value (i.e. data.get("key")) rather than the circuit breaker.

If bool(maybe_value) is True, the interpreter calls
type(maybe_value).__else__(maybe_value, y). The implementation of
types.CircuitBreaker.__else__ doesn't see anything that indicates
short-circuiting has taken place, and hence returns y.

Respecting De Morgan's Laws

Similar to and and or, the binary short-circuiting operators will permit
multiple ways of writing essentially the same expression. This seeming
redundancy is unfortunately an implied consequence of defining the
protocol as a full boolean algebra, as boolean algebras respect a pair
of properties known as "De Morgan's Laws": the ability to express the
results of and and or operations in terms of each other and a suitable
combination of not operations.

For and and or in Python, these invariants can be described as follows:

    assert bool(A and B) == bool(not (not A or not B))
    assert bool(A or B) == bool(not (not A and not B))

That is, if you take one of the operators, invert both operands, switch
to the other operator, and then invert the overall result, you'll get
the same answer (in a boolean sense) as you did from the original
operator. (This may seem redundant, but in many situations it actually
lets you eliminate double negatives and find tautologically true or
false subexpressions, thus reducing the overall expression size).

For circuit breakers, defining a suitable invariant is complicated by
the fact that they're often going to be designed to eliminate themselves
from the expression result when they're short-circuited, which is an
inherently asymmetric behaviour. Accordingly, that inherent asymmetry
needs to be accounted for when mapping De Morgan's Laws to the expected
behaviour of symmetric circuit breakers.

One way this complication can be addressed is to wrap the operand that
would otherwise short-circuit in operator.true, ensuring that when bool
is applied to the overall result, it uses the same definition of truth
that was used to decide which branch to evaluate, rather than applying
bool directly to the circuit breaker's input value.

Specifically, for the new short-circuiting operators, the following
properties would be reasonably expected to hold for any well-behaved
symmetric circuit breaker that implements both __bool__ and __not__:

    assert bool(B if true(A)) == bool(not (true(not A) else not B))
    assert bool(true(A) else B) == bool(not (not B if true(not A)))

Note the order of operations on the right hand side (applying true after
inverting the input circuit breaker) - this ensures that an assertion is
actually being made about type(A).__not__, rather than merely being
about the behaviour of type(true(A)).__not__.

At the very least, types.CircuitBreaker instances would respect this
logic, allowing existing boolean expression optimisations (like double
negative elimination) to continue to be applied.

Arbitrary sentinel objects

Unlike PEPs 505 and 531, the proposal in this PEP readily handles custom
sentinel objects:

    _MISSING = object()

    # Using the sentinel to check whether or not an argument was supplied
    def my_func(arg=_MISSING):
        arg = make_default() if is_sentinel(arg, _MISSING) # "else arg" implied

Implicitly defined circuit breakers in circuit breaking expressions

A never-posted draft of this PEP explored the idea of special casing the
is and is not binary operators such that they were automatically treated
as circuit breakers when used in the context of a circuit breaking
expression. Unfortunately, it turned out that this approach necessarily
resulted in one of two highly undesirable outcomes:

A.  the return type of these expressions changed universally from bool
    to types.CircuitBreaker, potentially creating a backwards
    compatibility problem (especially when working with extension module
    APIs that specifically look for a builtin boolean value with
    PyBool_Check rather than passing the supplied value through
    PyObject_IsTrue or using the p (predicate) format in one of the
    argument parsing functions)
B.  the return type of these expressions became context dependent,
    meaning that other routine refactorings (like pulling a comparison
    operation out into a local variable) could have a significant impact
    on the runtime semantics of a piece of code

Neither of those possible outcomes seems warranted by the proposal in
this PEP, so it reverted to the current design where circuit breaker
instances must be created explicitly via API calls, and are never
produced implicitly.

Implementation

As with PEP 505, actual implementation has been deferred pending
in-principle interest in the idea of making these changes.

...TBD...

Acknowledgements

Thanks go to Steven D'Aprano for his detailed critique[2] of the initial
draft of this PEP that inspired many of the changes in the second draft,
as well as to all of the other participants in that discussion
thread[3].

References

Copyright

This document has been placed in the public domain under the terms of
the CC0 1.0 license: https://creativecommons.org/publicdomain/zero/1.0/

[1] PEP 335 rejection notification
(https://mail.python.org/pipermail/python-dev/2012-March/117510.html)

[2] Steven D'Aprano's critique of the initial draft
(https://mail.python.org/pipermail/python-ideas/2016-November/043615.html)

[3] python-ideas thread discussing initial draft
(https://mail.python.org/pipermail/python-ideas/2016-November/043563.html)


text:
PEP: 652 Title: Maintaining the Stable ABI Author: Petr Viktorin
<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pre-pep-maintaining-the-stable-abi/6986/
Status: Final Type: Standards Track Content-Type: text/x-rst Created:
09-Feb-2021 Python-Version: 3.10 Resolution:
<EMAIL_ADDRESS>
python:stable (user docs) and devguide:c-api (development docs)

Abstract

CPython's Limited C-API and Stable ABI, introduced in PEP 384, will be
formalized in a single definitive file, tested, and documented.

Motivation

PEP 384 defined a Limited API and Stable ABI, which allow extenders and
embedders of CPython to compile extension modules that are
binary-compatible with any subsequent version of 3.x. In theory, this
brings several advantages:

-   A module can be built only once per platform and support multiple
    versions of Python, reducing time, power and maintainer attention
    needed for builds (in exchange for potentially worse performance).
-   Binary wheels using the Stable ABI work with new versions of CPython
    throughout the pre-release period, and can be tested in environments
    where building from source is not practical.
-   As a welcome side effect of the Limited API's hiding of
    implementation details, this API is becoming a viable target for
    alternate Python implementations that would be incompatible with the
    full C API.

However, in hindsight, PEP 384 and its implementation has several
issues:

-   It is ill-defined. According to PEP 384, functions are opt-out: all
    functions not specially marked are part of the Stable ABI. In
    practice, for Windows there's a list that's opt-in. For users there
    is a #define that should make only the Stable ABI available, but
    there is no process that ensures it is kept up-to date. Neither is
    there a process for updating the documentation.
-   Until recently, the Stable ABI was not tested at all. It tends to
    break. For example, changing a function to a macro can break the
    Stable ABI as the function symbol is removed.
-   There is no way to deprecate parts of the Limited API.
-   It is incomplete. Some operations are not available in the Stable
    ABI, with little reason except "we forgot". (This last point is one
    the PEP will not help with, however.)

This PEP defines the Limited API more clearly and introducess process
designed to make the Stable ABI and Limited API more useful and robust.

Rationale

This PEP contains a lot of clarifications and definitions, but just one
big technical change: the Stable ABI will be explicitly listed in a
human-maintained “manifest” file.

There have been efforts to collect such lists automatically, e.g. by
scanning the symbols exported from Python. Such automation might seem
easier to maintain than a handcrafted file, but has major issues: for
example, the set exported symbols has platform-specific variations.
Also, the cost of updating an explicit manifest is small compared to the
overall work that should go into changing API that will need to be
supported forever (or until Python 3 reaches end of life, if that comes
sooner).

This PEP proposes automatically generating things from the manifest:
initially documentation and DLL contents, with later possibilities for
also automating tests.

Stable ABI vs. Limited API

PEP 384 and this document deal with the Limited API and the Stable ABI,
two related but distinct concepts. In short:

-   The Stable ABI is a promise that certain extensions compiled with
    CPython 3.x will be binary compatible with all subsequent versions
    of CPython 3.x.
-   The Limited API is a subset of CPython's C API that produces such
    extensions.

This section clarifies these terms and defines some of their semantics
(either pre-existing or newly proposed here).

The word “Extensions” is used as a shorthand for all code that uses the
Python API, e.g. extension modules or software that embeds Python.

Stable ABI

The CPython Stable ABI is a promise that extensions built against a
specific Stable ABI version will be compatible with any newer
interpreter of the same major version.

The Stable ABI does not define a complete binary interface: important
details like the layout of structures in memory or function calling
conventions are determined by the platform and the compiler and its
settings. The Stable ABI promise only applies if these lower-details are
also stable.

For example, an extension built with the CPython 3.10 Stable ABI will be
usable with CPython 3.11, 3.12, etc. It will not necessarily be
compatible with CPython 4.0, nor with CPython 3.10 on a different
platform.

The Stable ABI is not generally forward-compatible: an extension built
and tested with CPython 3.10 will not generally be compatible with
CPython 3.9.

Note

For example, starting in Python 3.10, the Py_tp_doc slot may be set to
NULL, while in older versions, a NULL value will likely crash the
interpreter.

The Stable ABI trades performance for its stability. For example,
extensions built for a specific CPython version will automatically use
faster macros instead of functions in the Stable ABI.

Future Python versions may deprecate some members of the Stable ABI.
Deprecated members will still work, but may suffer from issues like
reduced performance or, in the most extreme cases, memory/resource
leaks.

Limited API

The Stable ABI promise holds for extensions compiled from code that
restricts itself to the Limited API (application programming interface).
The Limited API is a subset of CPython's C API.

Extensions that target the Limited API should define the preprocessor
macro Py_LIMITED_API to either 3 or the current PYTHON_API_VERSION. This
will enable Stable ABI versions of several functions and limit
definitions to the Limited API. (However, note that the macro is not
perfect: due to technical issues or oversights, some non-limited API
might be exposed even with it defined.)

The Limited API is not guaranteed to be stable. In the future, parts of
the Limited API may be deprecated. They may even be removed, as long as
the Stable ABI is kept stable and Python's general backwards
compatibility policy, PEP 387, is followed.

Note

For example, a function declaration might be removed from public header
files but kept in the library. This is currently a possibility for the
future; this PEP does not to propose a concrete process for deprecations
and removals.

The goal for the Limited API is to cover everything needed to interact
with the interpreter. The main reason to not include a public API in the
Limited subset should be that it needs implementation details that
change between CPython versions (like struct memory layouts) – usually
for performance reasons.

The Limited API is not limited to CPython. Other implementations are
encouraged to implement it and help drive its design.

Specification

To make the Stable ABI more useful and robust, the following changes are
proposed.

Stable ABI Manifest

All members of the Stable ABI – functions, typedefs, structs, data,
macros, and constants – will be explicitly listed in a single "manifest"
file, Misc/stable_abi.txt.

For structs, any fields that users of the Stable ABI are allowed to
access will be listed explicitly.

The manifest will also serve as the definitive list of the Limited API.
Members that are not part of the Limited API, but are part of the Stable
ABI (e.g. PyObject.ob_type, which is accessible by the Py_TYPE macro),
will be annotated as such.

For items that are only available on some systems, the manifest will
record the feature macro that determines their presence (such as
MS_WINDOWS or HAVE_FORK). To make the implementation (and usage from
non-C languages) easier, all such macros will be simple names. If a
future item needs a “negative” macro or complex expression (such as a
hypothetical #ifndef MACOSX or #if defined(POSIX) && !defined(LINUX)), a
new feature macro will be derived.

The format of the manifest will be subject to change whenever needed. It
should be consumed only by scripts in the CPython repository. If a
stable list is needed, a script can be added to generate it.

The following will be generated from the ABI manifest:

-   Source for the Windows shared library, PC/python3dll.c.
-   Input for documentation (see below).
-   Test case that checks the runtime availability of symbols (see
    below).

The following will be checked against the Stable ABI manifest as part of
continuous integration:

-   The reference count summary, Doc/data/refcounts.txt, includes all
    function in the Stable ABI (among others).
-   The functions/structs declared and constants/macros defined when
    Python.h is included with Py_LIMITED_API set. (Initially Linux only;
    checks on other systems may be added in the future.)

After the initial implementation, details such as function arguments
will be added and the manifest will be checked for internal consistency
(e.g. all types used in function signatures are part of the API).

Contents of the Stable ABI

The initial Stable ABI manifest will include:

-   The Stable ABI specified in PEP 384.
-   Everything listed in PC/python3dll.c.
-   All structs (struct typedefs) which these functions return or take
    as arguments. (Fields of such structs will not necessarily be
    added.)
-   New type slots, such as Py_am_aiter.
-   The type flags Py_TPFLAGS_DEFAULT, Py_TPFLAGS_BASETYPE,
    Py_TPFLAGS_HAVE_GC, Py_TPFLAGS_METHOD_DESCRIPTOR.
-   The calling conventions METH_* (except deprecated ones).
-   All API needed by macros is the Stable ABI (annotated as not being
    part of the Limited API).

Items that are no longer in CPython when this PEP is accepted will be
removed from the list.

Additional items may be added to the initial manifest according to the
checklist below.

Documenting the Limited API

Notes saying “Part of the Limited API” will be added to Python's
documentation automatically, in a way similar to the notes on functions
that return borrowed references.

A complete list of all members of the Limited API will also be added to
the documentation.

Testing the Stable ABI

An automatically generated test module will be added to ensure that all
symbols included in the Stable ABI are available at compile time.

Changing the Limited API

A checklist for changing the Limited API, including adding new items to
it and removing existing ones, will be added to the Devguide. The
checklist will 1) mention best practices and common pitfalls in Python C
API design and 2) guide the developer around the files that need
changing and scripts that need running when the Limited API is changed.

Below is the initial proposal for the checklist. (After the PEP is
accepted, see the Devguide for the current version.)

Note that the checklist applies to new changes; several items in the
existing Limited API are grandfathered and couldn't be added today.

Design considerations:

-   Make sure the change does not break the Stable ABI of any version of
    Python since 3.5.

-   Make sure no exposed names are private (i.e. begin with an
    underscore).

-   Make sure the new API is well documented.

-   Make sure the types of all parameters and return values of the added
    function(s) and all fields of the added struct(s) are be part of the
    Limited API (or standard C).

-   Make sure the new API and its intended use follows standard C, not
    just features of currently supported platforms. Specifically, follow
    the C dialect specified in PEP 7.

    -   Do not cast a function pointer to void* (a data pointer) or vice
        versa.

-   Make sure the new API follows reference counting conventions.
    (Following them makes the API easier to reason about, and easier use
    in other Python implementations.)

    -   Do not return borrowed references from functions.
    -   Do not steal references to function arguments.

-   Make sure the ownership rules and lifetimes of all applicable struct
    fields, arguments and return values are well defined.

-   Think about ease of use for the user. (In C, ease of use itself is
    not very important; what is useful is reducing boilerplate code
    needed to use the API. Bugs like to hide in boiler plates.)

    -   If a function will be often called with specific value for an
        argument, consider making it default (used when NULL is passed
        in).

-   Think about future extensions: for example, if it's possible that
    future Python versions will need to add a new field to your struct,
    how will that be done?

-   Make as few assumptions as possible about details that might change
    in future CPython versions or differ across C API implementations:

      -   The GIL
      -   Garbage collection
      -   Memory layout of PyObject, lists/tuples and other structures

If following these guidelines would hurt performance, add a fast
function (or macro) to the non-limited API and a stable equivalent to
the Limited API.

If anything is unclear, or you have a good reason to break the
guidelines, consider discussing the change at the capi-sig mailing list.

Procedure:

-   Move the declaration to a header file directly under Include/, into
    a #if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03yy0000
    block (with the yy corresponding to the target CPython version).
-   Make an entry in the Stable ABI manifest, Misc/stable_abi.txt.
-   Regenerate the autogenerated files using make regen-all. (or the
    alternative for non-make platforms)
-   Build Python and run checks using make check-abi. (or the
    alternative for non-make platforms)

Advice for Extenders and Embedders

The following notes will be added to documentation, along with better
information regarding this topic and what guarantees do we offer:

Extension authors should test with all Python versions they support, and
preferably build with the lowest such version.

Compiling with Py_LIMITED_API defined is not a guarantee that your code
conforms to the Limited API or the Stable ABI. Py_LIMITED_API only
covers definitions, but an API also includes other issues, such as
expected semantics.

Examples of issues that Py_LIMITED_API does not guard against are:

-   Calling a function with invalid arguments
-   A function that started accepting NULL values for an argument in
    Python 3.9 will fail if NULL is passed to it under Python 3.8. Only
    testing with 3.8 (or lower versions) will uncover this issue.
-   Some structs include a few fields that are part of the Stable ABI
    and other fields that aren't. Py_LIMITED_API does not filter out
    such “private” fields.
-   Code that uses something that is not documented as part of the
    Stable ABI, but exposed even with Py_LIMITED_API defined, may break
    in the future. Despite the team's best efforts, such issues may
    happen.

Note for Redistributors of Python

The Stable ABI promise relies on stable underlying ABI details, such as
the layout of structures in memory and function calling conventions,
which are affected by the compiler and its settings. For the promise to
hold, these details must not change between CPython 3.x releases on a
particular platform.

Backwards Compatibility

Backwards compatibility is one honking great idea!

This PEP aims at full compatibility with the existing Stable ABI and
Limited API, but defines them terms more explicitly. It might not be
consistent with some interpretations of what the existing Stable
ABI/Limited API is.

Security Implications

None known.

How to Teach This

Technical documentation will be provided in Doc/c-api/stable and linked
from the What's New document. Docs for CPython core developers will be
added to the devguide.

Reference Implementation

See issue 43795.

Ideas for the Future

The following issues are out of scope of this PEP, but show possible
future directions.

Defining a process for deprecations/removals

While this PEP acknowledges that parts of the Limited API might be
deprecated or removed in the future, a process to do this is not in
scope, and is left to a possible future PEP.

C syntax for the ABI manifest

It might be useful to have the ABI manifest be a C header file, or to
generate header files from the manifest. Again, either are options for
the future.

Open Issues

None so far.

References

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

license: Public Domain


text:
PEP: 714 Title: Rename dist-info-metadata in the Simple API Author:
Donald Stufft<EMAIL_ADDRESS>PEP-Delegate: Paul Moore
<EMAIL_ADDRESS>Discussions-To: https://discuss.python.org/t/27471
Status: Accepted Type: Standards Track Topic: Packaging Content-Type:
text/x-rst Created: 06-Jun-2023 Post-History: 06-Jun-2023 Resolution:
27-Jun-2023

Abstract

This PEP renames the metadata provided by PEP 658 in both HTML and JSON
formats of the Simple API and provides guidelines for both clients and
servers in how to handle the renaming.

Motivation

PEP 658 specified a mechanism to host the core metadata files from an
artifact available through the Simple API such that a client could fetch
the metadata and use it without having to download the entire artifact.
Later PEP 691 was written to add the ability to use JSON rather than
HTML on the Simple API, which included support for the PEP 658 metadata.

Unfortunately, PyPI did not support PEP 658 until just recently, which
released with a bug where the dist-info-metadata key from PEP 658 was
incorrectly named in the JSON representation, to be
data-dist-info-metadata. However, when attempting to fix that bug, it
was discovered that pip also had a bug, where any use of
dist-info-metadata in the JSON representation would cause pip to hard
fail with an exception.

The bug in pip has existed since at least v22.3, which means that it has
been released for approximately 8 months, long enough to have been
pulled into Python releases, downstream Linux releases, baked into
containers, virtual environments, etc.

This puts us in an awkward position of having a bug on PyPI that cannot
be fixed without breaking pip, due to a bug in pip, but that version of
pip is old enough to have been widely deployed. To make matters worse, a
version of pip that is broken in this way cannot install anything from
PyPI once it fixes its bug, including installing a new, fixed version of
pip.

Rationale

There are 3 main options for a path forward for fixing these bugs:

1.  Do not change the spec, fix the bug in pip, wait some amount of
    time, then fix the bug in PyPI, breaking anyone using an unfixed pip
    such that they cannot even install a new pip from PyPI.
2.  Do the same as (1), but special case PyPI so it does not emit the
    PEP 658 metadata for pip, even if it is available. This allows
    people to upgrade pip if they're on a broken version, but nothing
    else.
3.  Change the spec to avoid the key that pip can't handle currently,
    allowing PyPI to emit that key and a new version of pip to be
    released to take advantage of that key.

This PEP chooses (3), but goes a little further and also renames the key
in the HTML representation.

Typically we do not change specs because of bugs that only affect one
particular implementation, unless the spec itself is at fault, which
isn't the case here: the spec is fine and these are just genuine bugs in
pip and PyPI.

However, we choose to do this for 4 reasons:

1.  Bugs that affect pip and PyPI together represent an outsized amount
    of impact compared to any other client or repository combination.
2.  The impact of being broken is that installs do not function, at all,
    rather than degrading gracefully in some way.
3.  The feature that is being blocked by these bugs is of large
    importance to the ability to quickly and efficiently resolve
    dependencies from PyPI with pip, and having to delay it for a long
    period of time while we wait for the broken versions of pip to fall
    out of use would be of detriment to the entire ecosystem.
4.  The downsides of changing the spec are fairly limited, given that we
    do not believe that support for this is widespread, so it affects
    only a limited number of projects.

Specification

The keywords "**MUST**", "**MUST NOT**", "**REQUIRED**", "**SHALL**",
"**SHALL NOT**", "**SHOULD**", "**SHOULD NOT**", "**RECOMMENDED**",
"**MAY**", and "**OPTIONAL**"" in this document are to be interpreted as
described in RFC 2119 <2119>.

Servers

The PEP 658 metadata, when used in the HTML representation of the Simple
API, MUST be emitted using the attribute name data-core-metadata, with
the supported values remaining the same.

The PEP 658 metadata, when used in the PEP 691 JSON representation of
the Simple API, MUST be emitted using the key core-metadata, with the
supported values remaining the same.

To support clients that used the previous key names, the HTML
representation MAY also be emitted using the data-dist-info-metadata,
and if it does so it MUST match the value of data-core-metadata.

Clients

Clients consuming any of the HTML representations of the Simple API MUST
read the PEP 658 metadata from the key data-core-metadata if it is
present. They MAY optionally use the legacy data-dist-info-metadata if
it is present but data-core-metadata is not.

Clients consuming the JSON representation of the Simple API MUST read
the PEP 658 metadata from the key core-metadata if it is present. They
MAY optionally use the legacy dist-info-metadata key if it is present
but core-metadata is not.

Backwards Compatibility

There is a minor compatibility break in this PEP, in that clients that
currently correctly handle the existing metadata keys will not
automatically understand the newer metadata keys, but they should
degrade gracefully, and simply act as if the PEP 658 metadata does not
exist.

Otherwise there should be no compatibility problems with this PEP.

Rejected Ideas

Leave the spec unchanged, and cope with fixing in PyPI and/or pip

We believe that the improvements brought by PEP 658 are very important
to improving the performance of resolving dependencies from PyPI, and
would like to be able to deploy it as quickly as we can.

Unfortunately the nature of these bugs is that we cannot deploy them as
is without breaking widely deployed and used versions of pip. The
breakages in this case would be bad enough that affected users would not
even be able to directly upgrade their version of pip to fix it, but
would have to manually fetch pip another way first (e.g. get-pip.py).

This is something that PyPI would be unwilling to do without some way to
mitigate those breakages for those users. Without some reasonable
mitigation strategy, we would have to wait until those versions of pip
are no longer in use on PyPI, which would likely be 5+ years from now.

There are a few possible mitigation strategies that we could use, but
we've rejected them as well.

Mitigation: Special Case pip

The breakages are particularly bad in that it prevents users from even
upgrading pip to get an unbroken version of pip, so a command like
pip install --upgrade pip would fail. We could mitigate this by having
PyPI special case pip itself, so that the JSON endpoint never returns
the PEP 658 metadata and the above still works.

This PEP rejects this idea because while the simple command that only
upgrades pip would work, if the user included anything else in that
command to upgrade then the command would go back to failing, which we
consider to be still too large of a breakage.

Additionally, while this bug happens to be getting exposed right now
with PyPI, it is really a bug that would happen with any PEP 691
repository that correctly exposed the PEP 658 metadata. This would mean
that every repository would have to carry this special case for pip.

Mitigation: Have the server use User-Agent Detection

pip puts its version number into its User-Agent, which means that the
server could detect the version number and serve different responses
based on that version number so that we don't serve the PEP 658 metadata
to versions of pip that are broken.

This PEP rejects this idea because supporting User-Agent detection is
too difficult to implement in a reasonable way.

1.  On PyPI we rely heavily on caching the Simple API in our CDN. If we
    varied the responses based on User-Agent, then our CDN cache would
    have an explosion of cache keys for the same content, which would
    make it more likely that any particular request would not be cached
    and fall back to hitting our backend servers, which would have to
    scale much higher to support the load.
2.  PyPI could support the User-Agent detection idea by mutating the
    Accept header of the request so that those versions appear to only
    accept the HTML version, allowing us to maintain the CDNs cache
    keys. This doesn't affect any downstream caches of PyPI though,
    including pip's HTTP cache which would possibly have JSON versions
    cached for those requests and we wouldn't emit a Vary on User-Agent
    for them to know that it isn't acceptable to share those caches, and
    adding a Vary: User-Agent for downstream caches would have the same
    problem as (1), but for downstream caches instead of our CDN cache.
3.  The pip bug ultimately isn't PyPI specific, it affects any
    repository that implements PEP 691 and PEP 658 together. This would
    mean that workarounds that rely on implementation specific fixes
    have to be replicated for each repository that implements both,
    which may not be easy or possible in all cases (static mirrors may
    not be able to do this User-Agent detection for instance).

Only change the JSON key

The bug in pip only affects the JSON representation of the Simple API,
so we only need to actually change the key in the JSON, and we could
leave the existing HTML keys alone.

This PEP rejects doing that because we believe that in the long term,
having the HTML and JSON key names diverge would make mistakes like this
more likely and make implementing and understanding the spec more
confusing.

The main reason that we would want to not change the HTML keys is to not
lose PEP 658 support in any HTML only clients or repositories that might
already support it. This PEP mitigates that breakage by allowing both
clients and servers to continue to support both keys, with a
recommendation of when and how to do that.

Recommendations

The recommendations in this section, other than this notice itself, are
non-normative, and represent what the PEP authors believe to be the best
default implementation decisions for something implementing this PEP,
but it does not represent any sort of requirement to match these
decisions.

Servers

We recommend that servers only emit the newer keys, particularly for the
JSON representation of the Simple API since the bug itself only affected
JSON.

Servers that wish to support PEP 658 in clients that use HTML and have
it implemented, can safely emit both keys only in HTML.

Servers should not emit the old keys in JSON unless they know that no
broken versions of pip will be used to access their server.

Clients

We recommend that clients support both keys, for both HTML and JSON,
preferring the newer key as this PEP requires. This will allow clients
to support repositories that already have correctly implemented PEP 658
and PEP 691 but have not implemented this PEP.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 555 Title: Context-local variables (contextvars) Version:
$Revision$ Last-Modified: $Date$ Author: Koos Zevenhoven Status:
Withdrawn Type: Standards Track Content-Type: text/x-rst Created:
06-Sep-2017 Python-Version: 3.7 Post-History: 06-Sep-2017

Abstract

Sometimes, in special cases, it is desired that code can pass
information down the function call chain to the callees without having
to explicitly pass the information as arguments to each function in the
call chain. This proposal describes a construct which allows code to
explicitly switch in and out of a context where a certain context
variable has a given value assigned to it. This is a modern alternative
to some uses of things like global variables in traditional
single-threaded (or thread-unsafe) code and of thread-local storage in
traditional concurrency-unsafe code (single- or multi-threaded). In
particular, the proposed mechanism can also be used with more modern
concurrent execution mechanisms such as asynchronously executed
coroutines, without the concurrently executed call chains interfering
with each other's contexts.

The "call chain" can consist of normal functions, awaited coroutines, or
generators. The semantics of context variable scope are equivalent in
all cases, allowing code to be refactored freely into subroutines (which
here refers to functions, sub-generators or sub-coroutines) without
affecting the semantics of context variables. Regarding implementation,
this proposal aims at simplicity and minimum changes to the CPython
interpreter and to other Python interpreters.

Rationale

Consider a modern Python call chain (or call tree), which in this
proposal refers to any chained (nested) execution of subroutines, using
any possible combinations of normal function calls, or expressions using
await or yield from. In some cases, passing necessary information down
the call chain as arguments can substantially complicate the required
function signatures, or it can even be impossible to achieve in
practice. In these cases, one may search for another place to store this
information. Let us look at some historical examples.

The most naive option is to assign the value to a global variable or
similar, where the code down the call chain can access it. However, this
immediately makes the code thread-unsafe, because with multiple threads,
all threads assign to the same global variable, and another thread can
interfere at any point in the call chain. Sooner or later, someone will
probably find a reason to run the same code in parallel threads.

A somewhat less naive option is to store the information as per-thread
information in thread-local storage, where each thread has its own
"copy" of the variable which other threads cannot interfere with.
Although non-ideal, this has been the best solution in many cases.
However, thanks to generators and coroutines, the execution of the call
chain can be suspended and resumed, allowing code in other contexts to
run concurrently. Therefore, using thread-local storage is
concurrency-unsafe, because other call chains in other contexts may
interfere with the thread-local variable.

Note that in the above two historical approaches, the stored information
has the widest available scope without causing problems. For a third
solution along the same path, one would first define an equivalent of a
"thread" for asynchronous execution and concurrency. This could be seen
as the largest amount of code and nested calls that is guaranteed to be
executed sequentially without ambiguity in execution order. This might
be referred to as concurrency-local or task-local storage. In this
meaning of "task", there is no ambiguity in the order of execution of
the code within one task. (This concept of a task is close to equivalent
to a Task in asyncio, but not exactly.) In such concurrency-locals, it
is possible to pass information down the call chain to callees without
another code path interfering with the value in the background.

Common to the above approaches is that they indeed use variables with a
wide but just-narrow-enough scope. Thread-locals could also be called
thread-wide globals---in single-threaded code, they are indeed truly
global. And task-locals could be called task-wide globals, because tasks
can be very big.

The issue here is that neither global variables, thread-locals nor
task-locals are really meant to be used for this purpose of passing
information of the execution context down the call chain. Instead of the
widest possible variable scope, the scope of the variables should be
controlled by the programmer, typically of a library, to have the
desired scope---not wider. In other words, task-local variables (and
globals and thread-locals) have nothing to do with the kind of
context-bound information passing that this proposal intends to enable,
even if task-locals can be used to emulate the desired semantics.
Therefore, in the following, this proposal describes the semantics and
the outlines of an implementation for context-local variables (or
context variables, contextvars). In fact, as a side effect of this PEP,
an async framework can use the proposed feature to implement task-local
variables.

Proposal

Because the proposed semantics are not a direct extension to anything
already available in Python, this proposal is first described in terms
of semantics and API at a fairly high level. In particular, Python with
statements are heavily used in the description, as they are a good match
with the proposed semantics. However, the underlying __enter__ and
__exit__ methods correspond to functions in the lower-level
speed-optimized (C) API. For clarity of this document, the lower-level
functions are not explicitly named in the definition of the semantics.
After describing the semantics and high-level API, the implementation is
described, going to a lower level.

Semantics and higher-level API

Core concept

A context-local variable is represented by a single instance of
contextvars.Var, say cvar. Any code that has access to the cvar object
can ask for its value with respect to the current context. In the
high-level API, this value is given by the cvar.value property:

    cvar = contextvars.Var(default="the default value",
                           description="example context variable")

    assert cvar.value == "the default value"  # default still applies

    # In code examples, all ``assert`` statements should
    # succeed according to the proposed semantics.

No assignments to cvar have been applied for this context, so cvar.value
gives the default value. Assigning new values to contextvars is done in
a highly scope-aware manner:

    with cvar.assign(new_value):
        assert cvar.value is new_value
        # Any code here, or down the call chain from here, sees:
        #     cvar.value is new_value
        # unless another value has been assigned in a
        # nested context
        assert cvar.value is new_value
    # the assignment of ``cvar`` to ``new_value`` is no longer visible
    assert cvar.value == "the default value"

Here, cvar.assign(value) returns another object, namely
contextvars.Assignment(cvar, new_value). The essential part here is that
applying a context variable assignment (Assignment.__enter__) is paired
with a de-assignment (Assignment.__exit__). These operations set the
bounds for the scope of the assigned value.

Assignments to the same context variable can be nested to override the
outer assignment in a narrower context:

    assert cvar.value == "the default value"
    with cvar.assign("outer"):
        assert cvar.value == "outer"
        with cvar.assign("inner"):
            assert cvar.value == "inner"
        assert cvar.value == "outer"
    assert cvar.value == "the default value"

Also multiple variables can be assigned to in a nested manner without
affecting each other:

    cvar1 = contextvars.Var()
    cvar2 = contextvars.Var()

    assert cvar1.value is None # default is None by default
    assert cvar2.value is None

    with cvar1.assign(value1):
        assert cvar1.value is value1
        assert cvar2.value is None
        with cvar2.assign(value2):
            assert cvar1.value is value1
            assert cvar2.value is value2
        assert cvar1.value is value1
        assert cvar2.value is None
    assert cvar1.value is None
    assert cvar2.value is None

Or with more convenient Python syntax:

    with cvar1.assign(value1), cvar2.assign(value2):
        assert cvar1.value is value1
        assert cvar2.value is value2

In another context, in another thread or otherwise concurrently executed
task or code path, the context variables can have a completely different
state. The programmer thus only needs to worry about the context at
hand.

Refactoring into subroutines

Code using contextvars can be refactored into subroutines without
affecting the semantics. For instance:

    assi = cvar.assign(new_value)
    def apply():
        assi.__enter__()
    assert cvar.value == "the default value"
    apply()
    assert cvar.value is new_value
    assi.__exit__()
    assert cvar.value == "the default value"

Or similarly in an asynchronous context where await expressions are
used. The subroutine can now be a coroutine:

    assi = cvar.assign(new_value)
    async def apply():
        assi.__enter__()
    assert cvar.value == "the default value"
    await apply()
    assert cvar.value is new_value
    assi.__exit__()
    assert cvar.value == "the default value"

Or when the subroutine is a generator:

    def apply():
        yield
        assi.__enter__()

which is called using yield from apply() or with calls to next or .send.
This is discussed further in later sections.

Semantics for generators and generator-based coroutines

Generators, coroutines and async generators act as subroutines in much
the same way that normal functions do. However, they have the additional
possibility of being suspended by yield expressions. Assignment contexts
entered inside a generator are normally preserved across yields:

    def genfunc():
        with cvar.assign(new_value):
            assert cvar.value is new_value
            yield
            assert cvar.value is new_value
    g = genfunc()
    next(g)
    assert cvar.value == "the default value"
    with cvar.assign(another_value):
        next(g)

However, the outer context visible to the generator may change state
across yields:

    def genfunc():
        assert cvar.value is value2
        yield
        assert cvar.value is value1
        yield
        with cvar.assign(value3):
            assert cvar.value is value3

    with cvar.assign(value1):
        g = genfunc()
        with cvar.assign(value2):
            next(g)
        next(g)
        next(g)
        assert cvar.value is value1

Similar semantics apply to async generators defined by
async def ... yield ... ).

By default, values assigned inside a generator do not leak through
yields to the code that drives the generator. However, the assignment
contexts entered and left open inside the generator do become visible
outside the generator after the generator has finished with a
StopIteration or another exception:

    assi = cvar.assign(new_value)
    def genfunc():
        yield
        assi.__enter__():
        yield

    g = genfunc()
    assert cvar.value == "the default value"
    next(g)
    assert cvar.value == "the default value"
    next(g)  # assi.__enter__() is called here
    assert cvar.value == "the default value"
    next(g)
    assert cvar.value is new_value
    assi.__exit__()

Special functionality for framework authors

Frameworks, such as asyncio or third-party libraries, can use additional
functionality in contextvars to achieve the desired semantics in cases
which are not determined by the Python interpreter. Some of the
semantics described in this section are also afterwards used to describe
the internal implementation.

Leaking yields

Using the contextvars.leaking_yields decorator, one can choose to leak
the context through yield expressions into the outer context that drives
the generator:

    @contextvars.leaking_yields
    def genfunc():
        assert cvar.value == "outer"
        with cvar.assign("inner"):
            yield
            assert cvar.value == "inner"
        assert cvar.value == "outer"

    g = genfunc():
    with cvar.assign("outer"):
        assert cvar.value == "outer"
        next(g)
        assert cvar.value == "inner"
        next(g)
        assert cvar.value == "outer"

Capturing contextvar assignments

Using contextvars.capture(), one can capture the assignment contexts
that are entered by a block of code. The changes applied by the block of
code can then be reverted and subsequently reapplied, even in another
context:

    assert cvar1.value is None # default
    assert cvar2.value is None # default
    assi1 = cvar1.assign(value1)
    assi2 = cvar1.assign(value2)
    with contextvars.capture() as delta:
        assi1.__enter__()
        with cvar2.assign("not captured"):
            assert cvar2.value is "not captured"
        assi2.__enter__()
    assert cvar1.value is value2
    delta.revert()
    assert cvar1.value is None
    assert cvar2.value is None
    ...
    with cvar1.assign(1), cvar2.assign(2):
        delta.reapply()
        assert cvar1.value is value2
        assert cvar2.value == 2

However, reapplying the "delta" if its net contents include
deassignments may not be possible (see also Implementation and Open
Issues).

Getting a snapshot of context state

The function contextvars.get_local_state() returns an object
representing the applied assignments to all context-local variables in
the context where the function is called. This can be seen as equivalent
to using contextvars.capture() to capture all context changes from the
beginning of execution. The returned object supports methods .revert()
and reapply() as above.

Running code in a clean state

Although it is possible to revert all applied context changes using the
above primitives, a more convenient way to run a block of code in a
clean context is provided:

    with context_vars.clean_context():
        # here, all context vars start off with their default values
    # here, the state is back to what it was before the with block.

Implementation

This section describes to a variable level of detail how the described
semantics can be implemented. At present, an implementation aimed at
simplicity but sufficient features is described. More details will be
added later.

Alternatively, a somewhat more complicated implementation offers minor
additional features while adding some performance overhead and requiring
more code in the implementation.

Data structures and implementation of the core concept

Each thread of the Python interpreter keeps its own stack of
contextvars.Assignment objects, each having a pointer to the previous
(outer) assignment like in a linked list. The local state (also returned
by contextvars.get_local_state()) then consists of a reference to the
top of the stack and a pointer/weak reference to the bottom of the
stack. This allows efficient stack manipulations. An object produced by
contextvars.capture() is similar, but refers to only a part of the stack
with the bottom reference pointing to the top of the stack as it was in
the beginning of the capture block.

Now, the stack evolves according to the assignment __enter__ and
__exit__ methods. For example:

    cvar1 = contextvars.Var()
    cvar2 = contextvars.Var()
    # stack: []
    assert cvar1.value is None
    assert cvar2.value is None

    with cvar1.assign("outer"):
        # stack: [Assignment(cvar1, "outer")]
        assert cvar1.value == "outer"

        with cvar1.assign("inner"):
            # stack: [Assignment(cvar1, "outer"),
            #         Assignment(cvar1, "inner")]
            assert cvar1.value == "inner"

            with cvar2.assign("hello"):
                # stack: [Assignment(cvar1, "outer"),
                #         Assignment(cvar1, "inner"),
                #         Assignment(cvar2, "hello")]
                assert cvar2.value == "hello"

            # stack: [Assignment(cvar1, "outer"),
            #         Assignment(cvar1, "inner")]
            assert cvar1.value == "inner"
            assert cvar2.value is None

        # stack: [Assignment(cvar1, "outer")]
        assert cvar1.value == "outer"

    # stack: []
    assert cvar1.value is None
    assert cvar2.value is None

Getting a value from the context using cvar1.value can be implemented as
finding the topmost occurrence of a cvar1 assignment on the stack and
returning the value there, or the default value if no assignment is
found on the stack. However, this can be optimized to instead be an O(1)
operation in most cases. Still, even searching through the stack may be
reasonably fast since these stacks are not intended to grow very large.

The above description is already sufficient for implementing the core
concept. Suspendable frames require some additional attention, as
explained in the following.

Implementation of generator and coroutine semantics

Within generators, coroutines and async generators, assignments and
deassignments are handled in exactly the same way as anywhere else.
However, some changes are needed in the builtin generator methods send,
__next__, throw and close. Here is the Python equivalent of the changes
needed in send for a generator (here _old_send refers to the behavior in
Python 3.6):

    def send(self, value):
        if self.gi_contextvars is LEAK:
            # If decorated with contextvars.leaking_yields.
            # Nothing needs to be done to leak context through yields :)
            return self._old_send(value)
        try:
            with contextvars.capture() as delta:
                if self.gi_contextvars:
                    # non-zero captured content from previous iteration
                    self.gi_contextvars.reapply()
                ret = self._old_send(value)
        except Exception:
            raise  # back to the calling frame (e.g. StopIteration)
        else:
            # suspending, revert context changes but save them for later
            delta.revert()
            self.gi_contextvars = delta
        return ret

The corresponding modifications to the other methods is essentially
identical. The same applies to coroutines and async generators.

For code that does not use contextvars, the additions are O(1) and
essentially reduce to a couple of pointer comparisons. For code that
does use contextvars, the additions are still O(1) in most cases.

More on implementation

The rest of the functionality, including contextvars.leaking_yields,
contextvars.capture(), contextvars.get_local_state() and
contextvars.clean_context() are in fact quite straightforward to
implement, but their implementation will be discussed further in later
versions of this proposal. Caching of assigned values is somewhat more
complicated, and will be discussed later, but it seems that most cases
should achieve O(1) complexity.

Backwards compatibility

There are no direct backwards-compatibility concerns, since a completely
new feature is proposed.

However, various traditional uses of thread-local storage may need a
smooth transition to contextvars so they can be concurrency-safe. There
are several approaches to this, including emulating task-local storage
with a little bit of help from async frameworks. A fully general
implementation cannot be provided, because the desired semantics may
depend on the design of the framework.

Another way to deal with the transition is for code to first look for a
context created using contextvars. If that fails because a new-style
context has not been set or because the code runs on an older Python
version, a fallback to thread-local storage is used.

Open Issues

Out-of-order de-assignments

In this proposal, all variable deassignments are made in the opposite
order compared to the preceding assignments. This has two useful
properties: it encourages using with statements to define assignment
scope and has a tendency to catch errors early (forgetting a .__exit__()
call often results in a meaningful error. To have this as a requirement
is beneficial also in terms of implementation simplicity and
performance. Nevertheless, allowing out-of-order context exits is not
completely out of the question, and reasonable implementation strategies
for that do exist.

Rejected Ideas

Dynamic scoping linked to subroutine scopes

The scope of value visibility should not be determined by the way the
code is refactored into subroutines. It is necessary to have
per-variable control of the assignment scope.

Acknowledgements

To be added.

References

To be added.

Copyright

This document has been placed in the public domain.


text:
PEP: 602 Title: Annual Release Cycle for Python Author: Łukasz Langa
<EMAIL_ADDRESS>PEP-Delegate: Brett Cannon<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pep-602-annual-release-cycle-for-python/2296/
Status: Active Type: Process Created: 04-Jun-2019 Python-Version: 3.9
Post-History: 09-Oct-2023 Resolution:
<EMAIL_ADDRESS>
Abstract

This document describes a change in the release calendar for Python
starting with Python 3.9. This change accelerates the release cadence
such that feature versions are released predictably every twelve months,
in October every year.

Implementation

Seventeen months to develop a feature version

This PEP proposes that Python 3.X.0 will be developed for around 17
months:

-   The first five months overlap with Python 3.(X-1).0's beta and
    release candidate stages and are thus unversioned.
-   The next seven months are spent on versioned alpha releases where
    both new features are incrementally added and bug fixes are
    included.
-   The following three months are spent on four versioned beta releases
    where no new features can be added but bug fixes are still included.
-   The final two months are spent on two release candidates (or more,
    if necessary) and conclude with the release of the final release of
    Python 3.X.0.

2 years of full support, 3 more years of security fixes

After the release of Python 3.X.0, the 3.X series is maintained for five
years:

-   During the first twenty four months (2 years) it receives bugfix
    updates and full releases (sources and installers for Windows and
    macOS) are made approximately every other month.
-   For the next thirty six months (3 years) it receives security
    updates and source-only releases are made on an as-needed basis (no
    fixed cadence).
-   The final source-only release is made five years after 3.X.0.

Note: 2 years of full support start with Python 3.13 <719>. Python
versions 3.9 - 3.12 operate on a calendar with 1½ year of full support,
followed by 3½ more years of security fixes.

Annual release cadence

Feature development of Python 3.(X+1).0 starts as soon as Python 3.X.0
Beta 1 is released. This creates a twelve-month delta between Python
feature versions.

Example

-   3.9 development begins: Tuesday, 2019-06-04
-   3.9.0 alpha 1: Monday, 2019-10-14
-   3.9.0 alpha 2: Monday, 2019-11-18
-   3.9.0 alpha 3: Monday, 2019-12-16
-   3.9.0 alpha 4: Monday, 2020-01-13
-   3.9.0 alpha 5: Monday, 2020-02-17
-   3.9.0 alpha 6: Monday, 2020-03-16
-   3.9.0 alpha 7: Monday, 2020-04-13
-   3.9.0 beta 1: Monday, 2020-05-18 (No new features beyond this
    point.)
-   3.9.0 beta 2: Monday, 2020-06-08
-   3.9.0 beta 3: Monday, 2020-06-29
-   3.9.0 beta 4: Monday, 2020-07-20
-   3.9.0 candidate 1: Monday, 2020-08-10
-   3.9.0 candidate 2: Monday, 2020-09-14
-   3.9.0 final: Monday, 2020-10-05

[]

In comparison, if this PEP is rejected and Python keeps the current
release schedule:

-   3.9 development begins: Tuesday, 2019-06-04
-   3.9.0 alpha 1: Monday, 2020-08-03 (10 months later)
-   3.9.0 alpha 2: Monday, 2020-09-07
-   3.9.0 alpha 3: Monday, 2020-10-05
-   3.9.0 alpha 4: Monday, 2020-11-02
-   3.9.0 beta 1: Monday, 2020-11-30 (6 months later)
-   3.9.0 beta 2: Monday, 2021-01-04
-   3.9.0 beta 3: Monday, 2021-02-01
-   3.9.0 beta 4: Monday, 2021-03-01
-   3.9.0 candidate 1: Monday, 2021-03-29
-   3.9.0 candidate 2: Monday, 2021-04-05 (if necessary)
-   3.9.0 final: Monday, 2021-04-19 (6 months later)

Dependent Policies

Deprecations

The current policy around breaking changes assumes at least two releases
before a deprecated feature is removed from Python or a __future__
behavior is enabled by default. This is documented in PEP 387.

This PEP proposes to keep this policy of at least two releases before
making a breaking change.

The term of the Steering Council

The current wording of PEP 13 states that "a new council is elected
after each feature release". This PEP proposes to keep this policy as it
will lead to a consistent election schedule.

The term of the Release Manager

The current undocumented convention is for a single Release Manager to
handle two feature releases of Python. This PEP proposes to keep this
policy, allowing for the term to be extended to more releases with
approval from the Steering Council and the Cabal of Release Managers.

In particular, since this PEP is authored by the active Release Manager
and its effect would shorten the term of the Release Manager, the author
is open to managing the release of a third feature release to compensate
for the disruption.

Rationale and Goals

This change provides the following advantages:

-   makes releases smaller: since doubling the cadence doesn't double
    our available development resources, consecutive releases are going
    to be smaller in terms of features;
-   puts features and bug fixes in hands of users sooner;
-   creates a more gradual upgrade path for users, by decreasing the
    surface of change in any single release;
-   creates a predictable calendar for releases where the final release
    is always in October (so after the annual core sprint), and the beta
    phase starts in late May (so after PyCon US sprints), which is
    especially important for core developers who need to plan to include
    Python involvement in their calendar;
-   decreases the urge to rush features shortly before "Beta 1" due to
    the risk of them "slipping for 18 months";
-   allows for synchronizing the schedule of Python release management
    with external distributors like Fedora who've been historically very
    helpful in finding regressions early not only in core Python but
    also in third-party libraries, helping moving the community forward
    to support the latest version of Python from Day 1;
-   increases the explicit alpha release phase, which provides
    meaningful snapshots of progress on new features;
-   significantly cuts the implicit "alpha 0" release phase which
    provides limited use for new development anyway (it overlaps with
    the beta of the currently developed, still unreleased, version).

Non-goals

Adopting an annual release calendar allows for natural switching to
calendar versioning, for example by calling Python 3.9 "Python 3.20"
since it's released in October '20 and so on ("Python 3.23" would be the
one released in October '23).

While the ease of switching to calendar versioning can be treated as an
advantage of an annual release cycle, this PEP does not advocate for or
against a change in how Python is versioned. Should the annual release
cycle be adopted, the versioning question will be dealt with in a
separate PEP.

Non-risks

This change does not shorten the currently documented support calendar
for a Python release, both in terms of bugfix releases and security
fixes.

This change does not accelerate the velocity of development. Python is
not going to become incompatible faster or accrue new features faster.
It's just that features are going to be released more gradually as they
are developed.

Consequently, while this change introduces the ability for users to
upgrade much faster, it does not require them to do so. Say, if they
upgrade every second release, their experience with Python is going to
be similar to the current situation.

Risks

Python redistribution

This requires changes to how integrators, like Linux distributions,
release Python within their systems.

The testing matrix

This eventually increases the testing matrix for library and application
maintainers that want to support all actively supported Python versions
by one or two:

[]

The "extended bugfix support at the discretion of the Release Manager"
stage of the current release cycle is not codified. If fact, PEP 101
currently states that after the release of Python 3.(X+1).0 only one
last bugfix release is made for Python 3.X.0. However, in practice at
least the last four versions of Python 3 overlapped with stable releases
of the next version for around six months. Figure 2 is including this
information to demonstrate that overlap between stable version releases
with the 12-month release cadence will be nothing new.

Other policies may depend on the release cadence

Although identified dependent policies were addressed in a previous
section, it is entirely possible there are some other areas which
implicitly rely on the timing of Python releases.

Rejected Ideas

Keep the current 18 month release cadence

This is undesirable both for core developers and end users. From the
perspective of the core developer:

-   it makes contribution scheduling harder due to irregular release
    dates every year;
-   it creates a surge of rushed commits before (and even after!) Beta 1
    due to the stress involved with "missing a release";
-   ironically, after Beta 1 it creates a false sense of having "plenty
    of time" before the next release, time that passes quickly
    regardless;
-   it causes certain elements of the workflow to be executed so rarely
    that they are not explicitly documented, let alone automated.

More importantly, from the perspective of the user:

-   it creates releases with many new features, some being explicitly
    incompatible and some being accidentally incompatible, which makes
    the upgrade cost relatively high every time;
-   it sits on features and incompatible bug fixes for over a year
    before becoming available to the user; and more specifically
-   it causes every "point zero" release to be extra risky for users.
    While we provide and recommend testing with alphas and betas, "point
    zero" is the first release of a given Python version for many users.
    The bigger a release is feature-wise, the more potential problems
    are hiding in "point zero releases".

Double the release cadence to achieve 9 months between feature versions

This was originally proposed in PEP 596 and rejected as both too
irregular and too short. This would not give any of the benefits of a
regular release calendar but it would shorten all development phases,
especially the beta + RC phases. This was considered dangerous.

Keep "4 betas over 4 months and a final month for the release candidate"

While this would make the release calendar a bit cleaner, it would make
it very hard for external distributors like Fedora to release the newest
version of Python as soon as possible. We are adjusting Python's
calendar here in the hope that this will enable Fedora to integrate the
newest version of Python with the newest version of Fedora as both are
being developed which makes both projects better.

Slow down releases but don't freeze feature development with Beta 1

This is described in PEP 598. This proposal includes non-standard
concepts like the "incremental feature release" which makes it hard to
understand. The presented advantages are unclear while the unfamiliarity
of the scheme poses a real risk of user and integrator confusion.

Long-Term Support Releases

Each version of Python is effectively long-term support: it's supported
for five years, with the first eighteen months allowing regular bug
fixes and security updates. For the remaining time security updates are
accepted and promptly released.

No extended support in the vein of Python 2.7 is planned going forward.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.