text:
PEP: 487 Title: Simpler customisation of class creation Version:
$Revision$ Last-Modified: $Date$ Author: Martin Teichmann
<EMAIL_ADDRESS>Status: Final Type: Standards Track
Content-Type: text/x-rst Created: 27-Feb-2015 Python-Version: 3.6
Post-History: 27-Feb-2015, 05-Feb-2016, 24-Jun-2016, 02-Jul-2016,
13-Jul-2016 Replaces: 422 Resolution:
https://mail.python.org/pipermail/python-dev/2016-July/145629.html

Abstract

Currently, customising class creation requires the use of a custom
metaclass. This custom metaclass then persists for the entire lifecycle
of the class, creating the potential for spurious metaclass conflicts.

This PEP proposes to instead support a wide range of customisation
scenarios through a new __init_subclass__ hook in the class body, and a
hook to initialize attributes.

The new mechanism should be easier to understand and use than
implementing a custom metaclass, and thus should provide a gentler
introduction to the full power of Python's metaclass machinery.

Background

Metaclasses are a powerful tool to customize class creation. They have,
however, the problem that there is no automatic way to combine
metaclasses. If one wants to use two metaclasses for a class, a new
metaclass combining those two needs to be created, typically manually.

This need often occurs as a surprise to a user: inheriting from two base
classes coming from two different libraries suddenly raises the
necessity to manually create a combined metaclass, where typically one
is not interested in those details about the libraries at all. This
becomes even worse if one library starts to make use of a metaclass
which it has not done before. While the library itself continues to work
perfectly, suddenly every code combining those classes with classes from
another library fails.

Proposal

While there are many possible ways to use a metaclass, the vast majority
of use cases falls into just three categories: some initialization code
running after class creation, the initialization of descriptors and
keeping the order in which class attributes were defined.

The first two categories can easily be achieved by having simple hooks
into the class creation:

1.  An __init_subclass__ hook that initializes all subclasses of a given
    class.
2.  upon class creation, a __set_name__ hook is called on all the
    attribute (descriptors) defined in the class, and

The third category is the topic of another PEP, PEP 520.

As an example, the first use case looks as follows:

    >>> class QuestBase:
    ...    # this is implicitly a @classmethod (see below for motivation)
    ...    def __init_subclass__(cls, swallow, **kwargs):
    ...        cls.swallow = swallow
    ...        super().__init_subclass__(**kwargs)

    >>> class Quest(QuestBase, swallow="african"):
    ...    pass

    >>> Quest.swallow
    'african'

The base class object contains an empty __init_subclass__ method which
serves as an endpoint for cooperative multiple inheritance. Note that
this method has no keyword arguments, meaning that all methods which are
more specialized have to process all keyword arguments.

This general proposal is not a new idea (it was first suggested for
inclusion in the language definition more than 10 years ago, and a
similar mechanism has long been supported by Zope's ExtensionClass), but
the situation has changed sufficiently in recent years that the idea is
worth reconsidering for inclusion.

The second part of the proposal adds an __set_name__ initializer for
class attributes, especially if they are descriptors. Descriptors are
defined in the body of a class, but they do not know anything about that
class, they do not even know the name they are accessed with. They do
get to know their owner once __get__ is called, but still they do not
know their name. This is unfortunate, for example they cannot put their
associated value into their object's __dict__ under their name, since
they do not know that name. This problem has been solved many times, and
is one of the most important reasons to have a metaclass in a library.
While it would be easy to implement such a mechanism using the first
part of the proposal, it makes sense to have one solution for this
problem for everyone.

To give an example of its usage, imagine a descriptor representing weak
referenced values:

    import weakref

    class WeakAttribute:
        def __get__(self, instance, owner):
            return instance.__dict__[self.name]()

        def __set__(self, instance, value):
            instance.__dict__[self.name] = weakref.ref(value)

        # this is the new initializer:
        def __set_name__(self, owner, name):
            self.name = name

Such a WeakAttribute may, for example, be used in a tree structure where
one wants to avoid cyclic references via the parent:

    class TreeNode:
        parent = WeakAttribute()

        def __init__(self, parent):
            self.parent = parent

Note that the parent attribute is used like a normal attribute, yet the
tree contains no cyclic references and can thus be easily garbage
collected when out of use. The parent attribute magically becomes None
once the parent ceases existing.

While this example looks very trivial, it should be noted that until now
such an attribute cannot be defined without the use of a metaclass. And
given that such a metaclass can make life very hard, this kind of
attribute does not exist yet.

Initializing descriptors could simply be done in the __init_subclass__
hook. But this would mean that descriptors can only be used in classes
that have the proper hook, the generic version like in the example would
not work generally. One could also call __set_name__ from within the
base implementation of object.__init_subclass__. But given that it is a
common mistake to forget to call super(), it would happen too often that
suddenly descriptors are not initialized.

Key Benefits

Easier inheritance of definition time behaviour

Understanding Python's metaclasses requires a deep understanding of the
type system and the class construction process. This is legitimately
seen as challenging, due to the need to keep multiple moving parts (the
code, the metaclass hint, the actual metaclass, the class object,
instances of the class object) clearly distinct in your mind. Even when
you know the rules, it's still easy to make a mistake if you're not
being extremely careful.

Understanding the proposed implicit class initialization hook only
requires ordinary method inheritance, which isn't quite as daunting a
task. The new hook provides a more gradual path towards understanding
all of the phases involved in the class definition process.

Reduced chance of metaclass conflicts

One of the big issues that makes library authors reluctant to use
metaclasses (even when they would be appropriate) is the risk of
metaclass conflicts. These occur whenever two unrelated metaclasses are
used by the desired parents of a class definition. This risk also makes
it very difficult to add a metaclass to a class that has previously been
published without one.

By contrast, adding an __init_subclass__ method to an existing type
poses a similar level of risk to adding an __init__ method: technically,
there is a risk of breaking poorly implemented subclasses, but when that
occurs, it is recognised as a bug in the subclass rather than the
library author breaching backwards compatibility guarantees.

New Ways of Using Classes

Subclass registration

Especially when writing a plugin system, one likes to register new
subclasses of a plugin baseclass. This can be done as follows:

    class PluginBase:
        subclasses = []

        def __init_subclass__(cls, **kwargs):
            super().__init_subclass__(**kwargs)
            cls.subclasses.append(cls)

In this example, PluginBase.subclasses will contain a plain list of all
subclasses in the entire inheritance tree. One should note that this
also works nicely as a mixin class.

Trait descriptors

There are many designs of Python descriptors in the wild which, for
example, check boundaries of values. Often those "traits" need some
support of a metaclass to work. This is how this would look like with
this PEP:

    class Trait:
        def __init__(self, minimum, maximum):
            self.minimum = minimum
            self.maximum = maximum

        def __get__(self, instance, owner):
            return instance.__dict__[self.key]

        def __set__(self, instance, value):
            if self.minimum < value < self.maximum:
                instance.__dict__[self.key] = value
            else:
                raise ValueError("value not in range")

        def __set_name__(self, owner, name):
            self.key = name

Implementation Details

The hooks are called in the following order: type.__new__ calls the
__set_name__ hooks on the descriptor after the new class has been
initialized. Then it calls __init_subclass__ on the base class, on
super(), to be precise. This means that subclass initializers already
see the fully initialized descriptors. This way, __init_subclass__ users
can fix all descriptors again if this is needed.

Another option would have been to call __set_name__ in the base
implementation of object.__init_subclass__. This way it would be
possible even to prevent __set_name__ from being called. Most of the
times, however, such a prevention would be accidental, as it often
happens that a call to super() is forgotten.

As a third option, all the work could have been done in type.__init__.
Most metaclasses do their work in __new__, as this is recommended by the
documentation. Many metaclasses modify their arguments before they pass
them over to super().__new__. For compatibility with those kind of
classes, the hooks should be called from __new__.

Another small change should be done: in the current implementation of
CPython, type.__init__ explicitly forbids the use of keyword arguments,
while type.__new__ allows for its attributes to be shipped as keyword
arguments. This is weirdly incoherent, and thus it should be forbidden.
While it would be possible to retain the current behavior, it would be
better if this was fixed, as it is probably not used at all: the only
use case would be that at metaclass calls its super().__new__ with name,
bases and dict (yes, dict, not namespace or ns as mostly used with
modern metaclasses) as keyword arguments. This should not be done. This
little change simplifies the implementation of this PEP significantly,
while improving the coherence of Python overall.

As a second change, the new type.__init__ just ignores keyword
arguments. Currently, it insists that no keyword arguments are given.
This leads to a (wanted) error if one gives keyword arguments to a class
declaration if the metaclass does not process them. Metaclass authors
that do want to accept keyword arguments must filter them out by
overriding __init__.

In the new code, it is not __init__ that complains about keyword
arguments, but __init_subclass__, whose default implementation takes no
arguments. In a classical inheritance scheme using the method resolution
order, each __init_subclass__ may take out it's keyword arguments until
none are left, which is checked by the default implementation of
__init_subclass__.

For readers who prefer reading Python over English, this PEP proposes to
replace the current type and object with the following:

    class NewType(type):
        def __new__(cls, *args, **kwargs):
            if len(args) != 3:
                return super().__new__(cls, *args)
            name, bases, ns = args
            init = ns.get('__init_subclass__')
            if isinstance(init, types.FunctionType):
                ns['__init_subclass__'] = classmethod(init)
            self = super().__new__(cls, name, bases, ns)
            for k, v in self.__dict__.items():
                func = getattr(v, '__set_name__', None)
                if func is not None:
                    func(self, k)
            super(self, self).__init_subclass__(**kwargs)
            return self

        def __init__(self, name, bases, ns, **kwargs):
            super().__init__(name, bases, ns)

    class NewObject(object):
        @classmethod
        def __init_subclass__(cls):
            pass

Reference Implementation

The reference implementation for this PEP is attached to issue 27366.

Backward compatibility issues

The exact calling sequence in type.__new__ is slightly changed, raising
fears of backwards compatibility. It should be assured by tests that
common use cases behave as desired.

The following class definitions (except the one defining the metaclass)
continue to fail with a TypeError as superfluous class arguments are
passed:

    class MyMeta(type):
        pass

    class MyClass(metaclass=MyMeta, otherarg=1):
        pass

    MyMeta("MyClass", (), otherargs=1)

    import types
    types.new_class("MyClass", (), dict(metaclass=MyMeta, otherarg=1))
    types.prepare_class("MyClass", (), dict(metaclass=MyMeta, otherarg=1))

A metaclass defining only a __new__ method which is interested in
keyword arguments now does not need to define an __init__ method
anymore, as the default type.__init__ ignores keyword arguments. This is
nicely in line with the recommendation to override __new__ in
metaclasses instead of __init__. The following code does not fail
anymore:

    class MyMeta(type):
        def __new__(cls, name, bases, namespace, otherarg):
            return super().__new__(cls, name, bases, namespace)

    class MyClass(metaclass=MyMeta, otherarg=1):
        pass

Only defining an __init__ method in a metaclass continues to fail with
TypeError if keyword arguments are given:

    class MyMeta(type):
        def __init__(self, name, bases, namespace, otherarg):
            super().__init__(name, bases, namespace)

    class MyClass(metaclass=MyMeta, otherarg=1):
        pass

Defining both __init__ and __new__ continues to work fine.

About the only thing that stops working is passing the arguments of
type.__new__ as keyword arguments:

    class MyMeta(type):
        def __new__(cls, name, bases, namespace):
            return super().__new__(cls, name=name, bases=bases,
                                   dict=namespace)

    class MyClass(metaclass=MyMeta):
        pass

This will now raise TypeError, but this is weird code, and easy to fix
even if someone used this feature.

Rejected Design Options

Calling the hook on the class itself

Adding an __autodecorate__ hook that would be called on the class itself
was the proposed idea of PEP 422. Most examples work the same way or
even better if the hook is called only on strict subclasses. In general,
it is much easier to arrange to explicitly call the hook on the class in
which it is defined (to opt-in to such a behavior) than to opt-out (by
remember to check for cls is __class in the hook body), meaning that one
does not want the hook to be called on the class it is defined in.

This becomes most evident if the class in question is designed as a
mixin: it is very unlikely that the code of the mixin is to be executed
for the mixin class itself, as it is not supposed to be a complete class
on its own.

The original proposal also made major changes in the class
initialization process, rendering it impossible to back-port the
proposal to older Python versions.

When it's desired to also call the hook on the base class, two
mechanisms are available:

1.  Introduce an additional mixin class just to hold the
    __init_subclass__ implementation. The original "base" class can then
    list the new mixin as its first parent class.
2.  Implement the desired behaviour as an independent class decorator,
    and apply that decorator explicitly to the base class, and then
    implicitly to subclasses via __init_subclass__.

Calling __init_subclass__ explicitly from a class decorator will
generally be undesirable, as this will also typically call
__init_subclass__ a second time on the parent class, which is unlikely
to be desired behaviour.

Other variants of calling the hooks

Other names for the hook were presented, namely __decorate__ or
__autodecorate__. This proposal opts for __init_subclass__ as it is very
close to the __init__ method, just for the subclass, while it is not
very close to decorators, as it does not return the class.

For the __set_name__ hook other names have been proposed as well,
__set_owner__, __set_ownership__ and __init_descriptor__.

Requiring an explicit decorator on __init_subclass__

One could require the explicit use of @classmethod on the
__init_subclass__ decorator. It was made implicit since there's no
sensible interpretation for leaving it out, and that case would need to
be detected anyway in order to give a useful error message.

This decision was reinforced after noticing that the user experience of
defining __prepare__ and forgetting the @classmethod method decorator is
singularly incomprehensible (particularly since PEP 3115 documents it as
an ordinary method, and the current documentation doesn't explicitly say
anything one way or the other).

A more __new__-like hook

In PEP 422 the hook worked more like the __new__ method than the
__init__ method, meaning that it returned a class instead of modifying
one. This allows a bit more flexibility, but at the cost of much harder
implementation and undesired side effects.

Adding a class attribute with the attribute order

This got its own PEP 520.

History

This used to be a competing proposal to PEP 422 by Alyssa Coghlan and
Daniel Urban. PEP 422 intended to achieve the same goals as this PEP,
but with a different way of implementation. In the meantime, PEP 422 has
been withdrawn favouring this approach.

References

Copyright

This document has been placed in the public domain.


text:
PEP: 485 Title: A Function for testing approximate equality Version:
$Revision$ Last-Modified: $Date$ Author: Christopher Barker
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 20-Jan-2015 Python-Version: 3.5 Post-History:
Resolution:
https://mail.python.org/pipermail/python-dev/2015-February/138598.html

Abstract

This PEP proposes the addition of an isclose() function to the standard
library math module that determines whether one value is approximately
equal or "close" to another value.

Rationale

Floating point values contain limited precision, which results in their
being unable to exactly represent some values, and for errors to
accumulate with repeated computation. As a result, it is common advice
to only use an equality comparison in very specific situations. Often an
inequality comparison fits the bill, but there are times (often in
testing) where the programmer wants to determine whether a computed
value is "close" to an expected value, without requiring them to be
exactly equal. This is common enough, particularly in testing, and not
always obvious how to do it, that it would be useful addition to the
standard library.

Existing Implementations

The standard library includes the unittest.TestCase.assertAlmostEqual
method, but it:

-   Is buried in the unittest.TestCase class
-   Is an assertion, so you can't use it as a general test at the
    command line, etc. (easily)
-   Is an absolute difference test. Often the measure of difference
    requires, particularly for floating point numbers, a relative error,
    i.e. "Are these two values within x% of each-other?", rather than an
    absolute error. Particularly when the magnitude of the values is
    unknown a priori.

The numpy package has the allclose() and isclose() functions, but they
are only available with numpy.

The statistics package tests include an implementation, used for its
unit tests.

One can also find discussion and sample implementations on Stack
Overflow and other help sites.

Many other non-python systems provide such a test, including the Boost
C++ library and the APL language[1].

These existing implementations indicate that this is a common need and
not trivial to write oneself, making it a candidate for the standard
library.

Proposed Implementation

NOTE: this PEP is the result of extended discussions on the python-ideas
list[2].

The new function will go into the math module, and have the following
signature:

    isclose(a, b, rel_tol=1e-9, abs_tol=0.0)

a and b: are the two values to be tested to relative closeness

rel_tol: is the relative tolerance -- it is the amount of error allowed,
relative to the larger absolute value of a or b. For example, to set a
tolerance of 5%, pass tol=0.05. The default tolerance is 1e-9, which
assures that the two values are the same within about 9 decimal digits.
rel_tol must be greater than 0.0

abs_tol: is a minimum absolute tolerance level -- useful for comparisons
near zero.

Modulo error checking, etc, the function will return the result of:

    abs(a-b) <= max( rel_tol * max(abs(a), abs(b)), abs_tol )

The name, isclose, is selected for consistency with the existing isnan
and isinf.

Handling of non-finite numbers

The IEEE 754 special values of NaN, inf, and -inf will be handled
according to IEEE rules. Specifically, NaN is not considered close to
any other value, including NaN. inf and -inf are only considered close
to themselves.

Non-float types

The primary use-case is expected to be floating point numbers. However,
users may want to compare other numeric types similarly. In theory, it
should work for any type that supports abs(), multiplication,
comparisons, and subtraction. However, the implementation in the math
module is written in C, and thus can not (easily) use python's duck
typing. Rather, the values passed into the function will be converted to
the float type before the calculation is performed. Passing in types (or
values) that cannot be converted to floats will raise an appropriate
Exception (TypeError, ValueError, or OverflowError).

The code will be tested to accommodate at least some values of these
types:

-   Decimal
-   int
-   Fraction
-   complex: For complex, a companion function will be added to the
    cmath module. In cmath.isclose(), the tolerances are specified as
    floats, and the absolute value of the complex values will be used
    for scaling and comparison. If a complex tolerance is passed in, the
    absolute value will be used as the tolerance.

NOTE: it may make sense to add a Decimal.isclose() that works properly
and completely with the decimal type, but that is not included as part
of this PEP.

Behavior near zero

Relative comparison is problematic if either value is zero. By
definition, no value is small relative to zero. And computationally, if
either value is zero, the difference is the absolute value of the other
value, and the computed absolute tolerance will be rel_tol times that
value. When rel_tol is less than one, the difference will never be less
than the tolerance.

However, while mathematically correct, there are many use cases where a
user will need to know if a computed value is "close" to zero. This
calls for an absolute tolerance test. If the user needs to call this
function inside a loop or comprehension, where some, but not all, of the
expected values may be zero, it is important that both a relative
tolerance and absolute tolerance can be tested for with a single
function with a single set of parameters.

There is a similar issue if the two values to be compared straddle zero:
if a is approximately equal to -b, then a and b will never be computed
as "close".

To handle this case, an optional parameter, abs_tol can be used to set a
minimum tolerance used in the case of very small or zero computed
relative tolerance. That is, the values will be always be considered
close if the difference between them is less than abs_tol

The default absolute tolerance value is set to zero because there is no
value that is appropriate for the general case. It is impossible to know
an appropriate value without knowing the likely values expected for a
given use case. If all the values tested are on order of one, then a
value of about 1e-9 might be appropriate, but that would be far too
large if expected values are on order of 1e-9 or smaller.

Any non-zero default might result in user's tests passing totally
inappropriately. If, on the other hand, a test against zero fails the
first time with defaults, a user will be prompted to select an
appropriate value for the problem at hand in order to get the test to
pass.

NOTE: that the author of this PEP has resolved to go back over many of
his tests that use the numpy allclose() function, which provides a
default absolute tolerance, and make sure that the default value is
appropriate.

If the user sets the rel_tol parameter to 0.0, then only the absolute
tolerance will effect the result. While not the goal of the function, it
does allow it to be used as a purely absolute tolerance check as well.

Implementation

A sample implementation in python is available (as of Jan 22, 2015) on
gitHub:

https://github.com/PythonCHB/close_pep/blob/master/is_close.py

This implementation has a flag that lets the user select which relative
tolerance test to apply -- this PEP does not suggest that that be
retained, but rather that the weak test be selected.

There are also drafts of this PEP and test code, etc. there:

https://github.com/PythonCHB/close_pep

Relative Difference

There are essentially two ways to think about how close two numbers are
to each-other:

Absolute difference: simply abs(a-b)

Relative difference: abs(a-b)/scale_factor[3].

The absolute difference is trivial enough that this proposal focuses on
the relative difference.

Usually, the scale factor is some function of the values under
consideration, for instance:

1)  The absolute value of one of the input values
2)  The maximum absolute value of the two
3)  The minimum absolute value of the two.
4)  The absolute value of the arithmetic mean of the two

These lead to the following possibilities for determining if two values,
a and b, are close to each other.

1)  abs(a-b) <= tol*abs(a)
2)  abs(a-b) <= tol * max( abs(a), abs(b) )
3)  abs(a-b) <= tol * min( abs(a), abs(b) )
4)  abs(a-b) <= tol * abs(a + b)/2

NOTE: (2) and (3) can also be written as:

2)  (abs(a-b) <= abs(tol*a)) or (abs(a-b) <= abs(tol*b))
3)  (abs(a-b) <= abs(tol*a)) and (abs(a-b) <= abs(tol*b))

(Boost refers to these as the "weak" and "strong" formulations[4]) These
can be a tiny bit more computationally efficient, and thus are used in
the example code.

Each of these formulations can lead to slightly different results.
However, if the tolerance value is small, the differences are quite
small. In fact, often less than available floating point precision.

How much difference does it make?

When selecting a method to determine closeness, one might want to know
how much of a difference it could make to use one test or the other --
i.e. how many values are there (or what range of values) that will pass
one test, but not the other.

The largest difference is between options (2) and (3) where the
allowable absolute difference is scaled by either the larger or smaller
of the values.

Define delta to be the difference between the allowable absolute
tolerance defined by the larger value and that defined by the smaller
value. That is, the amount that the two input values need to be
different in order to get a different result from the two tests. tol is
the relative tolerance value.

Assume that a is the larger value and that both a and b are positive, to
make the analysis a bit easier. delta is therefore:

    delta = tol * (a-b)

or:

    delta / tol = (a-b)

The largest absolute difference that would pass the test: (a-b), equals
the tolerance times the larger value:

    (a-b) = tol * a

Substituting into the expression for delta:

    delta / tol = tol * a

so:

    delta = tol**2 * a

For example, for a = 10, b = 9, tol = 0.1 (10%):

maximum tolerance tol * a == 0.1 * 10 == 1.0

minimum tolerance tol * b == 0.1 * 9.0 == 0.9

delta = (1.0 - 0.9) = 0.1 or tol**2 * a = 0.1**2 * 10 = .1

The absolute difference between the maximum and minimum tolerance tests
in this case could be substantial. However, the primary use case for the
proposed function is testing the results of computations. In that case a
relative tolerance is likely to be selected of much smaller magnitude.

For example, a relative tolerance of 1e-8 is about half the precision
available in a python float. In that case, the difference between the
two tests is 1e-8**2 * a or 1e-16 * a, which is close to the limit of
precision of a python float. If the relative tolerance is set to the
proposed default of 1e-9 (or smaller), the difference between the two
tests will be lost to the limits of precision of floating point. That
is, each of the four methods will yield exactly the same results for all
values of a and b.

In addition, in common use, tolerances are defined to 1 significant
figure -- that is, 1e-9 is specifying about 9 decimal digits of
accuracy. So the difference between the various possible tests is well
below the precision to which the tolerance is specified.

Symmetry

A relative comparison can be either symmetric or non-symmetric. For a
symmetric algorithm:

isclose(a,b) is always the same as isclose(b,a)

If a relative closeness test uses only one of the values (such as (1)
above), then the result is asymmetric, i.e. isclose(a,b) is not
necessarily the same as isclose(b,a).

Which approach is most appropriate depends on what question is being
asked. If the question is: "are these two numbers close to each other?",
there is no obvious ordering, and a symmetric test is most appropriate.

However, if the question is: "Is the computed value within x% of this
known value?", then it is appropriate to scale the tolerance to the
known value, and an asymmetric test is most appropriate.

From the previous section, it is clear that either approach would yield
the same or similar results in the common use cases. In that case, the
goal of this proposal is to provide a function that is least likely to
produce surprising results.

The symmetric approach provide an appealing consistency -- it mirrors
the symmetry of equality, and is less likely to confuse people. A
symmetric test also relieves the user of the need to think about the
order in which to set the arguments. It was also pointed out that there
may be some cases where the order of evaluation may not be well defined,
for instance in the case of comparing a set of values all against each
other.

There may be cases when a user does need to know that a value is within
a particular range of a known value. In that case, it is easy enough to
simply write the test directly:

    if a-b <= tol*a:

(assuming a > b in this case). There is little need to provide a
function for this particular case.

This proposal uses a symmetric test.

Which symmetric test?

There are three symmetric tests considered:

The case that uses the arithmetic mean of the two values requires that
the value be either added together before dividing by 2, which could
result in extra overflow to inf for very large numbers, or require each
value to be divided by two before being added together, which could
result in underflow to zero for very small numbers. This effect would
only occur at the very limit of float values, but it was decided there
was no benefit to the method worth reducing the range of functionality
or adding the complexity of checking values to determine the order of
computation.

This leaves the boost "weak" test (2)-- or using the larger value to
scale the tolerance, or the Boost "strong" (3) test, which uses the
smaller of the values to scale the tolerance. For small tolerance, they
yield the same result, but this proposal uses the boost "weak" test
case: it is symmetric and provides a more useful result for very large
tolerances.

Large Tolerances

The most common use case is expected to be small tolerances -- on order
of the default 1e-9. However, there may be use cases where a user wants
to know if two fairly disparate values are within a particular range of
each other: "is a within 200% (rel_tol = 2.0) of b? In this case, the
strong test would never indicate that two values are within that range
of each other if one of them is zero. The weak case, however would use
the larger (non-zero) value for the test, and thus return true if one
value is zero. For example: is 0 within 200% of 10? 200% of ten is 20,
so the range within 200% of ten is -10 to +30. Zero falls within that
range, so it will return True.

Defaults

Default values are required for the relative and absolute tolerance.

Relative Tolerance Default

The relative tolerance required for two values to be considered "close"
is entirely use-case dependent. Nevertheless, the relative tolerance
needs to be greater than 1e-16 (approximate precision of a python
float). The value of 1e-9 was selected because it is the largest
relative tolerance for which the various possible methods will yield the
same result, and it is also about half of the precision available to a
python float. In the general case, a good numerical algorithm is not
expected to lose more than about half of available digits of accuracy,
and if a much larger tolerance is acceptable, the user should be
considering the proper value in that case. Thus 1e-9 is expected to
"just work" for many cases.

Absolute tolerance default

The absolute tolerance value will be used primarily for comparing to
zero. The absolute tolerance required to determine if a value is "close"
to zero is entirely use-case dependent. There is also essentially no
bounds to the useful range -- expected values would conceivably be
anywhere within the limits of a python float. Thus a default of 0.0 is
selected.

If, for a given use case, a user needs to compare to zero, the test will
be guaranteed to fail the first time, and the user can select an
appropriate value.

It was suggested that comparing to zero is, in fact, a common use case
(evidence suggest that the numpy functions are often used with zero). In
this case, it would be desirable to have a "useful" default. Values
around 1e-8 were suggested, being about half of floating point precision
for values of around value 1.

However, to quote The Zen: "In the face of ambiguity, refuse the
temptation to guess." Guessing that users will most often be concerned
with values close to 1.0 would lead to spurious passing tests when used
with smaller values -- this is potentially more damaging than requiring
the user to thoughtfully select an appropriate value.

Expected Uses

The primary expected use case is various forms of testing -- "are the
results computed near what I expect as a result?" This sort of test may
or may not be part of a formal unit testing suite. Such testing could be
used one-off at the command line, in an IPython notebook, part of
doctests, or simple asserts in an if __name__ == "__main__" block.

It would also be an appropriate function to use for the termination
criteria for a simple iterative solution to an implicit function:

    guess = something
    while True:
        new_guess = implicit_function(guess, *args)
        if isclose(new_guess, guess):
            break
        guess = new_guess

Inappropriate uses

One use case for floating point comparison is testing the accuracy of a
numerical algorithm. However, in this case, the numerical analyst
ideally would be doing careful error propagation analysis, and should
understand exactly what to test for. It is also likely that ULP (Unit in
the Last Place) comparison may be called for. While this function may
prove useful in such situations, It is not intended to be used in that
way without careful consideration.

Other Approaches

unittest.TestCase.assertAlmostEqual

(https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertAlmostEqual)

Tests that values are approximately (or not approximately) equal by
computing the difference, rounding to the given number of decimal places
(default 7), and comparing to zero.

This method is purely an absolute tolerance test, and does not address
the need for a relative tolerance test.

numpy isclose()

http://docs.scipy.org/doc/numpy-dev/reference/generated/numpy.isclose.html

The numpy package provides the vectorized functions isclose() and
allclose(), for similar use cases as this proposal:

isclose(a, b, rtol=1e-05, atol=1e-08, equal_nan=False)

  Returns a boolean array where two arrays are element-wise equal within
  a tolerance.

  The tolerance values are positive, typically very small numbers. The
  relative difference (rtol * abs(b)) and the absolute difference atol
  are added together to compare against the absolute difference between
  a and b

In this approach, the absolute and relative tolerance are added
together, rather than the or method used in this proposal. This is
computationally more simple, and if relative tolerance is larger than
the absolute tolerance, then the addition will have no effect. However,
if the absolute and relative tolerances are of similar magnitude, then
the allowed difference will be about twice as large as expected.

This makes the function harder to understand, with no computational
advantage in this context.

Even more critically, if the values passed in are small compared to the
absolute tolerance, then the relative tolerance will be completely
swamped, perhaps unexpectedly.

This is why, in this proposal, the absolute tolerance defaults to zero
-- the user will be required to choose a value appropriate for the
values at hand.

Boost floating-point comparison

The Boost project ([5] ) provides a floating point comparison function.
It is a symmetric approach, with both "weak" (larger of the two relative
errors) and "strong" (smaller of the two relative errors) options. This
proposal uses the Boost "weak" approach. There is no need to complicate
the API by providing the option to select different methods when the
results will be similar in most cases, and the user is unlikely to know
which to select in any case.

Alternate Proposals

A Recipe

The primary alternate proposal was to not provide a standard library
function at all, but rather, provide a recipe for users to refer to.
This would have the advantage that the recipe could provide and explain
the various options, and let the user select that which is most
appropriate. However, that would require anyone needing such a test to,
at the very least, copy the function into their code base, and select
the comparison method to use.

zero_tol

One possibility was to provide a zero tolerance parameter, rather than
the absolute tolerance parameter. This would be an absolute tolerance
that would only be applied in the case of one of the arguments being
exactly zero. This would have the advantage of retaining the full
relative tolerance behavior for all non-zero values, while allowing
tests against zero to work. However, it would also result in the
potentially surprising result that a small value could be "close" to
zero, but not "close" to an even smaller value. e.g., 1e-10 is "close"
to zero, but not "close" to 1e-11.

No absolute tolerance

Given the issues with comparing to zero, another possibility would have
been to only provide a relative tolerance, and let comparison to zero
fail. In this case, the user would need to do a simple absolute test:
abs(val) < zero_tol in the case where the comparison involved zero.

However, this would not allow the same call to be used for a sequence of
values, such as in a loop or comprehension. Making the function far less
useful. It is noted that the default abs_tol=0.0 achieves the same
effect if the default is not overridden.

Other tests

The other tests considered are all discussed in the Relative Error
section above.

References

  https://randomascii.wordpress.com/2012/02/25/comparing-floating-point-numbers-2012-edition/

Copyright

This document has been placed in the public domain.

[1] 1976. R. H. Lathwell. APL comparison tolerance. Proceedings of the
eighth international conference on APL Pages 255 - 258

http://dl.acm.org/citation.cfm?doid=800114.803685

[2] Python-ideas list discussion threads

https://mail.python.org/pipermail/python-ideas/2015-January/030947.html

https://mail.python.org/pipermail/python-ideas/2015-January/031124.html

https://mail.python.org/pipermail/python-ideas/2015-January/031313.html

[3] Wikipedia page on relative difference

http://en.wikipedia.org/wiki/Relative_change_and_difference

[4] Boost project floating-point comparison algorithms

http://www.boost.org/doc/libs/1_35_0/libs/test/doc/components/test_tools/floating_point_comparison.html

[5] Boost project floating-point comparison algorithms

http://www.boost.org/doc/libs/1_35_0/libs/test/doc/components/test_tools/floating_point_comparison.html

license: Public Domain


text:
PEP: 708 Title: Extending the Repository API to Mitigate Dependency
Confusion Attacks Author: Donald Stufft<EMAIL_ADDRESS>PEP-Delegate:
Paul Moore<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/24179 Status: Provisional Type: Standards
Track Topic: Packaging Content-Type: text/x-rst Created: 20-Feb-2023
Post-History: 01-Feb-2023, 23-Feb-2023 Resolution:
https://discuss.python.org/t/24179/72

Provisional Acceptance

This PEP has been provisionally accepted, with the following required
conditions before the PEP is made Final:

1.  An implementation of the PEP in PyPI (Warehouse) including any
    necessary UI elements to allow project owners to set the tracking
    data.
2.  An implementation of the PEP in at least one repository other than
    PyPI, as you can’t really test merging indexes without at least two
    indexes.
3.  An implementation of the PEP in pip, which supports the intended
    semantics and can be used to demonstrate that the expected security
    benefits are achieved. This implementation will need to be "off by
    default" initially, which means that users will have to opt in to
    testing it. Ideally, we should collect explicit positive reports
    from users (both project owners and project users) who have
    successfully tried out the new feature, rather than just assuming
    that "no news is good news".

Abstract

Dependency confusion attacks, in which a malicious package is installed
instead of the one the user expected, are an increasingly common supply
chain threat. Most such attacks against Python dependencies, including
the recent PyTorch incident, occur with multiple package repositories,
where a dependency expected to come from one repository (e.g. a custom
index) is installed from another (e.g. PyPI).

To help address this problem, this PEP proposes extending the
Simple Repository API <packaging:simple-repository-api> to allow
repository operators to indicate that a project found on their
repository "tracks" a project on different repositories, and allows
projects to extend their namespaces across multiple repositories.

These features will allow installers to determine when a project being
made available from a particular mix of repositories is expected and
should be allowed, and when it is not and should halt the install with
an error to protect the user.

Motivation

There is a long-standing class of attacks that are called "dependency
confusion" attacks, which roughly boil down to an individual user
expected to get package A, but instead they got B. In Python, this
almost always happens due to the configuration of multiple repositories
(possibly including the default of PyPI), where they expected package A
to come from repository X, but someone is able to publish package B to
repository Y under the same name.

Dependency Confusion attacks have long been possible, but they've
recently gained press with public examples of cases where these attacks
were successfully executed.

A specific example of this is the recent case where the PyTorch project
had an internal package named torchtriton which was only ever intended
to be installed from their repositories located at
https://download.pytorch.org/, but that repository was designed to be
used in conjunction with PyPI, and the name of torchtriton was not
claimed on PyPI, which allowed the attacker to use that name and publish
a malicious version.

There are a number of ways to mitigate against these attacks today, but
they all require that the end user go out of their way to protect
themselves, rather than being protected by default. This means that for
the vast bulk of users, they are likely to remain vulnerable, even if
they are ultimately aware of these types of attacks.

Ultimately the underlying cause of these attacks come from the fact that
there is no globally unique namespace that all Python package names come
from. Instead, each repository is its own distinct namespace, and when
given an "abstract" name such as spam to install, an installer has to
implicitly turn that into a "concrete" name such as pypi.org:spam or
example.com:spam. Currently the standard behavior in Python installation
tools is to implicitly flatten these multiple namespaces into one that
contains the files from all namespaces.

This assumption that collapsing the namespaces is what was expected
means that when packages with the same name in different repositories
are authored by different parties (such as in the torchtriton case)
dependency confusion attacks become possible.

This is made particularly tricky in that there is no "right" answer;
there are valid use cases both for wanting two repositories merged into
one namespace and for wanting two repositories to be treated as distinct
namespaces. This means that an installer needs some mechanism by which
to determine when it should merge the namespaces of multiple
repositories and when it should not, rather than a blanket always merge
or never merge rule.

This functionality could be pushed directly to the end user, since
ultimately the end user is the person whose expectations of what gets
installed from what repository actually matters. However, by extending
the repository specification to allow a repository to indicate when it
is safe, we can enable individual projects and repositories to "work by
default", even when their project naturally spans multiple distinct
namespaces, while maintaining the ability for an installer to be secure
by default.

On its own, this PEP does not solve dependency confusion attacks, but
what it does do is provide enough information so that installers can
prevent them without causing too much collateral damage to otherwise
valid and safe use cases.

Rationale

There are two broad use cases for merging names across repositories that
this PEP seeks to enable.

The first use case is when one repository is not defining its own names,
but rather is extending names defined in other repositories. This
commonly happens in cases where a project is being mirrored from one
repository to another (see Bandersnatch) or when a repository is
providing supplementary artifacts for a specific platform (see
Piwheels).

In this case neither the repositories nor the projects that are being
extended may have any knowledge that they are being extended or by whom,
so this cannot rely on any information that isn't present in the
"extending" repository itself.

The second use case is when the project wants to publish to one "main"
repository, but then have additional repositories that provide binaries
for additional platforms, GPUs, CPUs, etc. Currently wheel tags are not
sufficiently able to express these types of binary compatibility, so
projects that wish to rely on them are forced to set up multiple
repositories and have their users manually configure them to get the
correct binaries for their platform, GPU, CPU, etc.

This use case is similar to the first, but the important difference that
makes it a distinct use case on its own is who is providing the
information and what their level of trust is.

When a user configures a specific repository (or relies on the default)
there is no ambiguity as to what repository they mean. A repository is
identified by an URL, and through the domain system, URLs are globally
unique identifiers. This lack of ambiguity means that an installer can
assume that the repository operator is trustworthy and can trust
metadata that they provide without needing to validate it.

On the flip side, given an installer finds a name in multiple
repositories it is ambiguous which of them the installer should trust.
This ambiguity means that an installer cannot assume that the project
owner on either repository is trustworthy and needs to validate that
they are indeed the same project and that one isn't a dependency
confusion attack.

Without some way for the installer to validate the metadata between
multiple repositories, projects would be forced into becoming repository
operators to safely support this use case. That wouldn't be a
particularly wrong choice to make; however, there is a danger that if we
don't provide a way for repositories to let project owners express this
relationship safely, they will be incentivized to let them use the
repository operator's metadata instead which would reintroduce the
original insecurity.

Specification

This specification defines the changes in version 1.2 of the simple
repository API, adding new two new metadata items: Repository "Tracks"
and "Alternate Locations".

Repository "Tracks" Metadata

To enable one repository to host a project that is intended to "extend"
a project that is hosted at other repositories, this PEP allows the
extending repository to declare that a particular project "tracks" a
project at another repository or repositories by adding the URLs of the
project and repositories that it is extending.

This is exposed in JSON as the key meta.tracks and in HTML as a meta
element named pypi:tracks on the project specific URLs,
($root/$project/).

There are a few key properties that MUST be preserved when using this
metadata:

-   It MUST be under the control of the repository operators themselves,
    not any individual publisher using that repository.
    -   "Repository Operator" can also include anyone who managed the
        overall namespace for a particular repository, which may be the
        case in situations like hosted repository services where one
        entity operates the software but another owns/manages the entire
        namespace of that repository.
-   All URLs MUST represent the same "project" as the project in the
    extending repository.
    -   This does not mean that they need to serve the same files. It is
        valid for them to include binaries built on different platforms,
        copies with local patches being applied, etc. This is
        purposefully left vague as it's ultimately up to the
        expectations that the users have of the repository and its
        operators what exactly constitutes the "same" project.
-   It MUST point to the repositories that "own" the namespaces, not
    another repository that is also tracking that namespace.
-   It MUST point to a project with the exact same name (after
    normalization).
-   It MUST point to the actual URLs for that project, not the base URL
    for the extended repositories.

It is NOT required that every name in a repository tracks the same
repository, or that they all track a repository at all. Mixed use
repositories where some names track a repository and some names do not
are explicitly allowed.

JSON

    {
      "meta": {
        "api-version": "1.2",
        "tracks": ["https://pypi.org/simple/holygrail/", "https://test.pypi.org/simple/holygrail/"]
      },
      "name": "holygrail",
      "files": [
        {
          "filename": "holygrail-1.0.tar.gz",
          "url": "https://example.com/files/holygrail-1.0.tar.gz",
          "hashes": {"sha256": "...", "blake2b": "..."},
          "requires-python": ">=3.7",
          "yanked": "Had a vulnerability"
        },
        {
          "filename": "holygrail-1.0-py3-none-any.whl",
          "url": "https://example.com/files/holygrail-1.0-py3-none-any.whl",
          "hashes": {"sha256": "...", "blake2b": "..."},
          "requires-python": ">=3.7",
          "dist-info-metadata": true
        }
      ]
    }

HTML

    <!DOCTYPE html>

        <meta name="pypi:repository-version" content="1.2">
        <meta name="pypi:tracks" content="https://pypi.org/simple/holygrail/">
        <meta name="pypi:tracks" content="https://test.pypi.org/simple/holygrail/">

"Alternate Locations" Metadata

To enable a project to extend its namespace across multiple
repositories, this PEP allows a project owner to declare a list of
"alternate locations" for their project. This is exposed in JSON as the
key alternate-locations and in HTML as a meta element named
pypi-alternate-locations, which may be used multiple times.

There are a few key properties that MUST be observed when using this
metadata:

-   In order for this metadata to be trusted, there MUST be agreement
    between all locations where that project is found as to what the
    alternate locations are.
-   When using alternate locations, clients MUST implicitly assume that
    the url the response was fetched from was included in the list. This
    means that if you fetch from https://pypi.org/simple/foo/ and it has
    an alternate-locations metadata that has the value
    ["https://example.com/simple/foo/"], then you MUST treat it as if it
    had the value
    ["https://example.com/simple/foo/", "https://pypi.org/simple/foo/"].
-   Order of the elements within the array does not have any particular
    meaning.

When an installer encounters a project that is using the alternate
locations metadata it SHOULD consider that all repositories named are
extending the same namespace across multiple repositories.

Note

This alternate locations metadata is project level metadata, not
artifact level metadata, which means it doesn't get included as part of
the core metadata spec, but rather it is something that each repository
will have to provide a configuration option for (if they choose to
support it).

JSON

    {
      "meta": {
        "api-version": "1.2"
      },
      "name": "holygrail",
      "alternate-locations": ["https://pypi.org/simple/holygrail/", "https://test.pypi.org/simple/holygrail/"],
      "files": [
        {
          "filename": "holygrail-1.0.tar.gz",
          "url": "https://example.com/files/holygrail-1.0.tar.gz",
          "hashes": {"sha256": "...", "blake2b": "..."},
          "requires-python": ">=3.7",
          "yanked": "Had a vulnerability"
        },
        {
          "filename": "holygrail-1.0-py3-none-any.whl",
          "url": "https://example.com/files/holygrail-1.0-py3-none-any.whl",
          "hashes": {"sha256": "...", "blake2b": "..."},
          "requires-python": ">=3.7",
          "dist-info-metadata": true
        }
      ]
    }

HTML

    <!DOCTYPE html>

        <meta name="pypi:repository-version" content="1.2">
        <meta name="pypi:alternate-locations" content="https://pypi.org/simple/holygrail/">
        <meta name="pypi:alternate-locations" content="https://test.pypi.org/simple/holygrail/">

Recommendations

This section is non-normative; it provides recommendations to installers
in how to interpret this metadata that this PEP feels provides the best
tradeoff between protecting users by default and minimizing breakages to
existing workflows. These recommendations are not binding, and
installers are free to ignore them, or apply them selectively as they
make sense in their specific situations.

File Discovery Algorithm

Note

This algorithm is written based on how pip currently discovers files;
other installers may adapt this based on their own discovery procedures.

Currently the "standard" file discovery algorithm looks something like
this:

1.  Generate a list of all files across all configured repositories.
2.  Filter out any files that do not match known hashes from a lockfile
    or requirements file.
3.  Filter out any files that do not match the current platform, Python
    version, etc.
4.  Pass that list of files into the resolver where it will attempt to
    resolve the "best" match out of those files, irrespective of which
    repository it came from.

It is recommended that installers change their file discovery algorithm
to take into account the new metadata, and instead do:

1.  Generate a list of all files across all configured repositories.
2.  Filter out any files that do not match known hashes from a lockfile
    or requirements file.
3.  If the end user has explicitly told the installer to fetch the
    project from specific repositories, filter out all other
    repositories and skip to 5.
4.  Look to see if the discovered files span multiple repositories; if
    they do then determine if either "Tracks" or "Alternate Locations"
    metadata allows safely merging ALL of the repositories where files
    were discovered together. If that metadata does NOT allow that, then
    generate an error, otherwise continue.
    -   Note: This only applies to remote repositories; repositories
        that exist on the local filesystem SHOULD always be implicitly
        allowed to be merged to any remote repository.
5.  Filter out any files that do not match the current platform, Python
    version, etc.
6.  Pass that list of files into the resolver where it will attempt to
    resolve the "best" match out of those files, irrespective of what
    repository it came from.

This is somewhat subtle, but the key things in the recommendation are:

-   Users who are using lock files or requirements files that include
    specific hashes of artifacts that are "valid" are assumed to be
    protected by nature of those hashes, since the rest of these
    recommendations would apply during hash generation. Thus, we filter
    out unknown hashes up front.
-   If the user has explicitly told the installer that it wants to fetch
    a project from a certain set of repositories, then there is no
    reason to question that and we assume that they've made sure it is
    safe to merge those namespaces.
-   If the project in question only comes from a single repository, then
    there is no chance of dependency confusion, so there's no reason to
    do anything but allow.
-   We check for the metadata in this PEP before filtering out based on
    platform, Python version, etc., because we don't want errors that
    only show up on certain platforms, Python versions, etc.
-   If nothing tells us merging the namespaces is safe, we refuse to
    implicitly assume it is, and generate an error instead.
-   Otherwise we merge the namespaces, and continue on.

This algorithm ensures that an installer never assumes that two
disparate namespaces can be flattened into one, which for all practical
purposes eliminates the possibility of any kind of dependency confusion
attack, while still giving power throughout the stack in a safe way to
allow people to explicitly declare when those disparate namespaces are
actually one logical namespace that can be safely merged.

The above algorithm is mostly a conceptual model. In reality the
algorithm may end up being slightly different in order to be more
privacy preserving and faster, or even just adapted to fit a specific
installer better.

Explicit Configuration for End Users

This PEP avoids dictating or recommending a specific mechanism by which
an installer allows an end user to configure exactly what repositories
they want a specific package to be installed from. However, it does
recommend that installers do provide some mechanism for end users to
provide that configuration, as without it users can end up in a DoS
situation in cases like torchtriton where they're just completely broken
unless they resolve the namespace collision externally (get the name
taken down on one repository, stand up a personal repository that
handles the merging, etc).

This configuration also allows end users to pre-emptively secure
themselves during what is likely to be a long transition until the
default behavior is safe.

How to Communicate This

Note

This example is pip specific and assumes specifics about how pip will
choose to implement this PEP; it's included as an example of how we can
communicate this change, and not intended to constrain pip or any other
installer in how they implement this. This may ultimately be the actual
basis for communication, and if so will need be edited for accuracy and
clarity.

This section should be read as if it were an entire "post" to
communicate this change that could be used for a blog post, email, or
discourse post.

There's a long-standing class of attacks that are called "dependency
confusion" attacks, which roughly boil down to an individual expected to
get package A, but instead they got B. In Python, this almost always
happens due to the end user having configured multiple repositories,
where they expect package A to come from repository X, but someone is
able to publish package B with the same name as package A in repository
Y.

There are a number of ways to mitigate against these attacks today, but
they all require that the end user explicitly go out of their way to
protect themselves, rather than it being inherently safe.

In an effort to secure pip's users and protect them from these types of
attacks, we will be changing how pip discovers packages to install.

What is Changing?

When pip discovers that the same project is available from multiple
remote repositories, by default it will generate an error and refuse to
proceed rather than make a guess about which repository was the correct
one to install from.

Projects that natively publish to multiple repositories will be given
the ability to safely "link" their repositories together so that pip
does not error when those repositories are used together.

End users of pip will be given the ability to explicitly define one or
more repositories that are valid for a specific project, causing pip to
only consider those repositories for that project, and avoiding
generating an error altogether.

See TBD for more information.

Who is Affected?

Users who are installing from multiple remote (e.g. not present on the
local filesystem) repositories may be affected by having pip error
instead of successfully install if:

-   They install a project where the same "name" is being served by
    multiple remote repositories.
-   The project name that is available from multiple remote repositories
    has not used one of the defined mechanisms to link those
    repositories together.
-   The user invoking pip has not used the defined mechanism to
    explicitly control what repositories are valid for a particular
    project.

Users who are not using multiple remote repositories will not be
affected at all, which includes users who are only using a single remote
repository, plus a local filesystem "wheel house".

What do I need to do?

As a pip User?

If you're using only a single remote repository you do not have to do
anything.

If you're using multiple remote repositories, you can opt into the new
behavior by adding --use-feature=TBD to your pip invocation to see if
any of your dependencies are being served from multiple remote
repositories. If they are, you should audit them to determine why they
are, and what the best remediation step will be for you.

Once this behavior becomes the default, you can opt out of it
temporarily by adding --use-deprecated=TBD to your pip invocation.

If you're using projects that are not hosted on a public repository, but
you still have the public repository as a fallback, consider configuring
pip with a repository file to be explicit where that dependency is meant
to come from to prevent registration of that name in a public repository
to cause pip to error for you.

As a Project Owner?

If you only publish your project to a single repository, then you do not
have to do anything.

If you publish your project to multiple repositories that are intended
to be used together at the same time, configure all repositories to
serve the alternate repository metadata to prevent breakages for your
end users.

If you publish your project to a single repository, but it is commonly
used in conjunction with other repositories, consider preemptively
registering your names with those repositories to prevent a third party
from being able to cause your users pip install invocations to start
failing. This may not be available if your project name is too generic
or if the repositories have policies that prevent defensive name
squatting.

As a Repository Operator?

You'll need to decide how you intend for your repository to be used by
your end users and how you want them to use it.

For private repositories that host private projects, it is recommended
that you mirror the public projects that your users depend on into your
own repository, taking care not to let a public project merge with a
private project, and tell your users to use the --index-url option to
use only your repository.

For public repositories that host public projects, you should implement
the alternate repository mechanism and enable the owners of those
projects to configure the list of repositories that their project is
available from if they make it available from more than one repository.

For public repositories that "track" another repository, but provide
supplemental artifacts such as wheels built for a specific platform, you
should implement the "tracks" metadata for your repository. However,
this information MUST NOT be settable by end users who are publishing
projects to your repository. See TBD for more information.

Rejected Ideas

Note: Some of these are somewhat specific to pip, but any solution that
doesn't work for pip isn't a particularly useful solution.

Implicitly allow mirrors when the list of files are the same

If every repository returns the exact same list of files, then it is
safe to consider those repositories to be the same namespace and
implicitly merge them. This would possibly mean that mirrors would be
automatically allowed without any work on any user or repository
operator's part.

Unfortunately, this has two failings that make it undesirable:

-   It only solves the case of mirrors that are exact copies of each
    other, but not repositories that "track" another one, which ends up
    being a more generic solution.
-   Even in the case of exact mirrors, multiple repositories mirroring
    each other is a distributed system will not always be fully
    consistent with each other, effectively an eventually consistent
    system. This means that repositories that relied on this implicit
    heuristic to work would have sporadic failures due to drift between
    the source repository and the mirror repositories.

Provide a mechanism to order the repositories

Providing some mechanism to give the repositories an order, and then
short circuiting the discovery algorithm when it finds the first
repository that provides files for that project is another workable
solution that is safe if the order is specified correctly.

However, this has been rejected for a number of reasons:

-   We've spent 15+ years educating users that the ordering of
    repositories being specified is not meaningful, and they effectively
    have an undefined order. It would be difficult to backpedal on that
    and start saying that now order matters.
-   Users can easily rearrange the order that they specify their
    repositories in within a single location, but when loading
    repositories from multiple locations (env var, conf file,
    requirements file, cli arguments) the order is hard coded into pip.
    While it would be a deterministic and documented order, there's no
    reason to assume it's the order that the user wants their
    repositories to be defined in, forcing them to contort how they
    configure pip so that the implicit ordering ends up being the
    correct one.
-   The above can be mitigated by providing a way to explicitly declare
    the order rather than by implicitly using the order they were
    defined in; however, that then means that the protections are not
    provided unless the user does some explicit configuration.
-   Ordering assumes that one repository is always preferred over
    another repository without any way to decide on a project by project
    basis.
-   Relying on ordering is subtle; if I look at an ordering of
    repositories, I have no way of knowing or ensuring in advance what
    names are going to come from what repositories. I can only know in
    that moment what names are provided by which repositories.
-   Relying on ordering is fragile. There's no reason to assume that two
    disparate repositories are not going to have random naming
    collisions—what happens if I'm using a library from a lower priority
    repository and then a higher priority repository happens to start
    having a colliding name?
-   In cases where ordering does the wrong thing, it does so silently,
    with no feedback given to the user. This is by design because it
    doesn't actually know what the wrong or right thing is, it's just
    hoping that order will give the right thing, and if it does then
    users are protected without any breakage. However, when it does the
    wrong thing, users are left with a very confusing behavior coming
    from pip, where it's just silently installing the wrong thing.

There is a variant of this idea which effectively says that it's really
just PyPI's nature of open registration that causes the real problems,
so if we treat all repositories but the "default" one as equal priority,
and then treat the default one as a lower priority then we'll fix
things.

That is true in that it does improve things, but it has many of the same
problems as the general ordering idea (though not all of them).

It also assumes that PyPI, or whatever repository is configured as the
"default", is the only repository with open registration of names.
However, projects like Piwheels exist which users are expected to use in
addition to PyPI, which also effectively have open registration of names
since it tracks whatever names are registered on PyPI.

Rely on repository proxies

One possible solution is to instead of having the installer have to
solve this, to instead depend on repository proxies that can
intelligently merge multiple repositories safely. This could provide a
better experience for people with complex needs because they can have
configuration and features that are dedicated to the problem space.

However, that has been rejected because:

-   It requires users to opt into using them, unless we also remove the
    facilities to have more than one repository in installers to force
    users into using a repository proxy when they need multiple
    repositories.
    -   Removing facilities to have more than one repository configured
        has been rejected because it would be too disruptive to end
        users.
-   A user may need different outcomes of merging multiple repositories
    in different contexts, or may need to merge different, mutually
    exclusive repositories. This means they'll need to actually set up
    multiple repository proxies for each unique set of options.
-   It requires users to maintain infrastructure or it requires adding
    features in installers to automatically spin up a repository for
    each invocation.
-   It doesn't actually change the requirement to need to have a
    solution to these problems, it just shifts the responsibility of
    implementation from installers to some repository proxy, but in
    either case we still need something that figures out how to merge
    these disparate namespaces.
-   Ultimately, most users do not want to have to stand up a repository
    proxy just to safely interact with multiple repositories.

Rely only on hash checking

Another possible solution is to rely on hash checking, since with hash
checking enabled users cannot get an artifact that they didn't expect;
it doesn't matter if the namespaces are incorrectly merged or not.

This is certainly a solution; unfortunately it also suffers from
problems that make it unworkable:

-   It requires users to opt in to it, so users are still unprotected by
    default.
-   It requires users to do a bunch of labor to manage their hashes,
    which is something that most users are unlikely to be willing to do.
-   It is difficult and verbose to get the protection when users are not
    using a requirements.txt file as the source of their dependencies
    (this affects build time dependencies, and dependencies provided at
    the command line).
-   It only sort of solves the problem, in a way it just shifts the
    responsibility of the problem to be whatever system is generating
    the hashes that the installer would use. If that system isn't a
    human manually validating hashes, which it's unlikely it would be,
    then we've just shifted the question of how to merge these
    namespaces to whatever tool implements the maintenance of the
    hashes.

Require all projects to exist in the "default" repository

Another idea is that we can narrow the scope of --extra-index-url such
that its only supported use is to refer to supplemental repositories to
the default repository, effectively saying that the default repository
defines the namespace, and every additional repository just extends it
with extra packages.

The implementation of this would roughly be to require that the project
MUST be registered with the default repository in order for any
additional repositories to work.

This sort of works if you successfully narrow the scope in that way, but
ultimately it has been rejected because:

-   Users are unlikely to understand or accept this reduced scope, and
    thus are likely to attempt to continue to use it in the now
    unsupported fashion.
    -   This is complicated by the fact that with the scope now
        narrowed, users who have the excluded workflow no longer have
        any alternative besides setting up a repository proxy, which
        takes infrastructure and effort that they previously didn't have
        to do.
-   It assumes that just because a name in an "extra" repository is the
    same as in the default repository, that they are the same project.
    If we were starting from scratch in a brand new ecosystem then maybe
    we could make this assumption from the start and make it stick, but
    it's going to be incredibly difficult to get the ecosystem to adjust
    to that change.
    -   This is a fundamental issue with this approach; the underlying
        problem that drives dependency confusion is that we're taking
        disparate namespaces and flattening them into one. This approach
        essentially just declares that OK, and attempts to mitigate it
        by requiring everyone to register their names.
-   Because of the above assumption, in cases where a name in an extra
    repository collides by accident with the default repository, it's
    going to appear to work for those users, but they are going to be
    silently in a state of dependency confusion.
    -   This is made worse by the fact that the person who owns the name
        that is allowing this to work is going to be completely unaware
        of the role that they're playing for that user, and might
        possibly delete their project or hand it off to someone else,
        potentially allowing them to inadvertently allow a malicious
        user to take it over.
-   Users are likely to attempt to get back to a working state by
    registering their names in their default repository as a defensive
    name squat. Their ability to do this will depend on the specific
    policies of their default repository, whether someone already has
    that name, whether it's too generic, etc. As a best case scenario it
    will cause needless placeholder projects that serve no purpose other
    than to secure some internal use of a name.

Move to Globally Unique Names

The main reason this problem exists is that we don't have globally
unique names, we have locally unique names that exist under multiple
namespaces that we are attempting to merge into a single flat namespace.
If we could instead come up with a way to have globally unique names, we
could sidestep the entire issue.

This idea has been rejected because:

-   Generating globally unique but secure names that are also meaningful
    to humans is a nearly impossible feat without piggybacking off of
    some kind of centralized database. To my knowledge the only systems
    that have managed to do this end up piggybacking off of the domain
    system and refer to packages by URLs with domains etc.
-   Even if we come up with a mechanism to get globally unique names,
    our ability to retrofit that into our decades old system is
    practically zero without burning it all to the ground and starting
    over. The best we could probably do is declare that all non globally
    unique names are implicitly names on the PyPI domain name, and force
    everyone with a non PyPI package to rename their package.
-   This would upend so many core assumptions and fundamental parts of
    our current system it's hard to even know where to start to list
    them.

Only recommend that installers offer explicit configuration

One idea that has come up is to essentially just implement the explicit
configuration and don't make any other changes to anything else. The
specific proposal for a mapping policy is what actually inspired the
explicit configuration option, and created a file that looked something
like:

    {
      "repositories": {
        "PyTorch": ["https://download.pytorch.org/whl/nightly"],
        "PyPI": ["https://pypi.org/simple"]
      },
      "mapping": [
        {
          "paths": ["torch*"],
          "repositories": ["PyTorch"],
          "terminating": true
        },
        {
          "paths": ["*"],
          "repositories": ["PyPI"]
        }
      ]
    }

The recommendation to have explicit configuration pushes the decision on
how to implement that onto each installer, allowing them to choose what
works best for their users.

Ultimately only implementing some kind of explicit configuration was
rejected because by its nature it's opt in, so it doesn't protect
average users who are least capable to solve the problem with the
existing tools; by adding additional protections alongside the explicit
configuration, we are able to protect all users by default.

Additionally, relying on only explicit configuration also means that
every end user has to resolve the same problem over and over again, even
in cases like mirrors of PyPI, Piwheels, PyTorch, etc. In each and every
case they have to sit there and make decisions (or find some example to
cargo cult) in order to be secure. Adding extra features into the mix
allows us to centralize those protections where we can, while still
giving advanced end users the ability to completely control their own
destiny.

Scopes à la npm

There's been some suggestion that scopes similar to how npm has
implemented them may ultimately solve this. Ultimately scopes do not
change anything about this problem. As far as I know scopes in npm are
not globally unique, they're tied to a specific registry just like
unscoped names are. However what scopes do enable is an obvious
mechanism for grouping related projects and the ability for a user or
organization on npm.org to claim an entire scope, which makes explicit
configuration significantly easier to handle because you can be assured
that there's a whole little slice of the namespace that wholly belongs
to you, and you can easily write a rule that assigns an entire scope to
a specific non public registry.

Unfortunately, it basically ends up being an easier version of the idea
to only use explicit configuration, which works ok in npm because its
not particularly common for people to use their own registries, but in
Python we encourage you to do just that.

Define and Standardize the "Explicit Configuration"

This PEP recommends installers to have a mechanism for explicit
configuration of which repository a particular project comes from, but
it does not define what that mechanism is. We are purposefully leave
that undefined, as it is closely tied to the UX of each individual
installer and we want to allow each individual installer the ability to
expose that configuration in whatever way that they see fit for their
particular use cases.

Further, when the idea of defining that mechanism came up, none of the
other installers seemed particularly interested in having that mechanism
defined for them, suggesting that they were happy to treat that as part
of their UX.

Finally, that mechanism, if we did choose to define it, deserves it's
own PEP rather than baking it as part of the changes to the repository
API in this PEP and it can be a future PEP if we ultimately decide we do
want to go down the path of standardization for it.

Acknowledgements

Thanks to Trishank Kuppusamy for kick starting the discussion that lead
to this PEP with his proposal.

Thanks to Paul Moore, Pradyun Gedam, Steve Dower, and Trishank Kuppusamy
for providing early feedback and discussion on the ideas in this PEP.

Thanks to Jelle Zijlstra, C.A.M. Gerlach, Hugo van Kemenade, and Stefano
Rivera for copy editing and improving the structure and quality of this
PEP.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 9 Title: Sample Plaintext PEP Template Author: Barry Warsaw
<EMAIL_ADDRESS>Status: Withdrawn Type: Process Created: 14-Aug-2001
Post-History: Resolution:
<EMAIL_ADDRESS>
As of 05-Jan-2016, this PEP is officially deprecated and replaced by PEP
12. All PEPs should now use the reStructuredText format described by PEP
12, and plaintext PEPs will no longer be accepted.

    Abstract

        This PEP provides a boilerplate or sample template for creating
        your own plaintext PEPs.  In conjunction with the content
        guidelines in PEP 1 [1], this should make it easy for you to
        conform your own PEPs to the format outlined below.

        Note: if you are reading this PEP via the web, you should first
        grab the plaintext source of this PEP in order to complete the
        steps below.  DO NOT USE THE HTML FILE AS YOUR TEMPLATE!

        To get the source this (or any) PEP, look at the top of the HTML
        page and click on the date & time on the "Last-Modified" line.  It
        is a link to the source text in the Python repository.

        If you would prefer to use lightweight markup in your PEP, please
        see PEP 12, "Sample reStructuredText PEP Template" [2].

    Rationale

        PEP submissions come in a wide variety of forms, not all adhering
        to the format guidelines set forth below.  Use this template, in
        conjunction with the content guidelines in PEP 1, to ensure that
        your PEP submission won't get automatically rejected because of
        form.

    How to Use This Template

        To use this template you must first decide whether your PEP is
        going to be an Informational or Standards Track PEP.  Most PEPs
        are Standards Track because they propose a new feature for the
        Python language or standard library.  When in doubt, read PEP 1
        for details or contact the PEP editors<EMAIL_ADDRESS>
        Once you've decided which type of PEP yours is going to be, follow
        the directions below.

        - Make a copy of this file (.txt file, not HTML!) and perform the
          following edits.

        - Replace the "PEP: 9" header with "PEP: XXX" since you don't yet
          have a PEP number assignment.

        - Change the Title header to the title of your PEP.

        - Leave the Version and Last-Modified headers alone; we'll take
          care of those when we check your PEP into Python's Subversion
          repository.  These headers consist of keywords ("Revision" and
          "Date" enclosed in "$"-signs) which are automatically expanded
          by the repository.  Please do not edit the expanded date or
          revision text.

        - Change the Author header to include your name, and optionally
          your email address.  Be sure to follow the format carefully:
          your name must appear first, and it must not be contained in
          parentheses.  Your email address may appear second (or it can be
          omitted) and if it appears, it must appear in angle brackets.
          It is okay to obfuscate your email address.

        - If there is a mailing list for discussion of your new feature,
          add a Discussions-To header right after the Author header.  You
          should not add a Discussions-To header if the mailing list to be
          used is either<EMAIL_ADDRESS>or<EMAIL_ADDRESS>          or if discussions should be sent to you directly.  Most
          Informational PEPs don't have a Discussions-To header.

        - Change the Status header to "Draft".

        - For Standards Track PEPs, change the Type header to "Standards
          Track".

        - For Informational PEPs, change the Type header to
          "Informational".

        - For Standards Track PEPs, if your feature depends on the
          acceptance of some other currently in-development PEP, add a
          Requires header right after the Type header.  The value should
          be the PEP number of the PEP yours depends on.  Don't add this
          header if your dependent feature is described in a Final PEP.

        - Change the Created header to today's date.  Be sure to follow
          the format carefully: it must be in dd-mmm-yyyy format, where
          the mmm is the 3 English letter month abbreviation, e.g. one of
          Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct, Nov, Dec.

        - For Standards Track PEPs, after the Created header, add a
          Python-Version header and set the value to the next planned
          version of Python, i.e. the one your new feature will hopefully
          make its first appearance in.  Do not use an alpha or beta
          release designation here.  Thus, if the last version of Python
          was 2.2 alpha 1 and you're hoping to get your new feature into
          Python 2.2, set the header to:

          Python-Version: 2.2

        - Leave Post-History alone for now; you'll add dates to this
          header each time you post your PEP to<EMAIL_ADDRESS>or
<EMAIL_ADDRESS> E.g. if you posted your PEP to the lists
          on August 14, 2001 and September 3, 2001, the Post-History
          header would look like:

          Post-History: 14-Aug-2001, 03-Sept-2001

          You must manually add new dates and check them in.  If you don't
          have check-in privileges, send your changes to the PEP editor.

        - Add a Replaces header if your PEP obsoletes an earlier PEP.  The
          value of this header is the number of the PEP that your new PEP
          is replacing.  Only add this header if the older PEP is in
          "final" form, i.e. is either Accepted, Final, or Rejected.  You
          aren't replacing an older open PEP if you're submitting a
          competing idea.

        - Now write your Abstract, Rationale, and other content for your
          PEP, replacing all this gobbledygook with your own text. Be sure
          to adhere to the format guidelines below, specifically on the
          prohibition of tab characters and the indentation requirements.

        - Update your References and Copyright section.  Usually you'll
          place your PEP into the public domain, in which case just leave
          the "Copyright" section alone.  Alternatively, you can use the
          Open Publication License[3], but public domain is still strongly
          preferred.

        - Leave the little Emacs turd at the end of this file alone,
          including the formfeed character ("^L", or \f).

        - Send your PEP submission to the PEP editors (peps@python.org),
          along with $100k in unmarked pennies.  (Just kidding, I wanted
          to see if you were still awake. :)

    Plaintext PEP Formatting Requirements

        PEP headings must begin in column zero and the initial letter of
        each word must be capitalized as in book titles.  Acronyms should
        be in all capitals.  The body of each section must be indented 4
        spaces.  Code samples inside body sections should be indented a
        further 4 spaces, and other indentation can be used as required to
        make the text readable.  You must use two blank lines between the
        last line of a section's body and the next section heading.

        You must adhere to the Emacs convention of adding two spaces at
        the end of every sentence.  You should fill your paragraphs to
        column 70, but under no circumstances should your lines extend
        past column 79.  If your code samples spill over column 79, you
        should rewrite them.

        Tab characters must never appear in the document at all.  A PEP
        should include the standard Emacs stanza included by example at
        the bottom of this PEP.

        When referencing an external web page in the body of a PEP, you
        should include the title of the page in the text, with a
        footnote reference to the URL.  Do not include the URL in the body
        text of the PEP.  E.g.

            Refer to the Python Language web site [1] for more details.
            ...
            [1] http://www.python.org

        When referring to another PEP, include the PEP number in the body
        text, such as "PEP 1".  The title may optionally appear.  Add a
        footnote reference, a number in square brackets.  The footnote
        body should include the PEP's title and author.  It may optionally
        include the explicit URL on a separate line, but only in the
        References section.  Note that the pep2html.py script will
        calculate URLs automatically.  For example:

                ...
                Refer to PEP 1 [7] for more information about PEP style
                ...

            References

                [7] PEP 1, PEP Purpose and Guidelines, Warsaw, Hylton
                    http://peps.python.org/pep-0001/

        If you decide to provide an explicit URL for a PEP, please use
        this as the URL template:

            http://peps.python.org/pep-xxxx/

        PEP numbers in URLs must be padded with zeros from the left, so as
        to be exactly 4 characters wide, however PEP numbers in the text
        are never padded.

    References

        [1] PEP 1, PEP Purpose and Guidelines, Warsaw, Hylton
            http://peps.python.org/pep-0001/

        [2] PEP 12, Sample reStructuredText PEP Template, Goodger, Warsaw
            http://peps.python.org/pep-0012/

        [3] http://www.opencontent.org/openpub/

    Copyright

        This document has been placed in the public domain.


text:
PEP: 351 Title: The freeze protocol Version: 2.5 Last-Modified: $Date$
Author: Barry Warsaw<EMAIL_ADDRESS>Status: Rejected Type: Standards
Track Content-Type: text/x-rst Created: 14-Apr-2005 Post-History:

Abstract

This PEP describes a simple protocol for requesting a frozen, immutable
copy of a mutable object. It also defines a new built-in function which
uses this protocol to provide an immutable copy on any cooperating
object.

Rejection Notice

This PEP was rejected. For a rationale, see this thread on python-dev.

Rationale

Built-in objects such dictionaries and sets accept only immutable
objects as keys. This means that mutable objects like lists cannot be
used as keys to a dictionary. However, a Python programmer can convert a
list to a tuple; the two objects are similar, but the latter is
immutable, and can be used as a dictionary key.

It is conceivable that third party objects also have similar mutable and
immutable counterparts, and it would be useful to have a standard
protocol for conversion of such objects.

sets.Set objects expose a "protocol for automatic conversion to
immutable" so that you can create sets.Sets of sets.Sets. PEP 218
deliberately dropped this feature from built-in sets. This PEP advances
that the feature is still useful and proposes a standard mechanism for
its support.

Proposal

It is proposed that a new built-in function called freeze() is added.

If freeze() is passed an immutable object, as determined by hash() on
that object not raising a TypeError, then the object is returned
directly.

If freeze() is passed a mutable object (i.e. hash() of that object
raises a TypeError), then freeze() will call that object's __freeze__()
method to get an immutable copy. If the object does not have a
__freeze__() method, then a TypeError is raised.

Sample implementations

Here is a Python implementation of the freeze() built-in:

    def freeze(obj):
        try:
            hash(obj)
            return obj
        except TypeError:
            freezer = getattr(obj, '__freeze__', None)
            if freezer:
                return freezer()
            raise TypeError('object is not freezable')``

Here are some code samples which show the intended semantics:

    class xset(set):
        def __freeze__(self):
            return frozenset(self)

    class xlist(list):
        def __freeze__(self):
            return tuple(self)

    class imdict(dict):
        def __hash__(self):
            return id(self)

        def _immutable(self, *args, **kws):
            raise TypeError('object is immutable')

        __setitem__ = _immutable
        __delitem__ = _immutable
        clear       = _immutable
        update      = _immutable
        setdefault  = _immutable
        pop         = _immutable
        popitem     = _immutable

    class xdict(dict):
        def __freeze__(self):
            return imdict(self)

    >>> s = set([1, 2, 3])
    >>> {s: 4}
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    TypeError: set objects are unhashable
    >>> t = freeze(s)
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
      File "/usr/tmp/python-lWCjBK.py", line 9, in freeze
    TypeError: object is not freezable
    >>> t = xset(s)
    >>> u = freeze(t)
    >>> {u: 4}
    {frozenset([1, 2, 3]): 4}
    >>> x = 'hello'
    >>> freeze(x) is x
    True
    >>> d = xdict(a=7, b=8, c=9)
    >>> hash(d)
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    TypeError: dict objects are unhashable
    >>> hash(freeze(d))
    -1210776116
    >>> {d: 4}
    Traceback (most recent call last):
      File "<stdin>", line 1, in ?
    TypeError: dict objects are unhashable
    >>> {freeze(d): 4}
    {{'a': 7, 'c': 9, 'b': 8}: 4}

Reference implementation

Patch 1335812 provides the C implementation of this feature. It adds the
freeze() built-in, along with implementations of the __freeze__() method
for lists and sets. Dictionaries are not easily freezable in current
Python, so an implementation of dict.__freeze__() is not provided yet.

Open issues

-   Should we define a similar protocol for thawing frozen objects?
-   Should dicts and sets automatically freeze their mutable keys?
-   Should we support "temporary freezing" (perhaps with a method called
    __congeal__()) a la __as_temporarily_immutable__() in sets.Set?
-   For backward compatibility with sets.Set, should we support
    __as_immutable__()? Or should __freeze__() just be renamed to
    __as_immutable__()?

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 End:


text:
PEP: 619 Title: Python 3.10 Release Schedule Author: Pablo Galindo
Salgado<EMAIL_ADDRESS>Status: Active Type: Informational Topic:
Release Created: 25-May-2020 Python-Version: 3.10

Abstract

This document describes the development and release schedule for Python
3.10. The schedule primarily concerns itself with PEP-sized items.

Release Manager and Crew

-   3.10 Release Manager: Pablo Galindo Salgado
-   Windows installers: Steve Dower
-   Mac installers: Ned Deily
-   Documentation: Julien Palard

Release Schedule

3.10.0 schedule

Note: the dates below use a 17-month development period that results in
a 12-month release cadence between feature versions, as defined by PEP
602.

Actual:

-   3.10 development begins: Monday, 2020-05-18
-   3.10.0 alpha 1: Monday, 2020-10-05
-   3.10.0 alpha 2: Tuesday, 2020-11-03
-   3.10.0 alpha 3: Monday, 2020-12-07
-   3.10.0 alpha 4: Monday, 2021-01-04
-   3.10.0 alpha 5: Wednesday, 2021-02-03
-   3.10.0 alpha 6: Monday, 2021-03-01
-   3.10.0 alpha 7: Tuesday, 2021-04-06
-   3.10.0 beta 1: Monday, 2021-05-03 (No new features beyond this
    point.)
-   3.10.0 beta 2: Monday, 2021-05-31
-   3.10.0 beta 3: Thursday, 2021-06-17
-   3.10.0 beta 4: Saturday, 2021-07-10
-   3.10.0 candidate 1: Tuesday, 2021-08-03
-   3.10.0 candidate 2: Tuesday, 2021-09-07
-   3.10.0 final: Monday, 2021-10-04

Bugfix releases

Actual:

-   3.10.1: Monday, 2021-12-06
-   3.10.2: Friday, 2022-01-14
-   3.10.3: Wednesday, 2022-03-16
-   3.10.4: Thursday, 2022-03-24
-   3.10.5: Monday, 2022-06-06
-   3.10.6: Tuesday, 2022-08-02
-   3.10.7: Tuesday, 2022-09-06
-   3.10.8: Tuesday, 2022-10-11
-   3.10.9: Tuesday, 2022-12-06
-   3.10.10: Wednesday, 2023-02-08
-   3.10.11: Wednesday, 2023-04-05 (final regular bugfix release with
    binary installers)

Source-only security fix releases

Provided irregularly on an "as-needed" basis until October 2026.

-   3.10.12: Tuesday, 2023-06-06
-   3.10.13: Thursday, 2023-08-24
-   3.10.14: Tuesday, 2024-03-19
-   3.10.15: Saturday, 2024-09-07

3.10 Lifespan

3.10 will receive bugfix updates approximately every 2 months for
approximately 18 months. Some time after the release of 3.11.0 final,
the 11th and final 3.10 bugfix update will be released. After that, it
is expected that security updates (source only) will be released until 5
years after the release of 3.10 final, so until approximately October
2026.

Features for 3.10

Some of the notable features of Python 3.10 include:

-   PEP 604, Allow writing union types as X | Y
-   PEP 612, Parameter Specification Variables
-   PEP 613, Explicit Type Aliases
-   PEP 618, Add Optional Length-Checking To zip
-   PEP 626, Precise line numbers for debugging and other tools
-   PEP 634, PEP 635, PEP 636, Structural Pattern Matching
-   PEP 644, Require OpenSSL 1.1.1 or newer
-   PEP 624, Remove Py_UNICODE encoder APIs
-   PEP 597, Add optional EncodingWarning

Copyright

This document has been placed in the public domain.