text:
PEP: 384 Title: Defining a Stable ABI Author: Martin von Löwis
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 17-May-2009 Python-Version: 3.2 Post-History:

python:stable (user docs) and devguide:c-api (development docs)

Abstract

Currently, each feature release introduces a new name for the Python DLL
on Windows, and may cause incompatibilities for extension modules on
Unix. This PEP proposes to define a stable set of API functions which
are guaranteed to be available for the lifetime of Python 3, and which
will also remain binary-compatible across versions. Extension modules
and applications embedding Python can work with different feature
releases as long as they restrict themselves to this stable ABI.

Rationale

The primary source of ABI incompatibility are changes to the lay-out of
in-memory structures. For example, the way in which string interning
works, or the data type used to represent the size of an object, have
changed during the life of Python 2.x. As a consequence, extension
modules making direct access to fields of strings, lists, or tuples,
would break if their code is loaded into a newer version of the
interpreter without recompilation: offsets of other fields may have
changed, making the extension modules access the wrong data.

In some cases, the incompatibilities only affect internal objects of the
interpreter, such as frame or code objects. For example, the way line
numbers are represented has changed in the 2.x lifetime, as has the way
in which local variables are stored (due to the introduction of
closures). Even though most applications probably never used these
objects, changing them had required to change the PYTHON_API_VERSION.

On Linux, changes to the ABI are often not much of a problem: the system
will provide a default Python installation, and many extension modules
are already provided pre-compiled for that version. If additional
modules are needed, or additional Python versions, users can typically
compile them themselves on the system, resulting in modules that use the
right ABI.

On Windows, multiple simultaneous installations of different Python
versions are common, and extension modules are compiled by their
authors, not by end users. To reduce the risk of ABI incompatibilities,
Python currently introduces a new DLL name pythonXY.dll for each feature
release, whether or not ABI incompatibilities actually exist.

With this PEP, it will be possible to reduce the dependency of binary
extension modules on a specific Python feature release, and applications
embedding Python can be made work with different releases.

Specification

The ABI specification falls into two parts: an API specification,
specifying what function (groups) are available for use with the ABI,
and a linkage specification specifying what libraries to link with. The
actual ABI (layout of structures in memory, function calling
conventions) is not specified, but implied by the compiler. As a
recommendation, a specific ABI is recommended for selected platforms.

During evolution of Python, new ABI functions will be added.
Applications using them will then have a requirement on a minimum
version of Python; this PEP provides no mechanism for such applications
to fall back when the Python library is too old.

Terminology

Applications and extension modules that want to use this ABI are
collectively referred to as "applications" from here on.

Header Files and Preprocessor Definitions

Applications shall only include the header file Python.h (before
including any system headers), or, optionally, include pyconfig.h, and
then Python.h.

During the compilation of applications, the preprocessor macro
Py_LIMITED_API must be defined. Doing so will hide all definitions that
are not part of the ABI.

Structures

Only the following structures and structure fields are accessible to
applications:

-   PyObject (ob_refcnt, ob_type)
-   PyVarObject (ob_base, ob_size)
-   PyMethodDef (ml_name, ml_meth, ml_flags, ml_doc)
-   PyMemberDef (name, type, offset, flags, doc)
-   PyGetSetDef (name, get, set, doc, closure)
-   PyModuleDefBase (ob_base, m_init, m_index, m_copy)
-   PyModuleDef (m_base, m_name, m_doc, m_size, m_methods, m_traverse,
    m_clear, m_free)
-   PyStructSequence_Field (name, doc)
-   PyStructSequence_Desc (name, doc, fields, sequence)
-   PyType_Slot (see below)
-   PyType_Spec (see below)

The accessor macros to these fields (Py_REFCNT, Py_TYPE, Py_SIZE) are
also available to applications.

The following types are available, but opaque (i.e. incomplete):

-   PyThreadState
-   PyInterpreterState
-   struct _frame
-   struct symtable
-   struct _node
-   PyWeakReference
-   PyLongObject
-   PyTypeObject

Type Objects

The structure of type objects is not available to applications;
declaration of "static" type objects is not possible anymore (for
applications using this ABI). Instead, type objects get created
dynamically. To allow an easy creation of types (in particular, to be
able to fill out function pointers easily), the following structures and
functions are available:

    typedef struct{
      int slot;    /* slot id, see below */
      void *pfunc; /* function pointer */
    } PyType_Slot;

    typedef struct{
      const char* name;
      int basicsize;
      int itemsize;
      unsigned int flags;
      PyType_Slot *slots; /* terminated by slot==0. */
    } PyType_Spec;

    PyObject* PyType_FromSpec(PyType_Spec*);

To specify a slot, a unique slot id must be provided. New Python
versions may introduce new slot ids, but slot ids will never be
recycled. Slots may get deprecated, but continue to be supported
throughout Python 3.x.

The slot ids are named like the field names of the structures that hold
the pointers in Python 3.1, with an added Py_ prefix (i.e. Py_tp_dealloc
instead of just tp_dealloc):

-   tp_dealloc, tp_getattr, tp_setattr, tp_repr, tp_hash, tp_call,
    tp_str, tp_getattro, tp_setattro, tp_doc, tp_traverse, tp_clear,
    tp_richcompare, tp_iter, tp_iternext, tp_methods, tp_base,
    tp_descr_get, tp_descr_set, tp_init, tp_alloc, tp_new, tp_is_gc,
    tp_bases, tp_del
-   nb_add nb_subtract nb_multiply nb_remainder nb_divmod nb_power
    nb_negative nb_positive nb_absolute nb_bool nb_invert nb_lshift
    nb_rshift nb_and nb_xor nb_or nb_int nb_float nb_inplace_add
    nb_inplace_subtract nb_inplace_multiply nb_inplace_remainder
    nb_inplace_power nb_inplace_lshift nb_inplace_rshift nb_inplace_and
    nb_inplace_xor nb_inplace_or nb_floor_divide nb_true_divide
    nb_inplace_floor_divide nb_inplace_true_divide nb_index
-   sq_length sq_concat sq_repeat sq_item sq_ass_item sq_contains
    sq_inplace_concat sq_inplace_repeat
-   mp_length mp_subscript mp_ass_subscript

The following fields cannot be set during type definition: - tp_dict
tp_mro tp_cache tp_subclasses tp_weaklist tp_print - tp_weaklistoffset
tp_dictoffset

typedefs

In addition to the typedefs for structs listed above, the following
typedefs are available. Their inclusion in the ABI means that the
underlying type must not change on a platform (even though it may differ
across platforms).

-   Py_uintptr_t Py_intptr_t Py_ssize_t
-   unaryfunc binaryfunc ternaryfunc inquiry lenfunc ssizeargfunc
    ssizessizeargfunc ssizeobjargproc ssizessizeobjargproc objobjargproc
    objobjproc visitproc traverseproc destructor getattrfunc
    getattrofunc setattrfunc setattrofunc reprfunc hashfunc richcmpfunc
    getiterfunc iternextfunc descrgetfunc descrsetfunc initproc newfunc
    allocfunc
-   PyCFunction PyCFunctionWithKeywords PyNoArgsFunction
    PyCapsule_Destructor
-   getter setter
-   PyOS_sighandler_t
-   PyGILState_STATE
-   Py_UCS4

Most notably, Py_UNICODE is not available as a typedef, since the same
Python version may use different definitions of it on the same platform
(depending on whether it uses narrow or wide code units). Applications
that need to access the contents of a Unicode string can convert it to
wchar_t.

Functions and function-like Macros

By default, all functions are available, unless they are excluded below.
Whether a function is documented or not does not matter.

Function-like macros (in particular, field access macros) remain
available to applications, but get replaced by function calls (unless
their definition only refers to features of the ABI, such as the various
_Check macros)

ABI function declarations will not change their parameters or return
types. If a change to the signature becomes necessary, a new function
will be introduced. If the new function is source-compatible (e.g. if
just the return type changes), an alias macro may get added to redirect
calls to the new function when the applications is recompiled.

If continued provision of the old function is not possible, it may get
deprecated, then removed, causing applications that use that function to
break.

Excluded Functions

All functions starting with _Py are not available to applications. Also,
all functions that expect parameter types that are unavailable to
applications are excluded from the ABI, such as PyAST_FromNode (which
expects a node*).

Functions declared in the following header files are not part of the
ABI:

-   bytes_methods.h
-   cellobject.h
-   classobject.h
-   code.h
-   compile.h
-   datetime.h
-   dtoa.h
-   frameobject.h
-   funcobject.h
-   genobject.h
-   longintrepr.h
-   parsetok.h
-   pyarena.h
-   pyatomic.h
-   pyctype.h
-   pydebug.h
-   pytime.h
-   symtable.h
-   token.h
-   ucnhash.h

In addition, functions expecting FILE* are not part of the ABI, to avoid
depending on a specific version of the Microsoft C runtime DLL on
Windows.

Module and type initializer and finalizer functions are not available
(PyByteArray_Init, PyOS_FiniInterrupts and all functions ending in _Fini
or _ClearFreeList).

Several functions dealing with interpreter implementation details are
not available:

-   PyInterpreterState_Head, PyInterpreterState_Next,
    PyInterpreterState_ThreadHead, PyThreadState_Next
-   Py_SubversionRevision, Py_SubversionShortBranch

PyStructSequence_InitType is not available, as it requires the caller to
provide a static type object.

Py_FatalError will be moved from pydebug.h into some other header file
(e.g. pyerrors.h).

The exact list of functions being available is given in the Windows
module definition file for python3.dll[1].

Global Variables

Global variables representing types and exceptions are available to
applications. In addition, selected global variables referenced in
macros (such as Py_True and Py_False) are available.

A complete list of global variable definitions is given in the
python3.def file[2]; those declared DATA denote variables.

Other Macros

All macros defining symbolic constants are available to applications;
the numeric values will not change.

In addition, the following macros are available:

-   Py_BEGIN_ALLOW_THREADS, Py_BLOCK_THREADS, Py_UNBLOCK_THREADS,
    Py_END_ALLOW_THREADS

The Buffer Interface

The buffer interface (type Py_buffer, type slots bf_getbuffer and
bf_releasebuffer, etc) has been omitted from the ABI, since the
stability of the Py_buffer structure is not clear at this time.
Inclusion in the ABI can be considered in future releases.

Signature Changes

A number of functions currently expect a specific struct, even though
callers typically have PyObject* available. These have been changed to
expect PyObject* as the parameter; this will cause warnings in
applications that currently explicitly cast to the parameter type. These
functions are PySlice_GetIndices, PySlice_GetIndicesEx,
PyUnicode_AsWideChar, and PyEval_EvalCode.

Linkage

On Windows, applications shall link with python3.dll; an import library
python3.lib will be available. This DLL will redirect all of its API
functions through /export linker options to the full interpreter DLL,
i.e. python3y.dll.

On Unix systems, the ABI is typically provided by the python executable
itself. PyModule_Create is changed to pass 3 as the API version if the
extension module was compiled with Py_LIMITED_API; the version check for
the API version will accept either 3 or the current PYTHON_API_VERSION
as conforming. If Python is compiled as a shared library, it is
installed as both libpython3.so, and libpython3.y.so; applications
conforming to this PEP should then link to the former (extension modules
can continue to link with no libpython shared object, but rather rely on
runtime linking). The ABI version is symbolically available as
PYTHON_ABI_VERSION.

Also on Unix, the PEP 3149 tag abi<PYTHON_ABI_VERSION> is accepted in
file names of extension modules. No checking is performed that files
named in this way are actually restricted to the limited API, and no
support for building such files will be added to distutils due to the
distutils code freeze.

Implementation Strategy

This PEP will be implemented in a branch[3], allowing users to check
whether their modules conform to the ABI. To avoid users having to
rewrite their type definitions, a script to convert C source code
containing type definitions will be provided[4].

References

Copyright

This document has been placed in the public domain.

[1] "python3 module definition file":
http://svn.python.org/projects/python/branches/pep-0384/PC/python3.def

[2] "python3 module definition file":
http://svn.python.org/projects/python/branches/pep-0384/PC/python3.def

[3] "PEP 384 branch":
http://svn.python.org/projects/python/branches/pep-0384/

[4] "ABI type conversion script":
http://svn.python.org/projects/python/branches/pep-0384/Tools/scripts/abitype.py


text:
PEP: 505 Title: None-aware operators Author: Mark E. Haase
<EMAIL_ADDRESS>Steve Dower<EMAIL_ADDRESS>Status:
Deferred Type: Standards Track Content-Type: text/x-rst Created:
18-Sep-2015 Python-Version: 3.8

Abstract

Several modern programming languages have so-called "null-coalescing" or
"null- aware" operators, including C#[1], Dart[2], Perl, Swift, and PHP
(starting in version 7). There are also stage 3 draft proposals for
their addition to ECMAScript (a.k.a. JavaScript)[3][4]. These operators
provide syntactic sugar for common patterns involving null references.

-   The "null-coalescing" operator is a binary operator that returns its
    left operand if it is not null. Otherwise it returns its right
    operand.
-   The "null-aware member access" operator accesses an instance member
    only if that instance is non-null. Otherwise it returns null. (This
    is also called a "safe navigation" operator.)
-   The "null-aware index access" operator accesses an element of a
    collection only if that collection is non-null. Otherwise it returns
    null. (This is another type of "safe navigation" operator.)

This PEP proposes three None-aware operators for Python, based on the
definitions and other language's implementations of those above.
Specifically:

-   The "None coalescing" binary operator ?? returns the left hand side
    if it evaluates to a value that is not None, or else it evaluates
    and returns the right hand side. A coalescing ??= augmented
    assignment operator is included.
-   The "None-aware attribute access" operator ?. ("maybe dot")
    evaluates the complete expression if the left hand side evaluates to
    a value that is not None
-   The "None-aware indexing" operator ?[] ("maybe subscript") evaluates
    the complete expression if the left hand site evaluates to a value
    that is not None

See the Grammar changes section for specifics and examples of the
required grammar changes.

See the Examples section for more realistic examples of code that could
be updated to use the new operators.

Syntax and Semantics

Specialness of None

The None object denotes the lack of a value. For the purposes of these
operators, the lack of a value indicates that the remainder of the
expression also lacks a value and should not be evaluated.

A rejected proposal was to treat any value that evaluates as "false" in
a Boolean context as not having a value. However, the purpose of these
operators is to propagate the "lack of value" state, rather than the
"false" state.

Some argue that this makes None special. We contend that None is already
special, and that using it as both the test and the result of these
operators does not change the existing semantics in any way.

See the Rejected Ideas section for discussions on alternate approaches.

Grammar changes

The following rules of the Python grammar are updated to read:

    augassign: ('+=' | '-=' | '*=' | '@=' | '/=' | '%=' | '&=' | '|=' | '^=' |
                '<<=' | '>>=' | '**=' | '//=' | '??=')

    power: coalesce ['**' factor]
    coalesce: atom_expr ['??' factor]
    atom_expr: ['await'] atom trailer*
    trailer: ('(' [arglist] ')' |
              '[' subscriptlist ']' |
              '?[' subscriptlist ']' |
              '.' NAME |
              '?.' NAME)

The coalesce rule

The coalesce rule provides the ?? binary operator. Unlike most binary
operators, the right-hand side is not evaluated until the left-hand side
is determined to be None.

The ?? operator binds more tightly than other binary operators as most
existing implementations of these do not propagate None values (they
will typically raise TypeError). Expressions that are known to
potentially result in None can be substituted for a default value
without needing additional parentheses.

Some examples of how implicit parentheses are placed when evaluating
operator precedence in the presence of the ?? operator:

    a, b = None, None
    def c(): return None
    def ex(): raise Exception()

    (a ?? 2 ** b ?? 3) == a ?? (2 ** (b ?? 3))
    (a * b ?? c // d) == a * (b ?? c) // d
    (a ?? True and b ?? False) == (a ?? True) and (b ?? False)
    (c() ?? c() ?? True) == True
    (True ?? ex()) == True
    (c ?? ex)() == c()

Particularly for cases such as a ?? 2 ** b ?? 3, parenthesizing the
sub-expressions any other way would result in TypeError, as int.__pow__
cannot be called with None (and the fact that the ?? operator is used at
all implies that a or b may be None). However, as usual, while
parentheses are not required they should be added if it helps improve
readability.

An augmented assignment for the ?? operator is also added. Augmented
coalescing assignment only rebinds the name if its current value is
None. If the target name already has a value, the right-hand side is not
evaluated. For example:

    a = None
    b = ''
    c = 0

    a ??= 'value'
    b ??= undefined_name
    c ??= shutil.rmtree('/')    # don't try this at home, kids

    assert a == 'value'
    assert b == ''
    assert c == 0 and any(os.scandir('/'))

The maybe-dot and maybe-subscript operators

The maybe-dot and maybe-subscript operators are added as trailers for
atoms, so that they may be used in all the same locations as the regular
operators, including as part of an assignment target (more details
below). As the existing evaluation rules are not directly embedded in
the grammar, we specify the required changes below.

Assume that the atom is always successfully evaluated. Each trailer is
then evaluated from left to right, applying its own parameter (either
its arguments, subscripts or attribute name) to produce the value for
the next trailer. Finally, if present, await is applied.

For example, await a.b(c).d[e] is currently parsed as
['await', 'a', '.b', '(c)', '.d', '[e]'] and evaluated:

    _v = a
    _v = _v.b
    _v = _v(c)
    _v = _v.d
    _v = _v[e]
    await _v

When a None-aware operator is present, the left-to-right evaluation may
be short-circuited. For example, await a?.b(c).d?[e] is evaluated:

    _v = a
    if _v is not None:
        _v = _v.b
        _v = _v(c)
        _v = _v.d
        if _v is not None:
            _v = _v[e]
    await _v

Note

await will almost certainly fail in this context, as it would in the
case where code attempts await None. We are not proposing to add a
None-aware await keyword here, and merely include it in this example for
completeness of the specification, since the atom_expr grammar rule
includes the keyword. If it were in its own rule, we would have never
mentioned it.

Parenthesised expressions are handled by the atom rule (not shown
above), which will implicitly terminate the short-circuiting behaviour
of the above transformation. For example, (a?.b ?? c).d?.e is evaluated
as:

    # a?.b
    _v = a
    if _v is not None:
        _v = _v.b

    # ... ?? c
    if _v is None:
        _v = c

    # (...).d?.e
    _v = _v.d
    if _v is not None:
        _v = _v.e

When used as an assignment target, the None-aware operations may only be
used in a "load" context. That is, a?.b = 1 and a?[b] = 1 will raise
SyntaxError. Use earlier in the expression (a?.b.c = 1) is permitted,
though unlikely to be useful unless combined with a coalescing
operation:

    (a?.b ?? d).c = 1

Reading expressions

For the maybe-dot and maybe-subscript operators, the intention is that
expressions including these operators should be read and interpreted as
for the regular versions of these operators. In "normal" cases, the end
results are going to be identical between an expression such as a?.b?[c]
and a.b[c], and just as we do not currently read "a.b" as "read
attribute b from a if it has an attribute a or else it raises
AttributeError", there is no need to read "a?.b" as "read attribute b
from a if a is not None" (unless in a context where the listener needs
to be aware of the specific behaviour).

For coalescing expressions using the ?? operator, expressions should
either be read as "or ... if None" or "coalesced with". For example, the
expression a.get_value() ?? 100 would be read "call a dot get_value or
100 if None", or "call a dot get_value coalesced with 100".

Note

Reading code in spoken text is always lossy, and so we make no attempt
to define an unambiguous way of speaking these operators. These
suggestions are intended to add context to the implications of adding
the new syntax.

Examples

This section presents some examples of common None patterns and shows
what conversion to use None-aware operators may look like.

Standard Library

Using the find-pep505.py script[5] an analysis of the Python 3.7
standard library discovered up to 678 code snippets that could be
replaced with use of one of the None-aware operators:

    $ find /usr/lib/python3.7 -name '*.py' | xargs python3.7 find-pep505.py
    <snip>
    Total None-coalescing `if` blocks: 449
    Total [possible] None-coalescing `or`: 120
    Total None-coalescing ternaries: 27
    Total Safe navigation `and`: 13
    Total Safe navigation `if` blocks: 61
    Total Safe navigation ternaries: 8

Some of these are shown below as examples before and after converting to
use the new operators.

From bisect.py:

    def insort_right(a, x, lo=0, hi=None):
        # ...
        if hi is None:
            hi = len(a)
        # ...

After updating to use the ??= augmented assignment statement:

    def insort_right(a, x, lo=0, hi=None):
        # ...
        hi ??= len(a)
        # ...

From calendar.py:

    encoding = options.encoding
    if encoding is None:
        encoding = sys.getdefaultencoding()
    optdict = dict(encoding=encoding, css=options.css)

After updating to use the ?? operator:

    optdict = dict(encoding=options.encoding ?? sys.getdefaultencoding(),
                   css=options.css)

From email/generator.py (and importantly note that there is no way to
substitute or for ?? in this situation):

    mangle_from_ = True if policy is None else policy.mangle_from_

After updating:

    mangle_from_ = policy?.mangle_from_ ?? True

From asyncio/subprocess.py:

    def pipe_data_received(self, fd, data):
        if fd == 1:
            reader = self.stdout
        elif fd == 2:
            reader = self.stderr
        else:
            reader = None
        if reader is not None:
            reader.feed_data(data)

After updating to use the ?. operator:

    def pipe_data_received(self, fd, data):
        if fd == 1:
            reader = self.stdout
        elif fd == 2:
            reader = self.stderr
        else:
            reader = None
        reader?.feed_data(data)

From asyncio/tasks.py:

    try:
        await waiter
    finally:
        if timeout_handle is not None:
            timeout_handle.cancel()

After updating to use the ?. operator:

    try:
        await waiter
    finally:
        timeout_handle?.cancel()

From ctypes/_aix.py:

    if libpaths is None:
        libpaths = []
    else:
        libpaths = libpaths.split(":")

After updating:

    libpaths = libpaths?.split(":") ?? []

From os.py:

    if entry.is_dir():
        dirs.append(name)
        if entries is not None:
            entries.append(entry)
    else:
        nondirs.append(name)

After updating to use the ?. operator:

    if entry.is_dir():
        dirs.append(name)
        entries?.append(entry)
    else:
        nondirs.append(name)

From importlib/abc.py:

    def find_module(self, fullname, path):
        if not hasattr(self, 'find_spec'):
            return None
        found = self.find_spec(fullname, path)
        return found.loader if found is not None else None

After partially updating:

    def find_module(self, fullname, path):
        if not hasattr(self, 'find_spec'):
            return None
        return self.find_spec(fullname, path)?.loader

After extensive updating (arguably excessive, though that's for the
style guides to determine):

    def find_module(self, fullname, path):
        return getattr(self, 'find_spec', None)?.__call__(fullname, path)?.loader

From dis.py:

    def _get_const_info(const_index, const_list):
        argval = const_index
        if const_list is not None:
            argval = const_list[const_index]
        return argval, repr(argval)

After updating to use the ?[] and ?? operators:

    def _get_const_info(const_index, const_list):
        argval = const_list?[const_index] ?? const_index
        return argval, repr(argval)

jsonify

This example is from a Python web crawler that uses the Flask framework
as its front-end. This function retrieves information about a web site
from a SQL database and formats it as JSON to send to an HTTP client:

    class SiteView(FlaskView):
        @route('/site/<id_>', methods=['GET'])
        def get_site(self, id_):
            site = db.query('site_table').find(id_)

            return jsonify(
                first_seen=site.first_seen.isoformat() if site.first_seen is not None else None,
                id=site.id,
                is_active=site.is_active,
                last_seen=site.last_seen.isoformat() if site.last_seen is not None else None,
                url=site.url.rstrip('/')
            )

Both first_seen and last_seen are allowed to be null in the database,
and they are also allowed to be null in the JSON response. JSON does not
have a native way to represent a datetime, so the server's contract
states that any non-null date is represented as an ISO-8601 string.

Without knowing the exact semantics of the first_seen and last_seen
attributes, it is impossible to know whether the attribute can be safely
or performantly accessed multiple times.

One way to fix this code is to replace each conditional expression with
an explicit value assignment and a full if/else block:

    class SiteView(FlaskView):
        @route('/site/<id_>', methods=['GET'])
        def get_site(self, id_):
            site = db.query('site_table').find(id_)

            first_seen_dt = site.first_seen
            if first_seen_dt is None:
                first_seen = None
            else:
                first_seen = first_seen_dt.isoformat()

            last_seen_dt = site.last_seen
            if last_seen_dt is None:
                last_seen = None
            else:
                last_seen = last_seen_dt.isoformat()

            return jsonify(
                first_seen=first_seen,
                id=site.id,
                is_active=site.is_active,
                last_seen=last_seen,
                url=site.url.rstrip('/')
            )

This adds ten lines of code and four new code paths to the function,
dramatically increasing the apparent complexity. Rewriting using the
None-aware attribute operator results in shorter code with more clear
intent:

    class SiteView(FlaskView):
        @route('/site/<id_>', methods=['GET'])
        def get_site(self, id_):
            site = db.query('site_table').find(id_)

            return jsonify(
                first_seen=site.first_seen?.isoformat(),
                id=site.id,
                is_active=site.is_active,
                last_seen=site.last_seen?.isoformat(),
                url=site.url.rstrip('/')
            )

Grab

The next example is from a Python scraping library called Grab
<https://github.com/lorien/grab/blob/4c95b18dcb0fa88eeca81f5643c0ebfb114bf728/gr
ab/upload.py>:

    class BaseUploadObject(object):
        def find_content_type(self, filename):
            ctype, encoding = mimetypes.guess_type(filename)
            if ctype is None:
                return 'application/octet-stream'
            else:
                return ctype

    class UploadContent(BaseUploadObject):
        def __init__(self, content, filename=None, content_type=None):
            self.content = content
            if filename is None:
                self.filename = self.get_random_filename()
            else:
                self.filename = filename
            if content_type is None:
                self.content_type = self.find_content_type(self.filename)
            else:
                self.content_type = content_type

    class UploadFile(BaseUploadObject):
        def __init__(self, path, filename=None, content_type=None):
            self.path = path
            if filename is None:
                self.filename = os.path.split(path)[1]
            else:
                self.filename = filename
            if content_type is None:
                self.content_type = self.find_content_type(self.filename)
            else:
                self.content_type = content_type

This example contains several good examples of needing to provide
default values. Rewriting to use conditional expressions reduces the
overall lines of code, but does not necessarily improve readability:

    class BaseUploadObject(object):
        def find_content_type(self, filename):
            ctype, encoding = mimetypes.guess_type(filename)
            return 'application/octet-stream' if ctype is None else ctype

    class UploadContent(BaseUploadObject):
        def __init__(self, content, filename=None, content_type=None):
            self.content = content
            self.filename = (self.get_random_filename() if filename
                is None else filename)
            self.content_type = (self.find_content_type(self.filename)
                if content_type is None else content_type)

    class UploadFile(BaseUploadObject):
        def __init__(self, path, filename=None, content_type=None):
            self.path = path
            self.filename = (os.path.split(path)[1] if filename is
                None else filename)
            self.content_type = (self.find_content_type(self.filename)
                if content_type is None else content_type)

The first ternary expression is tidy, but it reverses the intuitive
order of the operands: it should return ctype if it has a value and use
the string literal as fallback. The other ternary expressions are
unintuitive and so long that they must be wrapped. The overall
readability is worsened, not improved.

Rewriting using the None coalescing operator:

    class BaseUploadObject(object):
        def find_content_type(self, filename):
            ctype, encoding = mimetypes.guess_type(filename)
            return ctype ?? 'application/octet-stream'

    class UploadContent(BaseUploadObject):
        def __init__(self, content, filename=None, content_type=None):
            self.content = content
            self.filename = filename ?? self.get_random_filename()
            self.content_type = content_type ?? self.find_content_type(self.filename)

    class UploadFile(BaseUploadObject):
        def __init__(self, path, filename=None, content_type=None):
            self.path = path
            self.filename = filename ?? os.path.split(path)[1]
            self.content_type = content_type ?? self.find_content_type(self.filename)

This syntax has an intuitive ordering of the operands. In
find_content_type, for example, the preferred value ctype appears before
the fallback value. The terseness of the syntax also makes for fewer
lines of code and less code to visually parse, and reading from
left-to-right and top-to-bottom more accurately follows the execution
flow.

Rejected Ideas

The first three ideas in this section are oft-proposed alternatives to
treating None as special. For further background on why these are
rejected, see their treatment in PEP 531 and PEP 532 and the associated
discussions.

No-Value Protocol

The operators could be generalised to user-defined types by defining a
protocol to indicate when a value represents "no value". Such a protocol
may be a dunder method __has_value__(self) that returns True if the
value should be treated as having a value, and False if the value should
be treated as no value.

With this generalization, object would implement a dunder method
equivalent to this:

    def __has_value__(self):
        return True

NoneType would implement a dunder method equivalent to this:

    def __has_value__(self):
        return False

In the specification section, all uses of x is None would be replaced
with not x.__has_value__().

This generalization would allow for domain-specific "no-value" objects
to be coalesced just like None. For example, the pyasn1 package has a
type called Null that represents an ASN.1 null:

    >>> from pyasn1.type import univ
    >>> univ.Null() ?? univ.Integer(123)
    Integer(123)

Similarly, values such as math.nan and NotImplemented could be treated
as representing no value.

However, the "no-value" nature of these values is domain-specific, which
means they should be treated as a value by the language. For example,
math.nan.imag is well defined (it's 0.0), and so short-circuiting
math.nan?.imag to return math.nan would be incorrect.

As None is already defined by the language as being the value that
represents "no value", and the current specification would not preclude
switching to a protocol in the future (though changes to built-in
objects would not be compatible), this idea is rejected for now.

Boolean-aware operators

This suggestion is fundamentally the same as adding a no-value protocol,
and so the discussion above also applies.

Similar behavior to the ?? operator can be achieved with an or
expression, however or checks whether its left operand is false-y and
not specifically None. This approach is attractive, as it requires fewer
changes to the language, but ultimately does not solve the underlying
problem correctly.

Assuming the check is for truthiness rather than None, there is no
longer a need for the ?? operator. However, applying this check to the
?. and ?[] operators prevents perfectly valid operations applying

Consider the following example, where get_log_list() may return either a
list containing current log messages (potentially empty), or None if
logging is not enabled:

    lst = get_log_list()
    lst?.append('A log message')

If ?. is checking for true values rather than specifically None and the
log has not been initialized with any items, no item will ever be
appended. This violates the obvious intent of the code, which is to
append an item. The append method is available on an empty list, as are
all other list methods, and there is no reason to assume that these
members should not be used because the list is presently empty.

Further, there is no sensible result to use in place of the expression.
A normal lst.append returns None, but under this idea lst?.append may
result in either [] or None, depending on the value of lst. As with the
examples in the previous section, this makes no sense.

As checking for truthiness rather than None results in apparently valid
expressions no longer executing as intended, this idea is rejected.

Exception-aware operators

Arguably, the reason to short-circuit an expression when None is
encountered is to avoid the AttributeError or TypeError that would be
raised under normal circumstances. As an alternative to testing for
None, the ?. and ?[] operators could instead handle AttributeError and
TypeError raised by the operation and skip the remainder of the
expression.

This produces a transformation for a?.b.c?.d.e similar to this:

    _v = a
    try:
        _v = _v.b
    except AttributeError:
        pass
    else:
        _v = _v.c
        try:
            _v = _v.d
        except AttributeError:
            pass
        else:
            _v = _v.e

One open question is which value should be returned as the expression
when an exception is handled. The above example simply leaves the
partial result, but this is not helpful for replacing with a default
value. An alternative would be to force the result to None, which then
raises the question as to why None is special enough to be the result
but not special enough to be the test.

Secondly, this approach masks errors within code executed implicitly as
part of the expression. For ?., any AttributeError within a property or
__getattr__ implementation would be hidden, and similarly for ?[] and
__getitem__ implementations.

Similarly, simple typing errors such as {}?.ietms() could go unnoticed.

Existing conventions for handling these kinds of errors in the form of
the getattr builtin and the .get(key, default) method pattern
established by dict show that it is already possible to explicitly use
this behaviour.

As this approach would hide errors in code, it is rejected.

None-aware Function Call

The None-aware syntax applies to attribute and index access, so it seems
natural to ask if it should also apply to function invocation syntax. It
might be written as foo?(), where foo is only called if it is not None.

This has been deferred on the basis of the proposed operators being
intended to aid traversal of partially populated hierarchical data
structures, not for traversal of arbitrary class hierarchies. This is
reflected in the fact that none of the other mainstream languages that
already offer this syntax have found it worthwhile to support a similar
syntax for optional function invocations.

A workaround similar to that used by C# would be to write
maybe_none?.__call__(arguments). If the callable is None, the expression
will not be evaluated. (The C# equivalent uses ?.Invoke() on its
callable type.)

? Unary Postfix Operator

To generalize the None-aware behavior and limit the number of new
operators introduced, a unary, postfix operator spelled ? was suggested.
The idea is that ? might return a special object that could would
override dunder methods that return self. For example, foo? would
evaluate to foo if it is not None, otherwise it would evaluate to an
instance of NoneQuestion:

    class NoneQuestion():
        def __call__(self, *args, **kwargs):
            return self

        def __getattr__(self, name):
            return self

        def __getitem__(self, key):
            return self

With this new operator and new type, an expression like foo?.bar[baz]
evaluates to NoneQuestion if foo is None. This is a nifty
generalization, but it's difficult to use in practice since most
existing code won't know what NoneQuestion is.

Going back to one of the motivating examples above, consider the
following:

    >>> import json
    >>> created = None
    >>> json.dumps({'created': created?.isoformat()})

The JSON serializer does not know how to serialize NoneQuestion, nor
will any other API. This proposal actually requires lots of specialized
logic throughout the standard library and any third party library.

At the same time, the ? operator may also be too general, in the sense
that it can be combined with any other operator. What should the
following expressions mean?:

    >>> x? + 1
    >>> x? -= 1
    >>> x? == 1
    >>> ~x?

This degree of generalization is not useful. The operators actually
proposed herein are intentionally limited to a few operators that are
expected to make it easier to write common code patterns.

Built-in maybe

Haskell has a concept called Maybe that encapsulates the idea of an
optional value without relying on any special keyword (e.g. null) or any
special instance (e.g. None). In Haskell, the purpose of Maybe is to
avoid separate handling of "something" and nothing".

A Python package called pymaybe provides a rough approximation. The
documentation shows the following example:

    >>> maybe('VALUE').lower()
    'value'

    >>> maybe(None).invalid().method().or_else('unknown')
    'unknown'

The function maybe() returns either a Something instance or a Nothing
instance. Similar to the unary postfix operator described in the
previous section, Nothing overrides dunder methods in order to allow
chaining on a missing value.

Note that or_else() is eventually required to retrieve the underlying
value from pymaybe's wrappers. Furthermore, pymaybe does not short
circuit any evaluation. Although pymaybe has some strengths and may be
useful in its own right, it also demonstrates why a pure Python
implementation of coalescing is not nearly as powerful as support built
into the language.

The idea of adding a builtin maybe type to enable this scenario is
rejected.

Just use a conditional expression

Another common way to initialize default values is to use the ternary
operator. Here is an excerpt from the popular Requests package
<https://github.com/kennethreitz/requests/blob/14a555ac716866678bf17e43e23230d81
a8149f5/requests/models.py#L212>:

    data = [] if data is None else data
    files = [] if files is None else files
    headers = {} if headers is None else headers
    params = {} if params is None else params
    hooks = {} if hooks is None else hooks

This particular formulation has the undesirable effect of putting the
operands in an unintuitive order: the brain thinks, "use data if
possible and use [] as a fallback," but the code puts the fallback
before the preferred value.

The author of this package could have written it like this instead:

    data = data if data is not None else []
    files = files if files is not None else []
    headers = headers if headers is not None else {}
    params = params if params is not None else {}
    hooks = hooks if hooks is not None else {}

This ordering of the operands is more intuitive, but it requires 4 extra
characters (for "not "). It also highlights the repetition of
identifiers: data if data, files if files, etc.

When written using the None coalescing operator, the sample reads:

    data = data ?? []
    files = files ?? []
    headers = headers ?? {}
    params = params ?? {}
    hooks = hooks ?? {}

References

Copyright

This document has been placed in the public domain.

[1] C# Reference: Operators
(https://learn.microsoft.com/en/dotnet/csharp/language-reference/operators/)

[2] A Tour of the Dart Language: Operators
(https://www.dartlang.org/docs/dart-up-and-running/ch02.html#operators)

[3] Proposal: Nullish Coalescing for JavaScript
(https://github.com/tc39/proposal-nullish-coalescing)

[4] Proposal: Optional Chaining for JavaScript
(https://github.com/tc39/proposal-optional-chaining)

[5] Associated scripts
(https://github.com/python/peps/tree/master/pep-0505/)

license: Public Domain

authors:
Mark E. Haase
Steve Dower


text:
PEP: 232 Title: Function Attributes Version: $Revision$ Last-Modified:
$Date$ Author: Barry Warsaw<EMAIL_ADDRESS>Status: Final Type:
Standards Track Content-Type: text/x-rst Created: 02-Dec-2000
Python-Version: 2.1 Post-History: 20-Feb-2001

Introduction

This PEP describes an extension to Python, adding attribute dictionaries
to functions and methods. This PEP tracks the status and ownership of
this feature. It contains a description of the feature and outlines
changes necessary to support the feature. This PEP summarizes
discussions held in mailing list forums, and provides URLs for further
information, where appropriate. The CVS revision history of this file
contains the definitive historical record.

Background

Functions already have a number of attributes, some of which are
writable, e.g. func_doc, a.k.a. func.__doc__. func_doc has the
interesting property that there is special syntax in function (and
method) definitions for implicitly setting the attribute. This
convenience has been exploited over and over again, overloading
docstrings with additional semantics.

For example, John Aycock has written a system where docstrings are used
to define parsing rules.[1] Zope's ZPublisher ORB[2] uses docstrings to
signal publishable methods, i.e. methods that can be called through the
web.

The problem with this approach is that the overloaded semantics may
conflict with each other. For example, if we wanted to add a doctest
unit test to a Zope method that should not be publishable through the
web.

Proposal

This proposal adds a new dictionary to function objects, called
func_dict (a.k.a. __dict__). This dictionary can be set and get using
ordinary attribute set and get syntax.

Methods also gain getter syntax, and they currently access the attribute
through the dictionary of the underlying function object. It is not
possible to set attributes on bound or unbound methods, except by doing
so explicitly on the underlying function object. See the Future
Directions discussion below for approaches in subsequent versions of
Python.

A function object's __dict__ can also be set, but only to a dictionary
object. Deleting a function's __dict__, or setting it to anything other
than a concrete dictionary object results in a TypeError. If no function
attributes have ever been set, the function's __dict__ will be empty.

Examples

Here are some examples of what you can do with this feature.

    def a():
        pass

    a.publish = 1
    a.unittest = '''...'''

    if a.publish:
        print a()

    if hasattr(a, 'unittest'):
        testframework.execute(a.unittest)

    class C:
        def a(self):
            'just a docstring'
            a.publish = 1

    c = C()
    if c.a.publish:
        publish(c.a())

Other Uses

Paul Prescod enumerated a bunch of other uses on the python-dev thread.

Future Directions

Here are a number of future directions to consider. Any adoption of
these ideas would require a new PEP, which referenced this one, and
would have to be targeted at a Python version subsequent to the 2.1
release.

-   A previous version of this PEP allowed for both setter and getter of
    attributes on unbound methods, and only getter on bound methods. A
    number of problems were discovered with this policy.

    Because method attributes were stored in the underlying function,
    this caused several potentially surprising results:

        class C:
            def a(self): pass

        c1 = C()
        c2 = C()
        c1.a.publish = 1
        # c2.a.publish would now be == 1 also!

    Because a change to a bound c1 also caused a change to a bound to
    c2, setting of attributes on bound methods was disallowed. However,
    even allowing setting of attributes on unbound methods has its
    ambiguities:

        class D(C): pass
        class E(C): pass

        D.a.publish = 1
        # E.a.publish would now be == 1 also!

    For this reason, the current PEP disallows setting attributes on
    either bound or unbound methods, but does allow for getting
    attributes on either -- both return the attribute value on the
    underlying function object.

    A future PEP might propose to implement setting (bound or unbound)
    method attributes by setting attributes on the instance or class,
    using special naming conventions. I.e.:

        class C:
            def a(self): pass

        C.a.publish = 1
        C.__a_publish__ == 1 # true

        c = C()
        c.a.publish = 2
        c.__a_publish__ == 2 # true

        d = C()
        d.__a_publish__ == 1 # true

    Here, a lookup on the instance would look to the instance's
    dictionary first, followed by a lookup on the class's dictionary,
    and finally a lookup on the function object's dictionary.

-   Currently, Python supports function attributes only on Python
    functions (i.e. those that are written in Python, not those that are
    built-in). Should it be worthwhile, a separate patch can be crafted
    that will add function attributes to built-ins.

-   __doc__ is the only function attribute that currently has syntactic
    support for conveniently setting. It may be worthwhile to eventually
    enhance the language for supporting easy function attribute setting.
    Here are some syntaxes suggested by PEP reviewers:[3]

        def a {
            'publish' : 1,
            'unittest': '''...''',
            }
            (args):
            # ...

        def a(args):
            """The usual docstring."""
            {'publish' : 1,
             'unittest': '''...''',
             # etc.
             }

        def a(args) having (publish = 1):
            # see reference [3]
            pass

    The BDFL is currently against any such special syntactic support for
    setting arbitrary function attributes. Any syntax proposals would
    have to be outlined in new PEPs.

Dissenting Opinion

When this was discussed on the python-dev mailing list in April 2000, a
number of dissenting opinions were voiced. For completeness, the
discussion thread starts on python-dev.

The dissenting arguments appear to fall under the following categories:

-   no clear purpose (what does it buy you?)
-   other ways to do it (e.g. mappings as class attributes)
-   useless until syntactic support is included

Countering some of these arguments is the observation that with vanilla
Python 2.0, __doc__ can in fact be set to any type of object, so some
semblance of writable function attributes are already feasible. But that
approach is yet another corruption of __doc__.

And while it is of course possible to add mappings to class objects (or
in the case of function attributes, to the function's module), it is
more difficult and less obvious how to extract the attribute values for
inspection.

Finally, it may be desirable to add syntactic support, much the same way
that __doc__ syntactic support exists. This can be considered separately
from the ability to actually set and get function attributes.

Reference Implementation

This PEP has been accepted and the implementation has been integrated
into Python 2.1.

References

Copyright

This document has been placed in the public domain.

[1] Aycock, "Compiling Little Languages in Python",
https://legacy.python.org/workshops/1998-11/proceedings/papers/aycock-little/aycock-little.html

[2] https://web.archive.org/web/20010307022153/http://classic.zope.org:8080/Documentation/Reference/ORB

[3] Hudson, Michael, SourceForge patch implementing this syntax,
https://web.archive.org/web/20010901050535/http://sourceforge.net/tracker/index.php?func=detail&aid=403441&group_id=5470&atid=305470


text:
PEP: 470 Title: Removing External Hosting Support on PyPI Version:
$Revision$ Last-Modified: $Date$ Author: Donald Stufft
<EMAIL_ADDRESS>BDFL-Delegate: Paul Moore<EMAIL_ADDRESS>Discussions-To<EMAIL_ADDRESS>Status: Final Type: Process
Topic: Packaging Content-Type: text/x-rst Created: 12-May-2014
Post-History: 14-May-2014, 05-Jun-2014, 03-Oct-2014, 13-Oct-2014,
26-Aug-2015 Replaces: 438 Resolution:
https://mail.python.org/pipermail/distutils-sig/2015-September/026789.html

Abstract

This PEP proposes the deprecation and removal of support for hosting
files externally to PyPI as well as the deprecation and removal of the
functionality added by PEP 438, particularly rel information to classify
different types of links and the meta-tag to indicate API version.

Rationale

Historically PyPI did not have any method of hosting files nor any
method of automatically retrieving installables, it was instead focused
on providing a central registry of names, to prevent naming collisions,
and as a means of discovery for finding projects to use. In the course
of time setuptools began to scrape these human facing pages, as well as
pages linked from those pages, looking for things it could automatically
download and install. Eventually this became the "Simple" API which used
a similar URL structure however it eliminated any of the extraneous
links and information to make the API more efficient. Additionally PyPI
grew the ability for a project to upload release files directly to PyPI
enabling PyPI to act as a repository in addition to an index.

This gives PyPI two equally important roles that it plays in the Python
ecosystem, that of index to enable easy discovery of Python projects and
central repository to enable easy hosting, download, and installation of
Python projects. Due to the history behind PyPI and the very organic
growth it has experienced the lines between these two roles are blurry,
and this blurring has caused confusion for the end users of both of
these roles and this has in turn caused ire between people attempting to
use PyPI in different capacities, most often when end users want to use
PyPI as a repository but the author wants to use PyPI solely as an
index.

This confusion comes down to end users of projects not realizing if a
project is hosted on PyPI or if it relies on an external service. This
often manifests itself when the external service is down but PyPI is
not. People will see that PyPI works, and other projects works, but this
one specific one does not. They oftentimes do not realize who they need
to contact in order to get this fixed or what their remediation steps
are.

PEP 438 attempted to solve this issue by allowing projects to explicitly
declare if they were using the repository features or not, and if they
were not, it had the installers classify the links it found as either
"internal", "verifiable external" or "unverifiable external". PEP 438
was accepted and implemented in pip 1.4 (released on Jul 23, 2013) with
the final transition implemented in pip 1.5 (released on Jan 2, 2014).

PEP 438 was successful in bringing about more people to utilize PyPI's
repository features, an altogether good thing given the global CDN
powering PyPI providing speed ups for a lot of people, however it did so
by introducing a new point of confusion and pain for both the end users
and the authors.

By moving to using explicit multiple repositories we can make the lines
between these two roles much more explicit and remove the "hidden"
surprises caused by the current implementation of handling people who do
not want to use PyPI as a repository.

Key User Experience Expectations

1.  Easily allow external hosting to "just work" when appropriately
    configured at the system, user or virtual environment level.
2.  Eliminate any and all references to the confusing "verifiable
    external" and "unverifiable external" distinction from the user
    experience (both when installing and when releasing packages).
3.  The repository aspects of PyPI should become just the default
    package hosting location (i.e. the only one that is treated as
    opt-out rather than opt-in by most client tools in their default
    configuration). Aside from that aspect, hosting on PyPI should not
    otherwise provide an enhanced user experience over hosting your own
    package repository.
4.  Do all of the above while providing default behaviour that is secure
    against most attackers below the nation state adversary level.

Why Additional Repositories?

The two common installer tools, pip and easy_install/setuptools, both
support the concept of additional locations to search for files to
satisfy the installation requirements and have done so for many years.
This means that there is no need to "phase" in a new flag or concept and
the solution to installing a project from a repository other than PyPI
will function regardless of how old (within reason) the end user's
installer is. Not only has this concept existed in the Python tooling
for some time, but it is a concept that exists across languages and even
extending to the OS level with OS package tools almost universally using
multiple repository support making it extremely likely that someone is
already familiar with the concept.

Additionally, the multiple repository approach is a concept that is
useful outside of the narrow scope of allowing projects that wish to be
included on the index portion of PyPI but do not wish to utilize the
repository portion of PyPI. This includes places where a company may
wish to host a repository that contains their internal packages or where
a project may wish to have multiple "channels" of releases, such as
alpha, beta, release candidate, and final release. This could also be
used for projects wishing to host files which cannot be uploaded to
PyPI, such as multi-gigabyte data files or, currently at least, Linux
Wheels.

Why Not PEP 438 or Similar?

While the additional search location support has existed in pip and
setuptools for quite some time support for PEP 438 has only existed in
pip since the 1.4 version, and still has yet to be implemented in
setuptools. The design of PEP 438 did mean that users still benefited
for projects which did not require external files even with older
installers, however for projects which did require external files, users
are still silently being given either potentially unreliable or, even
worse, unsafe files to download. This system is also unique to Python as
it arises out of the history of PyPI, this means that it is almost
certain that this concept will be foreign to most, if not all users,
until they encounter it while attempting to use the Python toolchain.

Additionally, the classification system proposed by PEP 438 has, in
practice, turned out to be extremely confusing to end users, so much so
that it is a position of this PEP that the situation as it stands is
completely untenable. The common pattern for a user with this system is
to attempt to install a project possibly get an error message (or maybe
not if the project ever uploaded something to PyPI but later switched
without removing old files), see that the error message suggests
--allow-external, they reissue the command adding that flag most likely
getting another error message, see that this time the error message
suggests also adding --allow-unverified, and again issue the command a
third time, this time finally getting the thing they wish to install.

This UX failure exists for several reasons.

1.  If pip can locate files at all for a project on the Simple API it
    will simply use that instead of attempting to locate more. This is
    generally the right thing to do as attempting to locate more would
    erase a large part of the benefit of PEP 438. This means that if a
    project ever uploaded a file that matches what the user has
    requested for install that will be used regardless of how old it is.

2.  PEP 438 makes an implicit assumption that most projects would either
    upload themselves to PyPI or would update themselves to directly
    linking to release files. While a large number of projects did
    ultimately decide to upload to PyPI, some of them did so only
    because the UX around what PEP 438 was so bad that they felt forced
    to do so. More concerning however, is the fact that very few
    projects have opted to directly and safely link to files and instead
    they still simply link to pages which must be scraped in order to
    find the actual files, thus rendering the safe variant
    (--allow-external) largely useless.

3.  Even if an author wishes to directly link to their files, doing so
    safely is non-obvious. It requires the inclusion of a MD5 hash (for
    historical reasons) in the hash of the URL. If they do not include
    this then their files will be considered "unverified".

4.  PEP 438 takes a security centric view and disallows any form of a
    global opt in for unverified projects. While this is generally a
    good thing, it creates extremely verbose and repetitive command
    invocations such as:

        $ pip install --allow-external myproject --allow-unverified myproject myproject
        $ pip install --allow-all-external --allow-unverified myproject myproject

Multiple Repository/Index Support

Installers SHOULD implement or continue to offer, the ability to point
the installer at multiple URL locations. The exact mechanisms for a user
to indicate they wish to use an additional location is left up to each
individual implementation.

Additionally the mechanism discovering an installation candidate when
multiple repositories are being used is also up to each individual
implementation, however once configured an implementation should not
discourage, warn, or otherwise cast a negative light upon the use of a
repository simply because it is not the default repository.

Currently both pip and setuptools implement multiple repository support
by using the best installation candidate it can find from either
repository, essentially treating it as if it were one large repository.

Installers SHOULD also implement some mechanism for removing or
otherwise disabling use of the default repository. The exact specifics
of how that is achieved is up to each individual implementation.

Installers SHOULD also implement some mechanism for whitelisting and
blacklisting which projects a user wishes to install from a particular
repository. The exact specifics of how that is achieved is up to each
individual implementation.

The Python packaging guide MUST be updated with a section detailing the
options for setting up their own repository so that any project that
wishes to not host on PyPI in the future can reference that
documentation. This should include the suggestion that projects relying
on hosting their own repositories should document in their project
description how to install their project.

Deprecation and Removal of Link Spidering

A new hosting mode will be added to PyPI. This hosting mode will be
called pypi-only and will be in addition to the three that PEP 438 has
already given us which are pypi-explicit, pypi-scrape,
pypi-scrape-crawl. This new hosting mode will modify a project's simple
api page so that it only lists the files which are directly hosted on
PyPI and will not link to anything else.

Upon acceptance of this PEP and the addition of the pypi-only mode, all
new projects will be defaulted to the PyPI only mode and they will be
locked to this mode and unable to change this particular setting.

An email will then be sent out to all of the projects which are hosted
only on PyPI informing them that in one month their project will be
automatically converted to the pypi-only mode. A month after these
emails have been sent any of those projects which were emailed, which
still are hosted only on PyPI will have their mode set permanently to
pypi-only.

At the same time, an email will be sent to projects which rely on
hosting external to PyPI. This email will warn these projects that
externally hosted files have been deprecated on PyPI and that in 3
months from the time of that email that all external links will be
removed from the installer APIs. This email MUST include instructions
for converting their projects to be hosted on PyPI and MUST include
links to a script or package that will enable them to enter their PyPI
credentials and package name and have it automatically download and
re-host all of their files on PyPI. This email MUST also include
instructions for setting up their own index page. This email must also
contain a link to the Terms of Service for PyPI as many users may have
signed up a long time ago and may not recall what those terms are.
Finally this email must also contain a list of the links registered with
PyPI where we were able to detect an installable file was located.

Two months after the initial email, another email must be sent to any
projects still relying on external hosting. This email will include all
of the same information that the first email contained, except that the
removal date will be one month away instead of three.

Finally a month later all projects will be switched to the pypi-only
mode and PyPI will be modified to remove the externally linked files
functionality.

Summary of Changes

Repository side

1.  Deprecate and remove the hosting modes as defined by PEP 438.
2.  Restrict simple API to only list the files that are contained within
    the repository.

Client side

1.  Implement multiple repository support.
2.  Implement some mechanism for removing/disabling the default
    repository.
3.  Deprecate / Remove PEP 438

Impact

To determine impact, we've looked at all projects using a method of
searching PyPI which is similar to what pip and setuptools use and
searched for all files available on PyPI, safely linked from PyPI,
unsafely linked from PyPI, and finally unsafely available outside of
PyPI. When the same file was found in multiple locations it was
deduplicated and only counted it in one location based on the following
preferences: PyPI > Safely Off PyPI > Unsafely Off PyPI. This gives us
the broadest possible definition of impact, it means that any single
file for this project may no longer be visible by default, however that
file could be years old, or it could be a binary file while there is a
sdist available on PyPI. This means that the real impact will likely be
much smaller, but in an attempt not to miscount we take the broadest
possible definition.

At the time of this writing there are 65,232 projects hosted on PyPI and
of those, 59 of them rely on external files that are safely hosted
outside of PyPI and 931 of them rely on external files which are
unsafely hosted outside of PyPI. This shows us that 1.5% of projects
will be affected in some way by this change while 98.5% will continue to
function as they always have. In addition, only 5% of the projects
affected are using the features provided by PEP 438 to safely host
outside of PyPI while 95% of them are exposing their users to Remote
Code Execution via a Man In The Middle attack.

Frequently Asked Questions

I can't host my project on PyPI because of <X>, what should I do?

First you should decide if <X> is something inherent to PyPI, or if PyPI
could grow a feature to solve <X> for you. If PyPI can add a feature to
enable you to host your project on PyPI then you should propose that
feature. However, if <X> is something inherent to PyPI, such as wanting
to maintain control over your own files, then you should setup your own
package repository and instruct your users in your project's description
to add it to the list of repositories their installer of choice will
use.

My users have a worse experience with this PEP than before, how do I explain that?

Part of this answer is going to be specific to each individual project,
you'll need to explain to your users what caused you to decide to host
in your own repository instead of utilizing one that they already have
in their installer's default list of repositories. However, part of this
answer will also be explaining that the previous behavior of
transparently including external links was both a security hazard (given
that in most cases it allowed a MITM to execute arbitrary Python code on
the end users machine) and a reliability concern and that PEP 438
attempted to resolve this by making them explicitly opt in, but that PEP
438 brought along with it a number of serious usability issues. PEP 470
represents a simplification of the model to a model that many users will
be familiar with, which is common amongst Linux distributions.

Switching to a repository structure breaks my workflow or isn't allowed by my host?

There are a number of cheap or free hosts that would gladly support what
is required for a repository. In particular you don't actually need to
upload your files anywhere differently as long as you can generate a
host with the correct structure that points to where your files are
actually located. Many of these hosts provide free HTTPS using a shared
domain name, and free HTTPS certificates can be gotten from StartSSL, or
in the near future LetsEncrypt or they may be gotten cheap from any
number of providers.

Why don't you provide <X>?

The answer here will depend on what <X> is, however the answers
typically are one of:

-   We hadn't been thought of it and nobody had suggested it before.
-   We don't have sufficient experience with <X> to properly design a
    solution for it and would welcome a domain expert to help us provide
    it.
-   We're an open source project and nobody has decided to volunteer to
    design and implement <X> yet.

Additional PEPs to propose additional features are always welcome,
however they would need someone with the time and expertise to
accurately design <X>. This particular PEP is intended to focus on
getting us to a point where the capabilities of PyPI are straightforward
with an easily understood baseline that is similar to existing models
such as Linux distribution repositories.

Why should I register on PyPI if I'm running my own repository anyways?

PyPI serves two critical functions for the Python ecosystem. One of
those is as a central repository for the actual files that get
downloaded and installed by pip or another package manager and it is
this function that this PEP is concerned with and that you'd be
replacing if you're running your own repository. However, it also
provides a central registry of who owns what name in order to prevent
naming collisions, think of it sort of as DNS but for Python packages.
In addition to making sure that names are handed out in a first-come,
first-served manner it also provides a single place for users to go to
look search for and discover new projects. So the simple answer is, you
should still register your project with PyPI to avoid naming collisions
and to make it so people can still easily discover your project.

Rejected Proposals

Allow easier discovery of externally hosted indexes

A previous version of this PEP included a new feature added to both PyPI
and installers that would allow project authors to enter into PyPI a
list of URLs that would instruct installers to ignore any files uploaded
to PyPI and instead return an error telling the end user about these
extra URLs that they can add to their installer to make the installation
work.

This feature has been removed from the scope of the PEP because it
proved too difficult to develop a solution that avoided UX issues
similar to those that caused so many problems with the PEP 438 solution.
If needed, a future PEP could revisit this idea.

Keep the current classification system but adjust the options

This PEP rejects several related proposals which attempt to fix some of
the usability problems with the current system but while still keeping
the general gist of PEP 438.

This includes:

-   Default to allowing safely externally hosted files, but disallow
    unsafely hosted.
-   Default to disallowing safely externally hosted files with only a
    global flag to enable them, but disallow unsafely hosted.
-   Continue on the suggested path of PEP 438 and remove the option to
    unsafely host externally but continue to allow the option to safely
    host externally.

These proposals are rejected because:

-   The classification system introduced in PEP 438 in an entirely
    unique concept to PyPI which is not generically applicable even in
    the context of Python packaging. Adding additional concepts comes at
    a cost.
-   The classification system itself is non-obvious to explain and to
    pre-determine what classification of link a project will require
    entails inspecting the project's /simple/<project>/ page, and
    possibly any URLs linked from that page.
-   The ability to host externally while still being linked for
    automatic discovery is mostly a historic relic which causes a fair
    amount of pain and complexity for little reward.
-   The installer's ability to optimize or clean up the user interface
    is limited due to the nature of the implicit link scraping which
    would need to be done. This extends to the --allow-* options as well
    as the inability to determine if a link is expected to fail or not.
-   The mechanism paints a very broad brush when enabling an option,
    while PEP 438 attempts to limit this with per package options.
    However a project that has existed for an extended period of time
    may oftentimes have several different URLs listed in their simple
    index. It is not unusual for at least one of these to no longer be
    under control of the project. While an unregistered domain will sit
    there relatively harmless most of the time, pip will continue to
    attempt to install from it on every discovery phase. This means that
    an attacker simply needs to look at projects which rely on unsafe
    external URLs and register expired domains to attack users.

Implement this PEP, but Do Not Remove the Existing Links

This is essentially the backwards compatible version of this PEP. It
attempts to allow people using older clients, or clients which do not
implement this PEP to continue on as if nothing had changed. This
proposal is rejected because the vast bulk of those scenarios are unsafe
uses of the deprecated features. It is the opinion of this PEP that
silently allowing unsafe actions to take place on behalf of end users is
simply not an acceptable solution.

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End: