text:
PEP: 630 Title: Isolating Extension Modules Author: Petr Viktorin
<EMAIL_ADDRESS>Discussions-To<EMAIL_ADDRESS>Status: Final
Type: Informational Content-Type: text/x-rst Created: 25-Aug-2020
Post-History: 16-Jul-2020

Isolating Extension Modules HOWTO

Abstract

Traditionally, state belonging to Python extension modules was kept in C
static variables, which have process-wide scope. This document describes
problems of such per-process state and efforts to make per-module
state—a better default—possible and easy to use.

The document also describes how to switch to per-module state where
possible. This transition involves allocating space for that state,
potentially switching from static types to heap types, and—perhaps most
importantly—accessing per-module state from code.

About This Document

As an informational PEP <1#pep-types>, this document does not introduce
any changes; those should be done in their own PEPs (or issues, if small
enough). Rather, it covers the motivation behind an effort that spans
multiple releases, and instructs early adopters on how to use the
finished features.

Once support is reasonably complete, this content can be moved to
Python's documentation as a HOWTO. Meanwhile, in the spirit of
documentation-driven development, gaps identified in this PEP can show
where to focus the effort, and it can be updated as new features are
implemented.

Whenever this PEP mentions extension modules, the advice also applies to
built-in modules.

Note

This PEP contains generic advice. When following it, always take into
account the specifics of your project.

For example, while much of this advice applies to the C parts of
Python's standard library, the PEP does not factor in stdlib specifics
(unusual backward compatibility issues, access to private API, etc.).

PEPs related to this effort are:

-   PEP 384 -- Defining a Stable ABI, which added a C API for creating
    heap types
-   PEP 489 -- Multi-phase extension module initialization
-   PEP 573 -- Module State Access from C Extension Methods

This document is concerned with Python's public C API, which is not
offered by all implementations of Python. However, nothing in this PEP
is specific to CPython.

As with any Informational PEP, this text does not necessarily represent
a Python community consensus or recommendation.

Motivation

An interpreter is the context in which Python code runs. It contains
configuration (e.g. the import path) and runtime state (e.g. the set of
imported modules).

Python supports running multiple interpreters in one process. There are
two cases to think about—users may run interpreters:

-   in sequence, with several Py_InitializeEx/Py_FinalizeEx cycles, and
-   in parallel, managing "sub-interpreters" using
    Py_NewInterpreter/Py_EndInterpreter.

Both cases (and combinations of them) would be most useful when
embedding Python within a library. Libraries generally shouldn't make
assumptions about the application that uses them, which includes
assuming a process-wide "main Python interpreter".

Currently, CPython doesn't handle this use case well. Many extension
modules (and even some stdlib modules) use per-process global state,
because C static variables are extremely easy to use. Thus, data that
should be specific to an interpreter ends up being shared between
interpreters. Unless the extension developer is careful, it is very easy
to introduce edge cases that lead to crashes when a module is loaded in
more than one interpreter in the same process.

Unfortunately, per-interpreter state is not easy to achieve—extension
authors tend to not keep multiple interpreters in mind when developing,
and it is currently cumbersome to test the behavior.

Rationale for Per-module State

Instead of focusing on per-interpreter state, Python's C API is evolving
to better support the more granular per-module state. By default,
C-level data will be attached to a module object. Each interpreter will
then create its own module object, keeping the data separate. For
testing the isolation, multiple module objects corresponding to a single
extension can even be loaded in a single interpreter.

Per-module state provides an easy way to think about lifetime and
resource ownership: the extension module will initialize when a module
object is created, and clean up when it's freed. In this regard, a
module is just like any other PyObject *; there are no "on interpreter
shutdown" hooks to think—or forget—about.

Goal: Easy-to-Use Module State

It is currently cumbersome or impossible to do everything the C API
offers while keeping modules isolated. Enabled by PEP 384, changes in
PEP 489 and PEP 573 (and future planned ones) aim to first make it
possible to build modules this way, and then to make it easy to write
new modules this way and to convert old ones, so that it can become a
natural default.

Even if per-module state becomes the default, there will be use cases
for different levels of encapsulation: per-process, per-interpreter,
per-thread or per-task state. The goal is to treat these as exceptional
cases: they should be possible, but extension authors will need to think
more carefully about them.

Non-goals: Speedups and the GIL

There is some effort to speed up CPython on multi-core CPUs by making
the GIL per-interpreter. While isolating interpreters helps that effort,
defaulting to per-module state will be beneficial even if no speedup is
achieved, as it makes supporting multiple interpreters safer by default.

Making Modules Safe with Multiple Interpreters

There are many ways to correctly support multiple interpreters in
extension modules. The rest of this text describes the preferred way to
write such a module, or to convert an existing one.

Note that support is a work in progress; the API for some features your
module needs might not yet be ready.

A full example module is available as xxlimited.

This section assumes that "*you*" are an extension module author.

Isolated Module Objects

The key point to keep in mind when developing an extension module is
that several module objects can be created from a single shared library.
For example:

    >>> import sys
    >>> import binascii
    >>> old_binascii = binascii
    >>> del sys.modules['binascii']
    >>> import binascii  # create a new module object
    >>> old_binascii == binascii
    False

As a rule of thumb, the two modules should be completely independent.
All objects and state specific to the module should be encapsulated
within the module object, not shared with other module objects, and
cleaned up when the module object is deallocated. Exceptions are
possible (see Managing Global State), but they will need more thought
and attention to edge cases than code that follows this rule of thumb.

While some modules could do with less stringent restrictions, isolated
modules make it easier to set clear expectations (and guidelines) that
work across a variety of use cases.

Surprising Edge Cases

Note that isolated modules do create some surprising edge cases. Most
notably, each module object will typically not share its classes and
exceptions with other similar modules. Continuing from the example
above, note that old_binascii.Error and binascii.Error are separate
objects. In the following code, the exception is not caught:

    >>> old_binascii.Error == binascii.Error
    False
    >>> try:
    ...     old_binascii.unhexlify(b'qwertyuiop')
    ... except binascii.Error:
    ...     print('boo')
    ...
    Traceback (most recent call last):
      File "<stdin>", line 2, in <module>
    binascii.Error: Non-hexadecimal digit found

This is expected. Notice that pure-Python modules behave the same way:
it is a part of how Python works.

The goal is to make extension modules safe at the C level, not to make
hacks behave intuitively. Mutating sys.modules "manually" counts as a
hack.

Managing Global State

Sometimes, state of a Python module is not specific to that module, but
to the entire process (or something else "more global" than a module).
For example:

-   The readline module manages the terminal.
-   A module running on a circuit board wants to control the on-board
    LED.

In these cases, the Python module should provide access to the global
state, rather than own it. If possible, write the module so that
multiple copies of it can access the state independently (along with
other libraries, whether for Python or other languages).

If that is not possible, consider explicit locking.

If it is necessary to use process-global state, the simplest way to
avoid issues with multiple interpreters is to explicitly prevent a
module from being loaded more than once per process—see Opt-Out:
Limiting to One Module Object per Process.

Managing Per-Module State

To use per-module state, use multi-phase extension module initialization
introduced in PEP 489. This signals that your module supports multiple
interpreters correctly.

Set PyModuleDef.m_size to a positive number to request that many bytes
of storage local to the module. Usually, this will be set to the size of
some module-specific struct, which can store all of the module's C-level
state. In particular, it is where you should put pointers to classes
(including exceptions, but excluding static types) and settings (e.g.
csv's field_size_limit) which the C code needs to function.

Note

Another option is to store state in the module's __dict__, but you must
avoid crashing when users modify __dict__ from Python code. This means
error- and type-checking at the C level, which is easy to get wrong and
hard to test sufficiently.

If the module state includes PyObject pointers, the module object must
hold references to those objects and implement the module-level hooks
m_traverse, m_clear and m_free. These work like tp_traverse, tp_clear
and tp_free of a class. Adding them will require some work and make the
code longer; this is the price for modules which can be unloaded
cleanly.

An example of a module with per-module state is currently available as
xxlimited; example module initialization shown at the bottom of the
file.

Opt-Out: Limiting to One Module Object per Process

A non-negative PyModuleDef.m_size signals that a module supports
multiple interpreters correctly. If this is not yet the case for your
module, you can explicitly make your module loadable only once per
process. For example:

    static int loaded = 0;

    static int
    exec_module(PyObject* module)
    {
        if (loaded) {
            PyErr_SetString(PyExc_ImportError,
                            "cannot load module more than once per process");
            return -1;
        }
        loaded = 1;
        // ... rest of initialization
    }

Module State Access from Functions

Accessing the state from module-level functions is straightforward.
Functions get the module object as their first argument; for extracting
the state, you can use PyModule_GetState:

    static PyObject *
    func(PyObject *module, PyObject *args)
    {
        my_struct *state = (my_struct*)PyModule_GetState(module);
        if (state == NULL) {
            return NULL;
        }
        // ... rest of logic
    }

Note

PyModule_GetState may return NULL without setting an exception if there
is no module state, i.e. PyModuleDef.m_size was zero. In your own
module, you're in control of m_size, so this is easy to prevent.

Heap Types

Traditionally, types defined in C code are static; that is,
static PyTypeObject structures defined directly in code and initialized
using PyType_Ready().

Such types are necessarily shared across the process. Sharing them
between module objects requires paying attention to any state they own
or access. To limit the possible issues, static types are immutable at
the Python level: for example, you can't set str.myattribute = 123.

Note

Sharing truly immutable objects between interpreters is fine, as long as
they don't provide access to mutable objects. However, in CPython, every
Python object has a mutable implementation detail: the reference count.
Changes to the refcount are guarded by the GIL. Thus, code that shares
any Python objects across interpreters implicitly depends on CPython's
current, process-wide GIL.

Because they are immutable and process-global, static types cannot
access "their" module state. If any method of such a type requires
access to module state, the type must be converted to a heap-allocated
type, or heap type for short. These correspond more closely to classes
created by Python's class statement.

For new modules, using heap types by default is a good rule of thumb.

Static types can be converted to heap types, but note that the heap type
API was not designed for "lossless" conversion from static types -- that
is, creating a type that works exactly like a given static type. Unlike
static types, heap type objects are mutable by default. Also, when
rewriting the class definition in a new API, you are likely to
unintentionally change a few details (e.g. pickle-ability or inherited
slots). Always test the details that are important to you.

Defining Heap Types

Heap types can be created by filling a PyType_Spec structure, a
description or "blueprint" of a class, and calling
PyType_FromModuleAndSpec() to construct a new class object.

Note

Other functions, like PyType_FromSpec(), can also create heap types, but
PyType_FromModuleAndSpec() associates the module with the class,
allowing access to the module state from methods.

The class should generally be stored in both the module state (for safe
access from C) and the module's __dict__ (for access from Python code).

Garbage Collection Protocol

Instances of heap types hold a reference to their type. This ensures
that the type isn't destroyed before all its instances are, but may
result in reference cycles that need to be broken by the garbage
collector.

To avoid memory leaks, instances of heap types must implement the
garbage collection protocol. That is, heap types should:

-   Have the Py_TPFLAGS_HAVE_GC flag.
-   Define a traverse function using Py_tp_traverse, which visits the
    type (e.g. using Py_VISIT(Py_TYPE(self));).

Please refer to the documentation of Py_TPFLAGS_HAVE_GC and tp_traverse
<https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_traverse>
for additional considerations.

If your traverse function delegates to the tp_traverse of its base class
(or another type), ensure that Py_TYPE(self) is visited only once. Note
that only heap type are expected to visit the type in tp_traverse.

For example, if your traverse function includes:

    base->tp_traverse(self, visit, arg)

...and base may be a static type, then it should also include:

    if (base->tp_flags & Py_TPFLAGS_HEAPTYPE) {
        // a heap type's tp_traverse already visited Py_TYPE(self)
    } else {
        Py_VISIT(Py_TYPE(self));
    }

It is not necessary to handle the type's reference count in tp_new and
tp_clear.

Module State Access from Classes

If you have a type object defined with PyType_FromModuleAndSpec(), you
can call PyType_GetModule to get the associated module, and then
PyModule_GetState to get the module's state.

To save a some tedious error-handling boilerplate code, you can combine
these two steps with PyType_GetModuleState, resulting in:

    my_struct *state = (my_struct*)PyType_GetModuleState(type);
    if (state === NULL) {
        return NULL;
    }

Module State Access from Regular Methods

Accessing the module-level state from methods of a class is somewhat
more complicated, but is possible thanks to the changes introduced in
PEP 573. To get the state, you need to first get the defining class, and
then get the module state from it.

The largest roadblock is getting the class a method was defined in, or
that method's "defining class" for short. The defining class can have a
reference to the module it is part of.

Do not confuse the defining class with Py_TYPE(self). If the method is
called on a subclass of your type, Py_TYPE(self) will refer to that
subclass, which may be defined in different module than yours.

Note

The following Python code can illustrate the concept.
Base.get_defining_class returns Base even if type(self) == Sub:

    class Base:
        def get_defining_class(self):
            return __class__

    class Sub(Base):
        pass

For a method to get its "defining class", it must use the
METH_METHOD | METH_FASTCALL | METH_KEYWORDS calling convention and the
corresponding PyCMethod signature:

    PyObject *PyCMethod(
        PyObject *self,               // object the method was called on
        PyTypeObject *defining_class, // defining class
        PyObject *const *args,        // C array of arguments
        Py_ssize_t nargs,             // length of "args"
        PyObject *kwnames)            // NULL, or dict of keyword arguments

Once you have the defining class, call PyType_GetModuleState to get the
state of its associated module.

For example:

    static PyObject *
    example_method(PyObject *self,
            PyTypeObject *defining_class,
            PyObject *const *args,
            Py_ssize_t nargs,
            PyObject *kwnames)
    {
        my_struct *state = (my_struct*)PyType_GetModuleState(defining_class);
        if (state === NULL) {
            return NULL;
        }
        ... // rest of logic
    }

    PyDoc_STRVAR(example_method_doc, "...");

    static PyMethodDef my_methods[] = {
        {"example_method",
          (PyCFunction)(void(*)(void))example_method,
          METH_METHOD|METH_FASTCALL|METH_KEYWORDS,
          example_method_doc}
        {NULL},
    }

Module State Access from Slot Methods, Getters and Setters

Note

This is new in Python 3.11.

  If you use the limited API
  <https://docs.python.org/3/c-api/stable.html>__, you must update
  Py_LIMITED_API to 0x030b0000`, losing ABI compatibility with earlier
  versions.

Slot methods -- the fast C equivalents for special methods, such as
nb_add for __add__ or tp_new for initialization -- have a very simple
API that doesn't allow passing in the defining class, unlike with
PyCMethod. The same goes for getters and setters defined with
PyGetSetDef.

To access the module state in these cases, use the PyType_GetModuleByDef
function, and pass in the module definition. Once you have the module,
call PyModule_GetState to get the state:

    PyObject *module = PyType_GetModuleByDef(Py_TYPE(self), &module_def);
    my_struct *state = (my_struct*)PyModule_GetState(module);
    if (state === NULL) {
        return NULL;
    }

PyType_GetModuleByDef works by searching the MRO (i.e. all superclasses)
for the first superclass that has a corresponding module.

Note

In very exotic cases (inheritance chains spanning multiple modules
created from the same definition), PyType_GetModuleByDef might not
return the module of the true defining class. However, it will always
return a module with the same definition, ensuring a compatible C memory
layout.

Lifetime of the Module State

When a module object is garbage-collected, its module state is freed.
For each pointer to (a part of) the module state, you must hold a
reference to the module object.

Usually this is not an issue, because types created with
PyType_FromModuleAndSpec, and their instances, hold a reference to the
module. However, you must be careful in reference counting when you
reference module state from other places, such as callbacks for external
libraries.

Open Issues

Several issues around per-module state and heap types are still open.

Discussions about improving the situation are best held on the capi-sig
mailing list.

Type Checking

Currently (as of Python 3.10), heap types have no good API to write
Py*_Check functions (like PyUnicode_Check exists for str, a static
type), and so it is not easy to ensure that instances have a particular
C layout.

Metaclasses

Currently (as of Python 3.10), there is no good API to specify the
metaclass of a heap type; that is, the ob_type field of the type object.

Per-Class Scope

It is also not possible to attach state to types. While PyHeapTypeObject
is a variable-size object (PyVarObject), its variable-size storage is
currently consumed by slots. Fixing this is complicated by the fact that
several classes in an inheritance hierarchy may need to reserve some
state.

Lossless Conversion to Heap Types

The heap type API was not designed for "lossless" conversion from static
types; that is, creating a type that works exactly like a given static
type. The best way to address it would probably be to write a guide that
covers known "gotchas".

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 361 Title: Python 2.6 and 3.0 Release Schedule Version: $Revision$
Last-Modified: $Date$ Author: Neal Norwitz, Barry Warsaw Status: Final
Type: Informational Topic: Release Content-Type: text/x-rst Created:
29-Jun-2006 Python-Version: 2.6, 3.0 Post-History: 17-Mar-2008

Abstract

This document describes the development and release schedule for Python
2.6 and 3.0. The schedule primarily concerns itself with PEP-sized
items. Small features may be added up to and including the first beta
release. Bugs may be fixed until the final release.

There will be at least two alpha releases, two beta releases, and one
release candidate. The releases are planned for October 2008.

Python 2.6 is not only the next advancement in the Python 2 series, it
is also a transitional release, helping developers begin to prepare
their code for Python 3.0. As such, many features are being backported
from Python 3.0 to 2.6. Thus, it makes sense to release both versions in
at the same time. The precedence for this was set with the Python 1.6
and 2.0 release.

Until rc, we will be releasing Python 2.6 and 3.0 in lockstep, on a
monthly release cycle. The releases will happen on the first Wednesday
of every month through the beta testing cycle. Because Python 2.6 is
ready sooner, and because we have outside deadlines we'd like to meet,
we've decided to split the rc releases. Thus Python 2.6 final is
currently planned to come out two weeks before Python 3.0 final.

Release Manager and Crew

-   2.6/3.0 Release Manager: Barry Warsaw
-   Windows installers: Martin v. Loewis
-   Mac installers: Ronald Oussoren
-   Documentation: Georg Brandl
-   RPMs: Sean Reifschneider

Release Lifespan

Python 3.0 is no longer being maintained for any purpose.

Python 2.6.9 is the final security-only source-only maintenance release
of the Python 2.6 series. With its release on October 29, 2013, all
official support for Python 2.6 has ended. Python 2.6 is no longer being
maintained for any purpose.

Release Schedule

-   Feb 29 2008: Python 2.6a1 and 3.0a3 are released
-   Apr 02 2008: Python 2.6a2 and 3.0a4 are released
-   May 08 2008: Python 2.6a3 and 3.0a5 are released
-   Jun 18 2008: Python 2.6b1 and 3.0b1 are released
-   Jul 17 2008: Python 2.6b2 and 3.0b2 are released
-   Aug 20 2008: Python 2.6b3 and 3.0b3 are released
-   Sep 12 2008: Python 2.6rc1 is released
-   Sep 17 2008: Python 2.6rc2 and 3.0rc1 released
-   Oct 01 2008: Python 2.6 final released
-   Nov 06 2008: Python 3.0rc2 released
-   Nov 21 2008: Python 3.0rc3 released
-   Dec 03 2008: Python 3.0 final released
-   Dec 04 2008: Python 2.6.1 final released
-   Apr 14 2009: Python 2.6.2 final released
-   Oct 02 2009: Python 2.6.3 final released
-   Oct 25 2009: Python 2.6.4 final released
-   Mar 19 2010: Python 2.6.5 final released
-   Aug 24 2010: Python 2.6.6 final released
-   Jun 03 2011: Python 2.6.7 final released (security-only)
-   Apr 10 2012: Python 2.6.8 final released (security-only)
-   Oct 29 2013: Python 2.6.9 final released (security-only)

Completed features for 3.0

See PEP 3000 and PEP 3100 for details on the Python 3.0 project.

Completed features for 2.6

PEPs:

-   PEP 352: Raising a string exception now triggers a TypeError.
    Attempting to catch a string exception raises DeprecationWarning.
    BaseException.message has been deprecated.
-   PEP 358: The "bytes" Object
-   PEP 366: Main module explicit relative imports
-   PEP 370: Per user site-packages directory
-   PEP 3112: Bytes literals in Python 3000
-   PEP 3127: Integer Literal Support and Syntax
-   PEP 371: Addition of the multiprocessing package

New modules in the standard library:

-   json
-   new enhanced turtle module
-   ast

Deprecated modules and functions in the standard library:

-   buildtools
-   cfmfile
-   commands.getstatus()
-   macostools.touched()
-   md5
-   MimeWriter
-   mimify
-   popen2, os.popen[234]()
-   posixfile
-   sets
-   sha

Modules removed from the standard library:

-   gopherlib
-   rgbimg
-   macfs

Warnings for features removed in Py3k:

-   builtins: apply, callable, coerce, dict.has_key, execfile, reduce,
    reload
-   backticks and <>
-   float args to xrange
-   coerce and all its friends
-   comparing by default comparison
-   {}.has_key()
-   file.xreadlines
-   softspace removal for print() function
-   removal of modules because of PEP 4/PEP 3100/PEP 3108

Other major features:

-   with/as will be keywords
-   a __dir__() special method to control dir() was added[1]
-   AtheOS support stopped.
-   warnings module implemented in C
-   compile() takes an AST and can convert to byte code

Possible features for 2.6

New features should be implemented prior to alpha2, particularly any C
modifications or behavioral changes. New features must be implemented
prior to beta1 or will require Release Manager approval.

The following PEPs are being worked on for inclusion in 2.6: None.

Each non-trivial feature listed here that is not a PEP must be discussed
on python-dev. Other enhancements include:

-   distutils replacement (requires a PEP)

New modules in the standard library:

-   winerror https://bugs.python.org/issue1505257 (Patch rejected,
    module should be written in C)
-   setuptools BDFL pronouncement for inclusion in 2.5:
    https://mail.python.org/pipermail/python-dev/2006-April/063964.html

PJE's withdrawal from 2.5 for inclusion in 2.6:
https://mail.python.org/pipermail/python-dev/2006-April/064145.html

Modules to gain a DeprecationWarning (as specified for Python 2.6 or
through negligence):

-   rfc822
-   mimetools
-   multifile
-   compiler package (or a Py3K warning instead?)
-   Convert Parser/*.c to use the C warnings module rather than printf
-   Add warnings for Py3k features removed:
    -   __getslice__/__setslice__/__delslice__
    -   float args to PyArgs_ParseTuple
    -   __cmp__?
    -   other comparison changes?
    -   int division?
    -   All PendingDeprecationWarnings (e.g. exceptions)
    -   using zip() result as a list
    -   the exec statement (use function syntax)
    -   function attributes that start with func_* (should use __*__)
    -   the L suffix for long literals
    -   renaming of __nonzero__ to __bool__
    -   multiple inheritance with classic classes? (MRO might change)
    -   properties and classic classes? (instance attrs shadow property)
-   use __bool__ method if available and there's no __nonzero__
-   Check the various bits of code in Demo/ and Tools/ all still work,
    update or remove the ones that don't.
-   All modules in Modules/ should be updated to be ssize_t clean.
-   All of Python (including Modules/) should compile cleanly with g++
-   Start removing deprecated features and generally moving towards Py3k
-   Replace all old style tests (operate on import) with unittest or
    docttest
-   Add tests for all untested modules
-   Document undocumented modules/features
-   bdist_deb in distutils package
    https://mail.python.org/pipermail/python-dev/2006-February/060926.html
-   bdist_egg in distutils package
-   pure python pgen module (Owner: Guido) Deferral to 2.6:
    https://mail.python.org/pipermail/python-dev/2006-April/064528.html
-   Remove the fpectl module?

Deferred until 2.7

None

Open issues

How should import warnings be handled?

-   https://mail.python.org/pipermail/python-dev/2006-June/066345.html
-   https://bugs.python.org/issue1515609
-   https://bugs.python.org/issue1515361

References

Copyright

This document has been placed in the public domain.

[1] Adding a __dir__() magic method
https://mail.python.org/pipermail/python-dev/2006-July/067139.html

license: Public Domain

authors:
Barry Warsaw
Neal Norwitz


text:
PEP: 482 Title: Literature Overview for Type Hints Version: $Revision$
Last-Modified: $Date$ Author: Łukasz Langa<EMAIL_ADDRESS>Discussions-To<EMAIL_ADDRESS>Status: Final Type:
Informational Topic: Typing Content-Type: text/x-rst Created:
08-Jan-2015 Post-History:

Abstract

This PEP is one of three related to type hinting. This PEP gives a
literature overview of related work. The main spec is PEP 484.

Existing Approaches for Python

mypy

(This section is a stub, since mypy is essentially what we're
proposing.)

Reticulated Python

Reticulated Python by Michael Vitousek is an example of a slightly
different approach to gradual typing for Python. It is described in an
actual academic paper written by Vitousek with Jeremy Siek and Jim Baker
(the latter of Jython fame).

PyCharm

PyCharm by JetBrains has been providing a way to specify and check types
for about four years. The type system suggested by PyCharm grew from
simple class types to tuple types, generic types, function types, etc.
based on feedback of many users who shared their experience of using
type hints in their code.

Others

TBD: Add sections on pyflakes, pylint, numpy, Argument Clinic,
pytypedecl, numba, obiwan.

Existing Approaches in Other Languages

ActionScript

ActionScript is a class-based, single inheritance, object-oriented
superset of ECMAScript. It supports interfaces and strong
runtime-checked static typing. Compilation supports a “strict dialect”
where type mismatches are reported at compile-time.

Example code with types:

    package {
      import flash.events.Event;

      public class BounceEvent extends Event {
        public static const BOUNCE:String = "bounce";
        private var _side:String = "none";

        public function get side():String {
          return _side;
        }

        public function BounceEvent(type:String, side:String){
          super(type, true);
          _side = side;
        }

        public override function clone():Event {
          return new BounceEvent(type, _side);
        }
      }
    }

Dart

Dart is a class-based, single inheritance, object-oriented language with
C-style syntax. It supports interfaces, abstract classes, reified
generics, and optional typing.

Types are inferred when possible. The runtime differentiates between two
modes of execution: checked mode aimed for development (catching type
errors at runtime) and production mode recommended for speed execution
(ignoring types and asserts).

Example code with types:

    class Point {
        final num x, y;

        Point(this.x, this.y);

        num distanceTo(Point other) {
            var dx = x - other.x;
            var dy = y - other.y;
            return math.sqrt(dx * dx + dy * dy);
        }
    }

Hack

Hack is a programming language that interoperates seamlessly with PHP.
It provides opt-in static type checking, type aliasing, generics,
nullable types, and lambdas.

Example code with types:

    <?hh
    class MyClass {
      private ?string $x = null;

      public function alpha(): int {
        return 1;
      }

      public function beta(): string {
        return 'hi test';
      }
    }

    function f(MyClass $my_inst): string {
      // Will generate a hh_client error
      return $my_inst->alpha();
    }

TypeScript

TypeScript is a typed superset of JavaScript that adds interfaces,
classes, mixins and modules to the language.

Type checks are duck typed. Multiple valid function signatures are
specified by supplying overloaded function declarations. Functions and
classes can use generics as type parameterization. Interfaces can have
optional fields. Interfaces can specify array and dictionary types.
Classes can have constructors that implicitly add arguments as fields.
Classes can have static fields. Classes can have private fields. Classes
can have getters/setters for fields (like property). Types are inferred.

Example code with types:

    interface Drivable {
        start(): void;
        drive(distance: number): boolean;
        getPosition(): number;
    }

    class Car implements Drivable {
        private _isRunning: boolean;
        private _distanceFromStart: number;

        constructor() {
            this._isRunning = false;
            this._distanceFromStart = 0;
        }

        public start() {
            this._isRunning = true;
        }

        public drive(distance: number): boolean {
            if (this._isRunning) {
                this._distanceFromStart += distance;
                return true;
            }
            return false;
        }

        public getPosition(): number {
            return this._distanceFromStart;
        }
    }

Copyright

This document has been placed in the public domain.


text:
PEP: 777 Title: How to Re-invent the Wheel Author: Ethan Smith
<EMAIL_ADDRESS>Sponsor: Barry Warsaw<EMAIL_ADDRESS>PEP-Delegate: Paul Moore<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pep-777-how-to-re-invent-the-wheel/67484
Status: Draft Type: Standards Track Topic: Packaging Created:
09-Oct-2024 Post-History: 10-Oct-2024

Abstract

The current wheel 1.0 specification <427> was written over a decade ago,
and has been extremely robust to changes in the Python packaging
ecosystem. Previous efforts to improve the wheel specification
were deferred <491#pep-deferral> to focus on other packaging
specifications. Meanwhile, the use of wheels has changed dramatically in
the last decade. There have been many requests for new wheel features
over the years; however, a fundamental obstacle to evolving the wheel
specification has been that there is no defined process for how to
handle adding backwards-incompatible features to wheels. Therefore, to
enable other PEPs to describe new enhancements to the wheel
specification, this PEP prescribes compatibility requirements on future
wheel revisions. This PEP does not specify a new wheel revision. The
specification of a new wheel format (“Wheel 2.0”) is left to a future
PEP.

Rationale

Currently, wheel specification changes that require new installer
behavior are backwards incompatible and require a major version increase
in the wheel metadata format. An increase of the wheel major version has
yet to happen, partially because such a change has the potential to be
catastrophically disruptive. Per the wheel specification, any installer
that does not support the new major version must abort at install time.
This means that if the major version were to be incremented without
further planning, many users would see installation failures as older
installers reject new wheels uploaded to public package indices like the
Python Package Index (PyPI). It is critically important to carefully
plan the interactions between build tools, package indices, and package
installers to avoid incompatibility issues, especially considering the
long tail of users who are slow to update their installers.

The backward compatibility concerns have prevented valuable improvements
to the wheel file format, such as better compression, wheel data format
improvements, better information about what is included in a wheel, and
JSON formatted metadata in the ".dist-info" folder.

This PEP describes constraints and behavior for new wheel revisions to
preserve stability for existing tools that do not support a new major
version of the wheel format. This ensures that backwards incompatible
changes to the wheel specification will only affect users and tools that
are properly set up to use the newer wheels. With a clear path for
evolving the wheel specification, future PEPs will be able to improve
the wheel format without needing to re-define a completely new
compatibility story.

Specification

Add Wheel-Version Metadata Field to Core Metadata

Currently, the wheel 1.0 PEP <427>, PEP 427, specifies that wheel files
must contain a WHEEL metadata file that contains the version of the
wheel specification that the file conforms to. PEP 427 stipulates that
installers MUST warn on installation of a wheel with a minor version
greater than supported, and MUST abort on installation of wheels with a
major version that is greater than what the installer supports. This
ensures that users do not get invalid installations from wheels that
installers cannot properly install.

However, resolvers do not currently exclude wheels with an incompatible
wheel version. There is also currently no way for a resolver to check a
wheel's version without downloading the wheel directly. To make wheel
version filtering easy for resolvers, the wheel version MUST be included
in the relevant metadata file (currently METADATA). This will allow
resolvers to efficiently check the wheel version using the PEP 658
metadata API without needing to download and inspect the
.dist-info/WHEEL file.

To accomplish this, a new field, Wheel-Version, will be added to the
Core Metadata Specification. This field is single use, and must contain
the exact same version specified as the Wheel-Version entry in the WHEEL
file, or any future replacement file defining metadata about the wheel
file. If Wheel-Version is absent from the metadata file, then tools MUST
infer the wheel file major version as 1.

Wheel-Version MUST NOT be included in source distribution metadata
(PKG-INFO) files. If a tool encounters Wheel-Version inside of a source
distribution metadata file, it SHOULD raise an error.

Wheel-Version MAY be included in the metadata file for wheels of version
1, but for wheels of version 2 or higher, the metadata file MUST include
Wheel-Version. This enforces that future revisions of the wheel
specification can rely on resolvers skipping incompatible wheels by
checking the Wheel-Version field. Build backends are encouraged to
include Wheel-Version in all wheels that they generate, regardless of
version.

Installers SHOULD copy the metadata file in a wheel unmodified during
installation. This prevents the need to update the RECORD file, which is
an error prone process. Tools reading installed core metadata SHOULD NOT
assume that the field is present, as other installation formats may omit
it.

When installing a wheel, installers MUST take the following steps:

1.  Check that the values of Wheel-Version in both the core metadata
    file and wheel metadata file match. If they do not match, installers
    MUST abort installation. Neither value takes precedence.
2.  Check that the installer is compatible with Wheel-Version. If
    Wheel-Version is absent, assume the version is 1.0. Warn if minor
    version is greater, abort if major version is greater. This
    procedure is identital to that in PEP 427.
3.  Proceed with installation as specified in the Binary Distribution
    Format specification.

Resolver Behavior Regarding Wheel-Version

Resolvers, in the process of selecting a wheel to install, MUST check a
candidate wheel's Wheel-Version, and ignore incompatible wheel files.
Without ignoring these files, older installers might select a wheel that
uses an unsupported wheel version for that installer, and force the
installer to abort per PEP 427. By skipping incompatible wheel files,
users will not see installation errors when a project adopts a new wheel
major version. As already specified in PEP 427, installers MUST abort if
a user tries to directly install a wheel that is incompatible. If, in
the process of resolving packages found in multiple indices, a resolver
comes across two wheels of the same distribution and version, resolvers
should prioritize the wheel of the highest compatible version.

While the above protects users from unexpected breakages, users may miss
a new release of a distribution if their installer does not support the
wheel version used in the release. Imagine in the future that a package
publishes 3.0 wheel files. Downstream users won't see that there is a
new release available if their installers only support 2.x wheels.
Therefore, installers SHOULD emit a warning if, in the process of
resolving packages, they come across an incompatible wheel and skip it.

First Major Version Bump Must Change File Extension

Unfortunately, existing resolvers do not check the compatibility of
wheels before selecting them as installation candidates. Until a
majority of users update to installers that properly check for wheel
compatibility, it is unsafe to allow publishing wheels of a new major
version that existing resolvers might select. It could take upwards of
four years before the majority of users are on updated resolvers, based
on current data about PyPI installer usage (See the
777-pypi-download-analysis, for details). To allow for experimentation
and faster adoption of 2.0 wheels, this PEP proposes a change to the
file extension of the wheel file format, from .whl to .whlx for all
future wheel versions. Note that x in whlx is the letter "x" and does
not specify the wheel major version. The change to extension name
resolves the initial transition issue of 2.0 wheels breaking users on
existing installers that do not implement Wheel-Version checks. By using
a different file extension, 2.0 wheels can immediately be uploaded to
PyPI, and users will be able to experiment with the new features right
away. Users on older installers will simply ignore these new files.

One rejected alternative would be to keep the .whl extension, but delay
the publishing of wheel 2.0 to PyPI. For more on that, please see
Rejected Ideas.

Recommended Build Backend Behavior with New Wheel Formats

Build backends are recommended to generate the most compatible wheel
based on features a project uses. For example, if a wheel does not use
symbolic links, and such a feature was introduced in wheel 5.0, the
build backend could generate a wheel of version 4.0. On the other hand,
some features will want to be adopted by default. For example, if wheel
3.0 introduces better compression, the build backend may wish to enable
this feature by default to improve the wheel size and download
performance.

Limitations on Future Wheel Revisions

While it is difficult to know what future features may be planned for
the wheel format, it is important that certain compatibility promises
are maintained.

Wheel files, when installed, MUST stay compatible with the Python
standard library's importlib.metadata for all supported CPython
versions. For example, replacing .dist-info/METADATA with a JSON
formatted metadata file MUST be a multi-major version migration with one
version introducing the new JSON file alongside the existing email
header format, and another future version removing the email header
format metadata file. The version to remove .dist-info/METADATA also
MUST be adopted only after the last CPython release that lacked support
for the new file reaches end of life. This ensures that code using
importlib.metadata will not break with wheel major version revisions.

Wheel files MUST remain ZIP format files as the outer container format.
Additionally, the .dist-info metadata directory MUST be placed at the
root of the archive without any compression, so that unpacking the wheel
file produces a normal .dist-info directory holding any metadata for the
wheel. Future wheel revisions MAY modify the layout, compression, and
other attributes about non-metadata components of a wheel such as data
and code. This assures that future wheel revisions remain compatible
with tools operating on package metadata, while allowing for
improvements to code storage in the wheel, such as adopting compression.

Package tooling MUST NOT assume that the contents and format of the
wheel file will remain the same for future wheel major versions beyond
the limitations above about metadata folder contents and outer container
format. For example, newer wheel major versions may add or remove
filename components, such as the build tag or the platform tag.
Therefore it is incumbent upon tooling to check the metadata for the
Wheel-Version before attempting to install a wheel.

Finally, future wheel revisions MUST NOT use any compression formats not
in the CPython standard library of at least the latest release. Wheels
generated using any new compression format should be tagged as requiring
at least the first released version of CPython to support the new
compression format, regardless of the Python API compatibility of the
code within the wheel.

Backwards Compatibility

Backwards compatibility is an incredibly important issue for evolving
the wheel format. If adopting a new wheel revision is painful for
downstream users, package creators will hesitate to adopt the new
standards, and users will be stuck with failed CI pipelines and other
installation woes.

Several choices in the above specification are made so that the adoption
of a new feature is less painful. For example, today wheels of an
incompatible major version are still selected by pip as installation
candidates, which causes installer failures if a project starts
publishing 2.0 wheels. To avoid this issue, this PEP requires resolvers
to filter out wheels with major versions or features incompatible with
the installer.

This PEP also defines constraints on future wheel revisions, with the
goal of maintaining compatibility with CPython, but allowing evolution
of wheel contents. Wheel revisions shouldn't cause package installations
to break on older CPython revisions, as not only would it be
frustrating, it would be incredibly hard to debug for users.

This PEP relies on resolvers being able to efficiently acquire package
metadata, usually through PEP 658. This might present a problem for
users of package indices that do not serve PEP 658 metadata. However,
today most installers fall back on using HTTP range requests to
efficiently acquire only the part of a wheel needed to read the
metadata, a feature most storage providers and servers include.
Furthermore, future improvements to wheels such as compression will make
up performance losses due to inspecting files in the wheel.

The main compatibility limitation of this PEP is for projects that start
publishing solely new wheels alongside a source distribution. If a user
on an older installer tries to install the package, it will fall back to
the source distribution, because the resolver will skip all newer
wheels. Users are often poorly set up to build projects from source, so
this could lead to some failed builds users would not see otherwise.
There are several approaches to resolving this issue, such as allowing
dual-publishing for the initial migration, or marking source
distributions as not intended to be built.

Rejected Ideas

The Wheel Format is Perfect and Does not Need to be Changed

The wheel format has been around for over 10 years, and in that time,
Python packages have changed a lot. It is much more common for packages
to include Rust or C extension modules, increasing the size of packages.
Better compression, such as lzma or zstd, could save a lot of time and
bandwidth for PyPI and its users. Compatibility tags cannot express the
wide variety of hardware used to accelerate Python code today, nor
encode shared library compatibility information. In order to address
these issues, evolution of the wheel package format is necessary.

Wheel Format Changes Should be Tied to CPython Releases

I do not believe that tying wheel revisions to CPython releases is
beneficial. The main benefit of doing so is to make adoption of new
wheels predictable - users with the latest CPython get the latest
package format! This choice has several issues however. First, tying the
new format to the latest CPython makes adoption much slower. Users on
LTS versions of Linux with older Python installations are free to update
their pip in a virtual environment, but cannot update the version of
Python as easily. While some changes to the wheel format must be tied to
CPython changes necessarily, such as adding new compression formats or
changing the metadata format, many changes do not need to be tied to the
Python version, such as symlinks, enhanced compatibility tags, and new
formats that use existing compression formats in the standard library.
Additionally, wheels are used across multiple different language
implementations, which lag behind the CPython version. It seems unfair
to prevent their users from using a feature due to the Python version.
Lastly, while this PEP does not suggest tying the wheel version to
CPython releases, a future PEP may still do so at any time, so this
choice does not need to be made in this PEP.

Keep Using .whl as the File Extension

While keeping the extension .whl is appealing for many reasons, it
presents several problems that are difficult to surmount. First, current
installers would still pick a new wheel and fail to install the package.
Furthermore, the file name of a wheel would not be able to change
without breaking existing installers that expect a set wheel file name
format. While the current filename specification for wheels is
sufficient for current usage, the optional build tag in the middle of
the file name makes any extensions ambiguous (i.e.
foo-0.3-py3-none-any-fancy_new_tag.whl would parse as the build tag
being py3). This limits changes to information stored in the wheel file
name.

Store the Wheel Major Version in the File Extension (.whl2)

Storing the wheel major version in the file extension has several nice
advantages. For one, there is no need to introduce the Wheel-Version
metadata field, since installers could simply filter based on file
extension. This would also allow future side-by-side packages. However,
changing the extension for wheels each major version has some downsides.
First, the version stored in the WHEEL file must match the file
extension, and this would need to be verified by installers.
Additionally, many systems associate file type by file extension (e.g.
executable associations, various web caching software), and these would
need to be updated every version that is released. Furthermore, part of
the brittleness of the current wheel specification is that so much
metadata is stored in the filename. Filenames are not well suited to
store structured data. Moving away from encoding information in the
filename should be a goal of future wheel revisions.

Another possibility is to use the file extension to encode the outer
container format (i.e. a ZIP file containing .dist-info) separate from
the inner wheel version. However, this could lead to confusion if the
file extension and inner Wheel-Version diverge. If an installer raises
an error due to an incompatible wheel 3.0 as obtained from the wheel
metadata, some users will be confused by the difference from the file
extension .whl2.

Wheel 2.0 Should Change the Outer Container Format

Since wheel 2.0 will change the extension of wheel files, it is the best
opportunity to modify the outer container format. Compatibility does not
need to be kept with a different file extension that tools will need to
opt-in to reading. The main use-case for a different exterior
compression format would be better compression. For example, the outer
container could be changed into a Zstandard tarfile, .tar.zst, which
would decompress faster and produce smaller wheels. However, there are
several practical issues with this. First, Zstandard is not part of the
Python standard library, so pure-Python packaging tools would need to
ship an extension to unpack these wheels. This could cause some
compatibility issues for several platforms where extension modules are
not easy to install. Furthermore, a future wheel revision could always
introduce a new layout of non-metadata files that uses a .tar.zst inside
the existing ZIP-based format.

Finally, it is not a good idea to change the wheel file format too much
at once. The goal of this PEP is to make evolving the specification
easier, and part of the rationale behind making wheel evolution easier
is to avoid "all at once" changes. Changing the outer file format for
wheels would require re-writing how package metadata is not only
discovered, but also installed.

Why not Specify Wheel 2.0 In This PEP?

There are many features that could be included as part of wheel 2.0, but
this PEP does not cover them. The goal of this PEP is to define a
compatibility story for the wheel file format. Changes that do not
pertain to compatibility for wheel versions do not need to be in this
PEP, and should be introducted in follow-up PEPs defining new wheel
features.

Discussion Topics

Should Indices Support Dual-publishing for the First Migration?

Since .whl and .whlx will look different in file name, they could be
uploaded side-by-side to package indices like PyPI. This has some nice
benefits, like dual-support for older and newer installers, so users who
can get the latest features, while users who don't upgrade still can
install the latest version of a package.

There are many complications however. Should we allow wheel 2 uploads to
existing wheel 1-only releases? Should we put any requirements on the
side-by-side wheels, such as:

Constraints on dual-published wheels

A given index may contain identical-content wheels with different wheel
versions, and installers should prefer the newest-available wheel
format, with all other factors held constant.

Should we only allow uploading both with PEP 694 allowing "atomic"
dual-publishing?

Acknowledgements

The author of this PEP is greatly indebted to the incredibly valuable
review, advice, and feedback of Barry Warsaw and Michael Sarahan.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 456 Title: Secure and interchangeable hash algorithm Version:
$Revision$ Last-Modified: $Date$ Author: Christian Heimes
<EMAIL_ADDRESS>BDFL-Delegate: Alyssa Coghlan Status: Final Type:
Standards Track Content-Type: text/x-rst Created: 27-Sep-2013
Python-Version: 3.4 Post-History: 06-Oct-2013, 14-Nov-2013, 20-Nov-2013
Resolution:
https://mail.python.org/pipermail/python-dev/2013-November/130400.html

Abstract

This PEP proposes SipHash as default string and bytes hash algorithm to
properly fix hash randomization once and for all. It also proposes
modifications to Python's C code in order to unify the hash code and to
make it easily interchangeable.

Rationale

Despite the last attempt [issue13703] CPython is still vulnerable to
hash collision DoS attacks [29c3] [issue14621]. The current hash
algorithm and its randomization is not resilient against attacks. Only a
proper cryptographic hash function prevents the extraction of secret
randomization keys. Although no practical attack against a Python-based
service has been seen yet, the weakness has to be fixed. Jean-Philippe
Aumasson and Daniel J. Bernstein have already shown how the seed for the
current implementation can be recovered [poc].

Furthermore, the current hash algorithm is hard-coded and implemented
multiple times for bytes and three different Unicode representations
UCS1, UCS2 and UCS4. This makes it impossible for embedders to replace
it with a different implementation without patching and recompiling
large parts of the interpreter. Embedders may want to choose a more
suitable hash function.

Finally the current implementation code does not perform well. In the
common case it only processes one or two bytes per cycle. On a modern
64-bit processor the code can easily be adjusted to deal with eight
bytes at once.

This PEP proposes three major changes to the hash code for strings and
bytes:

-   SipHash [sip] is introduced as default hash algorithm. It is fast
    and small despite its cryptographic properties. Due to the fact that
    it was designed by well known security and crypto experts, it is
    safe to assume that its secure for the near future.
-   The existing FNV code is kept for platforms without a 64-bit data
    type. The algorithm is optimized to process larger chunks per cycle.
-   Calculation of the hash of strings and bytes is moved into a single
    API function instead of multiple specialized implementations in
    Objects/object.c and Objects/unicodeobject.c. The function takes a
    void pointer plus length and returns the hash for it.
-   The algorithm can be selected at compile time. FNV is guaranteed to
    exist on all platforms. SipHash is available on the majority of
    modern systems.

Requirements for a hash function

-   It MUST be able to hash arbitrarily large blocks of memory from 1
    byte up to the maximum ssize_t value.
-   It MUST produce at least 32 bits on 32-bit platforms and at least 64
    bits on 64-bit platforms. (Note: Larger outputs can be compressed
    with e.g. v ^ (v >> 32).)
-   It MUST support hashing of unaligned memory in order to support
    hash(memoryview).
-   It is highly RECOMMENDED that the length of the input influences the
    outcome, so that hash(b'\00') != hash(b'\x00\x00').

The internal interface code between the hash function and the tp_hash
slots implements special cases for zero length input and a return value
of -1. An input of length 0 is mapped to hash value 0. The output -1 is
mapped to -2.

Current implementation with modified FNV

CPython currently uses a variant of the Fowler-Noll-Vo hash function
[fnv]. The variant is has been modified to reduce the amount and cost of
hash collisions for common strings. The first character of the string is
added twice, the first time with a bit shift of 7. The length of the
input string is XOR-ed to the final value. Both deviations from the
original FNV algorithm reduce the amount of hash collisions for short
strings.

Recently [issue13703] a random prefix and suffix were added as an
attempt to randomize the hash values. In order to protect the hash
secret the code still returns 0 for zero length input.

C code:

    Py_uhash_t x;
    Py_ssize_t len;
    /* p is either 1, 2 or 4 byte type */
    unsigned char *p;
    Py_UCS2 *p;
    Py_UCS4 *p;

    if (len == 0)
        return 0;
    x = (Py_uhash_t) _Py_HashSecret.prefix;
    x ^= (Py_uhash_t) *p << 7;
    for (i = 0; i < len; i++)
        x = (1000003 * x) ^ (Py_uhash_t) *p++;
    x ^= (Py_uhash_t) len;
    x ^= (Py_uhash_t) _Py_HashSecret.suffix;
    return x;

Which roughly translates to Python:

    def fnv(p):
        if len(p) == 0:
            return 0

        # bit mask, 2**32-1 or 2**64-1
        mask = 2 * sys.maxsize + 1

        x = hashsecret.prefix
        x = (x ^ (ord(p[0]) << 7)) & mask
        for c in p:
            x = ((1000003 * x) ^ ord(c)) & mask
        x = (x ^ len(p)) & mask
        x = (x ^ hashsecret.suffix) & mask

        if x == -1:
            x = -2

        return x

FNV is a simple multiply and XOR algorithm with no cryptographic
properties. The randomization was not part of the initial hash code, but
was added as counter measure against hash collision attacks as explained
in oCERT-2011-003 [ocert]. Because FNV is not a cryptographic hash
algorithm and the dict implementation is not fortified against side
channel analysis, the randomization secrets can be calculated by a
remote attacker. The author of this PEP strongly believes that the
nature of a non-cryptographic hash function makes it impossible to
conceal the secrets.

Examined hashing algorithms

The author of this PEP has researched several hashing algorithms that
are considered modern, fast and state-of-the-art.

SipHash

SipHash [sip] is a cryptographic pseudo random function with a 128-bit
seed and 64-bit output. It was designed by Jean-Philippe Aumasson and
Daniel J. Bernstein as a fast and secure keyed hash algorithm. It's used
by Ruby, Perl, OpenDNS, Rust, Redis, FreeBSD and more. The C reference
implementation has been released under CC0 license (public domain).

Quote from SipHash's site:

  SipHash is a family of pseudorandom functions (a.k.a. keyed hash
  functions) optimized for speed on short messages. Target applications
  include network traffic authentication and defense against
  hash-flooding DoS attacks.

siphash24 is the recommend variant with best performance. It uses 2
rounds per message block and 4 finalization rounds. Besides the
reference implementation several other implementations are available.
Some are single-shot functions, others use a Merkle–Damgård
construction-like approach with init, update and finalize functions.
Marek Majkowski C implementation csiphash [csiphash] defines the
prototype of the function. (Note: k is split up into two uint64_t):

    uint64_t siphash24(const void *src, unsigned long src_sz, const char k[16])

SipHash requires a 64-bit data type and is not compatible with pure C89
platforms.

MurmurHash

MurmurHash [murmur] is a family of non-cryptographic keyed hash function
developed by Austin Appleby. Murmur3 is the latest and fast variant of
MurmurHash. The C++ reference implementation has been released into
public domain. It features 32- or 128-bit output with a 32-bit seed.
(Note: The out parameter is a buffer with either 1 or 4 bytes.)

Murmur3's function prototypes are:

    void MurmurHash3_x86_32(const void *key, int len, uint32_t seed, void *out)

    void MurmurHash3_x86_128(const void *key, int len, uint32_t seed, void *out)

    void MurmurHash3_x64_128(const void *key, int len, uint32_t seed, void *out)

The 128-bit variants requires a 64-bit data type and are not compatible
with pure C89 platforms. The 32-bit variant is fully C89-compatible.

Aumasson, Bernstein and Boßlet have shown [sip] [ocert-2012-001] that
Murmur3 is not resilient against hash collision attacks. Therefore,
Murmur3 can no longer be considered as secure algorithm. It still may be
an alternative if hash collision attacks are of no concern.

CityHash

CityHash [city] is a family of non-cryptographic hash function developed
by Geoff Pike and Jyrki Alakuijala for Google. The C++ reference
implementation has been released under MIT license. The algorithm is
partly based on MurmurHash and claims to be faster. It supports 64- and
128-bit output with a 128-bit seed as well as 32-bit output without
seed.

The relevant function prototype for 64-bit CityHash with 128-bit seed
is:

    uint64 CityHash64WithSeeds(const char *buf, size_t len, uint64 seed0,
                               uint64 seed1)

CityHash also offers SSE 4.2 optimizations with CRC32 intrinsic for long
inputs. All variants except CityHash32 require 64-bit data types.
CityHash32 uses only 32-bit data types but it doesn't support seeding.

Like MurmurHash Aumasson, Bernstein and Boßlet have shown [sip] a
similar weakness in CityHash.

DJBX33A

DJBX33A is a very simple multiplication and addition algorithm by Daniel
J. Bernstein. It is fast and has low setup costs but it's not secure
against hash collision attacks. Its properties make it a viable choice
for small string hashing optimization.

Other

Crypto algorithms such as HMAC, MD5, SHA-1 or SHA-2 are too slow and
have high setup and finalization costs. For these reasons they are not
considered fit for this purpose. Modern AMD and Intel CPUs have AES-NI
(AES instruction set) [aes-ni] to speed up AES encryption. CMAC with
AES-NI might be a viable option but it's probably too slow for daily
operation. (testing required)

Conclusion

SipHash provides the best combination of speed and security. Developers
of other prominent projects have came to the same conclusion.

Small string optimization

Hash functions like SipHash24 have a costly initialization and
finalization code that can dominate speed of the algorithm for very
short strings. On the other hand, Python calculates the hash value of
short strings quite often. A simple and fast function for especially for
hashing of small strings can make a measurable impact on performance.
For example, these measurements were taken during a run of Python's
regression tests. Additional measurements of other code have shown a
similar distribution.

+-------+--------------+---------+
| bytes | hash() calls | portion |
+=======+==============+=========+
| 1     |   18709      |   0.2%  |
+-------+--------------+---------+
| 2     |   737480     |   9.5%  |
+-------+--------------+---------+
| 3     |   636178     |   17.6% |
+-------+--------------+---------+
| 4     |   1518313    |   36.7% |
+-------+--------------+---------+
| 5     |   643022     |   44.9% |
+-------+--------------+---------+
| 6     |   770478     |   54.6% |
+-------+--------------+---------+
| 7     |   525150     |   61.2% |
+-------+--------------+---------+
| 8     |   304873     |   65.1% |
+-------+--------------+---------+
| 9     |   297272     |   68.8% |
+-------+--------------+---------+
| 10    |   68191      |   69.7% |
+-------+--------------+---------+
| 11    |   1388484    |   87.2% |
+-------+--------------+---------+
| 12    |   480786     |   93.3% |
+-------+--------------+---------+
| 13    |   52730      |   93.9% |
+-------+--------------+---------+
| 14    |   65309      |   94.8% |
+-------+--------------+---------+
| 15    |   44245      |   95.3% |
+-------+--------------+---------+
| 16    |   85643      |   96.4% |
+-------+--------------+---------+
| Total |   7921678    |         |
+-------+--------------+---------+

However a fast function like DJBX33A is not as secure as SipHash24. A
cutoff at about 5 to 7 bytes should provide a decent safety margin and
speed up at the same time. The PEP's reference implementation provides
such a cutoff with Py_HASH_CUTOFF. The optimization is disabled by
default for several reasons. For one the security implications are
unclear yet and should be thoroughly studied before the optimization is
enabled by default. Secondly the performance benefits vary. On 64 bit
Linux system with Intel Core i7 multiple runs of Python's benchmark
suite [pybench] show an average speedups between 3% and 5% for
benchmarks such as django_v2, mako and etree with a cutoff of 7.
Benchmarks with X86 binaries and Windows X86_64 builds on the same
machine are a bit slower with small string optimization.

The state of small string optimization will be assessed during the beta
phase of Python 3.4. The feature will either be enabled with appropriate
values or the code will be removed before beta 2 is released.

C API additions

All C API extension modifications are not part of the stable API.

hash secret

The _Py_HashSecret_t type of Python 2.6 to 3.3 has two members with
either 32- or 64-bit length each. SipHash requires two 64-bit unsigned
integers as keys. The typedef will be changed to a union with a
guaranteed size of 24 bytes on all architectures. The union provides a
128 bit random key for SipHash24 and FNV as well as an additional value
of 64 bit for the optional small string optimization and pyexpat seed.
The additional 64 bit seed ensures that pyexpat or small string
optimization cannot reveal bits of the SipHash24 seed.

memory layout on 64 bit systems:

    cccccccc cccccccc cccccccc  uc -- unsigned char[24]
    pppppppp ssssssss ........  fnv -- two Py_hash_t
    k0k0k0k0 k1k1k1k1 ........  siphash -- two PY_UINT64_T
    ........ ........ ssssssss  djbx33a -- 16 bytes padding + one Py_hash_t
    ........ ........ eeeeeeee  pyexpat XML hash salt

memory layout on 32 bit systems:

    cccccccc cccccccc cccccccc  uc -- unsigned char[24]
    ppppssss ........ ........  fnv -- two Py_hash_t
    k0k0k0k0 k1k1k1k1 ........  siphash -- two PY_UINT64_T (if available)
    ........ ........ ssss....  djbx33a -- 16 bytes padding + one Py_hash_t
    ........ ........ eeee....  pyexpat XML hash salt

new type definition:

    typedef union {
        /* ensure 24 bytes */
        unsigned char uc[24];
        /* two Py_hash_t for FNV */
        struct {
            Py_hash_t prefix;
            Py_hash_t suffix;
        } fnv;
    #ifdef PY_UINT64_T
        /* two uint64 for SipHash24 */
        struct {
            PY_UINT64_T k0;
            PY_UINT64_T k1;
        } siphash;
    #endif
        /* a different (!) Py_hash_t for small string optimization */
        struct {
            unsigned char padding[16];
            Py_hash_t suffix;
        } djbx33a;
        struct {
            unsigned char padding[16];
            Py_hash_t hashsalt;
        } expat;
    } _Py_HashSecret_t;
    PyAPI_DATA(_Py_HashSecret_t) _Py_HashSecret;

_Py_HashSecret_t is initialized in Python/random.c:_PyRandom_Init()
exactly once at startup.

hash function definition

Implementation:

    typedef struct {
        /* function pointer to hash function, e.g. fnv or siphash24 */
        Py_hash_t (*const hash)(const void *, Py_ssize_t);
        const char *name;       /* name of the hash algorithm and variant */
        const int hash_bits;    /* internal size of hash value */
        const int seed_bits;    /* size of seed input */
    } PyHash_FuncDef;

    PyAPI_FUNC(PyHash_FuncDef*) PyHash_GetFuncDef(void);

autoconf

A new test is added to the configure script. The test sets
HAVE_ALIGNED_REQUIRED, when it detects a platform, that requires aligned
memory access for integers. Must current platforms such as X86, X86_64
and modern ARM don't need aligned data.

A new option --with-hash-algorithm enables the user to select a hash
algorithm in the configure step.

hash function selection

The value of the macro Py_HASH_ALGORITHM defines which hash algorithm is
used internally. It may be set to any of the three values
Py_HASH_SIPHASH24, Py_HASH_FNV or Py_HASH_EXTERNAL. If Py_HASH_ALGORITHM
is not defined at all, then the best available algorithm is selected. On
platforms which don't require aligned memory access
(HAVE_ALIGNED_REQUIRED not defined) and an unsigned 64 bit integer type
PY_UINT64_T, SipHash24 is used. On strict C89 platforms without a 64 bit
data type, or architectures such as SPARC, FNV is selected as fallback.
A hash algorithm can be selected with an autoconf option, for example
./configure --with-hash-algorithm=fnv.

The value Py_HASH_EXTERNAL allows 3rd parties to provide their own
implementation at compile time.

Implementation:

    #if Py_HASH_ALGORITHM == Py_HASH_EXTERNAL
    extern PyHash_FuncDef PyHash_Func;
    #elif Py_HASH_ALGORITHM == Py_HASH_SIPHASH24
    static PyHash_FuncDef PyHash_Func = {siphash24, "siphash24", 64, 128};
    #elif Py_HASH_ALGORITHM == Py_HASH_FNV
    static PyHash_FuncDef PyHash_Func = {fnv, "fnv", 8 * sizeof(Py_hash_t),
                                         16 * sizeof(Py_hash_t)};
    #endif

Python API addition

sys module

The sys module already has a hash_info struct sequence. More fields are
added to the object to reflect the active hash algorithm and its
properties.

    sys.hash_info(width=64,
                  modulus=2305843009213693951,
                  inf=314159,
                  nan=0,
                  imag=1000003,
                  # new fields:
                  algorithm='siphash24',
                  hash_bits=64,
                  seed_bits=128,
                  cutoff=0)

Necessary modifications to C code

_Py_HashBytes() (Objects/object.c)

_Py_HashBytes is an internal helper function that provides the hashing
code for bytes, memoryview and datetime classes. It currently implements
FNV for unsigned char *.

The function is moved to Python/pyhash.c and modified to use the hash
function through PyHash_Func.hash(). The function signature is altered
to take a const void * as first argument. _Py_HashBytes also takes care
of special cases: it maps zero length input to 0 and return value of -1
to -2.

bytes_hash() (Objects/bytesobject.c)

bytes_hash uses _Py_HashBytes to provide the tp_hash slot function for
bytes objects. The function will continue to use _Py_HashBytes but
without a type cast.

memory_hash() (Objects/memoryobject.c)

memory_hash provides the tp_hash slot function for read-only memory
views if the original object is hashable, too. It's the only function
that has to support hashing of unaligned memory segments in the future.
The function will continue to use _Py_HashBytes but without a type cast.

unicode_hash() (Objects/unicodeobject.c)

unicode_hash provides the tp_hash slot function for unicode. Right now
it implements the FNV algorithm three times for unsigned char*, Py_UCS2
and Py_UCS4. A reimplementation of the function must take care to use
the correct length. Since the macro PyUnicode_GET_LENGTH returns the
length of the unicode string and not its size in octets, the length must
be multiplied with the size of the internal unicode kind:

    if (PyUnicode_READY(u) == -1)
        return -1;
    x = _Py_HashBytes(PyUnicode_DATA(u),
                      PyUnicode_GET_LENGTH(u) * PyUnicode_KIND(u));

generic_hash() (Modules/_datetimemodule.c)

generic_hash acts as a wrapper around _Py_HashBytes for the tp_hash
slots of date, time and datetime types. timedelta objects are hashed by
their state (days, seconds, microseconds) and tzinfo objects are not
hashable. The data members of date, time and datetime types' struct are
not void* aligned. This can easily by fixed with memcpy()ing four to ten
bytes to an aligned buffer.

Performance

In general the PEP 456 code with SipHash24 is about as fast as the old
code with FNV. SipHash24 seems to make better use of modern compilers,
CPUs and large L1 cache. Several benchmarks show a small speed
improvement on 64 bit CPUs such as Intel Core i5 and Intel Core i7
processes. 32 bit builds and benchmarks on older CPUs such as an AMD
Athlon X2 are slightly slower with SipHash24. The performance increase
or decrease are so small that they should not affect any application
code.

The benchmarks were conducted on CPython default branch revision
b08868fd5994 and the PEP repository [pep-456-repos]. All upstream
changes were merged into the pep-456 branch. The "performance" CPU
governor was configured and almost all programs were stopped so the
benchmarks were able to utilize TurboBoost and the CPU caches as much as
possible. The raw benchmark results of multiple machines and platforms
are made available at [benchmarks].

Hash value distribution

A good distribution of hash values is important for dict and set
performance. Both SipHash24 and FNV take the length of the input into
account, so that strings made up entirely of NULL bytes don't have the
same hash value. The last bytes of the input tend to affect the least
significant bits of the hash value, too. That attribute reduces the
amount of hash collisions for strings with a common prefix.

Typical length

Serhiy Storchaka has shown in [issue16427] that a modified FNV
implementation with 64 bits per cycle is able to process long strings
several times faster than the current FNV implementation.

However, according to statistics [issue19183] a typical Python program
as well as the Python test suite have a hash ratio of about 50% small
strings between 1 and 6 bytes. Only 5% of the strings are larger than 16
bytes.

Grand Unified Python Benchmark Suite

Initial tests with an experimental implementation and the Grand Unified
Python Benchmark Suite have shown minimal deviations. The summarized
total runtime of the benchmark is within 1% of the runtime of an
unmodified Python 3.4 binary. The tests were run on an Intel i7-2860QM
machine with a 64-bit Linux installation. The interpreter was compiled
with GCC 4.7 for 64- and 32-bit.

More benchmarks will be conducted.

Backwards Compatibility

The modifications don't alter any existing API.

The output of hash() for strings and bytes are going to be different.
The hash values for ASCII Unicode and ASCII bytes will stay equal.

Alternative counter measures against hash collision DoS

Three alternative countermeasures against hash collisions were discussed
in the past, but are not subject of this PEP.

1.  Marc-Andre Lemburg has suggested that dicts shall count hash
    collisions. In case an insert operation causes too many collisions
    an exception shall be raised.
2.  Some applications (e.g. PHP) limit the amount of keys for GET and
    POST HTTP requests. The approach effectively leverages the impact of
    a hash collision attack. (XXX citation needed)
3.  Hash maps have a worst case of O(n) for insertion and lookup of
    keys. This results in a quadratic runtime during a hash collision
    attack. The introduction of a new and additional data structure with
    O(log n) worst case behavior would eliminate the root cause. A data
    structures like red-black-tree or prefix trees (trie [trie]) would
    have other benefits, too. Prefix trees with stringed keyed can
    reduce memory usage as common prefixes are stored within the tree
    structure.

Discussion

Pluggable

The first draft of this PEP made the hash algorithm pluggable at
runtime. It supported multiple hash algorithms in one binary to give the
user the possibility to select a hash algorithm at startup. The approach
was considered an unnecessary complication by several core committers
[pluggable]. Subsequent versions of the PEP aim for compile time
configuration.

Non-aligned memory access

The implementation of SipHash24 were criticized because it ignores the
issue of non-aligned memory and therefore doesn't work on architectures
that requires alignment of integer types. The PEP deliberately neglects
this special case and doesn't support SipHash24 on such platforms. It's
simply not considered worth the trouble until proven otherwise. All
major platforms like X86, X86_64 and ARMv6+ can handle unaligned memory
with minimal or even no speed impact. [alignmentmyth]

Almost every block is properly aligned anyway. At present bytes' and
str's data are always aligned. Only memoryviews can point to unaligned
blocks under rare circumstances. The PEP implementation is optimized and
simplified for the common case.

ASCII str / bytes hash collision

Since the implementation of PEP 393, bytes and ASCII text have the same
memory layout. Because of this the new hashing API will keep the
invariant:

    hash("ascii string") == hash(b"ascii string")

for ASCII string and ASCII bytes. Equal hash values result in a hash
collision and therefore cause a minor speed penalty for dicts and sets
with mixed keys. The cause of the collision could be removed by e.g.
subtracting 2 from the hash value of bytes. -2 because hash(b"") == 0
and -1 is reserved. The PEP doesn't change the hash value.

References

-   Issue 19183 [issue19183] contains a reference implementation.

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

29c3

    http://events.ccc.de/congress/2012/Fahrplan/events/5152.en.html

aes-ni

    http://en.wikipedia.org/wiki/AES_instruction_set

alignmentmyth

    http://lemire.me/blog/archives/2012/05/31/data-alignment-for-speed-myth-or-reality/

benchmarks

    https://bitbucket.org/tiran/pep-456-benchmarks/src

city

    http://code.google.com/p/cityhash/

csiphash

    https://github.com/majek/csiphash/

fnv

    http://en.wikipedia.org/wiki/Fowler-Noll-Vo_hash_function

issue13703

    http://bugs.python.org/issue13703

issue14621

    http://bugs.python.org/issue14621

issue16427

    http://bugs.python.org/issue16427

issue19183

    http://bugs.python.org/issue19183

murmur

    http://code.google.com/p/smhasher/

ocert

    http://www.nruns.com/_downloads/advisory28122011.pdf

ocert-2012-001

    http://www.ocert.org/advisories/ocert-2012-001.html

pep-456-repos

    http://hg.python.org/features/pep-456

pluggable

    https://mail.python.org/pipermail/python-dev/2013-October/129138.html

poc

    https://131002.net/siphash/poc.py

pybench

    http://hg.python.org/benchmarks/

sip

    https://131002.net/siphash/

trie

    http://en.wikipedia.org/wiki/Trie


text:
PEP: 488 Title: Elimination of PYO files Version: $Revision$
Last-Modified: $Date$ Author: Brett Cannon<EMAIL_ADDRESS>Status:
Final Type: Standards Track Content-Type: text/x-rst Created:
20-Feb-2015 Python-Version: 3.5 Post-History: 06-Mar-2015, 13-Mar-2015,
20-Mar-2015

Abstract

This PEP proposes eliminating the concept of PYO files from Python. To
continue the support of the separation of bytecode files based on their
optimization level, this PEP proposes extending the PYC file name to
include the optimization level in the bytecode repository directory when
there are optimizations applied.

Rationale

As of today, bytecode files come in two flavours: PYC and PYO. A PYC
file is the bytecode file generated and read from when no optimization
level is specified at interpreter startup (i.e., -O is not specified). A
PYO file represents the bytecode file that is read/written when any
optimization level is specified (i.e., when -O or -OO is specified).
This means that while PYC files clearly delineate the optimization level
used when they were generated -- namely no optimizations beyond the
peepholer -- the same is not true for PYO files. To put this in terms of
optimization levels and the file extension:

-   0: .pyc
-   1 (-O): .pyo
-   2 (-OO): .pyo

The reuse of the .pyo file extension for both level 1 and 2
optimizations means that there is no clear way to tell what optimization
level was used to generate the bytecode file. In terms of reading PYO
files, this can lead to an interpreter using a mixture of optimization
levels with its code if the user was not careful to make sure all PYO
files were generated using the same optimization level (typically done
by blindly deleting all PYO files and then using the compileall module
to compile all-new PYO files[1]). This issue is only compounded when
people optimize Python code beyond what the interpreter natively
supports, e.g., using the astoptimizer project[2].

In terms of writing PYO files, the need to delete all PYO files every
time one either changes the optimization level they want to use or are
unsure of what optimization was used the last time PYO files were
generated leads to unnecessary file churn. The change proposed by this
PEP also allows for all optimization levels to be pre-compiled for
bytecode files ahead of time, something that is currently impossible
thanks to the reuse of the .pyo file extension for multiple optimization
levels.

As for distributing bytecode-only modules, having to distribute both
.pyc and .pyo files is unnecessary for the common use-case of code
obfuscation and smaller file deployments. This means that bytecode-only
modules will only load from their non-optimized .pyc file name.

Proposal

To eliminate the ambiguity that PYO files present, this PEP proposes
eliminating the concept of PYO files and their accompanying .pyo file
extension. To allow for the optimization level to be unambiguous as well
as to avoid having to regenerate optimized bytecode files needlessly in
the __pycache__ directory, the optimization level used to generate the
bytecode file will be incorporated into the bytecode file name. When no
optimization level is specified, the pre-PEP .pyc file name will be used
(i.e., no optimization level will be specified in the file name). For
example, a source file named foo.py in CPython 3.5 could have the
following bytecode files based on the interpreter's optimization level
(none, -O, and -OO):

-   0: foo.cpython-35.pyc (i.e., no change)
-   1: foo.cpython-35.opt-1.pyc
-   2: foo.cpython-35.opt-2.pyc

Currently bytecode file names are created by
importlib.util.cache_from_source(), approximately using the following
expression defined by PEP 3147[3],[4]:

    '{name}.{cache_tag}.pyc'.format(name=module_name,
                                    cache_tag=sys.implementation.cache_tag)

This PEP proposes to change the expression when an optimization level is
specified to:

    '{name}.{cache_tag}.opt-{optimization}.pyc'.format(
            name=module_name,
            cache_tag=sys.implementation.cache_tag,
            optimization=str(sys.flags.optimize))

The "opt-" prefix was chosen so as to provide a visual separator from
the cache tag. The placement of the optimization level after the cache
tag was chosen to preserve lexicographic sort order of bytecode file
names based on module name and cache tag which will not vary for a
single interpreter. The "opt-" prefix was chosen over "o" so as to be
somewhat self-documenting. The "opt-" prefix was chosen over "O" so as
to not have any confusion in case "0" was the leading prefix of the
optimization level.

A period was chosen over a hyphen as a separator so as to distinguish
clearly that the optimization level is not part of the interpreter
version as specified by the cache tag. It also lends to the use of the
period in the file name to delineate semantically different concepts.

For example, if -OO had been passed to the interpreter then instead of
importlib.cpython-35.pyo the file name would be
importlib.cpython-35.opt-2.pyc.

Leaving out the new opt- tag when no optimization level is applied
should increase backwards-compatibility. This is also more understanding
of Python implementations which have no use for optimization levels
(e.g., PyPy[5]).

It should be noted that this change in no way affects the performance of
import. Since the import system looks for a single bytecode file based
on the optimization level of the interpreter already and generates a new
bytecode file if it doesn't exist, the introduction of potentially more
bytecode files in the __pycache__ directory has no effect in terms of
stat calls. The interpreter will continue to look for only a single
bytecode file based on the optimization level and thus no increase in
stat calls will occur.

The only potentially negative result of this PEP is the probable
increase in the number of .pyc files and thus increase in storage use.
But for platforms where this is an issue, sys.dont_write_bytecode exists
to turn off bytecode generation so that it can be controlled offline.

Implementation

An implementation of this PEP is available[6].

importlib

As importlib.util.cache_from_source() is the API that exposes bytecode
file paths as well as being directly used by importlib, it requires the
most critical change. As of Python 3.4, the function's signature is:

    importlib.util.cache_from_source(path, debug_override=None)

This PEP proposes changing the signature in Python 3.5 to:

    importlib.util.cache_from_source(path, debug_override=None, *, optimization=None)

The introduced optimization keyword-only parameter will control what
optimization level is specified in the file name. If the argument is
None then the current optimization level of the interpreter will be
assumed (including no optimization). Any argument given for optimization
will be passed to str() and must have str.isalnum() be true, else
ValueError will be raised (this prevents invalid characters being used
in the file name). If the empty string is passed in for optimization
then the addition of the optimization will be suppressed, reverting to
the file name format which predates this PEP.

It is expected that beyond Python's own two optimization levels,
third-party code will use a hash of optimization names to specify the
optimization level, e.g.
hashlib.sha256(','.join(['no dead code', 'const folding'])).hexdigest().
While this might lead to long file names, it is assumed that most users
never look at the contents of the __pycache__ directory and so this
won't be an issue.

The debug_override parameter will be deprecated. A False value will be
equivalent to optimization=1 while a True value will represent
optimization='' (a None argument will continue to mean the same as for
optimization). A deprecation warning will be raised when debug_override
is given a value other than None, but there are no plans for the
complete removal of the parameter at this time (but removal will be no
later than Python 4).

The various module attributes for importlib.machinery which relate to
bytecode file suffixes will be updated[7]. The DEBUG_BYTECODE_SUFFIXES
and OPTIMIZED_BYTECODE_SUFFIXES will both be documented as deprecated
and set to the same value as BYTECODE_SUFFIXES (removal of
DEBUG_BYTECODE_SUFFIXES and OPTIMIZED_BYTECODE_SUFFIXES is not currently
planned, but will be not later than Python 4).

All various finders and loaders will also be updated as necessary, but
updating the previous mentioned parts of importlib should be all that is
required.

Rest of the standard library

The various functions exposed by the py_compile and compileall functions
will be updated as necessary to make sure they follow the new bytecode
file name semantics[8],[9]. The CLI for the compileall module will not
be directly affected (the -b flag will be implicit as it will no longer
generate .pyo files when -O is specified).

Compatibility Considerations

Any code directly manipulating bytecode files from Python 3.2 on will
need to consider the impact of this change on their code (prior to
Python 3.2 -- including all of Python 2 -- there was no __pycache__
which already necessitates bifurcating bytecode file handling support).
If code was setting the debug_override argument to
importlib.util.cache_from_source() then care will be needed if they want
the path to a bytecode file with an optimization level of 2. Otherwise
only code not using importlib.util.cache_from_source() will need
updating.

As for people who distribute bytecode-only modules (i.e., use a bytecode
file instead of a source file), they will have to choose which
optimization level they want their bytecode files to be since
distributing a .pyo file with a .pyc file will no longer be of any use.
Since people typically only distribute bytecode files for code
obfuscation purposes or smaller distribution size then only having to
distribute a single .pyc should actually be beneficial to these
use-cases. And since the magic number for bytecode files changed in
Python 3.5 to support PEP 465 there is no need to support pre-existing
.pyo files[10].

Rejected Ideas

Completely dropping optimization levels from CPython

Some have suggested that instead of accommodating the various
optimization levels in CPython, we should instead drop them entirely.
The argument is that significant performance gains would occur from
runtime optimizations through something like a JIT and not through
pre-execution bytecode optimizations.

This idea is rejected for this PEP as that ignores the fact that there
are people who do find the pre-existing optimization levels for CPython
useful. It also assumes that no other Python interpreter would find what
this PEP proposes useful.

Alternative formatting of the optimization level in the file name

Using the "opt-" prefix and placing the optimization level between the
cache tag and file extension is not critical. All options which have
been considered are:

-   importlib.cpython-35.opt-1.pyc
-   importlib.cpython-35.opt1.pyc
-   importlib.cpython-35.o1.pyc
-   importlib.cpython-35.O1.pyc
-   importlib.cpython-35.1.pyc
-   importlib.cpython-35-O1.pyc
-   importlib.O1.cpython-35.pyc
-   importlib.o1.cpython-35.pyc
-   importlib.1.cpython-35.pyc

These were initially rejected either because they would change the sort
order of bytecode files, possible ambiguity with the cache tag, or were
not self-documenting enough. An informal poll was taken and people
clearly preferred the formatting proposed by the PEP[11]. Since this
topic is non-technical and of personal choice, the issue is considered
solved.

Embedding the optimization level in the bytecode metadata

Some have suggested that rather than embedding the optimization level of
bytecode in the file name that it be included in the file's metadata
instead. This would mean every interpreter had a single copy of bytecode
at any time. Changing the optimization level would thus require
rewriting the bytecode, but there would also only be a single file to
care about.

This has been rejected due to the fact that Python is often installed as
a root-level application and thus modifying the bytecode file for
modules in the standard library are always possible. In this situation
integrators would need to guess at what a reasonable optimization level
was for users for any/all situations. By allowing multiple optimization
levels to co-exist simultaneously it frees integrators from having to
guess what users want and allows users to utilize the optimization level
they want.

References

Copyright

This document has been placed in the public domain.

[1] The compileall module
(https://docs.python.org/3.5/library/compileall.html)

[2] The astoptimizer project
(https://web.archive.org/web/20150909225454/https://pypi.python.org/pypi/astoptimizer)

[3] importlib.util.cache_from_source()
(https://docs.python.org/3.5/library/importlib.html#importlib.util.cache_from_source)

[4] Implementation of importlib.util.cache_from_source() from CPython
3.4.3rc1
(https://github.com/python/cpython/blob/e55181f517bbfc875065ce86ed3e05cf0e0246fa/Lib/importlib/_bootstrap.py#L437)

[5] The PyPy Project (https://www.pypy.org/)

[6] Implementation of PEP 488
(https://github.com/python/cpython/issues/67919)

[7] The importlib.machinery module
(https://docs.python.org/3.5/library/importlib.html#module-importlib.machinery)

[8] The py_compile module
(https://docs.python.org/3.5/library/compileall.html)

[9] The compileall module
(https://docs.python.org/3.5/library/compileall.html)

[10] importlib.util.MAGIC_NUMBER
(https://docs.python.org/3.5/library/importlib.html#importlib.util.MAGIC_NUMBER)

[11] Informal poll of file name format options on Google+
(https://web.archive.org/web/20160925163500/https://plus.google.com/+BrettCannon/posts/fZynLNwHWGm)