text:
PEP: 754 Title: IEEE 754 Floating Point Special Values Version:
$Revision$ Last-Modified: $Date$ Author: Gregory R. Warnes
<EMAIL_ADDRESS>Status: Rejected Type: Standards
Track Content-Type: text/x-rst Created: 28-Mar-2003 Python-Version: 2.3
Post-History:

Rejection Notice

This PEP has been rejected. After sitting open for four years, it has
failed to generate sufficient community interest.

Several ideas of this PEP were implemented for Python 2.6. float('inf')
and repr(float('inf')) are now guaranteed to work on every supported
platform with IEEE 754 semantics. However the eval(repr(float('inf')))
roundtrip is still not supported unless you define inf and nan yourself:

    >>> inf = float('inf')
    >>> inf, 1E400
    (inf, inf)
    >>> neginf = float('-inf')
    >>> neginf, -1E400
    (-inf, -inf)
    >>> nan = float('nan')
    >>> nan, inf * 0.
    (nan, nan)

The math and the sys module also have gained additional features,
sys.float_info, math.isinf, math.isnan, math.copysign.

Abstract

This PEP proposes an API and a provides a reference module that
generates and tests for IEEE 754 double-precision special values:
positive infinity, negative infinity, and not-a-number (NaN).

Rationale

The IEEE 754 standard defines a set of binary representations and
algorithmic rules for floating point arithmetic. Included in the
standard is a set of constants for representing special values,
including positive infinity, negative infinity, and indeterminate or
non-numeric results (NaN). Most modern CPUs implement the IEEE 754
standard, including the (Ultra)SPARC, PowerPC, and x86 processor series.

Currently, the handling of IEEE 754 special values in Python depends on
the underlying C library. Unfortunately, there is little consistency
between C libraries in how or whether these values are handled. For
instance, on some systems "float('Inf')" will properly return the IEEE
754 constant for positive infinity. On many systems, however, this
expression will instead generate an error message.

The output string representation for an IEEE 754 special value also
varies by platform. For example, the expression "float(1e3000)", which
is large enough to generate an overflow, should return a string
representation corresponding to IEEE 754 positive infinity. Python 2.1.3
on x86 Debian Linux returns "inf". On Sparc Solaris 8 with Python 2.2.1,
this same expression returns "Infinity", and on MS-Windows 2000 with
Active Python 2.2.1, it returns "1.#INF".

Adding to the confusion, some platforms generate one string on
conversion from floating point and accept a different string for
conversion to floating point. On these systems :

    float(str(x))

will generate an error when "x" is an IEEE special value.

In the past, some have recommended that programmers use expressions
like:

    PosInf = 1e300**2
    NaN = PosInf/PosInf

to obtain positive infinity and not-a-number constants. However, the
first expression generates an error on current Python interpreters. A
possible alternative is to use:

    PosInf = 1e300000
    NaN = PosInf/PosInf

While this does not generate an error with current Python interpreters,
it is still an ugly and potentially non-portable hack. In addition,
defining NaN in this way does solve the problem of detecting such
values. First, the IEEE 754 standard provides for an entire set of
constant values for Not-a-Number. Second, the standard requires that :

    NaN != X

for all possible values of X, including NaN. As a consequence :

    NaN == NaN

should always evaluate to false. However, this behavior also is not
consistently implemented. [e.g. Cygwin Python 2.2.2]

Due to the many platform and library inconsistencies in handling IEEE
special values, it is impossible to consistently set or detect IEEE 754
floating point values in normal Python code without resorting to
directly manipulating bit-patterns.

This PEP proposes a standard Python API and provides a reference module
implementation which allows for consistent handling of IEEE 754 special
values on all supported platforms.

API Definition

Constants

NaN

    Non-signalling IEEE 754 "Not a Number" value

PosInf

    IEEE 754 Positive Infinity value

NegInf

    IEEE 754 Negative Infinity value

Functions

isNaN(value)

    Determine if the argument is an IEEE 754 NaN (Not a Number) value.

isPosInf(value)

    Determine if the argument is an IEEE 754 positive infinity value.

isNegInf(value)

    Determine if the argument is an IEEE 754 negative infinity value.

isFinite(value)

    Determine if the argument is a finite IEEE 754 value (i.e., is not
    NaN, positive, or negative infinity).

isInf(value)

    Determine if the argument is an infinite IEEE 754 value (positive or
    negative infinity)

Example

(Run under Python 2.2.1 on Solaris 8.)

>>> import fpconst >>> val = 1e30000 # should be cause overflow and
result in "Inf" >>> val Infinity >>> fpconst.isInf(val) 1 >>>
fpconst.PosInf Infinity >>> nval = val/val # should result in NaN >>>
nval NaN >>> fpconst.isNaN(nval) 1 >>> fpconst.isNaN(val) 0

Implementation

The reference implementation is provided in the module "fpconst"[1],
which is written in pure Python by taking advantage of the "struct"
standard module to directly set or test for the bit patterns that define
IEEE 754 special values. Care has been taken to generate proper results
on both big-endian and little-endian machines. The current
implementation is pure Python, but some efficiency could be gained by
translating the core routines into C.

Patch 1151323 "New fpconst module"[2] on SourceForge adds the fpconst
module to the Python standard library.

References

See http://babbage.cs.qc.edu/courses/cs341/IEEE-754references.html for
reference material on the IEEE 754 floating point standard.

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 End:

[1] Further information on the reference package is available at
http://research.warnes.net/projects/rzope/fpconst/

[2] http://sourceforge.net/tracker/?func=detail&aid=1151323&group_id=5470&atid=305470


text:
PEP: 375 Title: Python 3.1 Release Schedule Version: $Revision$
Last-Modified: $Date$ Author: Benjamin Peterson<EMAIL_ADDRESS>Status: Final Type: Informational Topic: Release Content-Type:
text/x-rst Created: 08-Feb-2009 Python-Version: 3.1

Abstract

This document describes the development and release schedule for Python
3.1. The schedule primarily concerns itself with PEP-sized items. Small
features may be added up to and including the first beta release. Bugs
may be fixed until the final release.

Release Manager and Crew

  Position              Name
  --------------------- -------------------
  3.1 Release Manager   Benjamin Peterson
  Windows installers    Martin v. Loewis
  Mac installers        Ronald Oussoren

Release Schedule

-   3.1a1 March 7, 2009
-   3.1a2 April 4, 2009
-   3.1b1 May 6, 2009
-   3.1rc1 May 30, 2009
-   3.1rc2 June 13, 2009
-   3.1 final June 27, 2009

Maintenance Releases

3.1 is no longer maintained. 3.1 received security fixes until June
2012.

Previous maintenance releases are:

-   v3.1.1rc1 2009-08-13
-   v3.1.1 2009-08-16
-   v3.1.2rc1 2010-03-06
-   v3.1.2 2010-03-20
-   v3.1.3rc1 2010-11-13
-   v3.1.3 2010-11-27
-   v3.1.4rc1 2011-05-29
-   v3.1.4 2011-06-11
-   v3.1.5rc1 2012-02-23
-   v3.1.5rc2 2012-03-15
-   v3.1.5 2012-04-06

Features for 3.1

-   importlib
-   io in C
-   Update simplejson to the latest external version[1].
-   Ordered dictionary for collections (PEP 372).
-   auto-numbered replacement fields in str.format() strings[2]
-   Nested with-statements in one with statement

Footnotes

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] http://bugs.python.org/issue4136

[2] http://bugs.python.org/issue5237

license: Public Domain


text:
PEP: 395 Title: Qualified Names for Modules Version: $Revision$
Last-Modified: $Date$ Author: Alyssa Coghlan<EMAIL_ADDRESS>Status: Withdrawn Type: Standards Track Content-Type: text/x-rst
Created: 04-Mar-2011 Python-Version: 3.4 Post-History: 05-Mar-2011,
19-Nov-2011

PEP Withdrawal

This PEP was withdrawn by the author in December 2013, as other
significant changes in the time since it was written have rendered
several aspects obsolete. Most notably PEP 420 namespace packages
rendered some of the proposals related to package detection unworkable
and PEP 451 module specifications resolved the multiprocessing issues
and provide a possible means to tackle the pickle compatibility issues.

A future PEP to resolve the remaining issues would still be appropriate,
but it's worth starting any such effort as a fresh PEP restating the
remaining problems in an updated context rather than trying to build on
this one directly.

Abstract

This PEP proposes new mechanisms that eliminate some longstanding traps
for the unwary when dealing with Python's import system, as well as
serialisation and introspection of functions and classes.

It builds on the "Qualified Name" concept defined in PEP 3155.

Relationship with Other PEPs

Most significantly, this PEP is currently deferred as it requires
significant changes in order to be made compatible with the removal of
mandatory __init__.py files in PEP 420 (which has been implemented and
released in Python 3.3).

This PEP builds on the "qualified name" concept introduced by PEP 3155,
and also shares in that PEP's aim of fixing some ugly corner cases when
dealing with serialisation of arbitrary functions and classes.

It also builds on PEP 366, which took initial tentative steps towards
making explicit relative imports from the main module work correctly in
at least some circumstances.

Finally, PEP 328 eliminated implicit relative imports from imported
modules. This PEP proposes that the de facto implicit relative imports
from main modules that are provided by the current initialisation
behaviour for sys.path[0] also be eliminated.

What's in a __name__?

Over time, a module's __name__ attribute has come to be used to handle a
number of different tasks.

The key use cases identified for this module attribute are:

1.  Flagging the main module in a program, using the
    if __name__ == "__main__": convention.
2.  As the starting point for relative imports
3.  To identify the location of function and class definitions within
    the running application
4.  To identify the location of classes for serialisation into pickle
    objects which may be shared with other interpreter instances

Traps for the Unwary

The overloading of the semantics of __name__, along with some
historically associated behaviour in the initialisation of sys.path[0],
has resulted in several traps for the unwary. These traps can be quite
annoying in practice, as they are highly unobvious (especially to
beginners) and can cause quite confusing behaviour.

Why are my imports broken?

There's a general principle that applies when modifying sys.path: never
put a package directory directly on sys.path. The reason this is
problematic is that every module in that directory is now potentially
accessible under two different names: as a top level module (since the
package directory is on sys.path) and as a submodule of the package (if
the higher level directory containing the package itself is also on
sys.path).

As an example, Django (up to and including version 1.3) is guilty of
setting up exactly this situation for site-specific applications - the
application ends up being accessible as both app and site.app in the
module namespace, and these are actually two different copies of the
module. This is a recipe for confusion if there is any meaningful
mutable module level state, so this behaviour is being eliminated from
the default site set up in version 1.4 (site-specific apps will always
be fully qualified with the site name).

However, it's hard to blame Django for this, when the same part of
Python responsible for setting __name__ = "__main__" in the main module
commits the exact same error when determining the value for sys.path[0].

The impact of this can be seen relatively frequently if you follow the
"python" and "import" tags on Stack Overflow. When I had the time to
follow it myself, I regularly encountered people struggling to
understand the behaviour of straightforward package layouts like the
following (I actually use package layouts along these lines in my own
projects):

    project/
        setup.py
        example/
            __init__.py
            foo.py
            tests/
                __init__.py
                test_foo.py

While I would often see it without the __init__.py files first, that's a
trivial fix to explain. What's hard to explain is that all of the
following ways to invoke test_foo.py probably won't work due to broken
imports (either failing to find example for absolute imports,
complaining about relative imports in a non-package or beyond the
toplevel package for explicit relative imports, or issuing even more
obscure errors if some other submodule happens to shadow the name of a
top-level module, such as an example.json module that handled
serialisation or an example.tests.unittest test runner):

    # These commands will most likely *FAIL*, even if the code is correct

    # working directory: project/example/tests
    ./test_foo.py
    python test_foo.py
    python -m package.tests.test_foo
    python -c "from package.tests.test_foo import main; main()"

    # working directory: project/package
    tests/test_foo.py
    python tests/test_foo.py
    python -m package.tests.test_foo
    python -c "from package.tests.test_foo import main; main()"

    # working directory: project
    example/tests/test_foo.py
    python example/tests/test_foo.py

    # working directory: project/..
    project/example/tests/test_foo.py
    python project/example/tests/test_foo.py
    # The -m and -c approaches don't work from here either, but the failure
    # to find 'package' correctly is easier to explain in this case

That's right, that long list is of all the methods of invocation that
will almost certainly break if you try them, and the error messages
won't make any sense if you're not already intimately familiar not only
with the way Python's import system works, but also with how it gets
initialised.

For a long time, the only way to get sys.path right with that kind of
setup was to either set it manually in test_foo.py itself (hardly
something a novice, or even many veteran, Python programmers are going
to know how to do) or else to make sure to import the module instead of
executing it directly:

    # working directory: project
    python -c "from package.tests.test_foo import main; main()"

Since the implementation of PEP 366 (which defined a mechanism that
allows relative imports to work correctly when a module inside a package
is executed via the -m switch), the following also works properly:

    # working directory: project
    python -m package.tests.test_foo

The fact that most methods of invoking Python code from the command line
break when that code is inside a package, and the two that do work are
highly sensitive to the current working directory is all thoroughly
confusing for a beginner. I personally believe it is one of the key
factors leading to the perception that Python packages are complicated
and hard to get right.

This problem isn't even limited to the command line - if test_foo.py is
open in Idle and you attempt to run it by pressing F5, or if you try to
run it by clicking on it in a graphical filebrowser, then it will fail
in just the same way it would if run directly from the command line.

There's a reason the general "no package directories on sys.path"
guideline exists, and the fact that the interpreter itself doesn't
follow it when determining sys.path[0] is the root cause of all sorts of
grief.

In the past, this couldn't be fixed due to backwards compatibility
concerns. However, scripts potentially affected by this problem will
already require fixes when porting to the Python 3.x (due to the
elimination of implicit relative imports when importing modules
normally). This provides a convenient opportunity to implement a
corresponding change in the initialisation semantics for sys.path[0].

Importing the main module twice

Another venerable trap is the issue of importing __main__ twice. This
occurs when the main module is also imported under its real name,
effectively creating two instances of the same module under different
names.

If the state stored in __main__ is significant to the correct operation
of the program, or if there is top-level code in the main module that
has non-idempotent side effects, then this duplication can cause obscure
and surprising errors.

In a bit of a pickle

Something many users may not realise is that the pickle module sometimes
relies on the __module__ attribute when serialising instances of
arbitrary classes. So instances of classes defined in __main__ are
pickled that way, and won't be unpickled correctly by another python
instance that only imported that module instead of running it directly.
This behaviour is the underlying reason for the advice from many Python
veterans to do as little as possible in the __main__ module in any
application that involves any form of object serialisation and
persistence.

Similarly, when creating a pseudo-module (see next paragraph), pickles
rely on the name of the module where a class is actually defined, rather
than the officially documented location for that class in the module
hierarchy.

For the purposes of this PEP, a "pseudo-module" is a package designed
like the Python 3.2 unittest and concurrent.futures packages. These
packages are documented as if they were single modules, but are in fact
internally implemented as a package. This is supposed to be an
implementation detail that users and other implementations don't need to
worry about, but, thanks to pickle (and serialisation in general), the
details are often exposed and can effectively become part of the public
API.

While this PEP focuses specifically on pickle as the principal
serialisation scheme in the standard library, this issue may also affect
other mechanisms that support serialisation of arbitrary class instances
and rely on __module__ attributes to determine how to handle
deserialisation.

Where's the source?

Some sophisticated users of the pseudo-module technique described above
recognise the problem with implementation details leaking out via the
pickle module, and choose to address it by altering __name__ to refer to
the public location for the module before defining any functions or
classes (or else by modifying the __module__ attributes of those objects
after they have been defined).

This approach is effective at eliminating the leakage of information via
pickling, but comes at the cost of breaking introspection for functions
and classes (as their __module__ attribute now points to the wrong
place).

Forkless Windows

To get around the lack of os.fork on Windows, the multiprocessing module
attempts to re-execute Python with the same main module, but skipping
over any code guarded by if __name__ == "__main__": checks. It does the
best it can with the information it has, but is forced to make
assumptions that simply aren't valid whenever the main module isn't an
ordinary directly executed script or top-level module. Packages and
non-top-level modules executed via the -m switch, as well as directly
executed zipfiles or directories, are likely to make multiprocessing on
Windows do the wrong thing (either quietly or noisily, depending on
application details) when spawning a new process.

While this issue currently only affects Windows directly, it also
impacts any proposals to provide Windows-style "clean process"
invocation via the multiprocessing module on other platforms.

Qualified Names for Modules

To make it feasible to fix these problems once and for all, it is
proposed to add a new module level attribute: __qualname__. This
abbreviation of "qualified name" is taken from PEP 3155, where it is
used to store the naming path to a nested class or function definition
relative to the top level module.

For modules, __qualname__ will normally be the same as __name__, just as
it is for top-level functions and classes in PEP 3155. However, it will
differ in some situations so that the above problems can be addressed.

Specifically, whenever __name__ is modified for some other purpose (such
as to denote the main module), then __qualname__ will remain unchanged,
allowing code that needs it to access the original unmodified value.

If a module loader does not initialise __qualname__ itself, then the
import system will add it automatically (setting it to the same value as
__name__).

Alternative Names

Two alternative names were also considered for the new attribute: "full
name" (__fullname__) and "implementation name" (__implname__).

Either of those would actually be valid for the use case in this PEP.
However, as a meta-issue, PEP 3155 is also adding a new attribute (for
functions and classes) that is "like __name__, but different in some
cases where __name__ is missing necessary information" and those terms
aren't accurate for the PEP 3155 function and class use case.

PEP 3155 deliberately omits the module information, so the term "full
name" is simply untrue, and "implementation name" implies that it may
specify an object other than that specified by __name__, and that is
never the case for PEP 3155 (in that PEP, __name__ and __qualname__
always refer to the same function or class, it's just that __name__ is
insufficient to accurately identify nested functions and classes).

Since it seems needlessly inconsistent to add two new terms for
attributes that only exist because backwards compatibility concerns keep
us from changing the behaviour of __name__ itself, this PEP instead
chose to adopt the PEP 3155 terminology.

If the relative inscrutability of "qualified name" and __qualname__
encourages interested developers to look them up at least once rather
than assuming they know what they mean just from the name and guessing
wrong, that's not necessarily a bad outcome.

Besides, 99% of Python developers should never need to even care these
extra attributes exist - they're really an implementation detail to let
us fix a few problematic behaviours exhibited by imports, pickling and
introspection, not something people are going to be dealing with on a
regular basis.

Eliminating the Traps

The following changes are interrelated and make the most sense when
considered together. They collectively either completely eliminate the
traps for the unwary noted above, or else provide straightforward
mechanisms for dealing with them.

A rough draft of some of the concepts presented here was first posted on
the python-ideas list ([1]), but they have evolved considerably since
first being discussed in that thread. Further discussion has
subsequently taken place on the import-sig mailing list ([2].[3]).

Fixing main module imports inside packages

To eliminate this trap, it is proposed that an additional filesystem
check be performed when determining a suitable value for sys.path[0].
This check will look for Python's explicit package directory markers and
use them to find the appropriate directory to add to sys.path.

The current algorithm for setting sys.path[0] in relevant cases is
roughly as follows:

    # Interactive prompt, -m switch, -c switch
    sys.path.insert(0, '')

    # Valid sys.path entry execution (i.e. directory and zip execution)
    sys.path.insert(0, sys.argv[0])

    # Direct script execution
    sys.path.insert(0, os.path.dirname(sys.argv[0]))

It is proposed that this initialisation process be modified to take
package details stored on the filesystem into account:

    # Interactive prompt, -m switch, -c switch
    in_package, path_entry, _ignored = split_path_module(os.getcwd(), '')
    if in_package:
        sys.path.insert(0, path_entry)
    else:
        sys.path.insert(0, '')

    # Start interactive prompt or run -c command as usual
    #   __main__.__qualname__ is set to "__main__"

    # The -m switches uses the same sys.path[0] calculation, but:
    #   modname is the argument to the -m switch
    #   modname is passed to ``runpy._run_module_as_main()`` as usual
    #   __main__.__qualname__ is set to modname

    # Valid sys.path entry execution (i.e. directory and zip execution)
    modname = "__main__"
    path_entry, modname = split_path_module(sys.argv[0], modname)
    sys.path.insert(0, path_entry)

    # modname (possibly adjusted) is passed to ``runpy._run_module_as_main()``
    # __main__.__qualname__ is set to modname

    # Direct script execution
    in_package, path_entry, modname = split_path_module(sys.argv[0])
    sys.path.insert(0, path_entry)
    if in_package:
        # Pass modname to ``runpy._run_module_as_main()``
    else:
        # Run script directly
    # __main__.__qualname__ is set to modname

The split_path_module() supporting function used in the above
pseudo-code would have the following semantics:

    def _splitmodname(fspath):
        path_entry, fname = os.path.split(fspath)
        modname = os.path.splitext(fname)[0]
        return path_entry, modname

    def _is_package_dir(fspath):
        return any(os.exists("__init__" + info[0]) for info
                       in imp.get_suffixes())

    def split_path_module(fspath, modname=None):
        """Given a filesystem path and a relative module name, determine an
           appropriate sys.path entry and a fully qualified module name.

           Returns a 3-tuple of (package_depth, fspath, modname). A reported
           package depth of 0 indicates that this would be a top level import.

           If no relative module name is given, it is derived from the final
           component in the supplied path with the extension stripped.
        """
        if modname is None:
            fspath, modname = _splitmodname(fspath)
        package_depth = 0
        while _is_package_dir(fspath):
            fspath, pkg = _splitmodname(fspath)
            modname = pkg + '.' + modname
        return package_depth, fspath, modname

This PEP also proposes that the split_path_module() functionality be
exposed directly to Python users via the runpy module.

With this fix in place, and the same simple package layout described
earlier, all of the following commands would invoke the test suite
correctly:

    # working directory: project/example/tests
    ./test_foo.py
    python test_foo.py
    python -m package.tests.test_foo
    python -c "from .test_foo import main; main()"
    python -c "from ..tests.test_foo import main; main()"
    python -c "from package.tests.test_foo import main; main()"

    # working directory: project/package
    tests/test_foo.py
    python tests/test_foo.py
    python -m package.tests.test_foo
    python -c "from .tests.test_foo import main; main()"
    python -c "from package.tests.test_foo import main; main()"

    # working directory: project
    example/tests/test_foo.py
    python example/tests/test_foo.py
    python -m package.tests.test_foo
    python -c "from package.tests.test_foo import main; main()"

    # working directory: project/..
    project/example/tests/test_foo.py
    python project/example/tests/test_foo.py
    # The -m and -c approaches still don't work from here, but the failure
    # to find 'package' correctly is pretty easy to explain in this case

With these changes, clicking Python modules in a graphical file browser
should always execute them correctly, even if they live inside a
package. Depending on the details of how it invokes the script, Idle
would likely also be able to run test_foo.py correctly with F5, without
needing any Idle specific fixes.

Optional addition: command line relative imports

With the above changes in place, it would be a fairly minor addition to
allow explicit relative imports as arguments to the -m switch:

    # working directory: project/example/tests
    python -m .test_foo
    python -m ..tests.test_foo

    # working directory: project/example/
    python -m .tests.test_foo

With this addition, system initialisation for the -m switch would change
as follows:

    # -m switch (permitting explicit relative imports)
    in_package, path_entry, pkg_name = split_path_module(os.getcwd(), '')
    qualname= <<arguments to -m switch>>
    if qualname.startswith('.'):
        modname = qualname
        while modname.startswith('.'):
            modname = modname[1:]
            pkg_name, sep, _ignored = pkg_name.rpartition('.')
            if not sep:
                raise ImportError("Attempted relative import beyond top level package")
        qualname = pkg_name + '.' modname
    if in_package:
        sys.path.insert(0, path_entry)
    else:
        sys.path.insert(0, '')

    # qualname is passed to ``runpy._run_module_as_main()``
    # _main__.__qualname__ is set to qualname

Compatibility with PEP 382

Making this proposal compatible with the PEP 382 namespace packaging PEP
is trivial. The semantics of _is_package_dir() are merely changed to be:

    def _is_package_dir(fspath):
        return (fspath.endswith(".pyp") or
                any(os.exists("__init__" + info[0]) for info
                        in imp.get_suffixes()))

Incompatibility with PEP 402

PEP 402 proposes the elimination of explicit markers in the file system
for Python packages. This fundamentally breaks the proposed concept of
being able to take a filesystem path and a Python module name and work
out an unambiguous mapping to the Python module namespace. Instead, the
appropriate mapping would depend on the current values in sys.path,
rendering it impossible to ever fix the problems described above with
the calculation of sys.path[0] when the interpreter is initialised.

While some aspects of this PEP could probably be salvaged if PEP 402
were adopted, the core concept of making import semantics from main and
other modules more consistent would no longer be feasible.

This incompatibility is discussed in more detail in the relevant
import-sig threads ([4],[5]).

Potential incompatibilities with scripts stored in packages

The proposed change to sys.path[0] initialisation may break some
existing code. Specifically, it will break scripts stored in package
directories that rely on the implicit relative imports from __main__ in
order to run correctly under Python 3.

While such scripts could be imported in Python 2 (due to implicit
relative imports) it is already the case that they cannot be imported in
Python 3, as implicit relative imports are no longer permitted when a
module is imported.

By disallowing implicit relatives imports from the main module as well,
such modules won't even work as scripts with this PEP. Switching them
over to explicit relative imports will then get them working again as
both executable scripts and as importable modules.

To support earlier versions of Python, a script could be written to use
different forms of import based on the Python version:

    if __name__ == "__main__" and sys.version_info < (3, 3):
        import peer # Implicit relative import
    else:
        from . import peer # explicit relative import

Fixing dual imports of the main module

Given the above proposal to get __qualname__ consistently set correctly
in the main module, one simple change is proposed to eliminate the
problem of dual imports of the main module: the addition of a
sys.metapath hook that detects attempts to import __main__ under its
real name and returns the original main module instead:

    class AliasImporter:
      def __init__(self, module, alias):
          self.module = module
          self.alias = alias

      def __repr__(self):
          fmt = "{0.__class__.__name__}({0.module.__name__}, {0.alias})"
          return fmt.format(self)

      def find_module(self, fullname, path=None):
          if path is None and fullname == self.alias:
              return self
          return None

      def load_module(self, fullname):
          if fullname != self.alias:
              raise ImportError("{!r} cannot load {!r}".format(self, fullname))
          return self.main_module

This metapath hook would be added automatically during import system
initialisation based on the following logic:

    main = sys.modules["__main__"]
    if main.__name__ != main.__qualname__:
        sys.metapath.append(AliasImporter(main, main.__qualname__))

This is probably the least important proposal in the PEP - it just
closes off the last mechanism that is likely to lead to module
duplication after the configuration of sys.path[0] at interpreter
startup is addressed.

Fixing pickling without breaking introspection

To fix this problem, it is proposed to make use of the new module level
__qualname__ attributes to determine the real module location when
__name__ has been modified for any reason.

In the main module, __qualname__ will automatically be set to the main
module's "real" name (as described above) by the interpreter.

Pseudo-modules that adjust __name__ to point to the public namespace
will leave __qualname__ untouched, so the implementation location
remains readily accessible for introspection.

If __name__ is adjusted at the top of a module, then this will
automatically adjust the __module__ attribute for all functions and
classes subsequently defined in that module.

Since multiple submodules may be set to use the same "public" namespace,
functions and classes will be given a new __qualmodule__ attribute that
refers to the __qualname__ of their module.

This isn't strictly necessary for functions (you could find out their
module's qualified name by looking in their globals dictionary), but it
is needed for classes, since they don't hold a reference to the globals
of their defining module. Once a new attribute is added to classes, it
is more convenient to keep the API consistent and add a new attribute to
functions as well.

These changes mean that adjusting __name__ (and, either directly or
indirectly, the corresponding function and class __module__ attributes)
becomes the officially sanctioned way to implement a namespace as a
package, while exposing the API as if it were still a single module.

All serialisation code that currently uses __name__ and __module__
attributes will then avoid exposing implementation details by default.

To correctly handle serialisation of items from the main module, the
class and function definition logic will be updated to also use
__qualname__ for the __module__ attribute in the case where
__name__ == "__main__".

With __name__ and __module__ being officially blessed as being used for
the public names of things, the introspection tools in the standard
library will be updated to use __qualname__ and __qualmodule__ where
appropriate. For example:

-   pydoc will report both public and qualified names for modules
-   inspect.getsource() (and similar tools) will use the qualified names
    that point to the implementation of the code
-   additional pydoc and/or inspect APIs may be provided that report all
    modules with a given public __name__.

Fixing multiprocessing on Windows

With __qualname__ now available to tell multiprocessing the real name of
the main module, it will be able to simply include it in the serialised
information passed to the child process, eliminating the need for the
current dubious introspection of the __file__ attribute.

For older Python versions, multiprocessing could be improved by applying
the split_path_module() algorithm described above when attempting to
work out how to execute the main module based on its __file__ attribute.

Explicit relative imports

This PEP proposes that __package__ be unconditionally defined in the
main module as __qualname__.rpartition('.')[0]. Aside from that, it
proposes that the behaviour of explicit relative imports be left alone.

In particular, if __package__ is not set in a module when an explicit
relative import occurs, the automatically cached value will continue to
be derived from __name__ rather than __qualname__. This minimises any
backwards incompatibilities with existing code that deliberately
manipulates relative imports by adjusting __name__ rather than setting
__package__ directly.

This PEP does not propose that __package__ be deprecated. While it is
technically redundant following the introduction of __qualname__, it
just isn't worth the hassle of deprecating it within the lifetime of
Python 3.x.

Reference Implementation

None as yet.

References

-   Elaboration of compatibility problems between this PEP and PEP 402

Copyright

This document has been placed in the public domain.

[1] Module aliases and/or "real names"

[2] PEP 395 (Module aliasing) and the namespace PEPs

[3] Updated PEP 395 (aka "Implicit Relative Imports Must Die!")

[4] PEP 395 (Module aliasing) and the namespace PEPs

[5] Updated PEP 395 (aka "Implicit Relative Imports Must Die!")


text:
PEP: 609 Title: Python Packaging Authority (PyPA) Governance Author:
Dustin Ingram<EMAIL_ADDRESS>Pradyun Gedam<EMAIL_ADDRESS>Sumana Harihareswara<EMAIL_ADDRESS>Sponsor: Paul Ganssle
<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pep-609-pypa-governance/2619 Status: Active
Type: Process Topic: Governance, Packaging Content-Type: text/x-rst
Created: 05-Nov-2019 Post-History: 05-Nov-2019

Abstract

This document describes a governance model for the Python Packaging
Authority (PyPA). The model is closely based on existing informal
practices, with the intent of providing clarity into the functioning of
the PyPA and formalizing transparent processes for the PyPA.

Rationale

The Python Packaging Authority (PyPA) is a collaborative community that
maintains and advances many of the relevant projects in Python
packaging. The software and standards developed through the PyPA are
used to package, share, and install Python software and to interact with
indexes of downloadable Python software such as PyPI, the Python Package
Index.

Currently, the PyPA is an informal and loosely defined organization that
lacks true authority, and the inclusion of a given project under the
PyPA umbrella or the creation of new projects has been done in an
ad-hoc, one-off manner. Similarly, individual membership in the PyPA is
not well-defined.

While this model has more or less worked for the PyPA in the past, it
results in an organization which is missing certain features of a stable
ecosystem, namely a clear and transparent decision-making process. This
PEP seeks to rectify this by defining a governance model for the PyPA.

Terminology

Relevant terms for groups of individual contributors used in this PEP:

PyPA members:

    Anyone with the triage bit or commit bit, on at least one project in
    the PyPA organization.

PyPA committers:

    Anyone with the commit bit on at least one project in the PyPA
    organization, which should correspond to everyone on the
    PyPA-Committers mailing list.

PyPA community:

    Anyone who is interested in PyPA activity and wants to follow along,
    contribute or make proposals.

Packaging-WG members:

    As described in the Packaging-WG Wiki page. For clarity: there is no
    formal relationship between the Packaging-WG and PyPA. This group is
    only included in this list to disambiguate it from PyPA.

Goals

The following section formalizes the goals (and non-goals) of the PyPA
and this governance model.

Goals of the PyPA

These goals are the primary motivation for the existence of the PyPA.
These goals are largely already being carried out, even though most have
not been explicitly defined.

Provide support for existing projects under the PyPA

In the event that a given project needs additional support, or no longer
has active maintainers, the PyPA will ensure that the given project will
continue to be supported for users to the extent necessary.

Foster the creation and acceptance of standards for PyPA projects

The PyPA should, as much as possible, strive for standardization and
coordination across PyPA projects, primarily though the governance
process outlined below. PyPA projects are expected to abide by
applicable specifications maintained by the PyPA.

Guide decisions which affect multiple PyPA projects

The PyPA community (especially PyPA members) should be expected to
provide opinions, insight and experience when ecosystem-wide changes are
being proposed.

Determine which projects should be under the guidance of the PyPA

For example: accepting new projects from the community, organically
creating projects within the PyPA, etc.

Enforce adherence to a Code of Conduct across all projects

Generally this means leading by example, but occasionally it may mean
more explicit moderation.

Non-goals of the PyPA

These are specific items that are explicitly _not goals of the PyPA.

Determine who is and isn't a PyPA member

This is for members of individual projects to decide, as they add new
members to their projects. Maintainership of a project that is under the
PyPA organization automatically transfers membership in the PyPA.

Micromanage individual projects

As long as the project is adhering to the Code of Conduct and following
specifications supported by the PyPA, the PyPA should only concerned
with large, ecosystem-wide changes.

Develop and maintain standalone Code of Conduct

PyPA projects follow the PSF Code of Conduct.

Goals of the PyPA's Governance Model

These are new goals which the governance model seeks to make possible.

Transparency in PyPA membership

Provide a transparent process for decisions taken, regarding project
membership in the PyPA.

Document PyPA's use of PEPs

Formally document how the PyPA uses Python Enhancement Proposals (PEPs),
for maintaining interoperability specifications defined by the PyPA.

Processes

The processes for the PyPA's activities are outlined below:

Specifications

The PyPA will use PEPs for defining, and making changes to, the
interoperability specifications maintained by the PyPA. Thus, the Python
Steering Council has the final say in the acceptance of these
interoperability specifications.

It is expected (but not required) that the Python Steering Council would
delegate authority to sponsor and/or approve/reject PEPs related to
packaging interoperability specifications, to individuals within the
PyPA community. At the time of writing (June 2020), the Python Steering
Council has standing delegations for currently active packaging
interoperability specifications.

The details of the process of proposing and updating the
interoperability specifications are described in the PyPA Specifications
document.

Governance

PyPA Committer Votes

A PyPA member can put forward a proposal and call for a vote on a public
PyPA communication channel. A PyPA committer vote is triggered when a
PyPA committer (not the proposer) seconds the proposal.

The proposal will be put to a vote on the PyPA-Committers mailing list,
over a 7-day period. Each PyPA committer can vote once, and can choose
one of +1 and -1. If at least two thirds of recorded votes are +1, then
the vote succeeds.

PyPA committer votes are required for, and limited to, the following
kinds of proposals:

Addition of a project to the PyPA

Proposing the acceptance of a project into the PyPA organization. This
proposal must not be opposed by the existing maintainers of the project.

Creation of a new project in the PyPA

Proposing the creation of a new tools / project in the PyPA
organization.

Removal of a project from PyPA

Proposing the removal of a project in the PyPA organization.

Updates to the Governance/Specification Processes

Proposing changes to how the PyPA operates, including but not limited to
changes to its specification and governance processes, and this PEP.

Leaving PyPA

A project that is a part of the PyPA organization, can request to leave
PyPA.

Such requests can made by a committer of the project, on the
PyPA-Committers mailing list and must clearly state the GitHub
user/organization to transfer the repository to.

If the request is not opposed by another committer of the same project
over a 7-day period, the project would leave the PyPA and be transferred
out of the PyPA organization as per the request.

Code of Conduct enforcement

Each project that is a part of the PyPA organization follows the PSF
Code of Conduct, including its incident reporting guidelines and
enforcement procedures.

PyPA members are responsible for leading by example. PyPA members
occasionally may need to more explicitly moderate behavior in their
projects, and each project that is a part of the PyPA organization must
designate at least one PyPA member as available to contact in case of a
Code of Conduct incident. If told of any Code of Conduct incidents
involving their projects, PyPA members are expected to report those
incidents up to the PSF Conduct WG, for recording purposes and for
potential assistance.

References

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 644 Title: Require OpenSSL 1.1.1 or newer Author: Christian Heimes
<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pep-644-require-openssl-1-1-or-newer/5584
Status: Final Type: Standards Track Content-Type: text/x-rst Created:
27-Oct-2020 Python-Version: 3.10 Post-History: 27-Oct-2020, 03-Mar-2021,
17-Mar-2021, 17-Apr-2021 Resolution:
<EMAIL_ADDRESS>
Abstract

This PEP proposes for CPython’s standard library to support only OpenSSL
1.1.1 LTS or newer. Support for OpenSSL versions past end-of-lifetime,
incompatible forks, and other TLS libraries are dropped.

Motivation

Python makes use of OpenSSL in hashlib, hmac, and ssl modules. OpenSSL
provides fast implementations of cryptographic primitives and a full TLS
stack including handling of X.509 certificates. The ssl module is used
by standard library modules like urllib and 3rd party modules like
urllib3 to implement secure variants of internet protocols. pip uses the
ssl module to securely download packages from PyPI. Any bug in the ssl
module's bindings to OpenSSL can lead to a severe security issue.

Over time OpenSSL's public API has evolved and changed. Version 1.0.2
introduced new APIs to verify and match hostnames. OpenSSL 1.1.0 made
internal structs opaque and introduced new APIs that replace direct
access of struct members. Version 3.0.0 will deprecate more APIs due to
internal reorganization that moves cryptographic algorithms out of the
core and into providers. Forks like LibreSSL and BoringSSL have diverged
in different directions.

Currently Python versions 3.6 to 3.9 are compatible with OpenSSL 1.0.2,
1.1.0, and 1.1.1. For the most part Python also works with LibreSSL >=
2.7.1 with some missing features and broken tests.

Due to limited resources and time it becomes increasingly hard to
support multiple versions and forks as well as test and verify
correctness. Besides multiple incompatible APIs there are build time
flags, distribution-specific patches, and local crypto-policy settings
that add to plethora of combinations. On the other hand, the Python core
team has only a couple of domain experts who are familiar with TLS and
OpenSSL internals and even fewer who are active maintainers.

Requiring OpenSSL 1.1.1 would allow us to give the vast majority of
users a better experience, reduce our maintenance overhead and thus free
resources to implement new features. Users would be able to rely on the
presence of new features and consistent behavior, ultimately resulting
in a more robust experience.

Impact

OpenSSL 1.1.1 is the default variant and version of OpenSSL on almost
all supported platforms and distributions. It’s also the only version
that still receives security support from upstream[1].

No macOS and Windows user will be affected by the deprecation. The
python.org installer and alternative distributions like Conda ship with
most recent OpenSSL version.

As of October 2020 and according to DistroWatch[2] most current BSD and
Linux distributions ship with OpenSSL 1.1.1 as well. Some older releases
of long-term support (LTS) and enterprise distributions have older
versions of OpenSSL or LibreSSL. By the time Python 3.10 will be
generally available, several of these distributions will have reached
end of lifetime, end of general support, or moved from LibreSSL to
OpenSSL.

Other software has dropped support for OpenSSL 1.0.2 as well. For
example, PyCA cryptography 3.2 (2020-10-25) removed compatibility with
OpenSSL 1.0.2.

OpenSSL 1.0.2 LTS

released: 2015-02 end of lifetime: 2019-12

OpenSSL 1.0.2 added hostname verification, ALPN support, and elliptic
curves.

-   CentOS 7 (EOL 2024-06)
-   Debian 8 Jessie (EOL 2020-07)
-   Linux Mint 18.3 (EOL 2021-04)
-   RHEL 7 (full support ends 2019-08, maintenance 2 support ends
    2024-06)
-   SUSE Enterprise Linux 12-SP5 (general supports ends 2024-10)
-   Ubuntu 16.04 LTS / Xenial (general support ends 2021-04)

OpenSSL 1.1.0

released: 2016-08 end of lifetime: 2019-09

OpenSSL 1.1.0 removed or disabled insecure ciphers by default and added
support for ChaCha20-Poly1305, BLAKE2 (basic features), X25519 and CT.
The majority of structs were made opaque and new APIs were introduced.
OpenSSL 1.1.0 is not API compatible with 1.0.2.

-   Debian 9 Stretch (security support ended 2020-07, LTS until 2022-06)
-   Ubuntu 18.04 LTS / Bionic (general support ends 2023-04)

OpenSSL 1.1.1 LTS

released: 2018-08 end of lifetime: 2023-09 (planned)

OpenSSL 1.1.1 added TLS 1.3, SHA-3, X448 and Ed448.

-   Alpine (switched back to OpenSSL in 2018[3])
-   Arch Linux current
-   CentOS 8.0+
-   Debian 10 Buster
-   Debian 11 Bullseye (ETA 2021-06)
-   Fedora 29+
-   FreeBSD 11.3+
-   Gentoo Linux stable (dropped LibreSSL as alternative in January
    2021[4])
-   HardenedBSD (switched back to OpenSSL in 2018[5])
-   Linux Mint 19.3+
-   macOS (python.org installer)
-   NetBSD 8.2+
-   openSUSE 15.2+
-   RHEL 8.0+
-   Slackware current
-   SUSE Enterprise Linux 15-SP2
-   Ubuntu 18.10+
-   Ubuntu 20.04 LTS / Focal
-   VoidLinux (switched back to OpenSSL in March 2021[6])
-   Windows (python.org installer, Conda)

Major CI providers provide images with OpenSSL 1.1.1.

-   AppVeyor (with image Ubuntu2004)
-   CircleCI (with recent cimg/base:stable or cimg/base:stable-20.04)
-   GitHub Actions (with runs-on: ubuntu-20.04)
-   Giblab CI (with Debian Stretch, Ubuntu Focal, CentOS 8, RHEL 8, or
    Fedora runner)
-   Packit
-   TravisCI (with dist: focal)
-   Zuul

OpenSSL 3.0.0

released: n/a (planned for mid/late 2021)

OpenSSL 3.0.0 is currently under development. Major changes include
relicensing to Apache License 2.0 and a new API for cryptographic
algorithms providers. Most changes are internal refactorings and don’t
affect public APIs.[7]

LibreSSL

created: 2014-04 (forked from OpenSSL 1.0.1g)

-   DragonFly BSD
-   Hyperbola GNU/Linux-libre
-   OpenBSD
-   OpenELEC (discontinued)
-   TrueOS (discontinued)

Some distributions like FreeBSD and OPNsense also feature LibreSSL
instead of OpenSSL as non-standard TLS libraries. Gentoo discontinued
LibreSSL as an alternative to OpenSSL in January 2021[8] due to
compatibility issues and little testing.

OpenBSD ports has a port security/openssl/1.1 which is documented as
"[...] is present to provide support for applications which cannot be
made compatible with LibReSSL"[9]. The package could be used by OpenBSD
to provide a working ssl module.

BoringSSL

created: 2014-06

BoringSSL is Google’s fork of OpenSSL. It’s not intended for general use
and therefore not supported by Python. There are no guarantees of API or
ABI stability. Vendored copies of BoringSSL are used in Chrome/Chromium
browser, Android, and on Apple platforms[10].

Benefits

TLS 1.3

OpenSSL 1.1.1 introduced support for the new TLS 1.3 version. The latest
version of the TLS protocol has a faster handshake and is more secure
than the previous versions.

Thread and fork safety

Starting with release 1.1.0c, OpenSSL is fully fork and thread safe.
Bindings no longer need any workarounds or additional callbacks to
support multithreading.

SHA-3

Since 1.1.0, OpenSSL ships with SHA-3 and SHAKE implementations.
Python's builtin SHA-3 support is based on the reference implementation.
The internal _sha3 code is fairly large and the resulting shared library
close to 0.5 MB. Python could drop the builtin implementation and rely
on OpenSSL's libcrypto instead.

So far LibreSSL upstream development has refused to add SHA-3
support.[11]

Compatibility

OpenSSL downstream patches and options

OpenSSL features more than 70 configure and build time options in the
form of OPENSSL_NO_* macros. Around 60 options affect the presence of
features like cryptographic algorithms and TLS versions. Some
distributions apply patches to alter settings. Furthermore, default
values for settings like security level, ciphers, TLS version range, and
signature algorithms can be set in OpenSSL config file.

The Python core team lacks resources to test all possible combinations.
This PEP proposes that Python only supports OpenSSL builds that have
standard features enabled. Vendors shall disable deprecated or insecure
algorithms and TLS versions with build time options like
OPENSSL_NO_TLS1_1_METHOD or OpenSSL config options like
MinProtocol = TLSv1.2.

Python assumes that OpenSSL is built with

-   hashlib’s default algorithms such as MD5, SHA-1, SHA-2 family,
    SHA-3/SHAKE family, BLAKE2
-   TLS 1.2 and TLS 1.3 protocols
-   current key agreement, signature, and encryption algorithms for TLS
    1.2 and 1.3 (ECDH, RSA, ECDSA, Curve25519, AES, Poly1309-ChaCha20,
    ...)
-   threading, file I/O, socket I/O, and error messages

Weak algorithms (MD5, SHA-1 signatures) and short keys (RSA < 2024 bits)
may be disabled at runtime. Algorithms may also be blocked when they are
disabled by a crypto policy such as FIPS. The PEP is not more specific
on purpose to give room for new features as well as countermeasures
against vulnerabilities. As a rule of thumb, Python should be able to
connect to PyPI and the test suite should pass.

LibreSSL support

LibreSSL is a fork of OpenSSL. The fork was created off OpenSSL 1.0.1g
by members of the OpenBSD team in 2014 in light of the heartbleed
vulnerability. Since its inception several features deemed problematic
or insecure were removed or replaced (SSL 2.0, SSL 3.0, improved CPRNG)
or backported from OpenSSL and BoringSSL.

At the moment LibreSSL is not fully API compatible with OpenSSL 1.1.1.
The latest release LibreSSL 3.3.2 is missing features and behaves
differently in some cases. Mentionable missing or incompatible features
include

-   SHA-3, SHAKE, BLAKE2
-   SSL_CERT_* environment variables
-   security level APIs
-   session handling APIs
-   key logging API
-   verified cert chain APIs
-   OPENSSL_VERSION macro

This PEP proposed to remove any and all LibreSSL related workarounds
from Python. In the future Python will not actively prohibit LibreSSL
support with configure and compile time checks. But Python will not
accept patches that add non-trivial workarounds or disable tests either.

BoringSSL

There are currently no plans to support BoringSSL.

Rejected Ideas

Formalize supported OpenSSL versions

This PEP does not provide a set of formal rules and conditions under
which an OpenSSL version is supported.

In general Python aims to be compatible with commonly used and
officially supported OpenSSL versions. Patch releases of Python may not
be compatible with new major releases of OpenSSL. Users should not
expect that a new major or minor release of Python works with an OpenSSL
version that is past its end-of-lifetime. Python core development may
backport fixes for new releases or extend compatibility with EOLed
releases as we see fit.

The new ABI stability and LTS policies of OpenSSL[12] should help, too.

Keep support for OpenSSL 1.1.0

It was suggested to keep support for OpenSSL 1.1.0 for compatibility
with Debian 9 (Stretch). The proposal was rejected since it would
complicated code cleanup and testing. Stretch is already out of regular
security support and close to end of long-term support. By the time of
Python 3.10 final release, Debian Buster and Debian Bullseye will be
available.

Instead Python 3.10 will gain additional documentation and a new
configure option --with-openssl-rpath=auto to simplify use of custom
OpenSSL builds[13].

Backwards Compatibility

Python 3.10 will no longer support TLS/SSL and fast hashing on platforms
with OpenSSL 1.0.2 or LibreSSL. The first draft of this PEP was
published at the beginning of the 3.10 release cycles to give vendors
like Linux distributors or CI providers sufficient time to plan.

Python's internal copy of the Keccak Code Package and the internal _sha3
module will be removed. This will reduce source code size by about 280kB
and code size by roughly 0.5MB. The hashlib will solely rely on
OpenSSL's SHA-3 implementation. SHA-3 and SHAKE will no longer be
available without OpenSSL.

Disclaimer and special thanks

The author of this PEP is a contributor to OpenSSL project and employed
by a major Linux distributor that uses OpenSSL.

Thanks to Alex Gaynor, Gregory P. Smith, Nathaniel J. Smith, Paul
Kehrer, and Seth Larson for their review and feedback on the initial
draft.

References

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

[1] https://www.openssl.org/policies/releasestrat.html

[2] https://distrowatch.com/

[3] https://lists.alpinelinux.org/~alpine/devel/%3CCA%2BT2pCGFeh30aEi43hAvJ3yoHBijABy_U62wfjhVmf3FmbNUUg%40mail.gmail.com%3E

[4] https://www.gentoo.org/support/news-items/2021-01-05-libressl-support-discontinued.html

[5] https://hardenedbsd.org/article/shawn-webb/2018-04-30/hardenedbsd-switching-back-openssl

[6] https://voidlinux.org/news/2021/02/OpenSSL.html

[7] https://www.openssl.org/docs/OpenSSL300Design.html

[8] https://www.gentoo.org/support/news-items/2021-01-05-libressl-support-discontinued.html

[9] https://openports.se/security/openssl/1.1

[10] https://forums.swift.org/t/rfc-moving-swiftnio-ssl-to-boringssl/18280

[11] https://github.com/libressl-portable/portable/issues/455

[12] https://www.openssl.org/policies/releasestrat.html

[13] https://bugs.python.org/issue43466


text:
PEP: 574 Title: Pickle protocol 5 with out-of-band data Version:
$Revision$ Last-Modified: $Date$ Author: Antoine Pitrou
<EMAIL_ADDRESS>BDFL-Delegate: Alyssa Coghlan Status: Final Type:
Standards Track Content-Type: text/x-rst Created: 23-Mar-2018
Python-Version: 3.8 Post-History: 28-Mar-2018, 30-Apr-2019 Resolution:
https://mail.python.org/pipermail/python-dev/2019-May/157284.html

Abstract

This PEP proposes to standardize a new pickle protocol version, and
accompanying APIs to take full advantage of it:

1.  A new pickle protocol version (5) to cover the extra metadata needed
    for out-of-band data buffers.
2.  A new PickleBuffer type for __reduce_ex__ implementations to return
    out-of-band data buffers.
3.  A new buffer_callback parameter when pickling, to handle out-of-band
    data buffers.
4.  A new buffers parameter when unpickling to provide out-of-band data
    buffers.

The PEP guarantees unchanged behaviour for anyone not using the new
APIs.

Rationale

The pickle protocol was originally designed in 1995 for on-disk
persistency of arbitrary Python objects. The performance of a 1995-era
storage medium probably made it irrelevant to focus on performance
metrics such as use of RAM bandwidth when copying temporary data before
writing it to disk.

Nowadays the pickle protocol sees a growing use in applications where
most of the data isn't ever persisted to disk (or, when it is, it uses a
portable format instead of Python-specific). Instead, pickle is being
used to transmit data and commands from one process to another, either
on the same machine or on multiple machines. Those applications will
sometimes deal with very large data (such as Numpy arrays or Pandas
dataframes) that need to be transferred around. For those applications,
pickle is currently wasteful as it imposes spurious memory copies of the
data being serialized.

As a matter of fact, the standard multiprocessing module uses pickle for
serialization, and therefore also suffers from this problem when sending
large data to another process.

Third-party Python libraries, such as Dask[1], PyArrow[2] and
IPyParallel[3], have started implementing alternative serialization
schemes with the explicit goal of avoiding copies on large data.
Implementing a new serialization scheme is difficult and often leads to
reduced generality (since many Python objects support pickle but not the
new serialization scheme). Falling back on pickle for unsupported types
is an option, but then you get back the spurious memory copies you
wanted to avoid in the first place. For example, dask is able to avoid
memory copies for Numpy arrays and built-in containers thereof (such as
lists or dicts containing Numpy arrays), but if a large Numpy array is
an attribute of a user-defined object, dask will serialize the
user-defined object as a pickle stream, leading to memory copies.

The common theme of these third-party serialization efforts is to
generate a stream of object metadata (which contains pickle-like
information about the objects being serialized) and a separate stream of
zero-copy buffer objects for the payloads of large objects. Note that,
in this scheme, small objects such as ints, etc. can be dumped together
with the metadata stream. Refinements can include opportunistic
compression of large data depending on its type and layout, like dask
does.

This PEP aims to make pickle usable in a way where large data is handled
as a separate stream of zero-copy buffers, letting the application
handle those buffers optimally.

Example

To keep the example simple and avoid requiring knowledge of third-party
libraries, we will focus here on a bytearray object (but the issue is
conceptually the same with more sophisticated objects such as Numpy
arrays). Like most objects, the bytearray object isn't immediately
understood by the pickle module and must therefore specify its
decomposition scheme.

Here is how a bytearray object currently decomposes for pickling:

    >>> b.__reduce_ex__(4)
    (<class 'bytearray'>, (b'abc',), None)

This is because the bytearray.__reduce_ex__ implementation reads morally
as follows:

    class bytearray:

       def __reduce_ex__(self, protocol):
          if protocol == 4:
             return type(self), bytes(self), None
          # Legacy code for earlier protocols omitted

In turn it produces the following pickle code:

    >>> pickletools.dis(pickletools.optimize(pickle.dumps(b, protocol=4)))
        0: \x80 PROTO      4
        2: \x95 FRAME      30
       11: \x8c SHORT_BINUNICODE 'builtins'
       21: \x8c SHORT_BINUNICODE 'bytearray'
       32: \x93 STACK_GLOBAL
       33: C    SHORT_BINBYTES b'abc'
       38: \x85 TUPLE1
       39: R    REDUCE
       40: .    STOP

(the call to pickletools.optimize above is only meant to make the pickle
stream more readable by removing the MEMOIZE opcodes)

We can notice several things about the bytearray's payload (the sequence
of bytes b'abc'):

-   bytearray.__reduce_ex__ produces a first copy by instantiating a new
    bytes object from the bytearray's data.
-   pickle.dumps produces a second copy when inserting the contents of
    that bytes object into the pickle stream, after the SHORT_BINBYTES
    opcode.
-   Furthermore, when deserializing the pickle stream, a temporary bytes
    object is created when the SHORT_BINBYTES opcode is encountered
    (inducing a data copy).

What we really want is something like the following:

-   bytearray.__reduce_ex__ produces a view of the bytearray's data.
-   pickle.dumps doesn't try to copy that data into the pickle stream
    but instead passes the buffer view to its caller (which can decide
    on the most efficient handling of that buffer).
-   When deserializing, pickle.loads takes the pickle stream and the
    buffer view separately, and passes the buffer view directly to the
    bytearray constructor.

We see that several conditions are required for the above to work:

-   __reduce__ or __reduce_ex__ must be able to return something that
    indicates a serializable no-copy buffer view.
-   The pickle protocol must be able to represent references to such
    buffer views, instructing the unpickler that it may have to get the
    actual buffer out of band.
-   The pickle.Pickler API must provide its caller with a way to receive
    such buffer views while serializing.
-   The pickle.Unpickler API must similarly allow its caller to provide
    the buffer views required for deserialization.
-   For compatibility, the pickle protocol must also be able to contain
    direct serializations of such buffer views, such that current uses
    of the pickle API don't have to be modified if they are not
    concerned with memory copies.

Producer API

We are introducing a new type pickle.PickleBuffer which can be
instantiated from any buffer-supporting object, and is specifically
meant to be returned from __reduce__ implementations:

    class bytearray:

       def __reduce_ex__(self, protocol):
          if protocol >= 5:
             return type(self), (PickleBuffer(self),), None
          # Legacy code for earlier protocols omitted

PickleBuffer is a simple wrapper that doesn't have all the memoryview
semantics and functionality, but is specifically recognized by the
pickle module if protocol 5 or higher is enabled. It is an error to try
to serialize a PickleBuffer with pickle protocol version 4 or earlier.

Only the raw data of the PickleBuffer will be considered by the pickle
module. Any type-specific metadata (such as shapes or datatype) must be
returned separately by the type's __reduce__ implementation, as is
already the case.

PickleBuffer objects

The PickleBuffer class supports a very simple Python API. Its
constructor takes a single PEP 3118-compatible object. PickleBuffer
objects themselves support the buffer protocol, so consumers can call
memoryview(...) on them to get additional information about the
underlying buffer (such as the original type, shape, etc.). In addition,
PickleBuffer objects have the following methods:

raw()

  Return a memoryview of the raw memory bytes underlying the
  PickleBuffer, erasing any shape, strides and format information. This
  is required to handle Fortran-contiguous buffers correctly in the pure
  Python pickle implementation.

release()

  Release the PickleBuffer's underlying buffer, making it unusable.

On the C side, a simple API will be provided to create and inspect
PickleBuffer objects:

PyObject *PyPickleBuffer_FromObject(PyObject *obj)

  Create a PickleBuffer object holding a view over the PEP
  3118-compatible obj.

PyPickleBuffer_Check(PyObject *obj)

  Return whether obj is a PickleBuffer instance.

const Py_buffer *PyPickleBuffer_GetBuffer(PyObject *picklebuf)

  Return a pointer to the internal Py_buffer owned by the PickleBuffer
  instance. An exception is raised if the buffer is released.

int PyPickleBuffer_Release(PyObject *picklebuf)

  Release the PickleBuffer instance's underlying buffer.

Buffer requirements

PickleBuffer can wrap any kind of buffer, including non-contiguous
buffers. However, it is required that __reduce__ only returns a
contiguous PickleBuffer (contiguity here is meant in the PEP 3118 sense:
either C-ordered or Fortran-ordered). Non-contiguous buffers will raise
an error when pickled.

This restriction is primarily an ease-of-implementation issue for the
pickle module but also other consumers of out-of-band buffers. The
simplest solution on the provider side is to return a contiguous copy of
a non-contiguous buffer; a sophisticated provider, though, may decide
instead to return a sequence of contiguous sub-buffers.

Consumer API

pickle.Pickler.__init__ and pickle.dumps are augmented with an
additional buffer_callback parameter:

    class Pickler:
       def __init__(self, file, protocol=None, ..., buffer_callback=None):
          """
          If *buffer_callback* is None (the default), buffer views are
          serialized into *file* as part of the pickle stream.

          If *buffer_callback* is not None, then it can be called any number
          of times with a buffer view.  If the callback returns a false value
          (such as None), the given buffer is out-of-band; otherwise the
          buffer is serialized in-band, i.e. inside the pickle stream.

          The callback should arrange to store or transmit out-of-band buffers
          without changing their order.

          It is an error if *buffer_callback* is not None and *protocol* is
          None or smaller than 5.
          """

    def pickle.dumps(obj, protocol=None, *, ..., buffer_callback=None):
       """
       See above for *buffer_callback*.
       """

pickle.Unpickler.__init__ and pickle.loads are augmented with an
additional buffers parameter:

    class Unpickler:
       def __init__(file, *, ..., buffers=None):
          """
          If *buffers* is not None, it should be an iterable of buffer-enabled
          objects that is consumed each time the pickle stream references
          an out-of-band buffer view.  Such buffers have been given in order
          to the *buffer_callback* of a Pickler object.

          If *buffers* is None (the default), then the buffers are taken
          from the pickle stream, assuming they are serialized there.
          It is an error for *buffers* to be None if the pickle stream
          was produced with a non-None *buffer_callback*.
          """

    def pickle.loads(data, *, ..., buffers=None):
       """
       See above for *buffers*.
       """

Protocol changes

Three new opcodes are introduced:

-   BYTEARRAY8 creates a bytearray from the data following it in the
    pickle stream and pushes it on the stack (just like BINBYTES8 does
    for bytes objects);
-   NEXT_BUFFER fetches a buffer from the buffers iterable and pushes it
    on the stack.
-   READONLY_BUFFER makes a readonly view of the top of the stack.

When pickling encounters a PickleBuffer, that buffer can be considered
in-band or out-of-band depending on the following conditions:

-   if no buffer_callback is given, the buffer is in-band;
-   if a buffer_callback is given, it is called with the buffer. If the
    callback returns a true value, the buffer is in-band; if the
    callback returns a false value, the buffer is out-of-band.

An in-band buffer is serialized as follows:

-   If the buffer is writable, it is serialized into the pickle stream
    as if it were a bytearray object.
-   If the buffer is readonly, it is serialized into the pickle stream
    as if it were a bytes object.

An out-of-band buffer is serialized as follows:

-   If the buffer is writable, a NEXT_BUFFER opcode is appended to the
    pickle stream.
-   If the buffer is readonly, a NEXT_BUFFER opcode is appended to the
    pickle stream, followed by a READONLY_BUFFER opcode.

The distinction between readonly and writable buffers is motivated below
(see "Mutability").

Side effects

Improved in-band performance

Even in-band pickling can be improved by returning a PickleBuffer
instance from __reduce_ex__, as one copy is avoided on the serialization
path[4][5].

Caveats

Mutability

PEP 3118 buffers can be readonly or writable. Some objects, such as
Numpy arrays, need to be backed by a mutable buffer for full operation.
Pickle consumers that use the buffer_callback and buffers arguments will
have to be careful to recreate mutable buffers. When doing I/O, this
implies using buffer-passing API variants such as readinto (which are
also often preferable for performance).

Data sharing

If you pickle and then unpickle an object in the same process, passing
out-of-band buffer views, then the unpickled object may be backed by the
same buffer as the original pickled object.

For example, it might be reasonable to implement reduction of a Numpy
array as follows (crucial metadata such as shapes is omitted for
simplicity):

    class ndarray:

       def __reduce_ex__(self, protocol):
          if protocol == 5:
             return numpy.frombuffer, (PickleBuffer(self), self.dtype)
          # Legacy code for earlier protocols omitted

Then simply passing the PickleBuffer around from dumps to loads will
produce a new Numpy array sharing the same underlying memory as the
original Numpy object (and, incidentally, keeping it alive):

    >>> import numpy as np
    >>> a = np.zeros(10)
    >>> a[0]
    0.0
    >>> buffers = []
    >>> data = pickle.dumps(a, protocol=5, buffer_callback=buffers.append)
    >>> b = pickle.loads(data, buffers=buffers)
    >>> b[0] = 42
    >>> a[0]
    42.0

This won't happen with the traditional pickle API (i.e. without passing
buffers and buffer_callback parameters), because then the buffer view is
serialized inside the pickle stream with a copy.

Rejected alternatives

Using the existing persistent load interface

The pickle persistence interface is a way of storing references to
designated objects in the pickle stream while handling their actual
serialization out of band. For example, one might consider the following
for zero-copy serialization of bytearrays:

    class MyPickle(pickle.Pickler):

        def __init__(self, *args, **kwargs):
            super().__init__(*args, **kwargs)
            self.buffers = []

        def persistent_id(self, obj):
            if type(obj) is not bytearray:
                return None
            else:
                index = len(self.buffers)
                self.buffers.append(obj)
                return ('bytearray', index)

    class MyUnpickle(pickle.Unpickler):

        def __init__(self, *args, buffers, **kwargs):
            super().__init__(*args, **kwargs)
            self.buffers = buffers

        def persistent_load(self, pid):
            type_tag, index = pid
            if type_tag == 'bytearray':
                return self.buffers[index]
            else:
                assert 0  # unexpected type

This mechanism has two drawbacks:

-   Each pickle consumer must reimplement Pickler and Unpickler
    subclasses, with custom code for each type of interest. Essentially,
    N pickle consumers end up each implementing custom code for M
    producers. This is difficult (especially for sophisticated types
    such as Numpy arrays) and poorly scalable.

-   Each object encountered by the pickle module (even simple built-in
    objects such as ints and strings) triggers a call to the user's
    persistent_id() method, leading to a possible performance drop
    compared to nominal.

    (the Python 2 cPickle module supported an undocumented
    inst_persistent_id() hook that was only called on non-built-in
    types; it was added in 1997 in order to alleviate the performance
    issue of calling persistent_id, presumably at ZODB's request)

Passing a sequence of buffers in buffer_callback

By passing a sequence of buffers, rather than a single buffer, we would
potentially save on function call overhead in case a large number of
buffers are produced during serialization. This would need additional
support in the Pickler to save buffers before calling the callback.
However, it would also prevent the buffer callback from returning a
boolean to indicate whether a buffer is to be serialized in-band or
out-of-band.

We consider that having a large number of buffers to serialize is an
unlikely case, and decided to pass a single buffer to the buffer
callback.

Allow serializing a PickleBuffer in protocol 4 and earlier

If we were to allow serializing a PickleBuffer in protocols 4 and
earlier, it would actually make a supplementary memory copy when the
buffer is mutable. Indeed, a mutable PickleBuffer would serialize as a
bytearray object in those protocols (that is a first copy), and
serializing the bytearray object would call bytearray.__reduce_ex__
which returns a bytes object (that is a second copy).

To prevent __reduce__ implementors from introducing involuntary
performance regressions, we decided to reject PickleBuffer when the
protocol is smaller than 5. This forces implementors to switch to
__reduce_ex__ and implement protocol-dependent serialization, taking
advantage of the best path for each protocol (or at least treat protocol
5 and upwards separately from protocols 4 and downwards).

Implementation

The PEP was initially implemented in the author's GitHub fork[6]. It was
later merged into Python 3.8[7].

A backport for Python 3.6 and 3.7 is downloadable from PyPI [8].

Support for pickle protocol 5 and out-of-band buffers was added to Numpy
[9].

Support for pickle protocol 5 and out-of-band buffers was added to the
Apache Arrow Python bindings[10].

Related work

Dask.distributed implements a custom zero-copy serialization with
fallback to pickle[11].

PyArrow implements zero-copy component-based serialization for a few
selected types[12].

PEP 554 proposes hosting multiple interpreters in a single process, with
provisions for transferring buffers between interpreters as a
communication scheme.

Acknowledgements

Thanks to the following people for early feedback: Alyssa Coghlan,
Olivier Grisel, Stefan Krah, MinRK, Matt Rocklin, Eric Snow.

Thanks to Pierre Glaser and Olivier Grisel for experimenting with the
implementation.

References

Copyright

This document has been placed into the public domain.

[1] Dask.distributed -- A lightweight library for distributed computing
in Python https://distributed.readthedocs.io/

[2] PyArrow -- A cross-language development platform for in-memory data
https://arrow.apache.org/docs/python/

[3] IPyParallel -- Using IPython for parallel computing
https://ipyparallel.readthedocs.io/

[4] Benchmark zero-copy pickling in Apache Arrow
https://github.com/apache/arrow/pull/2161#issuecomment-407859213

[5] Benchmark pickling Numpy arrays with different pickle protocols
https://github.com/numpy/numpy/issues/11161#issuecomment-424035962

[6] pickle5 branch on GitHub
https://github.com/pitrou/cpython/tree/pickle5

[7] PEP 574 Pull Request on GitHub
https://github.com/python/cpython/pull/7076

[8] pickle5 project on PyPI https://pypi.org/project/pickle5/

[9] Pull request: Support pickle protocol 5 in Numpy
https://github.com/numpy/numpy/pull/12011

[10] Pull request: Experimental zero-copy pickling in Apache Arrow
https://github.com/apache/arrow/pull/2161

[11] Dask.distributed custom serialization
https://distributed.readthedocs.io/en/latest/serialization.html

[12] PyArrow IPC and component-based serialization
https://arrow.apache.org/docs/python/ipc.html#component-based-serialization