text:
PEP: 401 Title: BDFL Retirement Version: $Revision$ Last-Modified:
$Date: 2009-04-01 00:00:00 -0400 (Wed, 1 Apr 2009)$ Author: Barry
Warsaw, Brett Cannon Status: April Fool! Type: Process Content-Type:
text/x-rst Created: 01-Apr-2009 Post-History: 01-Apr-2009

Abstract

The BDFL, having shepherded Python development for 20 years, officially
announces his retirement, effective immediately. Following a unanimous
vote, his replacement is named.

Rationale

Guido wrote the original implementation of Python in 1989, and after
nearly 20 years of leading the community, has decided to step aside as
its Benevolent Dictator For Life. His official title is now Benevolent
Dictator Emeritus Vacationing Indefinitely from the Language (BDEVIL).
Guido leaves Python in the good hands of its new leader and its vibrant
community, in order to train for his lifelong dream of climbing Mount
Everest.

After unanimous vote of the Python Steering Union (not to be confused
with the Python Secret Underground, which emphatically does not exist)
at the 2009 Python Conference (PyCon 2009), Guido's successor has been
chosen: Barry Warsaw, or as he is affectionately known, Uncle Barry.
Uncle Barry's official title is Friendly Language Uncle For Life
(FLUFL).

Official Acts of the FLUFL

FLUFL Uncle Barry enacts the following decisions, in order to
demonstrate his intention to lead the community in the same responsible
and open manner as his predecessor, whose name escapes him:

-   Recognized that the selection of Hg as the DVCS of choice was clear
    proof of the onset of the BDEVIL's insanity, and reverting this
    decision to switch to Bzr instead, the only true choice.
-   Recognized that the != inequality operator in Python 3.0 was a
    horrible, finger pain inducing mistake, the FLUFL reinstates the <>
    diamond operator as the sole spelling. This change is important
    enough to be implemented for, and released in Python 3.1. To help
    transition to this feature, a new future statement,
    from __future__ import barry_as_FLUFL has been added.
-   Recognized that the print function in Python 3.0 was a horrible,
    pain-inducing mistake, the FLUFL reinstates the print statement.
    This change is important enough to be implemented for, and released
    in Python 3.0.2.
-   Recognized that the disappointing adoption curve of Python 3.0
    signals its abject failure, all work on Python 3.1 and subsequent
    Python 3.x versions is hereby terminated. All features in Python 3.0
    shall be back ported to Python 2.7 which will be the official and
    sole next release. The Python 3.0 string and bytes types will be
    back ported to Python 2.6.2 for the convenience of developers.
-   Recognized that C is a 20th-century language with almost universal
    rejection by programmers under the age of 30, the CPython
    implementation will terminate with the release of Python 2.6.2 and
    3.0.2. Thereafter, the reference implementation of Python will
    target the Parrot[1] virtual machine. Alternative implementations of
    Python (e.g. Jython[2], IronPython[3], and PyPy[4]) are officially
    discouraged but tolerated.
-   Recognized that the Python Software Foundation[5] having fulfilled
    its mission admirably, is hereby disbanded. The Python Steering
    Union[6] (not to be confused with the Python Secret Underground,
    which emphatically does not exist), is now the sole steward for all
    of Python's intellectual property. All PSF funds are hereby
    transferred to the PSU (not that PSU, the other PSU).

References

Copyright

This document is the property of the Python Steering Union (not to be
confused with the Python Secret Underground, which emphatically does not
exist). We suppose it's okay for you to read this, but don't even think
about quoting, copying, modifying, or distributing it.

[1] http://www.parrot.org

[2] http://www.jython.org

[3] http://www.ironpython.com

[4] http://www.codespeak.net/pypy

[5] http://www.python.org/psf

[6] http://www.pythonlabs.com

authors:
Barry Warsaw
Brett Cannon


text:
PEP: 601 Title: Forbid return/break/continue breaking out of finally
Author: Damien George, Batuhan Taskaya Sponsor: Alyssa Coghlan
Discussions-To:
https://discuss.python.org/t/pep-601-forbid-return-break-continue-breaking-out-of-finally/2239
Status: Rejected Type: Standards Track Content-Type: text/x-rst Created:
26-Aug-2019 Python-Version: 3.8 Post-History: 26-Aug-2019, 23-Sep-2019
Resolution:
https://discuss.python.org/t/pep-601-forbid-return-break-continue-breaking-out-of-finally/2239/32

Rejection Note

This PEP was rejected by the Steering Council by a vote of 4/4.

Guido's arguments for rejecting the PEP are: "it seems to me that most
languages implement this kind of construct but have style guides and/or
linters that reject it. I would support a proposal to add this to PEP
8", and "I note that the toy examples are somewhat misleading – the
functionality that may be useful is a conditional return (or break etc.)
inside a finally block.".

Abstract

This PEP proposes to forbid return, break and continue statements within
a finally suite where they would break out of the finally. Their use in
such a location silently cancels any active exception being raised
through the finally, leading to unclear code and possible bugs.

continue is currently not supported in a finally in Python 3.7 (due to
implementation issues) and the proposal is to not add support for it in
Python 3.8. For return and break the proposal is to deprecate their use
in Python 3.9, emit a compilation warning in Python 3.10 and then forbid
their use after that.

Motivation

The use of return, break and continue within a finally suite leads to
behaviour which is not at all obvious. Consider the following function:

    def foo():
        try:
            foo()
        finally:
            return

This will return cleanly (without an exception) even though it has
infinite recursion and raises an exception within the try. The reason is
that the return within the finally will silently cancel any exception
that propagates through the finally suite. Such behaviour is unexpected
and not at all obvious. This function is equivalent to:

    def foo():
        try:
            foo()
        except:
            pass
        return

break and continue have similar behaviour (they silence exceptions) if
they jump to code outside the finally suite. For example:

    def bar():
        while True:
            try:
                1 / 0
            finally:
                break

This behaviour goes against the following parts of The Zen of Python:

-   Explicit is better than implicit - exceptions are implicitly
    silenced
-   Readability counts - the intention of the code is not obvious
-   Errors should never pass silently; Unless explicitly silenced -
    exceptions are implicitly silenced

If this behaviour of silencing exceptions is really needed then the
explicit form of a try-except can be used instead, and this makes the
code clearer.

Independent to the semantics, implementing return/break/continue within
a finally suite is non-trivial as it requires to correctly track any
active exceptions at runtime (an executing finally suite may or may not
have an active exception) and cancel them as appropriate. CPython did
have a bug in this for the case of continue and so originally disallowed
it[1]. Requiring correct behaviour for return/break/continue within a
finally puts an unnecessary burden on alternative implementations of
Python.

Other languages

Java allows to return from within a finally block, but its use is
discouraged according to[2],[3],[4]. The Java compiler later on included
a linting option -Xlint:finally to warn against the use of return within
a finally block. The Eclipse editor also warns about this use.

Ruby allows return from inside ensure (Python's finally), but it should
be an explicit return. It is discouraged and handled by linters[5],[6].

Like Ruby, JavaScript also allows use of return/break/continue within a
finally but it is seen as unsafe and it is handled by eslint[7].

C# forbids the use of ending statements like return/goto/break within a
finally [8],[9].

Rationale

Since the behaviour of return/break/continue within a finally is
unclear, the pattern is rarely used, and there is a simple alternative
to writing equivalent code (which is more explicit), forbidding the
syntax is the most straightforward approach.

Specification

This is a change to the compiler, not the grammar. The compiler should
check for the following in a finally suite:

-   A return in any statement, at any level of nesting.
-   A break/continue in any statement, at any level of nesting, that
    would transfer control flow outside the finally suite.

Upon finding such a case it should emit the appropriate exception:

-   For continue, a SyntaxError (this is the current behaviour of 3.7).
-   For return/break, a SyntaxWarning in 3.10, and a SyntaxError after
    that.

For example, the following are all forbidden by this proposal:

    def f():
        try:
            pass
        finally:
            return

    def g():
        try:
            pass
        finally:
            try:
                return
            finally:
                pass

    def h():
        try:
            pass
        finally:
            try:
                pass
            finally:
                for x in range(10):
                    return

The following is still allowed because the continue doesn't escape the
finally:

    try:
        pass
    finally:
        for x in range(10):
            continue

Note that yielding from within a finally remains acceptable by this PEP
because resuming the generator will resume the finally and eventually
raise any active exceptions (so they are never silenced by yielding).

Backwards Compatibility

This is a backwards incompatible change, for return and break.

The following locations in the CPython standard library (at
v3.8.0b1-651-g7fcc2088a5) use return within finally:

-   Lib/subprocess.py:921 - the use here looks like a bug
-   Lib/multiprocessing/connection.py:316 - the use here looks
    legitimate but the intention is not clear
-   Lib/multiprocessing/connection.py:318 - the use here looks
    legitimate but the intention is not clear
-   Lib/test/test_sys_settrace.py:837 - a test for return within finally
-   Lib/test/test_sys_settrace.py:1346 - a test for return within
    finally

There are no uses of break within a finally (that break out of the
finally) in the standard library.

Security Implications

This is a simplification of the language, and removal of associated
code, so should not introduce any new paths for a security exploit.

How to Teach This

This feature is very rarely used so forbidding it will likely only
impact advanced users, not beginners and probably not any existing
teaching material. Since this is the removal of a feature teaching users
will be one by the raising of a SyntaxError if/when the forbidden
feature is used.

Reference Implementation

There is currently no reference implementation, although the way
continue is currently handled in a finally (raising a SyntaxError) can
be extended to return and break.

References

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

[1] https://github.com/python/cpython/issues/82011

[2] https://stackoverflow.com/questions/48088/returning-from-a-finally-block-in-java

[3] https://web.archive.org/web/20070922061412/http://weblogs.java.net/blog/staufferjames/archive/2007/06/_dont_return_in.html

[4] https://wiki.sei.cmu.edu/confluence/display/java/ERR04-J.+Do+not+complete+abruptly+from+a+finally+block

[5] https://github.com/rubocop/rubocop/issues/5949

[6] https://www.rubydoc.info/gems/rubocop/0.74.0/RuboCop/Cop/Lint/EnsureReturn

[7] https://eslint.org/docs/rules/no-unsafe-finally

[8] https://social.msdn.microsoft.com/Forums/vstudio/en-US/87faf259-3c54-4f3a-8d2b-ff82de44992f/return-statement-in-finally-block?forum=netfxbcl

[9] https://stackoverflow.com/a/5788268

license: Public Domain

authors:
Batuhan Taskaya
Damien George


text:
PEP: 3122 Title: Delineation of the main module Version: $Revision$
Last-Modified: $Date$ Author: Brett Cannon Status: Rejected Type:
Standards Track Content-Type: text/x-rst Created: 27-Apr-2007
Post-History:

Attention

This PEP has been rejected. Guido views running scripts within a package
as an anti-pattern[1].

Abstract

Because of how name resolution works for relative imports in a world
where PEP 328 is implemented, the ability to execute modules within a
package ceases being possible. This failing stems from the fact that the
module being executed as the "main" module replaces its __name__
attribute with "__main__" instead of leaving it as the absolute name of
the module. This breaks import's ability to resolve relative imports
from the main module into absolute names.

In order to resolve this issue, this PEP proposes to change how the main
module is delineated. By leaving the __name__ attribute in a module
alone and setting sys.main to the name of the main module this will
allow at least some instances of executing a module within a package
that uses relative imports.

This PEP does not address the idea of introducing a module-level
function that is automatically executed like PEP 299 proposes.

The Problem

With the introduction of PEP 328, relative imports became dependent on
the __name__ attribute of the module performing the import. This is
because the use of dots in a relative import are used to strip away
parts of the calling module's name to calculate where in the package
hierarchy an import should fall (prior to PEP 328 relative imports could
fail and would fall back on absolute imports which had a chance of
succeeding).

For instance, consider the import from .. import spam made from the
bacon.ham.beans module (bacon.ham.beans is not a package itself, i.e.,
does not define __path__). Name resolution of the relative import takes
the caller's name (bacon.ham.beans), splits on dots, and then slices off
the last n parts based on the level (which is 2). In this example both
ham and beans are dropped and spam is joined with what is left (bacon).
This leads to the proper import of the module bacon.spam.

This reliance on the __name__ attribute of a module when handling
relative imports becomes an issue when executing a script within a
package. Because the executing script has its name set to '__main__',
import cannot resolve any relative imports, leading to an ImportError.

For example, assume we have a package named bacon with an __init__.py
file containing:

    from . import spam

Also create a module named spam within the bacon package (it can be an
empty file). Now if you try to execute the bacon package (either through
python bacon/__init__.py or python -m bacon) you will get an ImportError
about trying to do a relative import from within a non-package.
Obviously the import is valid, but because of the setting of __name__ to
'__main__' import thinks that bacon/__init__.py is not in a package
since no dots exist in __name__. To see how the algorithm works in more
detail, see importlib.Import._resolve_name() in the sandbox [2].

Currently a work-around is to remove all relative imports in the module
being executed and make them absolute. This is unfortunate, though, as
one should not be required to use a specific type of resource in order
to make a module in a package be able to be executed.

The Solution

The solution to the problem is to not change the value of __name__ in
modules. But there still needs to be a way to let executing code know it
is being executed as a script. This is handled with a new attribute in
the sys module named main.

When a module is being executed as a script, sys.main will be set to the
name of the module. This changes the current idiom of:

    if __name__ == '__main__':
        ...

to:

    import sys
    if __name__ == sys.main:
        ...

The newly proposed solution does introduce an added line of boilerplate
which is a module import. But as the solution does not introduce a new
built-in or module attribute (as discussed in Rejected Ideas) it has
been deemed worth the extra line.

Another issue with the proposed solution (which also applies to all
rejected ideas as well) is that it does not directly solve the problem
of discovering the name of a file. Consider python bacon/spam.py. By the
file name alone it is not obvious whether bacon is a package. In order
to properly find this out both the current direction must exist on
sys.path as well as bacon/__init__.py existing.

But this is the simple example. Consider python ../spam.py. From the
file name alone it is not at all clear if spam.py is in a package or
not. One possible solution is to find out what the absolute name of ..,
check if a file named __init__.py exists, and then look if the directory
is on sys.path. If it is not, then continue to walk up the directory
until no more __init__.py files are found or the directory is found on
sys.path.

This could potentially be an expensive process. If the package depth
happens to be deep then it could require a large amount of disk access
to discover where the package is anchored on sys.path, if at all. The
stat calls alone can be expensive if the file system the executed script
is on is something like NFS.

Because of these issues, only when the -m command-line argument
(introduced by PEP 338) is used will __name__ be set. Otherwise the
fallback semantics of setting __name__ to "__main__" will occur.
sys.main will still be set to the proper value, regardless of what
__name__ is set to.

Implementation

When the -m option is used, sys.main will be set to the argument passed
in. sys.argv will be adjusted as it is currently. Then the equivalent of
__import__(self.main) will occur. This differs from current semantics as
the runpy module fetches the code object for the file specified by the
module name in order to explicitly set __name__ and other attributes.
This is no longer needed as import can perform its normal operation in
this situation.

If a file name is specified, then sys.main will be set to "__main__".
The specified file will then be read and have a code object created and
then be executed with __name__ set to "__main__". This mirrors current
semantics.

Transition Plan

In order for Python 2.6 to be able to support both the current semantics
and the proposed semantics, sys.main will always be set to "__main__".
Otherwise no change will occur for Python 2.6. This unfortunately means
that no benefit from this change will occur in Python 2.6, but it
maximizes compatibility for code that is to work as much as possible
with 2.6 and 3.0.

To help transition to the new idiom, 2to3[3] will gain a rule to
transform the current if __name__ == '__main__': ... idiom to the new
one. This will not help with code that checks __name__ outside of the
idiom, though.

Rejected Ideas

__main__ built-in

A counter-proposal to introduce a built-in named __main__. The value of
the built-in would be the name of the module being executed (just like
the proposed sys.main). This would lead to a new idiom of:

    if __name__ == __main__:
        ...

A drawback is that the syntactic difference is subtle; the dropping of
quotes around "__main". Some believe that for existing Python
programmers bugs will be introduced where the quotation marks will be
put on by accident. But one could argue that the bug would be discovered
quickly through testing as it is a very shallow bug.

While the name of built-in could obviously be different (e.g., main) the
other drawback is that it introduces a new built-in. With a simple
solution such as sys.main being possible without adding another built-in
to Python, this proposal was rejected.

__main__ module attribute

Another proposal was to add a __main__ attribute to every module. For
the one that was executing as the main module, the attribute would have
a true value while all other modules had a false value. This has a nice
consequence of simplify the main module idiom to:

    if __main__:
        ...

The drawback was the introduction of a new module attribute. It also
required more integration with the import machinery than the proposed
solution.

Use __file__ instead of __name__

Any of the proposals could be changed to use the __file__ attribute on
modules instead of __name__, including the current semantics. The
problem with this is that with the proposed solutions there is the issue
of modules having no __file__ attribute defined or having the same value
as other modules.

The problem that comes up with the current semantics is you still have
to try to resolve the file path to a module name for the import to work.

Special string subclass for __name__ that overrides __eq__

One proposal was to define a subclass of str that overrode the __eq__
method so that it would compare equal to "__main__" as well as the
actual name of the module. In all other respects the subclass would be
the same as str.

This was rejected as it seemed like too much of a hack.

References

Copyright

This document has been placed in the public domain.

[1] Python-Dev email: "PEP to change how the main module is delineated"
(https://mail.python.org/pipermail/python-3000/2007-April/006793.html)

[2] importlib
(http://svn.python.org/view/sandbox/trunk/import_in_py/importlib.py?view=markup)
[ViewVC]

[3] 2to3 tool (http://svn.python.org/view/sandbox/trunk/2to3/) [ViewVC]


text:
PEP: 522 Title: Allow BlockingIOError in security sensitive APIs
Version: $Revision$ Last-Modified: $Date$ Author: Alyssa Coghlan
<EMAIL_ADDRESS>Nathaniel J. Smith<EMAIL_ADDRESS>Status:
Rejected Type: Standards Track Content-Type: text/x-rst Requires: 506
Created: 16-Jun-2016 Python-Version: 3.6 Resolution:
https://mail.python.org/pipermail/security-sig/2016-August/000101.html

Abstract

A number of APIs in the standard library that return random values
nominally suitable for use in security sensitive operations currently
have an obscure operating system dependent failure mode that allows them
to return values that are not, in fact, suitable for such operations.

This is due to some operating system kernels (most notably the Linux
kernel) permitting reads from /dev/urandom before the system random
number generator is fully initialized, whereas most other operating
systems will implicitly block on such reads until the random number
generator is ready.

For the lower level os.urandom and random.SystemRandom APIs, this PEP
proposes changing such failures in Python 3.6 from the current silent,
hard to detect, and hard to debug, errors to easily detected and
debugged errors by raising BlockingIOError with a suitable error
message, allowing developers the opportunity to unambiguously specify
their preferred approach for handling the situation.

For the new high level secrets API, it proposes to block implicitly if
needed whenever random number is generated by that module, as well as to
expose a new secrets.wait_for_system_rng() function to allow code
otherwise using the low level APIs to explicitly wait for the system
random number generator to be available.

This change will impact any operating system that offers the getrandom()
system call, regardless of whether the default behaviour of the
/dev/urandom device is to return potentially predictable results when
the system random number generator is not ready (e.g. Linux, NetBSD) or
to block (e.g. FreeBSD, Solaris, Illumos). Operating systems that
prevent execution of userspace code prior to the initialization of the
system random number generator, or do not offer the getrandom() syscall,
will be entirely unaffected by the proposed change (e.g. Windows, Mac OS
X, OpenBSD).

The new exception or the blocking behaviour in the secrets module would
potentially be encountered in the following situations:

-   Python code calling these APIs during Linux system initialization
-   Python code running on improperly initialized Linux systems (e.g.
    embedded hardware without adequate sources of entropy to seed the
    system random number generator, or Linux VMs that aren't configured
    to accept entropy from the VM host)

Relationship with other PEPs

This PEP depends on the Accepted PEP 506, which adds the secrets module.

This PEP competes with Victor Stinner's PEP 524, which proposes to make
os.urandom itself implicitly block when the system RNG is not ready.

PEP Rejection

For the reference implementation, Guido rejected this PEP in favour of
the unconditional implicit blocking proposal in PEP 524 (which brings
CPython's behaviour on Linux into line with its behaviour on other
operating systems).

This means any further discussion of appropriate default behaviour for
os.urandom() in system Python installations in Linux distributions
should take place on the respective distro mailing lists, rather than on
the upstream CPython mailing lists.

Changes independent of this PEP

CPython interpreter initialization and random module initialization have
already been updated to gracefully fall back to alternative seeding
options if the system random number generator is not ready.

This PEP does not compete with the proposal in PEP 524 to add an
os.getrandom() API to expose the getrandom syscall on platforms that
offer it. There is sufficient motive for adding that API in the os
module's role as a thin wrapper around potentially platform dependent
operating system features that it can be added regardless of what
happens to the default behaviour of os.urandom() on these systems.

Proposal

Changing os.urandom() on platforms with the getrandom() system call

This PEP proposes that in Python 3.6+, os.urandom() be updated to call
the getrandom() syscall in non-blocking mode if available and raise
BlockingIOError: system random number generator is not ready; see secrets.token_bytes()
if the kernel reports that the call would block.

This behaviour will then propagate through to the existing
random.SystemRandom, which provides a relatively thin wrapper around
os.urandom() that matches the random.Random() API.

However, the new secrets module introduced by PEP 506 will be updated to
catch the new exception and implicitly wait for the system random number
generator if the exception is ever encountered.

In all cases, as soon as a call to one of these security sensitive APIs
succeeds, all future calls to these APIs in that process will succeed
without blocking (once the operating system random number generator is
ready after system boot, it remains ready).

On Linux and NetBSD, this will replace the previous behaviour of
returning potentially predictable results read from /dev/urandom.

On FreeBSD, Solaris, and Illumos, this will replace the previous
behaviour of implicitly blocking until the system random number
generator is ready. However, it is not clear if these operating systems
actually allow userspace code (and hence Python) to run before the
system random number generator is ready.

Note that in all cases, if calling the underlying getrandom() API
reports ENOSYS rather than returning a successful response or reporting
EAGAIN, CPython will continue to fall back to reading from /dev/urandom
directly.

Adding secrets.wait_for_system_rng()

A new exception shouldn't be added without a straightforward
recommendation for how to resolve that error when encountered (however
rare encountering the new error is expected to be in practice). For
security sensitive code that actually does need to use the lower level
interfaces to the system random number generator (rather than the new
secrets module), and does receive live bug reports indicating this is a
real problem for the userbase of that particular application rather than
a theoretical one, this PEP's recommendation will be to add the
following snippet (directly or indirectly) to the __main__ module:

    import secrets
    secrets.wait_for_system_rng()

Or, if compatibility with versions prior to Python 3.6 is needed:

    try:
        import secrets
    except ImportError:
        pass
    else:
        secrets.wait_for_system_rng()

Within the secrets module itself, this will then be used in
token_bytes() to block implicitly if the new exception is encountered:

    def token_bytes(nbytes=None):
        if nbytes is None:
            nbytes = DEFAULT_ENTROPY
        try:
            result = os.urandom(nbytes)
        except BlockingIOError:
            wait_for_system_rng()
            result = os.urandom(nbytes)
        return result

Other parts of the module will then be updated to use token_bytes() as
their basic random number generation building block, rather than calling
os.urandom() directly.

Application frameworks covering use cases where access to the system
random number generator is almost certain to be needed (e.g. web
frameworks) may choose to incorporate a call to
secrets.wait_for_system_rng() implicitly into the commands that start
the application such that existing calls to os.urandom() will be
guaranteed to never raise the new exception when using those frameworks.

For cases where the error is encountered for an application which cannot
be modified directly, then the following command can be used to wait for
the system random number generator to initialize before starting that
application:

    python3 -c "import secrets; secrets.wait_for_system_rng()"

For example, this snippet could be added to a shell script or a systemd
ExecStartPre hook (and may prove useful in reliably waiting for the
system random number generator to be ready, even if the subsequent
command is not itself an application running under Python 3.6)

Given the changes proposed to os.urandom() above, and the inclusion of
an os.getrandom() API on systems that support it, the suggested
implementation of this function would be:

    if hasattr(os, "getrandom"):
        # os.getrandom() always blocks waiting for the system RNG by default
        def wait_for_system_rng():
            """Block waiting for system random number generator to be ready"""
            os.getrandom(1)
            return
    else:
       # As far as we know, other platforms will never get BlockingIOError
       # below but the implementation makes pessimistic assumptions
        def wait_for_system_rng():
            """Block waiting for system random number generator to be ready"""
            # If the system RNG is already seeded, don't wait at all
            try:
                os.urandom(1)
                return
            except BlockingIOError:
                pass
            # Avoid the below busy loop if possible
            try:
                block_on_system_rng = open("/dev/random", "rb")
            except FileNotFoundError:
                pass
            else:
                with block_on_system_rng:
                    block_on_system_rng.read(1)
            # Busy loop until the system RNG is ready
            while True:
                try:
                    os.urandom(1)
                    break
                except BlockingIOError:
                    # Only check once per millisecond
                    time.sleep(0.001)

On systems where it is possible to wait for the system RNG to be ready,
this function will do so without a busy loop if os.getrandom() is
defined, os.urandom() itself implicitly blocks, or the /dev/random
device is available. If the system random number generator is ready,
this call is guaranteed to never block, even if the system's /dev/random
device uses a design that permits it to block intermittently during
normal system operation.

Limitations on scope

No changes are proposed for Windows or Mac OS X systems, as neither of
those platforms provides any mechanism to run Python code before the
operating system random number generator has been initialized. Mac OS X
goes so far as to kernel panic and abort the boot process if it can't
properly initialize the random number generator (although Apple's
restrictions on the supported hardware platforms make that exceedingly
unlikely in practice).

Similarly, no changes are proposed for other *nix systems that do not
offer the getrandom() syscall. On these systems, os.urandom() will
continue to block waiting for the system random number generator to be
initialized.

While other *nix systems that offer a non-blocking API (other than
getrandom()) for requesting random numbers suitable for use in security
sensitive applications could potentially receive a similar update to the
one proposed for getrandom() in this PEP, such changes are out of scope
for this particular proposal.

Python's behaviour on older versions of affected platforms that do not
offer the new getrandom() syscall will also remain unchanged.

Rationale

Ensuring the secrets module implicitly blocks when needed

This is done to help encourage the meme that arises for folks that want
the simplest possible answer to the right way to generate security
sensitive random numbers to be "Use the secrets module when available or
your application might crash unexpectedly", rather than the more
boilerplate heavy "Always call secrets.wait_for_system_rng() when
available or your application might crash unexpectedly".

It's also done due to the BDFL having a higher tolerance for APIs that
might block unexpectedly than he does for APIs that might throw an
unexpected exception[1].

Raising BlockingIOError in os.urandom() on Linux

For several years now, the security community's guidance has been to use
os.urandom() (or the random.SystemRandom() wrapper) when implementing
security sensitive operations in Python.

To help improve API discoverability and make it clearer that secrecy and
simulation are not the same problem (even though they both involve
random numbers), PEP 506 collected several of the one line recipes based
on the lower level os.urandom() API into a new secrets module.

However, this guidance has also come with a longstanding caveat:
developers writing security sensitive software at least for Linux, and
potentially for some other *BSD systems, may need to wait until the
operating system's random number generator is ready before relying on it
for security sensitive operations. This generally only occurs if
os.urandom() is read very early in the system initialization process, or
on systems with few sources of available entropy (e.g. some kinds of
virtualized or embedded systems), but unfortunately the exact conditions
that trigger this are difficult to predict, and when it occurs then
there is no direct way for userspace to tell it has happened without
querying operating system specific interfaces.

On *BSD systems (if the particular *BSD variant allows the problem to
occur at all) and potentially also Solaris and Illumos, encountering
this situation means os.urandom() will either block waiting for the
system random number generator to be ready (the associated symptom would
be for the affected script to pause unexpectedly on the first call to
os.urandom()) or else will behave the same way as it does on Linux.

On Linux, in Python versions up to and including Python 3.4, and in
Python 3.5 maintenance versions following Python 3.5.2, there's no clear
indicator to developers that their software may not be working as
expected when run early in the Linux boot process, or on hardware
without good sources of entropy to seed the operating system's random
number generator: due to the behaviour of the underlying /dev/urandom
device, os.urandom() on Linux returns a result either way, and it takes
extensive statistical analysis to show that a security vulnerability
exists.

By contrast, if BlockingIOError is raised in those situations, then
developers using Python 3.6+ can easily choose their desired behaviour:

1.  Wait for the system RNG at or before application startup (security
    sensitive)
2.  Switch to using the random module (non-security sensitive)

Making secrets.wait_for_system_rng() public

Earlier versions of this PEP proposed a number of recipes for wrapping
os.urandom() to make it suitable for use in security sensitive use
cases.

Discussion of the proposal on the security-sig mailing list prompted the
realization[2] that the core assumption driving the API design in this
PEP was that choosing between letting the exception cause the
application to fail, blocking waiting for the system RNG to be ready and
switching to using the random module instead of os.urandom is an
application and use-case specific decision that should take into account
application and use-case specific details.

There is no way for the interpreter runtime or support libraries to
determine whether a particular use case is security sensitive or not,
and while it's straightforward for application developer to decide how
to handle an exception thrown by a particular API, they can't readily
workaround an API blocking when they expected it to be non-blocking.

Accordingly, the PEP was updated to add secrets.wait_for_system_rng() as
an API for applications, scripts and frameworks to use to indicate that
they wanted to ensure the system RNG was available before continuing,
while library developers could continue to call os.urandom() without
worrying that it might unexpectedly start blocking waiting for the
system RNG to be available.

Backwards Compatibility Impact Assessment

Similar to PEP 476, this is a proposal to turn a previously silent
security failure into a noisy exception that requires the application
developer to make an explicit decision regarding the behaviour they
desire.

As no changes are proposed for operating systems that don't provide the
getrandom() syscall, os.urandom() retains its existing behaviour as a
nominally blocking API that is non-blocking in practice due to the
difficulty of scheduling Python code to run before the operating system
random number generator is ready. We believe it may be possible to
encounter problems akin to those described in this PEP on at least some
*BSD variants, but nobody has explicitly demonstrated that. On Mac OS X
and Windows, it appears to be straight up impossible to even try to run
a Python interpreter that early in the boot process.

On Linux and other platforms with similar /dev/urandom behaviour,
os.urandom() retains its status as a guaranteed non-blocking API.
However, the means of achieving that status changes in the specific case
of the operating system random number generator not being ready for use
in security sensitive operations: historically it would return
potentially predictable random data, with this PEP it would change to
raise BlockingIOError.

Developers of affected applications would then be required to make one
of the following changes to gain forward compatibility with Python 3.6,
based on the kind of application they're developing.

Unaffected Applications

The following kinds of applications would be entirely unaffected by the
change, regardless of whether or not they perform security sensitive
operations:

-   applications that don't support Linux
-   applications that are only run on desktops or conventional servers
-   applications that are only run after the system RNG is ready
    (including those where an application framework calls
    secrets.wait_for_system_rng() on their behalf)

Applications in this category simply won't encounter the new exception,
so it will be reasonable for developers to wait and see if they receive
Python 3.6 compatibility bugs related to the new runtime behaviour,
rather than attempting to pre-emptively determine whether or not they're
affected.

Affected security sensitive applications

Security sensitive applications would need to either change their system
configuration so the application is only started after the operating
system random number generator is ready for security sensitive
operations, change the application startup code to invoke
secrets.wait_for_system_rng(), or else switch to using the new
secrets.token_bytes() API.

As an example for components started via a systemd unit file, the
following snippet would delay activation until the system RNG was ready:

  ExecStartPre=python3 -c "import secrets;
  secrets.wait_for_system_rng()"

Alternatively, the following snippet will use secrets.token_bytes() if
available, and fall back to os.urandom() otherwise:

  try:

      import secrets.token_bytes as _get_random_bytes

  except ImportError:

      import os.urandom as _get_random_bytes

Affected non-security sensitive applications

Non-security sensitive applications should be updated to use the random
module rather than os.urandom:

    def pseudorandom_bytes(num_bytes):
        return random.getrandbits(num_bytes*8).to_bytes(num_bytes, "little")

Depending on the details of the application, the random module may offer
other APIs that can be used directly, rather than needing to emulate the
raw byte sequence produced by the os.urandom() API.

Additional Background

Why propose this now?

The main reason is because the Python 3.5.0 release switched to using
the new Linux getrandom() syscall when available in order to avoid
consuming a file descriptor[3], and this had the side effect of making
the following operations block waiting for the system random number
generator to be ready:

-   os.urandom (and APIs that depend on it)
-   importing the random module
-   initializing the randomized hash algorithm used by some builtin
    types

While the first of those behaviours is arguably desirable (and
consistent with the existing behaviour of os.urandom on other operating
systems), the latter two behaviours are unnecessary and undesirable, and
the last one is now known to cause a system level deadlock when
attempting to run Python scripts during the Linux init process with
Python 3.5.0 or 3.5.1[4], while the second one can cause problems when
using virtual machines without robust entropy sources configured[5].

Since decoupling these behaviours in CPython will involve a number of
implementation changes more appropriate for a feature release than a
maintenance release, the relatively simple resolution applied in Python
3.5.2 was to revert all three of them to a behaviour similar to that of
previous Python versions: if the new Linux syscall indicates it will
block, then Python 3.5.2 will implicitly fall back on reading
/dev/urandom directly[6].

However, this bug report also resulted in a range of proposals to add
new APIs like os.getrandom()[7], os.urandom_block()[8],
os.pseudorandom() and os.cryptorandom()[9], or adding new optional
parameters to os.urandom() itself[10], and then attempting to educate
users on when they should call those APIs instead of just using a plain
os.urandom() call.

These proposals arguably represent overreactions, as the question of
reliably obtaining random numbers suitable for security sensitive work
on Linux is a relatively obscure problem of interest mainly to operating
system developers and embedded systems programmers, that may not justify
expanding the Python standard library's cross-platform APIs with new
Linux-specific concerns. This is especially so with the secrets module
already being added as the "use this and don't worry about the low level
details" option for developers writing security sensitive software that
for some reason can't rely on even higher level domain specific APIs
(like web frameworks) and also don't need to worry about Python versions
prior to Python 3.6.

That said, it's also the case that low cost ARM devices are becoming
increasingly prevalent, with a lot of them running Linux, and a lot of
folks writing Python applications that run on those devices. That
creates an opportunity to take an obscure security problem that
currently requires a lot of knowledge about Linux boot processes and
provably unpredictable random number generation to diagnose and resolve,
and instead turn it into a relatively mundane and
easy-to-find-in-an-internet-search runtime exception.

The cross-platform behaviour of os.urandom()

On operating systems other than Linux and NetBSD, os.urandom() may
already block waiting for the operating system's random number generator
to be ready. This will happen at most once in the lifetime of the
process, and the call is subsequently guaranteed to be non-blocking.

Linux and NetBSD are outliers in that, even when the operating system's
random number generator doesn't consider itself ready for use in
security sensitive operations, reading from the /dev/urandom device will
return random values based on the entropy it has available.

This behaviour is potentially problematic, so Linux 3.17 added a new
getrandom() syscall that (amongst other benefits) allows callers to
either block waiting for the random number generator to be ready, or
else request an error return if the random number generator is not
ready. Notably, the new API does not support the old behaviour of
returning data that is not suitable for security sensitive use cases.

Versions of Python prior up to and including Python 3.4 access the Linux
/dev/urandom device directly.

Python 3.5.0 and 3.5.1 (when build on a system that offered the new
syscall) called getrandom() in blocking mode in order to avoid the use
of a file descriptor to access /dev/urandom. While there were no
specific problems reported due to os.urandom() blocking in user code,
there were problems due to CPython implicitly invoking the blocking
behaviour during interpreter startup and when importing the random
module.

Rather than trying to decouple SipHash initialization from the
os.urandom() implementation, Python 3.5.2 switched to calling
getrandom() in non-blocking mode, and falling back to reading from
/dev/urandom if the syscall indicates it will block.

As a result of the above, os.urandom() in all Python versions up to and
including Python 3.5 propagate the behaviour of the underling
/dev/urandom device to Python code.

Problems with the behaviour of /dev/urandom on Linux

The Python os module has largely co-evolved with Linux APIs, so having
os module functions closely follow the behaviour of their Linux
operating system level counterparts when running on Linux is typically
considered to be a desirable feature.

However, /dev/urandom represents a case where the current behaviour is
acknowledged to be problematic, but fixing it unilaterally at the kernel
level has been shown to prevent some Linux distributions from booting
(at least in part due to components like Python currently using it for
non-security-sensitive purposes early in the system initialization
process).

As an analogy, consider the following two functions:

    def generate_example_password():
        """Generates passwords solely for use in code examples"""
        return generate_unpredictable_password()

    def generate_actual_password():
        """Generates actual passwords for use in real applications"""
        return generate_unpredictable_password()

If you think of an operating system's random number generator as a
method for generating unpredictable, secret passwords, then you can
think of Linux's /dev/urandom as being implemented like:

    # Oversimplified artist's conception of the kernel code
    # implementing /dev/urandom
    def generate_unpredictable_password():
        if system_rng_is_ready:
            return use_system_rng_to_generate_password()
        else:
            # we can't make an unpredictable password; silently return a
            # potentially predictable one instead:
            return "p4ssw0rd"

In this scenario, the author of generate_example_password is fine - even
if "p4ssw0rd" shows up a bit more often than they expect, it's only used
in examples anyway. However, the author of generate_actual_password has
a problem - how do they prove that their calls to
generate_unpredictable_password never follow the path that returns a
predictable answer?

In real life it's slightly more complicated than this, because there
might be some level of system entropy available -- so the fallback might
be more like return random.choice(["p4ssword", "passw0rd", "p4ssw0rd"])
or something even more variable and hence only statistically predictable
with better odds than the author of generate_actual_password was
expecting. This doesn't really make things more provably secure, though;
mostly it just means that if you try to catch the problem in the obvious
way --if returned_password == "p4ssw0rd": raise UhOh -- then it doesn't
work, because returned_password might instead be p4ssword or even
pa55word, or just an arbitrary 64 bit sequence selected from fewer than
2**64 possibilities. So this rough sketch does give the right general
idea of the consequences of the "more predictable than expected"
fallback behaviour, even though it's thoroughly unfair to the Linux
kernel team's efforts to mitigate the practical consequences of this
problem without resorting to breaking backwards compatibility.

This design is generally agreed to be a bad idea. As far as we can tell,
there are no use cases whatsoever in which this is the behavior you
actually want. It has led to the use of insecure ssh keys on real
systems, and many *nix-like systems (including at least Mac OS X,
OpenBSD, and FreeBSD) have modified their /dev/urandom implementations
so that they never return predictable outputs, either by making reads
block in this case, or by simply refusing to run any userspace programs
until the system RNG has been initialized. Unfortunately, Linux has so
far been unable to follow suit, because it's been empirically determined
that enabling the blocking behavior causes some currently extant
distributions to fail to boot.

Instead, the new getrandom() syscall was introduced, making it possible
for userspace applications to access the system random number generator
safely, without introducing hard to debug deadlock problems into the
system initialization processes of existing Linux distros.

Consequences of getrandom() availability for Python

Prior to the introduction of the getrandom() syscall, it simply wasn't
feasible to access the Linux system random number generator in a
provably safe way, so we were forced to settle for reading from
/dev/urandom as the best available option. However, with getrandom()
insisting on raising an error or blocking rather than returning
predictable data, as well as having other advantages, it is now the
recommended method for accessing the kernel RNG on Linux, with reading
/dev/urandom directly relegated to "legacy" status. This moves Linux
into the same category as other operating systems like Windows, which
doesn't provide a /dev/urandom device at all: the best available option
for implementing os.urandom() is no longer simply reading bytes from the
/dev/urandom device.

This means that what used to be somebody else's problem (the Linux
kernel development team's) is now Python's problem -- given a way to
detect that the system RNG is not initialized, we have to choose how to
handle this situation whenever we try to use the system RNG.

It could simply block, as was somewhat inadvertently implemented in
3.5.0, and as is proposed in Victor Stinner's competing PEP:

    # artist's impression of the CPython 3.5.0-3.5.1 behavior
    def generate_unpredictable_bytes_or_block(num_bytes):
        while not system_rng_is_ready:
            wait
        return unpredictable_bytes(num_bytes)

Or it could raise an error, as this PEP proposes (in some cases):

    # artist's impression of the behavior proposed in this PEP
    def generate_unpredictable_bytes_or_raise(num_bytes):
        if system_rng_is_ready:
            return unpredictable_bytes(num_bytes)
        else:
            raise BlockingIOError

Or it could explicitly emulate the /dev/urandom fallback behavior, as
was implemented in 3.5.2rc1 and is expected to remain for the rest of
the 3.5.x cycle:

    # artist's impression of the CPython 3.5.2rc1+ behavior
    def generate_unpredictable_bytes_or_maybe_not(num_bytes):
        if system_rng_is_ready:
            return unpredictable_bytes(num_bytes)
        else:
            return (b"p4ssw0rd" * (num_bytes // 8 + 1))[:num_bytes]

(And the same caveats apply to this sketch as applied to the
generate_unpredictable_password sketch of /dev/urandom above.)

There are five places where CPython and the standard library attempt to
use the operating system's random number generator, and thus five places
where this decision has to be made:

-   initializing the SipHash used to protect str.__hash__ and friends
    against DoS attacks (called unconditionally at startup)
-   initializing the random module (called when random is imported)
-   servicing user calls to the os.urandom public API
-   the higher level random.SystemRandom public API
-   the new secrets module public API added by PEP 506

Previously, these five places all used the same underlying code, and
thus made this decision in the same way.

This whole problem was first noticed because 3.5.0 switched that
underlying code to the generate_unpredictable_bytes_or_block behavior,
and it turns out that there are some rare cases where Linux boot scripts
attempted to run a Python program as part of system initialization, the
Python startup sequence blocked while trying to initialize SipHash, and
then this triggered a deadlock because the system stopped doing anything
-- including gathering new entropy -- until the Python script was
forcibly terminated by an external timer. This is particularly
unfortunate since the scripts in question never processed untrusted
input, so there was no need for SipHash to be initialized with provably
unpredictable random data in the first place. This motivated the change
in 3.5.2rc1 to emulate the old /dev/urandom behavior in all cases (by
calling getrandom() in non-blocking mode, and then falling back to
reading /dev/urandom if the syscall indicates that the /dev/urandom pool
is not yet fully initialized.)

We don't know whether such problems may also exist in the
Fedora/RHEL/CentOS ecosystem, as the build systems for those
distributions use chroots on servers running an older operating system
kernel that doesn't offer the getrandom() syscall, which means CPython's
current build configuration compiles out the runtime check for that
syscall[11].

A similar problem was found due to the random module calling os.urandom
as a side-effect of import in order to seed the default global
random.Random() instance.

We have not received any specific complaints regarding direct calls to
os.urandom() or random.SystemRandom() blocking with 3.5.0 or 3.5.1 -only
problem reports due to the implicit blocking on interpreter startup and
as a side-effect of importing the random module.

Independently of this PEP, the first two cases have already been updated
to never block, regardless of the behaviour of os.urandom().

Where PEP 524 proposes to make all 3 of the latter cases block
implicitly, this PEP proposes that approach only for the last case (the
secrets) module, with os.urandom() and random.SystemRandom() instead
raising an exception when they detect that the underlying operating
system call would block.

References

For additional background details beyond those captured in this PEP and
Victor's competing PEP, also see Victor's prior collection of relevant
information and links at
https://haypo-notes.readthedocs.io/summary_python_random_issue.html

Copyright

This document has been placed into the public domain.

[1] Take a decision for os.urandom() in Python 3.6
(https://mail.python.org/pipermail/security-sig/2016-August/000084.htm)

[2] Application level vs library level design decisions
(https://mail.python.org/pipermail/security-sig/2016-June/000057.html)

[3] os.urandom() should use Linux 3.17 getrandom() syscall
(http://bugs.python.org/issue22181)

[4] Python 3.5 running on Linux kernel 3.17+ can block at startup or on
importing the random module on getrandom()
(http://bugs.python.org/issue26839)

[5] "import random" blocks on entropy collection on Linux with low
entropy (http://bugs.python.org/issue25420)

[6] os.urandom() doesn't block on Linux anymore
(https://hg.python.org/cpython/rev/9de508dc4837)

[7] Proposal to add os.getrandom()
(http://bugs.python.org/issue26839#msg267803)

[8] Add os.urandom_block() (http://bugs.python.org/issue27250)

[9] Add random.cryptorandom() and random.pseudorandom, deprecate
os.urandom() (http://bugs.python.org/issue27279)

[10] Always use getrandom() in os.random() on Linux and add block=False
parameter to os.urandom() (http://bugs.python.org/issue27266)

[11] Does the HAVE_GETRANDOM_SYSCALL config setting make sense?
(https://mail.python.org/pipermail/security-sig/2016-June/000060.html)

authors:
Alyssa Coghlan
Nathaniel J. Smith


text:
PEP: 542 Title: Dot Notation Assignment In Function Header Version:
$Revision$ Last-Modified: $Date$ Author: Markus Meskanen
<EMAIL_ADDRESS>Status: Rejected Type: Standards Track
Content-Type: text/x-rst Created: 10-Feb-2017 Resolution:
https://mail.python.org/pipermail/python-dev/2019-March/156695.html

Abstract

Function definitions only allow simple function names to be used, even
though functions are assignable first class objects.

This PEP proposes adding support for assigning a function to a class or
instance attribute directly in the function definition's header by using
the dot notation to separate the object from the function's name.

Although a similar feature, this PEP does not address general assignment
to anything that supports assignment, such as dict keys and list
indexes.

Rationale

Currently if a function needs to be assigned to a class or instance
attribute, it requires an additional assignment statement to be made:

    class MyClass:
        ...

    my_instance = MyClass()

    def my_function(self):
        ...

    # Assign to class attribute
    MyClass.my_function = my_function

    # Or assign to instance attribute
    my_instance.my_function = my_function

While this isn't usually an inconvenience, using dot notation to assign
directly in the function's header would greatly simplify this:

    class MyClass:
        ...

    my_instance = MyClass()

    # Assign to class attribute
    def MyClass.my_function(self):
        ...

    # Or assign to instance attribute
    def my_instance.my_function(self):
        ...

There are multiple reasons to use this functionality over a standard
class method, for example when the class is referenced inside the
function's header (such as with decorators and typing). This is also
useful when an instance requires a callback attribute:

    class Menu:
        def __init__(self, items=None, select_callback=None):
            self.items = items if items is not None else []
            self.select_callback = select_callback

    my_menu = Menu([item1, item2])

    def my_menu.select_callback(item_index, menu):
        print(menu.items[item_index])

As opposed to:

    my_menu = Menu([item1, item2])

    def select_callback(item_index, menu):
        print(menu.items[item_index])
    my_menu.select_callback = select_callback

Or defining them in an "unnatural" order:

    def select_callback(item_index, menu):
        print(menu.items[item_index])

    my_menu = Menu([item1, item2], select_callback)

It reads better than the "unnatural" way, since you already know at the
time of the function definition what it's goig to be used for. It also
saves one line of code while removing visual complexity.

The feature would also avoid leaving the function's name into the global
namespace:

    eggs = 'something'

    def Spam.eggs(self):
        ...

    def Cheese.eggs(self):
        ...

    assert eggs == 'something'

Ideally this would be just syntastic sugar:

    def x.y():
        ...

    # Equals to

    def y():
        ...
    x.y = y

Similar to how decorators are syntastic sugar:

    @decorate
    def f():
        ...

    # Equals to

    def f():
        ...
    f = decorate(f)

Implementation

The __name__ would follow the principles of a normal function:

    class MyClass:
        def my_function1(self):
            ...

    def MyClass.my_function2(self):
        ...

    assert my_function1.__name__ == 'my_function1'
    assert my_function2.__name__ == 'my_function2'

The grammar would use dotted_name to support chaining of attributes:

    def Person.name.fset(self, value):
        self._name = value

Backwards Compatibility

This PEP is fully backwards compatible.

Copyright

This document has been placed in the public domain.


text:
PEP: 352 Title: Required Superclass for Exceptions Version: $Revision$
Last-Modified: $Date$ Author: Brett Cannon, Guido van Rossum Status:
Final Type: Standards Track Content-Type: text/x-rst Created:
27-Oct-2005 Python-Version: 2.5 Post-History:

Abstract

In Python 2.4 and before, any (classic) class can be raised as an
exception. The plan for 2.5 was to allow new-style classes, but this
makes the problem worse -- it would mean any class (or instance) can be
raised! This is a problem as it prevents any guarantees from being made
about the interface of exceptions. This PEP proposes introducing a new
superclass that all raised objects must inherit from. Imposing the
restriction will allow a standard interface for exceptions to exist that
can be relied upon. It also leads to a known hierarchy for all
exceptions to adhere to.

One might counter that requiring a specific base class for a particular
interface is unPythonic. However, in the specific case of exceptions
there's a good reason (which has generally been agreed to on
python-dev): requiring hierarchy helps code that wants to catch
exceptions by making it possible to catch all exceptions explicitly by
writing except BaseException: instead of except *:.[1]

Introducing a new superclass for exceptions also gives us the chance to
rearrange the exception hierarchy slightly for the better. As it
currently stands, all exceptions in the built-in namespace inherit from
Exception. This is a problem since this includes two exceptions
(KeyboardInterrupt and SystemExit) that often need to be excepted from
the application's exception handling: the default behavior of shutting
the interpreter down without a traceback is usually more desirable than
whatever the application might do (with the possible exception of
applications that emulate Python's interactive command loop with >>>
prompt). Changing it so that these two exceptions inherit from the
common superclass instead of Exception will make it easy for people to
write except clauses that are not overreaching and not catch exceptions
that should propagate up.

This PEP is based on previous work done for PEP 348.

Requiring a Common Superclass

This PEP proposes introducing a new exception named BaseException that
is a new-style class and has a single attribute, args. Below is the code
as the exception will work in Python 3.0 (how it will work in Python 2.x
is covered in the Transition Plan section):

    class BaseException(object):

        """Superclass representing the base of the exception hierarchy.

        Provides an 'args' attribute that contains all arguments passed
        to the constructor.  Suggested practice, though, is that only a
        single string argument be passed to the constructor.

        """

        def __init__(self, *args):
            self.args = args

        def __str__(self):
            if len(self.args) == 1:
                return str(self.args[0])
            else:
                return str(self.args)

        def __repr__(self):
            return "%s(*%s)" % (self.__class__.__name__, repr(self.args))

No restriction is placed upon what may be passed in for args for
backwards-compatibility reasons. In practice, though, only a single
string argument should be used. This keeps the string representation of
the exception to be a useful message about the exception that is
human-readable; this is why the __str__ method special-cases on length-1
args value. Including programmatic information (e.g., an error code
number) should be stored as a separate attribute in a subclass.

The raise statement will be changed to require that any object passed to
it must inherit from BaseException. This will make sure that all
exceptions fall within a single hierarchy that is anchored at
BaseException[2]. This also guarantees a basic interface that is
inherited from BaseException. The change to raise will be enforced
starting in Python 3.0 (see the Transition Plan below).

With BaseException being the root of the exception hierarchy, Exception
will now inherit from it.

Exception Hierarchy Changes

With the exception hierarchy now even more important since it has a
basic root, a change to the existing hierarchy is called for. As it
stands now, if one wants to catch all exceptions that signal an error
and do not mean the interpreter should be allowed to exit, you must
specify all but two exceptions specifically in an except clause or catch
the two exceptions separately and then re-raise them and have all other
exceptions fall through to a bare except clause:

    except (KeyboardInterrupt, SystemExit):
        raise
    except:
        ...

That is needlessly explicit. This PEP proposes moving KeyboardInterrupt
and SystemExit to inherit directly from BaseException.

    - BaseException
      |- KeyboardInterrupt
      |- SystemExit
      |- Exception
         |- (all other current built-in exceptions)

Doing this makes catching Exception more reasonable. It would catch only
exceptions that signify errors. Exceptions that signal that the
interpreter should exit will not be caught and thus be allowed to
propagate up and allow the interpreter to terminate.

KeyboardInterrupt has been moved since users typically expect an
application to exit when they press the interrupt key (usually Ctrl-C).
If people have overly broad except clauses the expected behaviour does
not occur.

SystemExit has been moved for similar reasons. Since the exception is
raised when sys.exit() is called the interpreter should normally be
allowed to terminate. Unfortunately overly broad except clauses can
prevent the explicitly requested exit from occurring.

To make sure that people catch Exception most of the time, various parts
of the documentation and tutorials will need to be updated to strongly
suggest that Exception be what programmers want to use. Bare except
clauses or catching BaseException directly should be discouraged based
on the fact that KeyboardInterrupt and SystemExit almost always should
be allowed to propagate up.

Transition Plan

Since semantic changes to Python are being proposed, a transition plan
is needed. The goal is to end up with the new semantics being used in
Python 3.0 while providing a smooth transition for 2.x code. All
deprecations mentioned in the plan will lead to the removal of the
semantics starting in the version following the initial deprecation.

Here is BaseException as implemented in the 2.x series:

    class BaseException(object):

        """Superclass representing the base of the exception hierarchy.

        The __getitem__ method is provided for backwards-compatibility
        and will be deprecated at some point.  The 'message' attribute
        is also deprecated.

        """

        def __init__(self, *args):
            self.args = args

        def __str__(self):
            return str(self.args[0]
                       if len(self.args) <= 1
                       else self.args)

        def __repr__(self):
            func_args = repr(self.args) if self.args else "()"
            return self.__class__.__name__ + func_args

        def __getitem__(self, index):
            """Index into arguments passed in during instantiation.

            Provided for backwards-compatibility and will be
            deprecated.

            """
            return self.args[index]

        def _get_message(self):
            """Method for 'message' property."""
            warnings.warn("the 'message' attribute has been deprecated "
                            "since Python 2.6")
            return self.args[0] if len(args) == 1 else ''

        message = property(_get_message,
                            doc="access the 'message' attribute; "
                                "deprecated and provided only for "
                                "backwards-compatibility")

Deprecation of features in Python 2.9 is optional. This is because it is
not known at this time if Python 2.9 (which is slated to be the last
version in the 2.x series) will actively deprecate features that will
not be in 3.0. It is conceivable that no deprecation warnings will be
used in 2.9 since there could be such a difference between 2.9 and 3.0
that it would make 2.9 too "noisy" in terms of warnings. Thus the
proposed deprecation warnings for Python 2.9 will be revisited when
development of that version begins, to determine if they are still
desired.

-   Python 2.5 [done]
    -   all standard exceptions become new-style classes [done]
    -   introduce BaseException [done]
    -   Exception, KeyboardInterrupt, and SystemExit inherit from
        BaseException [done]
    -   deprecate raising string exceptions [done]
-   Python 2.6 [done]
    -   deprecate catching string exceptions [done]
    -   deprecate message attribute (see Retracted Ideas) [done]
-   Python 2.7 [done]
    -   deprecate raising exceptions that do not inherit from
        BaseException
-   Python 3.0 [done]
    -   drop everything that was deprecated above:
        -   string exceptions (both raising and catching) [done]
        -   all exceptions must inherit from BaseException [done]
        -   drop __getitem__, message [done]

Retracted Ideas

A previous version of this PEP that was implemented in Python 2.5
included a 'message' attribute on BaseException. Its purpose was to
begin a transition to BaseException accepting only a single argument.
This was to tighten the interface and to force people to use attributes
in subclasses to carry arbitrary information with an exception instead
of cramming it all into args.

Unfortunately, while implementing the removal of the args attribute in
Python 3.0 at the PyCon 2007 sprint [3], it was discovered that the
transition was very painful, especially for C extension modules. It was
decided that it would be better to deprecate the message attribute in
Python 2.6 (and remove it in Python 2.7 and Python 3.0) and consider a
more long-term transition strategy in Python 3.0 to remove
multiple-argument support in BaseException in preference of accepting
only a single argument. Thus the introduction of message and the
original deprecation of args has been retracted.

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 End:

[1] python-dev Summary for 2004-08-01 through 2004-08-15
http://www.python.org/dev/summary/2004-08-01_2004-08-15.html#an-exception-is-an-exception-unless-it-doesn-t-inherit-from-exception

[2] python-dev Summary for 2004-08-01 through 2004-08-15
http://www.python.org/dev/summary/2004-08-01_2004-08-15.html#an-exception-is-an-exception-unless-it-doesn-t-inherit-from-exception

[3] python-3000 email ("How far to go with cleaning up exceptions")
https://mail.python.org/pipermail/python-3000/2007-March/005911.html

authors:
Brett Cannon
Guido van Rossum


text:
PEP: 658 Title: Serve Distribution Metadata in the Simple Repository API
Author: Tzu-ping Chung<EMAIL_ADDRESS>Sponsor: Brett Cannon
<EMAIL_ADDRESS>PEP-Delegate: Donald Stufft<EMAIL_ADDRESS>Discussions-To: https://discuss.python.org/t/8651 Status: Accepted Type:
Standards Track Topic: Packaging Content-Type: text/x-rst Created:
10-May-2021 Post-History: 10-May-2021 Resolution:
https://discuss.python.org/t/8651/48

Abstract

This PEP proposes adding an anchor tag to expose the METADATA file from
distributions in the PEP 503 "simple" repository API. A
data-dist-info-metadata attribute is introduced to indicate that the
file from a given distribution can be independently fetched.

Motivation

Package management workflows made popular by recent tooling increase the
need to inspect distribution metadata without intending to install the
distribution, and download multiple distributions of a project to choose
from based on their metadata. This means they end up discarding much
downloaded data, which is inefficient and results in a bad user
experience.

Rationale

Tools have been exploring methods to reduce the download size by
partially downloading wheels with HTTP range requests. This, however,
adds additional run-time requirements to the repository server. It also
still adds additional overhead, since a separate request is needed to
fetch the wheel's file listing to find the correct offset to fetch the
metadata file. It is therefore desired to make the server extract the
metadata file in advance, and serve it as an independent file to avoid
the need to perform additional requests and ZIP inspection.

The metadata file defined by the Core Metadata Specification
[core-metadata] will be served directly by repositories since it
contains the necessary information for common use cases. The metadata
must only be served for standards-compliant distributions such as wheels
[wheel] and sdists [sdist], and must be identical to the distribution's
canonical metadata file, such as a wheel's METADATA file in the
.dist-info directory [dist-info].

An HTML attribute on the distribution file's anchor link is needed to
indicate whether a client is able to choose the separately served
metadata file. The attribute is also used to provide the metadata
content's hash for client-side verification. The attribute's absence
indicates that a separate metadata entry is not available for the
distribution, either because of the distribution's content, or lack of
repository support.

Specification

In a simple repository's project page, each anchor tag pointing to a
distribution MAY have a data-dist-info-metadata attribute. The presence
of the attribute indicates the distribution represented by the anchor
tag MUST contain a Core Metadata file that will not be modified when the
distribution is processed and/or installed.

If a data-dist-info-metadata attribute is present, the repository MUST
serve the distribution's Core Metadata file alongside the distribution
with a .metadata appended to the distribution's file name. For example,
the Core Metadata of a distribution served at
/files/distribution-1.0-py3.none.any.whl would be located at
/files/distribution-1.0-py3.none.any.whl.metadata. This is similar to
how PEP 503 specifies the GPG signature file's location.

The repository SHOULD provide the hash of the Core Metadata file as the
data-dist-info-metadata attribute's value using the syntax
<hashname>=<hashvalue>, where <hashname> is the lower cased name of the
hash function used, and <hashvalue> is the hex encoded digest. The
repository MAY use true as the attribute's value if a hash is
unavailable.

Backwards Compatibility

If an anchor tag lacks the data-dist-info-metadata attribute, tools are
expected to revert to their current behaviour of downloading the
distribution to inspect the metadata.

Older tools not supporting the new data-dist-info-metadata attribute are
expected to ignore the attribute and maintain their current behaviour of
downloading the distribution to inspect the metadata. This is similar to
how prior data- attribute additions expect existing tools to operate.

Rejected Ideas

Put metadata content on the project page

Since tools generally only need dependency information from a
distribution in addition to what's already available on the project
page, it was proposed that repositories may directly include the
information on the project page, like the data-requires-python attribute
specified in PEP 503.

This approach was abandoned since a distribution may contain arbitrarily
long lists of dependencies (including required and optional), and it is
unclear whether including the information for every distribution in a
project would result in net savings since the information for most
distributions generally ends up unneeded. By serving the metadata
separately, performance can be better estimated since data usage will be
more proportional to the number of distributions inspected.

Expose more files in the distribution

It was proposed to provide the entire .dist-info directory as a separate
part, instead of only the metadata file. However, serving multiple files
in one entity through HTTP requires re-archiving them separately after
they are extracted from the original distribution by the repository
server, and there are no current use cases for files other than METADATA
when the distribution itself is not going to be installed.

It should also be noted that the approach taken here does not preclude
other files from being introduced in the future, whether we want to
serve them together or individually.

Explicitly specify the metadata file's URL on the project page

An early version of this draft proposed putting the metadata file's URL
in the data-dist-info-metadata attribute. But people feel it is better
for discoverability to require the repository to serve the metadata file
at a determined location instead. The current approach also has an
additional benefit of making the project page smaller.

References

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

core-metadata

    https://packaging.python.org/specifications/core-metadata/

dist-info

    https://packaging.python.org/specifications/recording-installed-packages/

sdist

    https://packaging.python.org/specifications/source-distribution-format/

wheel

    https://packaging.python.org/specifications/binary-distribution-format/


text:
PEP: 215 Title: String Interpolation Author: Ka-Ping Yee<EMAIL_ADDRESS>Status: Superseded Type: Standards Track Created: 24-Jul-2000
Python-Version: 2.1 Post-History: Superseded-By: 292

292

Abstract

This document proposes a string interpolation feature for Python to
allow easier string formatting. The suggested syntax change is the
introduction of a '$' prefix that triggers the special interpretation of
the '$' character within a string, in a manner reminiscent to the
variable interpolation found in Unix shells, awk, Perl, or Tcl.

Copyright

This document is in the public domain.

Specification

Strings may be preceded with a '$' prefix that comes before the leading
single or double quotation mark (or triplet) and before any of the other
string prefixes ('r' or 'u'). Such a string is processed for
interpolation after the normal interpretation of backslash-escapes in
its contents. The processing occurs just before the string is pushed
onto the value stack, each time the string is pushed. In short, Python
behaves exactly as if '$' were a unary operator applied to the string.
The operation performed is as follows:

The string is scanned from start to end for the '$' character (\x24 in
8-bit strings or \u0024 in Unicode strings). If there are no '$'
characters present, the string is returned unchanged.

Any '$' found in the string, followed by one of the two kinds of
expressions described below, is replaced with the value of the
expression as evaluated in the current namespaces. The value is
converted with str() if the containing string is an 8-bit string, or
with unicode() if it is a Unicode string.

1.  A Python identifier optionally followed by any number of trailers,
    where a trailer consists of:

    -   a dot and an identifier,
    -   an expression enclosed in square brackets, or

    - an argument list enclosed in parentheses (This is exactly the
    pattern expressed in the Python grammar by "NAME trailer*", using
    the definitions in Grammar/Grammar.)

2.  Any complete Python expression enclosed in curly braces.

Two dollar-signs ("$$") are replaced with a single "$".

Examples

Here is an example of an interactive session exhibiting the expected
behaviour of this feature. :

    >>> a, b = 5, 6
    >>> print $'a = $a, b = $b'
    a = 5, b = 6
    >>> $u'uni${a}ode'
    u'uni5ode'
    >>> print $'\$a'
    5
    >>> print $r'\$a'
    \5
    >>> print $'$$$a.$b'
    $5.6
    >>> print $'a + b = ${a + b}'
    a + b = 11
    >>> import sys
    >>> print $'References to $a: $sys.getrefcount(a)'
    References to 5: 15
    >>> print $"sys = $sys, sys = $sys.modules['sys']"
    sys = <module 'sys' (built-in)>, sys = <module 'sys' (built-in)>
    >>> print $'BDFL = $sys.copyright.split()[4].upper()'
    BDFL = GUIDO

Discussion

'$' is chosen as the interpolation character within the string for the
sake of familiarity, since it is already used for this purpose in many
other languages and contexts.

It is then natural to choose '$' as a prefix, since it is a mnemonic for
the interpolation character.

Trailers are permitted to give this interpolation mechanism even more
power than the interpolation available in most other languages, while
the expression to be interpolated remains clearly visible and free of
curly braces.

'$' works like an operator and could be implemented as an operator, but
that prevents the compile-time optimization and presents security
issues. So, it is only allowed as a string prefix.

Security Issues

"$" has the power to eval, but only to eval a literal. As described here
(a string prefix rather than an operator), it introduces no new security
issues since the expressions to be evaluated must be literally present
in the code.

Implementation

The Itpl module at[1] provides a prototype of this feature. It uses the
tokenize module to find the end of an expression to be interpolated,
then calls eval() on the expression each time a value is needed. In the
prototype, the expression is parsed and compiled again each time it is
evaluated.

As an optimization, interpolated strings could be compiled directly into
the corresponding bytecode; that is, :

    $'a = $a, b = $b'

could be compiled as though it were the expression :

    ('a = ' + str(a) + ', b = ' + str(b))

so that it only needs to be compiled once.

References

[1] http://www.lfw.org/python/Itpl.py