text:
PEP: 356 Title: Python 2.5 Release Schedule Version: $Revision$
Last-Modified: $Date$ Author: Neal Norwitz, Guido van Rossum, Anthony
Baxter Status: Final Type: Informational Topic: Release Content-Type:
text/x-rst Created: 07-Feb-2006 Python-Version: 2.5 Post-History:

Abstract

This document describes the development and release schedule for Python
2.5. The schedule primarily concerns itself with PEP-sized items. Small
features may be added up to and including the first beta release. Bugs
may be fixed until the final release.

There will be at least two alpha releases, two beta releases, and one
release candidate. The release date is planned for 12 September 2006.

Release Manager

-   Anthony Baxter has volunteered to be Release Manager.
-   Martin von Loewis is building the Windows installers,
-   Ronald Oussoren is building the Mac installers,
-   Fred Drake the doc packages and
-   Sean Reifschneider the RPMs.

Release Schedule

-   alpha 1: April 5, 2006 [completed]
-   alpha 2: April 27, 2006 [completed]
-   beta 1: June 20, 2006 [completed]
-   beta 2: July 11, 2006 [completed]
-   beta 3: August 3, 2006 [completed]
-   rc 1: August 17, 2006 [completed]
-   rc 2: September 12, 2006 [completed]
-   final: September 19, 2006 [completed]

Completed features for 2.5

-   PEP 308: Conditional Expressions
-   PEP 309: Partial Function Application
-   PEP 314: Metadata for Python Software Packages v1.1
-   PEP 328: Absolute/Relative Imports
-   PEP 338: Executing Modules as Scripts
-   PEP 341: Unified try-except/try-finally to try-except-finally
-   PEP 342: Coroutines via Enhanced Generators
-   PEP 343: The "with" Statement (still need updates in Doc/ref and for
    the contextlib module)
-   PEP 352: Required Superclass for Exceptions
-   PEP 353: Using ssize_t as the index type
-   PEP 357: Allowing Any Object to be Used for Slicing
-   ASCII became the default coding
-   AST-based compiler
-   Access to C AST from Python through new _ast module
-   any()/all() builtin truth functions

New standard library modules:

-   cProfile -- suitable for profiling long running applications with
    minimal overhead
-   ctypes -- optional component of the windows installer
-   ElementTree and cElementTree -- by Fredrik Lundh
-   hashlib -- adds support for SHA-224, -256, -384, and -512 (replaces
    old md5 and sha modules)
-   msilib -- for creating MSI files and bdist_msi in distutils.
-   pysqlite
-   uuid
-   wsgiref

Other notable features:

-   Added support for reading shadow passwords[1]
-   Added support for the Unicode 4.1 UCD
-   Added PEP 302 zipfile/__loader__ support to the following modules:
    warnings, linecache, inspect, traceback, site, and doctest
-   Added pybench Python benchmark suite -- by Marc-Andre Lemburg
-   Add write support for mailboxes from the code in sandbox/mailbox.
    (Owner: A.M. Kuchling. It would still be good if another person
    would take a look at the new code.)
-   Support for building "fat" Mac binaries (Intel and PPC)
-   Add new icons for Windows with the new Python logo?
-   New utilities in functools to help write wrapper functions that
    support naive introspection (e.g. having f.__name__ return the
    original function name).
-   Upgrade pyexpat to use expat 2.0.
-   Python core now compiles cleanly with g++

Possible features for 2.5

Each feature below should implemented prior to beta1 or will require
BDFL approval for inclusion in 2.5.

-   Modules under consideration for inclusion:
-   Add new icons for MacOS and Unix with the new Python logo? (Owner:
    ???) MacOS:
    http://hcs.harvard.edu/~jrus/python/prettified-py-icons.png
-   Check the various bits of code in Demo/ all still work, update or
    remove the ones that don't. (Owner: Anthony)
-   All modules in Modules/ should be updated to be ssize_t clean.
    (Owner: Neal)

Deferred until 2.6

-   bdist_deb in distutils package[2]
-   bdist_egg in distutils package
-   pure python pgen module (Owner: Guido)
-   Remove the fpectl module?
-   Make everything in Modules/ build cleanly with g++

Open issues

-   Bugs that need resolving before release, ie, they block release:

    None

-   Bugs deferred until 2.5.1 (or later):

    -   https://bugs.python.org/issue1544279 - Socket module is not
        thread-safe
    -   https://bugs.python.org/issue1541420 - tools and demo missing
        from windows
    -   https://bugs.python.org/issue1542451 - crash with continue in
        nested try/finally
    -   https://bugs.python.org/issue1475523 - gettext.py bug (owner:
        Martin v. Loewis)
    -   https://bugs.python.org/issue1467929 - %-formatting and dicts
    -   https://bugs.python.org/issue1446043 - unicode() does not raise
        LookupError

-   The PEP 302 changes to (at least) pkgutil, runpy and pydoc must be
    documented.

-   test_zipfile64 takes too long and too much disk space for most of
    the buildbots. How should this be handled? It is currently disabled.

-   should C modules listed in "Undocumented modules" be removed too?
    "timing" (listed as obsolete), "cl" (listed as possibly not
    up-to-date), and "sv" (listed as obsolete hardware specific).

References

Copyright

This document has been placed in the public domain.

[1] Shadow Password Support Module https://bugs.python.org/issue579435

[2] Joe Smith, bdist* to stdlib?
https://mail.python.org/pipermail/python-dev/2006-February/060926.html

authors:
Anthony Baxter
Guido van Rossum
Neal Norwitz


text:
PEP: 248 Title: Python Database API Specification v1.0 Author: Greg
Stein<EMAIL_ADDRESS>Marc-André Lemburg<EMAIL_ADDRESS>Discussions-To<EMAIL_ADDRESS>Status: Final Type: Informational
Content-Type: text/x-rst Created: 08-May-1996 Post-History:
Superseded-By: 249

Introduction

This API has been defined to encourage similarity between the Python
modules that are used to access databases. By doing this, we hope to
achieve a consistency leading to more easily understood modules, code
that is generally more portable across databases, and a broader reach of
database connectivity from Python.

This interface specification consists of several items:

-   Module Interface
-   Connection Objects
-   Cursor Objects
-   DBI Helper Objects

Comments and questions about this specification may be directed to the
SIG on Tabular Databases in Python (http://www.python.org/sigs/db-sig).

This specification document was last updated on: April 9, 1996. It will
be known as Version 1.0 of this specification.

Module Interface

The database interface modules should typically be named with something
terminated by db. Existing examples are: oracledb, informixdb, and
pg95db. These modules should export several names:

modulename(connection_string)

    Constructor for creating a connection to the database. Returns a
    Connection Object.

error

    Exception raised for errors from the database module.

Connection Objects

Connection Objects should respond to the following methods:

close()

    Close the connection now (rather than whenever __del__ is called).
    The connection will be unusable from this point forward; an
    exception will be raised if any operation is attempted with the
    connection.

commit()

    Commit any pending transaction to the database.

rollback()

    Roll the database back to the start of any pending transaction.

cursor()

    Return a new Cursor Object. An exception may be thrown if the
    database does not support a cursor concept.

callproc([params])

    (Note: this method is not well-defined yet.) Call a stored database
    procedure with the given (optional) parameters. Returns the result
    of the stored procedure.

(all Cursor Object attributes and methods)

    For databases that do not have cursors and for simple applications
    that do not require the complexity of a cursor, a Connection Object
    should respond to each of the attributes and methods of the Cursor
    Object. Databases that have cursor can implement this by using an
    implicit, internal cursor.

Cursor Objects

These objects represent a database cursor, which is used to manage the
context of a fetch operation.

Cursor Objects should respond to the following methods and attributes:

arraysize

    This read/write attribute specifies the number of rows to fetch at a
    time with fetchmany(). This value is also used when inserting
    multiple rows at a time (passing a tuple/list of tuples/lists as the
    params value to execute()). This attribute will default to a single
    row.

    Note that the arraysize is optional and is merely provided for
    higher performance database interactions. Implementations should
    observe it with respect to the fetchmany() method, but are free to
    interact with the database a single row at a time.

description

    This read-only attribute is a tuple of 7-tuples. Each 7-tuple
    contains information describing each result column: (name,
    type_code, display_size, internal_size, precision, scale, null_ok).
    This attribute will be None for operations that do not return rows
    or if the cursor has not had an operation invoked via the execute()
    method yet.

    The 'type_code' is one of the 'dbi' values specified in the section
    below.

    Note: this is a bit in flux. Generally, the first two items of the
    7-tuple will always be present; the others may be database specific.

close()

    Close the cursor now (rather than whenever __del__ is called). The
    cursor will be unusable from this point forward; an exception will
    be raised if any operation is attempted with the cursor.

execute(operation [,params])

    Execute (prepare) a database operation (query or command).
    Parameters may be provided (as a sequence (e.g. tuple/list)) and
    will be bound to variables in the operation. Variables are specified
    in a database-specific notation that is based on the index in the
    parameter tuple (position-based rather than name-based).

    The parameters may also be specified as a sequence of sequences
    (e.g. a list of tuples) to insert multiple rows in a single
    operation.

    A reference to the operation will be retained by the cursor. If the
    same operation object is passed in again, then the cursor can
    optimize its behavior. This is most effective for algorithms where
    the same operation is used, but different parameters are bound to it
    (many times).

    For maximum efficiency when reusing an operation, it is best to use
    the setinputsizes() method to specify the parameter types and sizes
    ahead of time. It is legal for a parameter to not match the
    predefined information; the implementation should compensate,
    possibly with a loss of efficiency.

    Using SQL terminology, these are the possible result values from the
    execute() method:

    -   If the statement is DDL (e.g. CREATE TABLE), then 1 is returned.
    -   If the statement is DML (e.g. UPDATE or INSERT), then the number
        of rows affected is returned (0 or a positive integer).
    -   If the statement is DQL (e.g. SELECT), None is returned,
        indicating that the statement is not really complete until you
        use one of the 'fetch' methods.

fetchone()

    Fetch the next row of a query result, returning a single tuple.

fetchmany([size])

    Fetch the next set of rows of a query result, returning as a list of
    tuples. An empty list is returned when no more rows are available.
    The number of rows to fetch is specified by the parameter. If it is
    None, then the cursor's arraysize determines the number of rows to
    be fetched.

    Note there are performance considerations involved with the size
    parameter. For optimal performance, it is usually best to use the
    arraysize attribute. If the size parameter is used, then it is best
    for it to retain the same value from one fetchmany() call to the
    next.

fetchall()

    Fetch all rows of a query result, returning as a list of tuples.
    Note that the cursor's arraysize attribute can affect the
    performance of this operation.

setinputsizes(sizes)

    (Note: this method is not well-defined yet.) This can be used before
    a call to execute() to predefine memory areas for the operation's
    parameters. sizes is specified as a tuple -- one item for each input
    parameter. The item should be a Type object that corresponds to the
    input that will be used, or it should be an integer specifying the
    maximum length of a string parameter. If the item is None, then no
    predefined memory area will be reserved for that column (this is
    useful to avoid predefined areas for large inputs).

    This method would be used before the execute() method is invoked.

    Note that this method is optional and is merely provided for higher
    performance database interaction. Implementations are free to do
    nothing and users are free to not use it.

setoutputsize(size [,col])

    (Note: this method is not well-defined yet.)

    Set a column buffer size for fetches of large columns (e.g. LONG).
    The column is specified as an index into the result tuple. Using a
    column of None will set the default size for all large columns in
    the cursor.

    This method would be used before the execute() method is invoked.

    Note that this method is optional and is merely provided for higher
    performance database interaction. Implementations are free to do
    nothing and users are free to not use it.

DBI Helper Objects

Many databases need to have the input in a particular format for binding
to an operation's input parameters. For example, if an input is destined
for a DATE column, then it must be bound to the database in a particular
string format. Similar problems exist for "Row ID" columns or large
binary items (e.g. blobs or RAW columns). This presents problems for
Python since the parameters to the execute() method are untyped. When
the database module sees a Python string object, it doesn't know if it
should be bound as a simple CHAR column, as a raw binary item, or as a
DATE.

To overcome this problem, the 'dbi' module was created. This module
specifies some basic database interface types for working with
databases. There are two classes: 'dbiDate' and 'dbiRaw'. These are
simple container classes that wrap up a value. When passed to the
database modules, the module can then detect that the input parameter is
intended as a DATE or a RAW. For symmetry, the database modules will
return DATE and RAW columns as instances of these classes.

A Cursor Object's 'description' attribute returns information about each
of the result columns of a query. The 'type_code' is defined to be one
of five types exported by this module: STRING, RAW, NUMBER, DATE, or
ROWID.

The module exports the following names:

dbiDate(value)

    This function constructs a 'dbiDate' instance that holds a date
    value. The value should be specified as an integer number of seconds
    since the "epoch" (e.g. time.time()).

dbiRaw(value)

    This function constructs a 'dbiRaw' instance that holds a raw
    (binary) value. The value should be specified as a Python string.

STRING

    This object is used to describe columns in a database that are
    string-based (e.g. CHAR).

RAW

    This object is used to describe (large) binary columns in a database
    (e.g. LONG RAW, blobs).

NUMBER

    This object is used to describe numeric columns in a database.

DATE

    This object is used to describe date columns in a database.

ROWID

    This object is used to describe the "Row ID" column in a database.

Acknowledgements

Many thanks go to Andrew Kuchling who converted the Python Database API
Specification 1.0 from the original HTML format into the PEP format in
2001.

Greg Stein is the original author of the Python Database API
Specification 1.0. Marc-André later continued maintenance of the API as
an editor.

Copyright

This document has been placed in the Public Domain.

license: Public Domain

authors:
Greg Stein
Marc-André Lemburg


text:
PEP: 3151 Title: Reworking the OS and IO exception hierarchy Version:
$Revision$ Last-Modified: $Date$ Author: Antoine Pitrou
<EMAIL_ADDRESS>BDFL-Delegate: Barry Warsaw Status: Final Type:
Standards Track Content-Type: text/x-rst Created: 21-Jul-2010
Python-Version: 3.3 Post-History: Resolution:
https://mail.python.org/pipermail/python-dev/2011-October/114033.html

Abstract

The standard exception hierarchy is an important part of the Python
language. It has two defining qualities: it is both generic and
selective. Generic in that the same exception type can be raised - and
handled - regardless of the context (for example, whether you are trying
to add something to an integer, to call a string method, or to write an
object on a socket, a TypeError will be raised for bad argument types).
Selective in that it allows the user to easily handle (silence, examine,
process, store or encapsulate...) specific kinds of error conditions
while letting other errors bubble up to higher calling contexts. For
example, you can choose to catch ZeroDivisionErrors without affecting
the default handling of other ArithmeticErrors (such as OverflowErrors).

This PEP proposes changes to a part of the exception hierarchy in order
to better embody the qualities mentioned above: the errors related to
operating system calls (OSError, IOError, mmap.error, select.error, and
all their subclasses).

Rationale

Confusing set of OS-related exceptions

OS-related (or system call-related) exceptions are currently a diversity
of classes, arranged in the following sub-hierarchies:

    +-- EnvironmentError
        +-- IOError
            +-- io.BlockingIOError
            +-- io.UnsupportedOperation (also inherits from ValueError)
            +-- socket.error
                +-- socket.gaierror
                +-- socket.herror
                +-- socket.timeout
        +-- OSError
            +-- VMSError
            +-- WindowsError
        +-- mmap.error
    +-- select.error

While some of these distinctions can be explained by implementation
considerations, they are often not very logical at a higher level. The
line separating OSError and IOError, for example, is often blurry.
Consider the following:

    >>> os.remove("fff")
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    OSError: [Errno 2] No such file or directory: 'fff'
    >>> open("fff")
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    IOError: [Errno 2] No such file or directory: 'fff'

The same error condition (a non-existing file) gets cast as two
different exceptions depending on which library function was called. The
reason for this is that the os module exclusively raises OSError (or its
subclass WindowsError) while the io module mostly raises IOError.
However, the user is interested in the nature of the error, not in which
part of the interpreter it comes from (since the latter is obvious from
reading the traceback message or application source code).

In fact, it is hard to think of any situation where OSError should be
caught but not IOError, or the reverse.

A further proof of the ambiguity of this segmentation is that the
standard library itself sometimes has problems deciding. For example, in
the select module, similar failures will raise select.error, OSError or
IOError depending on whether you are using select(), a poll object, a
kqueue object, or an epoll object. This makes user code uselessly
complicated since it has to be prepared to catch various exception
types, depending on which exact implementation of a single primitive it
chooses to use at runtime.

As for WindowsError, it seems to be a pointless distinction. First, it
only exists on Windows systems, which requires tedious compatibility
code in cross-platform applications (such code can be found in
Lib/shutil.py). Second, it inherits from OSError and is raised for
similar errors as OSError is raised for on other systems. Third, the
user wanting access to low-level exception specifics has to examine the
errno or winerror attribute anyway.

Note

Appendix B surveys the use of the various exception types across the
interpreter and the standard library.

Lack of fine-grained exceptions

The current variety of OS-related exceptions doesn't allow the user to
filter easily for the desired kinds of failures. As an example, consider
the task of deleting a file if it exists. The Look Before You Leap
(LBYL) idiom suffers from an obvious race condition:

    if os.path.exists(filename):
        os.remove(filename)

If a file named as filename is created by another thread or process
between the calls to os.path.exists and os.remove, it won't be deleted.
This can produce bugs in the application, or even security issues.

Therefore, the solution is to try to remove the file, and ignore the
error if the file doesn't exist (an idiom known as Easier to Ask
Forgiveness than to get Permission, or EAFP). Careful code will read
like the following (which works under both POSIX and Windows systems):

    try:
        os.remove(filename)
    except OSError as e:
        if e.errno != errno.ENOENT:
            raise

or even:

    try:
        os.remove(filename)
    except EnvironmentError as e:
        if e.errno != errno.ENOENT:
            raise

This is a lot more to type, and also forces the user to remember the
various cryptic mnemonics from the errno module. It imposes an
additional cognitive burden and gets tiresome rather quickly.
Consequently, many programmers will instead write the following code,
which silences exceptions too broadly:

    try:
        os.remove(filename)
    except OSError:
        pass

os.remove can raise an OSError not only when the file doesn't exist, but
in other possible situations (for example, the filename points to a
directory, or the current process doesn't have permission to remove the
file), which all indicate bugs in the application logic and therefore
shouldn't be silenced. What the programmer would like to write instead
is something such as:

    try:
        os.remove(filename)
    except FileNotFoundError:
        pass

Compatibility strategy

Reworking the exception hierarchy will obviously change the exact
semantics of at least some existing code. While it is not possible to
improve on the current situation without changing exact semantics, it is
possible to define a narrower type of compatibility, which we will call
useful compatibility.

For this we first must explain what we will call careful and careless
exception handling. Careless (or "naïve") code is defined as code which
blindly catches any of OSError, IOError, socket.error, mmap.error,
WindowsError, select.error without checking the errno attribute. This is
because such exception types are much too broad to signify anything. Any
of them can be raised for error conditions as diverse as: a bad file
descriptor (which will usually indicate a programming error), an
unconnected socket (ditto), a socket timeout, a file type mismatch, an
invalid argument, a transmission failure, insufficient permissions, a
non-existent directory, a full filesystem, etc.

(moreover, the use of certain of these exceptions is irregular; Appendix
B exposes the case of the select module, which raises different
exceptions depending on the implementation)

Careful code is defined as code which, when catching any of the above
exceptions, examines the errno attribute to determine the actual error
condition and takes action depending on it.

Then we can define useful compatibility as follows:

-   useful compatibility doesn't make exception catching any narrower,
    but it can be broader for careless exception-catching code. Given
    the following kind of snippet, all exceptions caught before this PEP
    will also be caught after this PEP, but the reverse may be false
    (because the coalescing of OSError, IOError and others means the
    except clause throws a slightly broader net):

        try:
            ...
            os.remove(filename)
            ...
        except OSError:
            pass

-   useful compatibility doesn't alter the behaviour of careful
    exception-catching code. Given the following kind of snippet, the
    same errors should be silenced or re-raised, regardless of whether
    this PEP has been implemented or not:

        try:
            os.remove(filename)
        except OSError as e:
            if e.errno != errno.ENOENT:
                raise

The rationale for this compromise is that careless code can't really be
helped, but at least code which "works" won't suddenly raise errors and
crash. This is important since such code is likely to be present in
scripts used as cron tasks or automated system administration programs.

Careful code, on the other hand, should not be penalized. Actually, one
purpose of this PEP is to ease writing careful code.

Step 1: coalesce exception types

The first step of the resolution is to coalesce existing exception
types. The following changes are proposed:

-   alias both socket.error and select.error to OSError
-   alias mmap.error to OSError
-   alias both WindowsError and VMSError to OSError
-   alias IOError to OSError
-   coalesce EnvironmentError into OSError

Each of these changes doesn't preserve exact compatibility, but it does
preserve useful compatibility (see "compatibility" section above).

Each of these changes can be accepted or refused individually, but of
course it is considered that the greatest impact can be achieved if this
first step is accepted in full. In this case, the IO exception
sub-hierarchy would become:

    +-- OSError   (replacing IOError, WindowsError, EnvironmentError, etc.)
        +-- io.BlockingIOError
        +-- io.UnsupportedOperation (also inherits from ValueError)
        +-- socket.gaierror
        +-- socket.herror
        +-- socket.timeout

Justification

Not only does this first step present the user a simpler landscape as
explained in the rationale section, but it also allows for a better and
more complete resolution of Step 2 (see Prerequisite).

The rationale for keeping OSError as the official name for generic
OS-related exceptions is that it, precisely, is more generic than
IOError. EnvironmentError is more tedious to type and also much
lesser-known.

The survey in Appendix B shows that IOError is the dominant error today
in the standard library. As for third-party Python code, Google Code
Search shows IOError being ten times more popular than EnvironmentError
in user code, and three times more popular than OSError [1]. However,
with no intention to deprecate IOError in the middle term, the lesser
popularity of OSError is not a problem.

Exception attributes

Since WindowsError is coalesced into OSError, the latter gains a
winerror attribute under Windows. It is set to None under situations
where it is not meaningful, as is already the case with the errno,
filename and strerror attributes (for example when OSError is raised
directly by Python code).

Deprecation of names

The following paragraphs outline a possible deprecation strategy for old
exception names. However, it has been decided to keep them as aliases
for the time being. This decision could be revised in time for Python
4.0.

built-in exceptions

Deprecating the old built-in exceptions cannot be done in a
straightforward fashion by intercepting all lookups in the builtins
namespace, since these are performance-critical. We also cannot work at
the object level, since the deprecated names will be aliased to
non-deprecated objects.

A solution is to recognize these names at compilation time, and then
emit a separate LOAD_OLD_GLOBAL opcode instead of the regular
LOAD_GLOBAL. This specialized opcode will handle the output of a
DeprecationWarning (or PendingDeprecationWarning, depending on the
policy decided upon) when the name doesn't exist in the globals
namespace, but only in the builtins one. This will be enough to avoid
false positives (for example if someone defines their own OSError in a
module), and false negatives will be rare (for example when someone
accesses OSError through the builtins module rather than directly).

module-level exceptions

The above approach cannot be used easily, since it would require
special-casing some modules when compiling code objects. However, these
names are by construction much less visible (they don't appear in the
builtins namespace), and lesser-known too, so we might decide to let
them live in their own namespaces.

Step 2: define additional subclasses

The second step of the resolution is to extend the hierarchy by defining
subclasses which will be raised, rather than their parent, for specific
errno values. Which errno values is subject to discussion, but a survey
of existing exception matching practices (see Appendix A) helps us
propose a reasonable subset of all values. Trying to map all errno
mnemonics, indeed, seems foolish, pointless, and would pollute the root
namespace.

Furthermore, in a couple of cases, different errno values could raise
the same exception subclass. For example, EAGAIN, EALREADY, EWOULDBLOCK
and EINPROGRESS are all used to signal that an operation on a
non-blocking socket would block (and therefore needs trying again
later). They could therefore all raise an identical subclass and let the
user examine the errno attribute if (s)he so desires (see below
"exception attributes").

Prerequisite

Step 1 is a loose prerequisite for this.

Prerequisite, because some errnos can currently be attached to different
exception classes: for example, ENOENT can be attached to both OSError
and IOError, depending on the context. If we don't want to break useful
compatibility, we can't make an except OSError (or IOError) fail to
match an exception where it would succeed today.

Loose, because we could decide for a partial resolution of step 2 if
existing exception classes are not coalesced: for example, ENOENT could
raise a hypothetical FileNotFoundError where an IOError was previously
raised, but continue to raise OSError otherwise.

The dependency on step 1 could be totally removed if the new subclasses
used multiple inheritance to match with all of the existing superclasses
(or, at least, OSError and IOError, which are arguable the most
prevalent ones). It would, however, make the hierarchy more complicated
and therefore harder to grasp for the user.

New exception classes

The following tentative list of subclasses, along with a description and
the list of errnos mapped to them, is submitted to discussion:

-   FileExistsError: trying to create a file or directory which already
    exists (EEXIST)
-   FileNotFoundError: for all circumstances where a file and directory
    is requested but doesn't exist (ENOENT)
-   IsADirectoryError: file-level operation (open(), os.remove()...)
    requested on a directory (EISDIR)
-   NotADirectoryError: directory-level operation requested on something
    else (ENOTDIR)
-   PermissionError: trying to run an operation without the adequate
    access rights - for example filesystem permissions (EACCES, EPERM)
-   BlockingIOError: an operation would block on an object (e.g. socket)
    set for non-blocking operation (EAGAIN, EALREADY, EWOULDBLOCK,
    EINPROGRESS); this is the existing io.BlockingIOError with an
    extended role
-   BrokenPipeError: trying to write on a pipe while the other end has
    been closed, or trying to write on a socket which has been shutdown
    for writing (EPIPE, ESHUTDOWN)
-   InterruptedError: a system call was interrupted by an incoming
    signal (EINTR)
-   ConnectionAbortedError: connection attempt aborted by peer
    (ECONNABORTED)
-   ConnectionRefusedError: connection reset by peer (ECONNREFUSED)
-   ConnectionResetError: connection reset by peer (ECONNRESET)
-   TimeoutError: connection timed out (ETIMEDOUT); this can be re-cast
    as a generic timeout exception, replacing socket.timeout and also
    useful for other types of timeout (for example in Lock.acquire())
-   ChildProcessError: operation on a child process failed (ECHILD);
    this is raised mainly by the wait() family of functions.
-   ProcessLookupError: the given process (as identified by, e.g., its
    process id) doesn't exist (ESRCH).

In addition, the following exception class is proposed for inclusion:

-   ConnectionError: a base class for ConnectionAbortedError,
    ConnectionRefusedError and ConnectionResetError

The following drawing tries to sum up the proposed additions, along with
the corresponding errno values (where applicable). The root of the
sub-hierarchy (OSError, assuming Step 1 is accepted in full) is not
shown:

    +-- BlockingIOError        EAGAIN, EALREADY, EWOULDBLOCK, EINPROGRESS
    +-- ChildProcessError                                          ECHILD
    +-- ConnectionError
        +-- BrokenPipeError                              EPIPE, ESHUTDOWN
        +-- ConnectionAbortedError                           ECONNABORTED
        +-- ConnectionRefusedError                           ECONNREFUSED
        +-- ConnectionResetError                               ECONNRESET
    +-- FileExistsError                                            EEXIST
    +-- FileNotFoundError                                          ENOENT
    +-- InterruptedError                                            EINTR
    +-- IsADirectoryError                                          EISDIR
    +-- NotADirectoryError                                        ENOTDIR
    +-- PermissionError                                     EACCES, EPERM
    +-- ProcessLookupError                                          ESRCH
    +-- TimeoutError                                            ETIMEDOUT

Naming

Various naming controversies can arise. One of them is whether all
exception class names should end in "Error". In favour is consistency
with the rest of the exception hierarchy, against is concision
(especially with long names such as ConnectionAbortedError).

Exception attributes

In order to preserve useful compatibility, these subclasses should still
set adequate values for the various exception attributes defined on the
superclass (for example errno, filename, and optionally winerror).

Implementation

Since it is proposed that the subclasses are raised based purely on the
value of errno, little or no changes should be required in extension
modules (either standard or third-party).

The first possibility is to adapt the PyErr_SetFromErrno() family of
functions (PyErr_SetFromWindowsErr() under Windows) to raise the
appropriate OSError subclass. This wouldn't cover, however, Python code
raising OSError directly, using the following idiom (seen in
Lib/tempfile.py):

    raise IOError(_errno.EEXIST, "No usable temporary file name found")

A second possibility, suggested by Marc-Andre Lemburg, is to adapt
OSError.__new__ to instantiate the appropriate subclass. This has the
benefit of also covering Python code such as the above.

Possible objections

Namespace pollution

Making the exception hierarchy finer-grained makes the root (or
builtins) namespace larger. This is to be moderated, however, as:

-   only a handful of additional classes are proposed;
-   while standard exception types live in the root namespace, they are
    visually distinguished by the fact that they use the CamelCase
    convention, while almost all other builtins use lowercase naming
    (except True, False, None, Ellipsis and NotImplemented)

An alternative would be to provide a separate module containing the
finer-grained exceptions, but that would defeat the purpose of
encouraging careful code over careless code, since the user would first
have to import the new module instead of using names already accessible.

Earlier discussion

While this is the first time such as formal proposal is made, the idea
has received informal support in the past[2]; both the introduction of
finer-grained exception classes and the coalescing of OSError and
IOError.

The removal of WindowsError alone has been discussed and rejected as
part of another PEP <348#removing-windowserror>, but there seemed to be
a consensus that the distinction with OSError wasn't meaningful. This
supports at least its aliasing with OSError.

Implementation

The reference implementation has been integrated into Python 3.3. It was
formerly developed in http://hg.python.org/features/pep-3151/ in branch
pep-3151, and also tracked on the bug tracker at
http://bugs.python.org/issue12555. It has been successfully tested on a
variety of systems: Linux, Windows, OpenIndiana and FreeBSD buildbots.

One source of trouble has been with the respective constructors of
OSError and WindowsError, which were incompatible. The way it is solved
is by keeping the OSError signature and adding a fourth optional
argument to allow passing the Windows error code (which is different
from the POSIX errno). The fourth argument is stored as winerror and its
POSIX translation as errno. The PyErr_SetFromWindowsErr* functions have
been adapted to use the right constructor call.

A slight complication is when the PyErr_SetExcFromWindowsErr* functions
are called with OSError rather than WindowsError: the errno attribute of
the exception object would store the Windows error code (such as 109 for
ERROR_BROKEN_PIPE) rather than its POSIX translation (such as 32 for
EPIPE), which it does now. For non-socket error codes, this only occurs
in the private _multiprocessing module for which there is no
compatibility concern.

Note

For socket errors, the "POSIX errno" as reflected by the errno module is
numerically equal to the Windows Socket error code returned by the
WSAGetLastError system call:

    >>> errno.EWOULDBLOCK

10035 >>> errno.WSAEWOULDBLOCK 10035

Possible alternative

Pattern matching

Another possibility would be to introduce an advanced pattern matching
syntax when catching exceptions. For example:

    try:
        os.remove(filename)
    except OSError as e if e.errno == errno.ENOENT:
        pass

Several problems with this proposal:

-   it introduces new syntax, which is perceived by the author to be a
    heavier change compared to reworking the exception hierarchy
-   it doesn't decrease typing effort significantly
-   it doesn't relieve the programmer from the burden of having to
    remember errno mnemonics

Exceptions ignored by this PEP

This PEP ignores EOFError, which signals a truncated input stream in
various protocol and file format implementations (for example GzipFile).
EOFError is not OS- or IO-related, it is a logical error raised at a
higher level.

This PEP also ignores SSLError, which is raised by the ssl module in
order to propagate errors signalled by the OpenSSL library. Ideally,
SSLError would benefit from a similar but separate treatment since it
defines its own constants for error types (ssl.SSL_ERROR_WANT_READ,
etc.). In Python 3.2, SSLError is already replaced with socket.timeout
when it signals a socket timeout (see issue 10272).

Endly, the fate of socket.gaierror and socket.herror is not settled.
While they would deserve less cryptic names, this can be handled
separately from the exception hierarchy reorganization effort.

Appendix A: Survey of common errnos

This is a quick inventory of the various errno mnemonics checked for in
the standard library and its tests, as part of except clauses.

Common errnos with OSError

-   EBADF: bad file descriptor (usually means the file descriptor was
    closed)
-   EEXIST: file or directory exists
-   EINTR: interrupted function call
-   EISDIR: is a directory
-   ENOTDIR: not a directory
-   ENOENT: no such file or directory
-   EOPNOTSUPP: operation not supported on socket (possible confusion
    with the existing io.UnsupportedOperation)
-   EPERM: operation not permitted (when using e.g. os.setuid())

Common errnos with IOError

-   EACCES: permission denied (for filesystem operations)
-   EBADF: bad file descriptor (with select.epoll); read operation on a
    write-only GzipFile, or vice-versa
-   EBUSY: device or resource busy
-   EISDIR: is a directory (when trying to open())
-   ENODEV: no such device
-   ENOENT: no such file or directory (when trying to open())
-   ETIMEDOUT: connection timed out

Common errnos with socket.error

All these errors may also be associated with a plain IOError, for
example when calling read() on a socket's file descriptor.

-   EAGAIN: resource temporarily unavailable (during a non-blocking
    socket call except connect())
-   EALREADY: connection already in progress (during a non-blocking
    connect())
-   EINPROGRESS: operation in progress (during a non-blocking connect())
-   EINTR: interrupted function call
-   EISCONN: the socket is connected
-   ECONNABORTED: connection aborted by peer (during an accept() call)
-   ECONNREFUSED: connection refused by peer
-   ECONNRESET: connection reset by peer
-   ENOTCONN: socket not connected
-   ESHUTDOWN: cannot send after transport endpoint shutdown
-   EWOULDBLOCK: same reasons as EAGAIN

Common errnos with select.error

-   EINTR: interrupted function call

Appendix B: Survey of raised OS and IO errors

About VMSError

VMSError is completely unused by the interpreter core and the standard
library. It was added as part of the OpenVMS patches submitted in 2002
by Jean-François Piéronne[3]; the motivation for including VMSError was
that it could be raised by third-party packages.

Interpreter core

Handling of PYTHONSTARTUP raises IOError (but the error gets discarded):

    $ PYTHONSTARTUP=foox ./python
    Python 3.2a0 (py3k:82920M, Jul 16 2010, 22:53:23)
    [GCC 4.4.3] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    Could not open PYTHONSTARTUP
    IOError: [Errno 2] No such file or directory: 'foox'

PyObject_Print() raises IOError when ferror() signals an error on the
FILE * parameter (which, in the source tree, is always either stdout or
stderr).

Unicode encoding and decoding using the mbcs encoding can raise
WindowsError for some error conditions.

Standard library

bz2

Raises IOError throughout (OSError is unused):

    >>> bz2.BZ2File("foox", "rb")
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    IOError: [Errno 2] No such file or directory
    >>> bz2.BZ2File("LICENSE", "rb").read()
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    IOError: invalid data stream
    >>> bz2.BZ2File("/tmp/zzz.bz2", "wb").read()
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    IOError: file is not ready for reading

curses

Not examined.

dbm.gnu, dbm.ndbm

_dbm.error and _gdbm.error inherit from IOError:

    >>> dbm.gnu.open("foox")
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    _gdbm.error: [Errno 2] No such file or directory

fcntl

Raises IOError throughout (OSError is unused).

imp module

Raises IOError for bad file descriptors:

    >>> imp.load_source("foo", "foo", 123)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    IOError: [Errno 9] Bad file descriptor

io module

Raises IOError when trying to open a directory under Unix:

    >>> open("Python/", "r")
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    IOError: [Errno 21] Is a directory: 'Python/'

Raises IOError or io.UnsupportedOperation (which inherits from the
former) for unsupported operations:

    >>> open("LICENSE").write("bar")
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    IOError: not writable
    >>> io.StringIO().fileno()
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    io.UnsupportedOperation: fileno
    >>> open("LICENSE").seek(1, 1)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    IOError: can't do nonzero cur-relative seeks

Raises either IOError or TypeError when the inferior I/O layer
misbehaves (i.e. violates the API it is expected to implement).

Raises IOError when the underlying OS resource becomes invalid:

    >>> f = open("LICENSE")
    >>> os.close(f.fileno())
    >>> f.read()
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    IOError: [Errno 9] Bad file descriptor

...or for implementation-specific optimizations:

    >>> f = open("LICENSE")
    >>> next(f)
    'A. HISTORY OF THE SOFTWARE\n'
    >>> f.tell()
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    IOError: telling position disabled by next() call

Raises BlockingIOError (inheriting from IOError) when a call on a
non-blocking object would block.

mmap

Under Unix, raises its own mmap.error (inheriting from EnvironmentError)
throughout:

    >>> mmap.mmap(123, 10)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    mmap.error: [Errno 9] Bad file descriptor
    >>> mmap.mmap(os.open("/tmp", os.O_RDONLY), 10)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    mmap.error: [Errno 13] Permission denied

Under Windows, however, it mostly raises WindowsError (the source code
also shows a few occurrences of mmap.error):

    >>> fd = os.open("LICENSE", os.O_RDONLY)
    >>> m = mmap.mmap(fd, 16384)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    WindowsError: [Error 5] Accès refusé
    >>> sys.last_value.errno
    13
    >>> errno.errorcode[13]
    'EACCES'

    >>> m = mmap.mmap(-1, 4096)
    >>> m.resize(16384)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    WindowsError: [Error 87] Paramètre incorrect
    >>> sys.last_value.errno
    22
    >>> errno.errorcode[22]
    'EINVAL'

multiprocessing

Not examined.

os / posix

The os (or posix) module raises OSError throughout, except under Windows
where WindowsError can be raised instead.

ossaudiodev

Raises IOError throughout (OSError is unused):

    >>> ossaudiodev.open("foo", "r")
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    IOError: [Errno 2] No such file or directory: 'foo'

readline

Raises IOError in various file-handling functions:

    >>> readline.read_history_file("foo")
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    IOError: [Errno 2] No such file or directory
    >>> readline.read_init_file("foo")
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    IOError: [Errno 2] No such file or directory
    >>> readline.write_history_file("/dev/nonexistent")
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    IOError: [Errno 13] Permission denied

select

-   select() and poll objects raise select.error, which doesn't inherit
    from anything (but poll.modify() raises IOError);
-   epoll objects raise IOError;
-   kqueue objects raise both OSError and IOError.

As a side-note, not deriving from EnvironmentError means select.error
does not get the useful errno attribute. User code must check args[0]
instead:

    >>> signal.alarm(1); select.select([], [], [])
    0
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    select.error: (4, 'Interrupted system call')
    >>> e = sys.last_value
    >>> e
    error(4, 'Interrupted system call')
    >>> e.errno == errno.EINTR
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    AttributeError: 'error' object has no attribute 'errno'
    >>> e.args[0] == errno.EINTR
    True

signal

signal.ItimerError inherits from IOError.

socket

socket.error inherits from IOError.

sys

sys.getwindowsversion() raises WindowsError with a bogus error number if
the GetVersionEx() call fails.

time

Raises IOError for internal errors in time.time() and time.sleep().

zipimport

zipimporter.get_data() can raise IOError.

Acknowledgments

Significant input has been received from Alyssa Coghlan.

References

Copyright

This document has been placed in the public domain.

[1] Google Code Search of IOError in Python code: around 40000 results;
OSError: around 15200 results; EnvironmentError: around 3000 results

[2] "IO module precisions and exception hierarchy":
https://mail.python.org/pipermail/python-dev/2009-September/092130.html

[3] http://bugs.python.org/issue614055


text:
PEP: 729 Title: Typing governance process Author: Jelle Zijlstra
<EMAIL_ADDRESS>Shantanu Jain <hauntsaninja at gmail.com>
Discussions-To:
https://discuss.python.org/t/pep-729-typing-governance-process/35362
Status: Active Type: Process Topic: Governance, Typing Created:
19-Sep-2023 Post-History: 04-Oct-2023, 20-Sep-2023 Resolution:
20-Nov-2023

Abstract

This PEP proposes a new way to govern the Python type system: a council
that is responsible for maintaining and developing the Python type
system. The council will maintain a specification and conformance test
suite and will initially be appointed by the Python Steering Council.

Motivation

The Python type system was created by PEP 484, almost ten years ago. The
type system is now widely used, and typing has become an important tool
for writing good, maintainable Python code. Many changes have been made
to the type system to cover more use cases and improve usability.
Several type checkers have been created, each with their own strengths.
The type annotation syntax has driven several major innovations in the
Python ecosystem, such as the popular dataclasses module, runtime type
checking and validation by packages such as Pydantic, and static
compilation by tools such as mypyc.

However, as the type system has grown, several interrelated problems
with the current way to manage the type system have become apparent.

PEPs are the only specification

The Python type system was initially created by a PEP (PEP 484), and
changes to the type system are still made by PEPs. The specification for
the Python type system, to the extent there is one, consists of this
series of PEPs. But Standards Track PEPs aren't meant to be living
documents or specifications; they are change proposals.

An example may illustrate the problem here. Around the same time as the
introduction of the typing module by PEP 484, PEP 3156 introduced the
asyncio module, another major new feature that has been instrumental to
the success of Python 3. Both modules have evolved greatly since their
initial creation and inspired changes to the core language.

However, asyncio and typing are different in an essential aspect: a user
who uses asyncio interacts only with the standard library itself, while
a user of typing has to also think about an external tool, the type
checker. The Python language reference covers the symbols in the typing
module, but does not (and should not) go into detail on how the full
type system should be interpreted by type checkers. That material
currently exists only in the PEPs.

This problem is shared by the packaging ecosystem, which attempts to
solve it by maintaining a separate set of PyPA specifications.

It's hard to evolve the specification

Because the PEPs are the only specification we have, anything that could
be seen as a change to the specification would theoretically require a
new PEP. But that is often too heavy a process for a small change.
Sometimes changes are made directly to old PEPs instead, but that goes
against the idea that accepted and implemented PEPs become historical
documents that should no longer be changed.

Some concrete examples include:

-   PEP 484 explicitly says that typing.NoReturn cannot be used in
    argument annotations. Nevertheless, type checkers have long accepted
    such usage.
-   A 2023 discussion noted that PEP 561's description of partial stubs
    is unclear, and major type checkers did not implement it exactly as
    specified.
-   The widely used third-party typing_extensions package provides
    backports of new type system features. Type checkers are expected to
    treat symbols in this module the same as symbols in typing, but this
    is not explicitly specified in any of the PEPs.

The type system is underspecified

While the PEPs provide a specification, they are often not sufficiently
precise (sometimes intentionally so). This is especially true as the
combinatorial complexity of the type system has grown.

It ends up falling to individual type checkers to decide how to navigate
underspecified areas. In cases where type checkers informally
coordinate, this results in de facto standards that aren't clearly
recorded anywhere, making the type system less accessible to newcomers.
For example:

-   How @overload matching works
-   How ParamSpec should work with methods
-   The concept of recursive aliases
-   Semantics of variable initialization
-   Reachability semantics of annotations on __exit__
-   Symbol visibility
-   Use of NoReturn for exhaustiveness checking

The Steering Council is not well-placed to solve the above problems

The SC has the entire language in its remit, and is not well-placed to
make decisions that are purely about the type system -- if only because
they don't have the time to deal with type system arcana alongside their
other responsibilities. This is similar in spirit to the reasons why the
Steering Council sometimes uses PEP delegation.

Endorsements

This PEP was endorsed by maintainers of all major type checkers,
including Rebecca Chen (pytype), Eric Traut (Pyright), and privately by
maintainers of mypy and Pyre.

Specification

We propose the creation of a new group, the Typing Council. This group
will be responsible for developing and maintaining the Python type
system, and for solving the above problems.

The "operations and process" section describes how this group would
operate and be governed.

The more exciting "projects" section describes solutions to the above
problems that the Typing Council could shepherd.

Mandate

The Typing Council's mandate is to ensure that the Python type system
is:

-   Useful: The type system should serve common use cases. As identified
    by PEP 484, the primary use case is static analysis, but there are
    others, such as runtime type checking, static compilation, IDE
    support, and documentation. The Typing Council should consider all
    of these use cases when making decisions, and be open to supporting
    additional use cases as they come up.
-   Usable: The type system should be easy to use for Python developers.
    It should be ergonomic to write well-typed Python code that is
    accepted by type checkers. There should be good documentation for
    the type system.
-   Stable: As the type system matures, users should be able to rely on
    their typed code continuing to work and be able to trust their
    mental model for the type system. Changes should be made with care
    and in a way that minimizes disruption. Nevertheless, the type
    system should be able to evolve, and it does not make sense to use
    the same compatibility guidelines for type checker behavior as for
    Python itself. Of course, the existence and runtime behavior of
    objects in the typing module does follow Python's standard
    compatibility policy in PEP 387.

Operations and process

The council would have three to five members, comprised of prominent
community members, such as Python core developers and maintainers of
major type checkers. The members should include people affiliated with a
variety of projects related to type checking, which may include type
checkers, CPython, typeshed, or other projects.

The initial members of the council are:

-   Eric Traut (Pyright; author of PEP 647, PEP 681, and PEP 695)
-   Guido van Rossum (core developer; author of PEP 484 and PEP 526)
-   Jelle Zijlstra (core developer; typeshed; pyanalyze; author of PEP
    688 and PEP 702)
-   Rebecca Chen (pytype)
-   Shantanu Jain (core developer; typeshed; mypy)

Current membership of the council is recorded in the
python/typing-council repository.

There is no term limit for council members. Council members may resign
their position at any time. There is an expectation that each member
serves at most five consecutive years before resigning.

If there is a vacancy and there are three or more remaining members, it
is up to the Council to decide whether to appoint a new member. To
determine replacements, nominations will be collected from the typing
community. Self-nominations are allowed. The existing Typing Council
will then decide the replacement member(s) from the nominees. The
expectation is that this would be done by fiat, but the Typing Council
can choose a replacement by any means they see fit, including a vote.

The Typing Council remains accountable to the Steering Council. At any
point, for any reason, the Steering Council could (publicly or
privately) make a specific change or request a non-specific change to
the composition of the Typing Council.

We acknowledge that this is a not particularly democratic structure and
puts a lot of faith in the Typing Council. However, the Python community
has a long history of success with not particularly democratic
structures! We believe self-governance, cycling of membership, and
accountability to the Steering Council will be sufficient to ensure that
the Typing Council is meeting the needs of the community.

The council would operate primarily through reviews of GitHub PRs.
Regular meetings are likely not necessary, but the council may set up
video calls, a private chat, or whatever other mechanism they decide
upon internally.

The council should aim for transparency, posting all decisions publicly
on discuss.python.org, with a rationale if possible. Before making a
decision, the council should give all interested community members a
chance to weigh in. There should be at least a week between the start of
a discussion and the council's decision.

Members of the council will be eligible to sponsor PEPs. If this PEP is
accepted, PEP 1 should be amended to note this fact.

Relationship with the Steering Council

Just like today, the Python Steering Council would remain responsible
for the overall direction of the Python language and would continue to
decide on typing-related PEPs. The Typing Council would provide written
opinions and recommendations to the Steering Council on typing-related
PEPs.

However, smaller changes to the type system could be made by the Typing
Council directly. The Steering Council could also choose to delegate
decisions on some PEPs to the Typing Council (exactly as any other PEP
delegation).

Some examples of how past and recent issues could have been handled
under this model:

-   A PEP like PEP 695 (type parameter syntax), which changes the
    language syntax, would need to be decided upon by the Steering
    Council; the Typing Council would merely provide opinion or
    endorsement. Similarly, PEPs like PEP 702 (deprecations) would be
    decided upon by the Steering Council, because it concerns runtime
    behaviour beyond pure typing. Other examples that would need to be
    decided by the SC include PEP 718 (subscriptable functions) and PEP
    727 (documentation metadata).
-   A PEP like PEP 698 (@override), which affects only users of type
    checkers and does not change the overall language, would also by
    default be decided upon by the Steering Council. However, such PEPs
    could be delegated to the Typing Council for a decision (like any
    other PEP delegation). Other examples of PEPs that could potentially
    be delegated include PEP 647 (type guards), PEP 655 (individual
    required TypedDict items), PEP 673 (Self), and PEP 675 (Literal).
-   Adding a smaller feature, such as typing.Never as an alias for
    typing.NoReturn, would be done by means of a PR to the spec and
    conformance test suite. The Typing Council would then decide whether
    or not to merge the PR. They may ask for the feature to be specified
    and discussed in a PEP if they feel that is warranted.
-   If there is confusion about the interpretation of some part of the
    spec, like happened recently with partial stubs in PEP 561, somebody
    would make a PR to the typing specification to clarify the spec, and
    then the Typing Council would decide on the spec change.

The runtime typing module will continue to be maintained by the CPython
core developer team. However, any changes to the runtime module that
affect type checker behavior should be made in conjunction with a change
to the specification (see below) and should be approved by the Typing
Council. For example, in Python 3.11 the core developers added the new
function typing.assert_type. If the Typing Council had been in place,
this change would require a matching change to the specification and
approval by the Typing Council. On the other hand, Python 3.11 also
added the typing.get_overloads introspection helper. As this function
does not affect type checker behavior, it would not require approval by
the Typing Council. However, as support for runtime type checkers is
within the remit of the Council, they should monitor such changes and
provide feedback when appropriate.

Relationship with type checkers

The Typing Council has no direct authority over type checkers; it cannot
force them to implement particular features or make behavior changes.
Type checkers are incentivized to follow the specification set out by
the Council because it allows them to take advantage of shared
resources, such as libraries that expose typing information that follows
the specification, the stub files in typeshed, the typing standard
library module, and user documentation that covers the standard type
system. Type checkers are free to extend the type system or deviate from
the specification, but they should document such differences clearly.

The fact that type checkers need to implement any decisions made by the
Typing Council acts as a useful brake on the Council, ensuring that its
decisions are conservative and well-considered. Individual type checkers
remain free to innovate as they see fit, and successful innovations can
be incorporated into the standard type system.

Projects

Here are some efforts a Typing Council would be responsible for.

Conformance test suite

A conformance test suite would provide machine checkable documentation
for how type checkers should check Python code, accompanied by the
results of major type checker implementations on the test suite. A rough
sketch for what this could look like was created by Shantanu.

This would contain prescriptive tests from behavior prescribed by
previous PEPs and descriptive tests that let us document behavior of
existing implementations in areas that are not prescribed by any
standard. These descriptions would be useful to inform efforts below and
to identify areas of focus for standardization.

Specification for the type system

A specification would initially be created by stitching together the
specification sections from the existing PEPs, and then gradually
improved to clarify points of confusion and cover more areas. A draft of
such a stitched-together spec was created by Jelle.

The specification has a few audiences:

-   For type checkers, it provides a description of how an idealized
    type checker should behave. Individual type checkers have different
    goals and technical constraints and they are free to deviate from
    the spec if they do not have the resources to fully implement it or
    if they believe a different behavior better serves their users.
    However, they should document such deviations from the spec.
-   For projects such as typeshed, or libraries that want to be
    compatible with multiple type checkers, it provides a set of rules
    that they can follow to make their code understood by type checkers.
-   For people who want to propose changes to the type system, it
    provides a foundation for any new proposals.

Notably, the specification is not aimed at application developers who
use typing. Such users typically do not need to worry about
compatibility across type checkers. They are better served by a more
informal user-facing reference, which is discussed in the next section.

There are different opinions within the community about how formal such
a specification should be. While this document recommends an incremental
approach that builds off existing specification, it does not aim to
prescribe a final state. The Typing Council would provide a mechanism to
allow the specification to evolve to meet the level of formality that
the community desires, for instance, by incorporating parts of Kevin
Millikin's document on "Python Static Types" as a means to achieve a
better formalisation of the spec.

Proposed changes to the specification, including PEPs, should generally
be accompanied by the following:

-   Buy-in from type checker maintainers to confirm that the change can
    be implemented and maintained within their type checkers.
-   For changes to existing features, a survey of the behavior of
    existing type checkers. If existing type checkers behave roughly
    similarly, that is evidence that their shared behavior should be
    made part of the specification.
-   Changes to the conformance test suite that demonstrate the specified
    behavior.

User-facing reference for the type system

Documentation is important for the success of the Python type system, so
the Typing Council should ensure that there is good documentation for
the type system.

As mentioned previously, PEPs are point in time change proposals aimed
at multiple audiences that are hard to clarify. This makes them
ill-suited as user documentation. The specification discussed in the
previous section would be a living document, but it would likely be too
technical to serve as documentation for normal usage.

Therefore, a separate user-facing reference for the type system would be
useful. Such an effort could expand the documentation on
typing.readthedocs.io and reuse material from the documentation sections
of individual type checkers and the CPython documentation.

Amendments

This PEP serves as a charter for the Typing Council. Changes to its
operation can be made either through a new PEP or through a change to
this PEP. In either case, the change would be decided upon by the
Steering Council after discussion in the community.

Rejected ideas

Writing the specification from scratch

This PEP proposes creating the typing specification by starting from the
existing PEPs, then clarifying and improving the specification as
necessary. Some members of the community prefer to start from scratch,
writing a new, more formal specification covering the entire type
system. This could provide a more solid basis for the specification.

However, this would be a much larger undertaking. The existing
formalization effort by Kevin Millikin is a good start, but so far
covers only a subset of PEP 484. Covering the rest of the type system
would likely require several times more effort when we consider that
major type system features such as typing.Protocol, typing.Literal, and
typing.TypedDict were introduced only after PEP 484. It is not clear
that there is even energy in the community for such a huge undertaking.
Even if someone steps up to do all the work of putting together a
specification, lots of effort would be required from community members
and type checker maintainers to consider whether the specification
accurately reflects current behavior, and if not, whether the
specification or the type checkers should change.

Starting with the existing PEPs creates a lower-quality specification,
but it means that the Typing Council can immediately start making a
difference anywhere in the type system by improving and clarifying the
specification. A formalization effort can still proceed by gradually
replacing sections of the specification.

Alternate governance mechanisms

An earlier draft of this PEP suggested that the Steering Council appoint
members of the Typing Council each year. The current Steering Council
suggested that it would be better to have the Typing Council
self-organise and avoid the need for the Steering Council to
continuously supervise the Typing Council.

Alternate governance mechanisms are possible, including more democratic
ones, but these typically raise several thorny questions, require much
heavier amounts of process and are potentially more divisive. For
example, see the PEP 8000 series, or recent discussions about
alternative governance in other Python subcommunities. Ultimately, the
Typing Council exists under the authority of the Steering Council, and
so can rely on it to bootstrap governance and serve as an accountability
mechanism.

Do nothing

We are hopeful substantial progress will be made on projects that
improve the type system regardless of whether this PEP is accepted. We
anticipate projects like specification or the potential for PEP
delegation would benefit more from a Typing Council, and projects like
end user documentation would benefit less. Certainly the bottleneck is
likely to be contributor effort, not governance.

However, currently the tools available to the community to resolve
potential contention are either establishment of approximate consensus
or the exercise of power by individual projects or contributors. While
very valuable, the former is a slow process that can often end in
inaction. The latter can result in a less consistent ecosystem. Finally,
easily legible governance structures make the community more accessible
and equitable.

Contact

To ask the Typing Council for a decision, community members may open an
issue in the python/typing-council repository.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

authors:
Jelle Zijlstra
Shantanu Jain


text:
PEP: 541 Title: Package Index Name Retention Version: $Revision$
Last-Modified: $Date$ Author: Łukasz Langa<EMAIL_ADDRESS>BDFL-Delegate: Mark Mangoba<EMAIL_ADDRESS>Discussions-To:
<EMAIL_ADDRESS>Status: Final Type: Process Topic: Packaging
Content-Type: text/x-rst Created: 12-Jan-2017 Post-History: Resolution:
https://mail.python.org/pipermail/distutils-sig/2018-March/032089.html

Abstract

This PEP proposes an extension to the Terms of Use[1] of the Package
Index[2], clarifying expectations of package owners regarding ownership
of a package name on the Package Index, specifically with regards to
conflict resolution.

Existing package repositories such as CPAN[3], NPM[4], and GitHub[5]
will be investigated as prior art in this field.

Rationale

Given that package names on the Index are sharing a single flat
namespace, a unique name is a finite resource. The growing age of the
Package Index causes a constant rise of situations of conflict between
the current use of the name and a different suggested use of the same
name.

This document aims to provide general guidelines for solving the most
typical cases of such conflicts.

Approval Process

As the application of this policy has potential legal ramifications for
the Python Software Foundation, the approval process used is more formal
than that used for most PEPs.

Rather than accepting the PEP directly, the assigned BDFL-Delegate will
instead recommend its acceptance to the PSF's Packaging Working Group.
After consultation with the PSF's General Counsel, adoption of the
policy will then be subject to a formal vote within the working group.

This formal approval process will be used for both initial adoption of
the policy, and for adoption of any future amendments.

Specification

The main idea behind this document is that the Package Index serves the
community. Every user is invited to upload content to the Package Index
under the Terms of Use, understanding that it is at the sole risk of the
user.

While the Package Index is not a backup service, the maintainers of the
Package Index do their best to keep that content accessible indefinitely
in its published form. However, in certain edge cases the greater
community's needs might overweigh the individual's expectation of
ownership of a package name.

The use cases covered by this document are:

-   Abandoned projects:
    -   continued maintenance by a different set of users; or
    -   removal from the Index for use with a different project.
-   Active projects:
    -   resolving disputes over a name.
-   Invalid projects:
    -   projects subject to a claim of intellectual property
        infringement.

The proposed extension to the Terms of Use, as expressed in the
Implementation section, will be published as a separate document on the
Package Index, linked next to existing Terms of Use in the front page
footer.

Implementation

Reachability

The user of the Package Index is solely responsible for being reachable
by the Package Index maintainers for matters concerning projects that
the user owns. In every case where contacting the user is necessary, the
maintainers will try to do so at least three times, using the following
means of contact:

-   the e-mail address on file in the user's profile on the Package
    Index;
-   the e-mail address listed in the Author field for a given project
    uploaded to the Index; and
-   any e-mail addresses found in the given project's documentation on
    the Index or on the listed Home Page.

The maintainers stop trying to reach the user after six weeks.

Abandoned projects

A project is considered abandoned when ALL of the following are met:

-   owner not reachable (see Reachability above);
-   no releases within the past twelve months; and
-   no activity from the owner on the project's home page (or no home
    page listed).

All other projects are considered active.

Continued maintenance of an abandoned project

If a candidate appears willing to continue maintenance on an abandoned
project, ownership of the name is transferred when ALL of the following
are met:

-   the project has been determined abandoned by the rules described
    above;
-   the candidate is able to demonstrate their own failed attempts to
    contact the existing owner;
-   the candidate is able to demonstrate improvements made on the
    candidate's own fork of the project;
-   the candidate is able to demonstrate why a fork under a different
    name is not an acceptable workaround; and
-   the maintainers of the Package Index don't have any additional
    reservations.

Under no circumstances will a name be reassigned against the wishes of a
reachable owner.

Removal of an abandoned project

Projects are never removed from the Package Index solely on the basis of
abandonment. Artifacts uploaded to the Package Index hold inherent
historical value.

An abandoned project can be transferred to a new owner for purposes of
reusing the name when ALL of the following are met:

-   the project has been determined abandoned by the rules described
    above;
-   the candidate is able to demonstrate their own failed attempts to
    contact the existing owner;
-   the candidate is able to demonstrate that the project suggested to
    reuse the name already exists and meets notability requirements;
-   the candidate is able to demonstrate why a fork under a different
    name is not an acceptable workaround;
-   download statistics on the Package Index for the existing package
    indicate project is not being used; and
-   the maintainers of the Package Index don't have any additional
    reservations.

Name conflict resolution for active projects

The maintainers of the Package Index are not arbiters in disputes around
active projects. There are many possible scenarios here, a non-exclusive
list describing some real-world examples is presented below. None of the
following qualify for package name ownership transfer:

1.  User A and User B share project X. After some time they part ways
    and each of them wants to continue the project under name X.
2.  User A owns a project X outside the Package Index. User B creates a
    package under the name X on the Index. After some time, User A wants
    to publish project X on the Index but realizes name is taken. This
    is true even if User A's project X gains notability and the User B's
    project X is not notable.
3.  User A publishes project X to the Package Index. After some time
    User B proposes bug fixes to the project but no new release is
    published by User A. This is true even if User A agrees to publish a
    new version and later doesn't, even if User B's changes are merged
    to the source code repository for project X.

Again, the list above is not exclusive. The maintainers of the Package
Index recommend users to get in touch with each other and solve the
issue by respectful communication (see the PSF Code of Conduct[6]).

Invalid projects

A project published on the Package Index meeting ANY of the following is
considered invalid and will be removed from the Index:

-   project does not conform to Terms of Use;
-   project is malware (designed to exploit or harm systems or users
    directly, to facilitate command-and-control attacks, or perform data
    exfiltration);
-   project is spam (designed to advertise or solicit goods or
    services);
-   project contains illegal content;
-   project violates copyright, trademarks, patents, or licenses;
-   project is name squatting (package has no functionality or is
    empty);
-   project name, description, or content violates the Code of Conduct;
-   project uses obfuscation to hide or mask functionality; or
-   project is abusing the Package Index for purposes it was not
    intended.

The Package Index maintainers pre-emptively declare certain package
names as unavailable for security reasons.

Intellectual property policy

It is the policy of Python Software Foundation and the Package Index
maintainers to be appropriately responsive to claims of intellectual
property infringement by third parties. It is not the policy of the
Python Software Foundation nor the Package Index maintainers to
pre-screen uploaded packages for any type of intellectual property
infringement.

Possibly-infringing packages should be reported to<EMAIL_ADDRESS>and
counsel to the Python Software Foundation will determine an appropriate
response. A package can be removed or transferred to a new owner at the
sole discretion of the Python Software Foundation to address a claim of
infringement.

A project published on the Package Index meeting ANY of the following
may be considered infringing and subject to removal from the Index or
transferral to a new owner:

-   project contains unlicensed copyrighted material from a third party,
    and is subject to a properly made claim under the DMCA;
-   project uses a third party's trademark in a way not covered by
    nominal or fair use guidelines;
-   project clearly implicates a patented system or process, and is the
    subject of a complaint; or
-   project is subject to an active lawsuit.

In the event of a complaint for intellectual property infringement, a
copy of the complaint will be sent to the package owner. In some cases,
action may be taken by the Package Index maintainers before the owner
responds.

The role of the Python Software Foundation

The Python Software Foundation[7] is the non-profit legal entity that
provides the Package Index as a community service.

The Package Index maintainers can escalate issues covered by this
document for resolution by the Packaging Workgroup if the matter is not
clear enough. Some decisions require additional judgement by the Board,
especially in cases of Code of Conduct violations or legal claims.
Recommendations made by the Board are sent to the Packaging Workgroup[8]
for review.

The Packaging Workgroup has the final say in any disputes covered by
this document and can decide to reassign or remove a project from the
Package Index after careful consideration even when not all requirements
listed here are met.

How to request a name transfer

If you want to take over an existing project name on PyPI, these are the
steps to follow:

1.  Try to contact the current owner(s) directly: email them and open an
    issue if you can find a related repository. The processes described
    here are meant as a last resort if the owner cannot be contacted.
2.  Check the criteria above to see when a transfer is allowed. In
    particular, the criteria for reusing a name for a different project
    are more stringent than for continuing maintenance of the same
    project - although it's not easy to get a name transferred in either
    case.
3.  Search the PyPI Support issues to see if anyone else is already
    requesting the same name.
4.  If all the criteria are met to transfer ownership of the name, open
    a new issue to request it, detailing why you believe each relevant
    criterion is satisfied.

Prior art

NPM contains a separate section linked from the front page called
Package Name Disputes. It is described as a "living document", as of
January 2017 its contents might be summarized as follows:

-   package name squatting is prohibited;
-   users wanting to reuse a project name are required to contact the
    existing author, with cc to<EMAIL_ADDRESS>-   all contact must conform to the NPM Code of Conduct;
-   in case of no resolution after a few weeks, npm inc. holds the right
    to the final decision in the matter.

CPAN lets any user upload modules with the same name. PAUSE, a related
index, only lists modules uploaded by the primary maintainer or listed
co-maintainers. CPAN documentation doesn't address disputes otherwise.

GitHub's terms of service contain an exhaustive list of behavior not
meeting general conditions of use. While not codified anywhere, GitHub
does agree for users to reclaim abandoned account names by archiving the
abandoned account and letting the other user or organization rename
their account. This is done on a case-by-case basis.

Rejected Proposals

The original approach was to hope for the best and solve issues as they
arise without written policy. This is not sustainable. The lack of
generally available guidelines in writing on package name conflict
resolution is causing unnecessary tensions. From the perspective of
users, decisions made by the Package Index maintainers without written
guidelines may appear arbitrary. From the perspective of the Package
Index maintainers, solving name conflicts is a stressful task due to
risk of unintentional harm due to lack of defined policy.

References

Copyright

This document has been placed in the public domain.

Acknowledgements

The many participants of the Distutils and Catalog SIGs for their ideas
over the years.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 End:

[1] Terms of Use of the Python Package Index
(https://pypi.org/policy/terms-of-use/)

[2] The Python Package Index (https://pypi.org/)

[3] The Comprehensive Perl Archive Network (http://www.cpan.org/)

[4] Node Package Manager (https://www.npmjs.com/package/left-pad)

[5] GitHub (https://github.com/)

[6] Python Community Code of Conduct
(https://www.python.org/psf/codeofconduct/)

[7] Python Software Foundation (https://www.python.org/psf/)

[8] Python Packaging Working Group
(https://wiki.python.org/psf/PackagingWG/)


text:
PEP: 244 Title: The directive statement Author: Martin von Löwis
<EMAIL_ADDRESS>Status: Rejected Type: Standards Track
Content-Type: text/x-rst Created: 20-Mar-2001 Python-Version: 2.1
Post-History:

Motivation

From time to time, Python makes an incompatible change to the advertised
semantics of core language constructs, or changes their accidental
(implementation-dependent) behavior in some way. While this is never
done capriciously, and is always done with the aim of improving the
language over the long term, over the short term it's contentious and
disrupting.

PEP 5, Guidelines for Language Evolution suggests ways to ease the pain,
and this PEP introduces some machinery in support of that.

PEP 227, Statically Nested Scopes is the first application, and will be
used as an example here.

When a new, potentially incompatible language feature is added, some
modules and libraries may chose to use it, while others may not. This
specification introduces a syntax where a module author can denote
whether a certain language feature is used in the module or not.

In discussion of this PEP, readers commented that there are two kinds of
"settable" language features:

-   those that are designed to eventually become the only option, at
    which time specifying use of them is not necessary anymore. The
    features for which the syntax of the "Back to the __future__" PEP
    236, Back to the __future__ was proposed fall into this category.
    This PEP supports declaring such features, and supports phasing out
    the "old" meaning of constructs whose semantics has changed under
    the new feature. However, it defines no policy as to what features
    must be phased out eventually.
-   those which are designed to stay optional forever, e.g. if they
    change some default setting in the interpreter. An example for such
    settings might be the request to always emit line-number
    instructions for a certain module; no specific flags of that kind
    are proposed in this specification.

Since a primary goal of this PEP is to support new language constructs
without immediately breaking old libraries, special care was taken not
to break old libraries by introducing the new syntax.

Syntax

A directive_statement is a statement of the form:

    directive_statement: 'directive' ``NAME`` [atom] [';'] NEWLINE

The name in the directive indicates the kind of the directive; it
defines whether the optional atom can be present, and whether there are
further syntactical or semantical restrictions to the atom. In addition,
depending on the name of the directive, certain additional syntactical
or semantical restrictions may be placed on the directive (e.g.
placement of the directive in the module may be restricted to the top of
the module).

In the directive_statement, directive is a new keyword. According to PEP
5, this keyword is initially considered as a keyword only when used in a
directive statement, see "Backwards Compatibility" below.

Semantics

A directive statement instructs the Python interpreter to process a
source file in a different way; the specific details of that processing
depend on the directive name. The optional atom is typically interpreted
when the source code is processed; details of that interpretation depend
on the directive.

Specific Directives: transitional

If a syntactical or semantical change is added to Python which is
incompatible, PEP 5 mandates a transitional evolution of the language,
where the new feature is initially available alongside with the old one.
Such a transition is possible by means of the transitional directive.

In a transitional directive, the NAME is 'transitional'. The atom MUST
be present, and it MUST be a NAME. The possible values for that name are
defined when the language change is defined. One example for such a
directive is:

    directive transitional nested_scopes

The transitional directive MUST occur at before any other statement in a
module, except for the documentation string (i.e. it may appear as the
second statement of a module only if the first statement is a STRING+).

Backwards Compatibility

Introducing directive as a new keyword might cause incompatibilities
with existing code. Following the guideline in PEP 5, in the initial
implementation of this specification, directive is a new keyword only if
it was used in a valid directive_statement (i.e. if it appeared as the
first non-string token in a module).

Unresolved Problems: directive as the first identifier

Using directive in a module as:

    directive = 1

(i.e. the name directive appears as the first thing in a module) will
treat it as keyword, not as identifier. It would be possible to classify
it as a NAME with an additional look-ahead token, but such look-ahead is
not available in the Python tokenizer.

Questions and Answers

Q: It looks like this PEP was written to allow definition of source code
character sets. Is that true?

A: No. Even though the directive facility can be extended to allow
source code encodings, no specific directive is proposed.

Q: Then why was this PEP written at all?

A: It acts as a counter-proposal to PEP 236, which proposes to overload
the import statement with a new meaning. This PEP allows to solve the
problem in a more general way.

Q: But isn't mixing source encodings and language changes like mixing
apples and oranges?

A: Perhaps. To address the difference, the predefined "transitional"
directive has been defined.

Copyright

This document has been placed in the public domain.


text:
PEP: 3132 Title: Extended Iterable Unpacking Version: $Revision$
Last-Modified: $Date$ Author: Georg Brandl<EMAIL_ADDRESS>Status:
Final Type: Standards Track Content-Type: text/x-rst Created:
30-Apr-2007 Python-Version: 3.0 Post-History:

Abstract

This PEP proposes a change to iterable unpacking syntax, allowing to
specify a "catch-all" name which will be assigned a list of all items
not assigned to a "regular" name.

An example says more than a thousand words:

    >>> a, *b, c = range(5)
    >>> a
    0
    >>> c
    4
    >>> b
    [1, 2, 3]

Rationale

Many algorithms require splitting a sequence in a "first, rest" pair.
With the new syntax, :

    first, rest = seq[0], seq[1:]

is replaced by the cleaner and probably more efficient:

    first, *rest = seq

For more complex unpacking patterns, the new syntax looks even cleaner,
and the clumsy index handling is not necessary anymore.

Also, if the right-hand value is not a list, but an iterable, it has to
be converted to a list before being able to do slicing; to avoid
creating this temporary list, one has to resort to :

    it = iter(seq)
    first = it.next()
    rest = list(it)

Specification

A tuple (or list) on the left side of a simple assignment (unpacking is
not defined for augmented assignment) may contain at most one expression
prepended with a single asterisk (which is henceforth called a "starred"
expression, while the other expressions in the list are called
"mandatory"). This designates a subexpression that will be assigned a
list of all items from the iterable being unpacked that are not assigned
to any of the mandatory expressions, or an empty list if there are no
such items.

For example, if seq is a sliceable sequence, all the following
assignments are equivalent if seq has at least two elements:

    a, b, c = seq[0], list(seq[1:-1]), seq[-1]
    a, *b, c = seq
    [a, *b, c] = seq

It is an error (as it is currently) if the iterable doesn't contain
enough items to assign to all the mandatory expressions.

It is also an error to use the starred expression as a lone assignment
target, as in :

    *a = range(5)

This, however, is valid syntax:

    *a, = range(5)

Note that this proposal also applies to tuples in implicit assignment
context, such as in a for statement:

    for a, *b in [(1, 2, 3), (4, 5, 6, 7)]:
        print(b)

would print out :

    [2, 3]
    [5, 6, 7]

Starred expressions are only allowed as assignment targets, using them
anywhere else (except for star-args in function calls, of course) is an
error.

Implementation

Grammar change

This feature requires a new grammar rule:

    star_expr: ['*'] expr

In these two rules, expr is changed to star_expr:

    comparison: star_expr (comp_op star_expr)*
    exprlist: star_expr (',' star_expr)* [',']

Changes to the Compiler

A new ASDL expression type Starred is added which represents a starred
expression. Note that the starred expression element introduced here is
universal and could later be used for other purposes in non-assignment
context, such as the yield *iterable proposal.

The compiler is changed to recognize all cases where a starred
expression is invalid and flag them with syntax errors.

A new bytecode instruction, UNPACK_EX, is added, whose argument has the
number of mandatory targets before the starred target in the lower 8
bits and the number of mandatory targets after the starred target in the
upper 8 bits. For unpacking sequences without starred expressions, the
old UNPACK_ITERABLE opcode is kept.

Changes to the Bytecode Interpreter

The function unpack_iterable() in ceval.c is changed to handle the
extended unpacking, via an argcntafter parameter. In the UNPACK_EX case,
the function will do the following:

-   collect all items for mandatory targets before the starred one
-   collect all remaining items from the iterable in a list
-   pop items for mandatory targets after the starred one from the list
-   push the single items and the resized list on the stack

Shortcuts for unpacking iterables of known types, such as lists or
tuples, can be added.

The current implementation can be found at the SourceForge Patch tracker
[SFPATCH]. It now includes a minimal test case.

Acceptance

After a short discussion on the python-3000 list[1], the PEP was
accepted by Guido in its current form. Possible changes discussed were:

-   Only allow a starred expression as the last item in the exprlist.
    This would simplify the unpacking code a bit and allow for the
    starred expression to be assigned an iterator. This behavior was
    rejected because it would be too surprising.
-   Try to give the starred target the same type as the source iterable,
    for example, b in a, *b = 'hello' would be assigned the string
    'ello'. This may seem nice, but is impossible to get right
    consistently with all iterables.
-   Make the starred target a tuple instead of a list. This would be
    consistent with a function's *args, but make further processing of
    the result harder.

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

SFPATCH

    https://bugs.python.org/issue1711529

[1] https://mail.python.org/pipermail/python-3000/2007-May/007198.html