text:
PEP: 428 Title: The pathlib module -- object-oriented filesystem paths
Version: $Revision$ Last-Modified: $Date$ Author: Antoine Pitrou
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 30-Jul-2012 Python-Version: 3.4 Post-History:
05-Oct-2012 Resolution:
https://mail.python.org/pipermail/python-dev/2013-November/130424.html

Abstract

This PEP proposes the inclusion of a third-party module, pathlib, in the
standard library. The inclusion is proposed under the provisional label,
as described in PEP 411. Therefore, API changes can be done, either as
part of the PEP process, or after acceptance in the standard library
(and until the provisional label is removed).

The aim of this library is to provide a simple hierarchy of classes to
handle filesystem paths and the common operations users do over them.

Related work

An object-oriented API for filesystem paths has already been proposed
and rejected in PEP 355. Several third-party implementations of the idea
of object-oriented filesystem paths exist in the wild:

-   The historical path.py module by Jason Orendorff, Jason R. Coombs
    and others, which provides a str-subclassing Path class;
-   Twisted's slightly specialized FilePath class;
-   An AlternativePathClass proposal, subclassing tuple rather than str;
-   Unipath, a variation on the str-subclassing approach with two public
    classes, an AbstractPath class for operations which don't do I/O and
    a Path class for all common operations.

This proposal attempts to learn from these previous attempts and the
rejection of PEP 355.

Implementation

The implementation of this proposal is tracked in the pep428 branch of
pathlib's Mercurial repository.

Why an object-oriented API

The rationale to represent filesystem paths using dedicated classes is
the same as for other kinds of stateless objects, such as dates, times
or IP addresses. Python has been slowly moving away from strictly
replicating the C language's APIs to providing better, more helpful
abstractions around all kinds of common functionality. Even if this PEP
isn't accepted, it is likely that another form of filesystem handling
abstraction will be adopted one day into the standard library.

Indeed, many people will prefer handling dates and times using the
high-level objects provided by the datetime module, rather than using
numeric timestamps and the time module API. Moreover, using a dedicated
class allows to enable desirable behaviours by default, for example the
case insensitivity of Windows paths.

Proposal

Class hierarchy

The pathlib module implements a simple hierarchy of classes:

    +----------+
    |          |
    ---------| PurePath |--------
    |        |          |       |
    |        +----------+       |
    |             |             |
    |             |             |
    v             |             v
    +---------------+    |    +-----------------+
    |               |    |    |                 |
    | PurePosixPath |    |    | PureWindowsPath |
    |               |    |    |                 |
    +---------------+    |    +-----------------+
    |             v             |
    |          +------+         |
    |          |      |         |
    |   -------| Path |------   |
    |   |      |      |     |   |
    |   |      +------+     |   |
    |   |                   |   |
    |   |                   |   |
    v   v                   v   v
    +-----------+           +-------------+
    |           |           |             |
    | PosixPath |           | WindowsPath |
    |           |           |             |
    +-----------+           +-------------+

This hierarchy divides path classes along two dimensions:

-   a path class can be either pure or concrete: pure classes support
    only operations that don't need to do any actual I/O, which are most
    path manipulation operations; concrete classes support all the
    operations of pure classes, plus operations that do I/O.
-   a path class is of a given flavour according to the kind of
    operating system paths it represents. pathlib implements two
    flavours: Windows paths for the filesystem semantics embodied in
    Windows systems, POSIX paths for other systems.

Any pure class can be instantiated on any system: for example, you can
manipulate PurePosixPath objects under Windows, PureWindowsPath objects
under Unix, and so on. However, concrete classes can only be
instantiated on a matching system: indeed, it would be error-prone to
start doing I/O with WindowsPath objects under Unix, or vice-versa.

Furthermore, there are two base classes which also act as
system-dependent factories: PurePath will instantiate either a
PurePosixPath or a PureWindowsPath depending on the operating system.
Similarly, Path will instantiate either a PosixPath or a WindowsPath.

It is expected that, in most uses, using the Path class is adequate,
which is why it has the shortest name of all.

No confusion with builtins

In this proposal, the path classes do not derive from a builtin type.
This contrasts with some other Path class proposals which were derived
from str. They also do not pretend to implement the sequence protocol:
if you want a path to act as a sequence, you have to lookup a dedicated
attribute (the parts attribute).

The key reasoning behind not inheriting from str is to prevent
accidentally performing operations with a string representing a path and
a string that doesn't, e.g. path + an_accident. Since operations with a
string will not necessarily lead to a valid or expected file system
path, "explicit is better than implicit" by avoiding accidental
operations with strings by not subclassing it. A blog post by a Python
core developer goes into more detail on the reasons behind this specific
design decision.

Immutability

Path objects are immutable, which makes them hashable and also prevents
a class of programming errors.

Sane behaviour

Little of the functionality from os.path is reused. Many os.path
functions are tied by backwards compatibility to confusing or plain
wrong behaviour (for example, the fact that os.path.abspath() simplifies
".." path components without resolving symlinks first).

Comparisons

Paths of the same flavour are comparable and orderable, whether pure or
not:

    >>> PurePosixPath('a') == PurePosixPath('b')
    False
    >>> PurePosixPath('a') < PurePosixPath('b')
    True
    >>> PurePosixPath('a') == PosixPath('a')
    True

Comparing and ordering Windows path objects is case-insensitive:

    >>> PureWindowsPath('a') == PureWindowsPath('A')
    True

Paths of different flavours always compare unequal, and cannot be
ordered:

    >>> PurePosixPath('a') == PureWindowsPath('a')
    False
    >>> PurePosixPath('a') < PureWindowsPath('a')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: unorderable types: PurePosixPath() < PureWindowsPath()

Paths compare unequal to, and are not orderable with instances of
builtin types (such as str) and any other types.

Useful notations

The API tries to provide useful notations all the while avoiding magic.
Some examples:

    >>> p = Path('/home/antoine/pathlib/setup.py')
    >>> p.name
    'setup.py'
    >>> p.suffix
    '.py'
    >>> p.root
    '/'
    >>> p.parts
    ('/', 'home', 'antoine', 'pathlib', 'setup.py')
    >>> p.relative_to('/home/antoine')
    PosixPath('pathlib/setup.py')
    >>> p.exists()
    True

Pure paths API

The philosophy of the PurePath API is to provide a consistent array of
useful path manipulation operations, without exposing a hodge-podge of
functions like os.path does.

Definitions

First a couple of conventions:

-   All paths can have a drive and a root. For POSIX paths, the drive is
    always empty.
-   A relative path has neither drive nor root.
-   A POSIX path is absolute if it has a root. A Windows path is
    absolute if it has both a drive and a root. A Windows UNC path (e.g.
    \\host\share\myfile.txt) always has a drive and a root (here,
    \\host\share and \, respectively).
-   A path which has either a drive or a root is said to be anchored.
    Its anchor is the concatenation of the drive and root. Under POSIX,
    "anchored" is the same as "absolute".

Construction

We will present construction and joining together since they expose
similar semantics.

The simplest way to construct a path is to pass it its string
representation:

    >>> PurePath('setup.py')
    PurePosixPath('setup.py')

Extraneous path separators and "." components are eliminated:

    >>> PurePath('a///b/c/./d/')
    PurePosixPath('a/b/c/d')

If you pass several arguments, they will be automatically joined:

    >>> PurePath('docs', 'Makefile')
    PurePosixPath('docs/Makefile')

Joining semantics are similar to os.path.join, in that anchored paths
ignore the information from the previously joined components:

    >>> PurePath('/etc', '/usr', 'bin')
    PurePosixPath('/usr/bin')

However, with Windows paths, the drive is retained as necessary:

    >>> PureWindowsPath('c:/foo', '/Windows')
    PureWindowsPath('c:/Windows')
    >>> PureWindowsPath('c:/foo', 'd:')
    PureWindowsPath('d:')

Also, path separators are normalized to the platform default:

    >>> PureWindowsPath('a/b') == PureWindowsPath('a\\b')
    True

Extraneous path separators and "." components are eliminated, but not
".." components:

    >>> PurePosixPath('a//b/./c/')
    PurePosixPath('a/b/c')
    >>> PurePosixPath('a/../b')
    PurePosixPath('a/../b')

Multiple leading slashes are treated differently depending on the path
flavour. They are always retained on Windows paths (because of the UNC
notation):

    >>> PureWindowsPath('//some/path')
    PureWindowsPath('//some/path/')

On POSIX, they are collapsed except if there are exactly two leading
slashes, which is a special case in the POSIX specification on pathname
resolution (this is also necessary for Cygwin compatibility):

    >>> PurePosixPath('///some/path')
    PurePosixPath('/some/path')
    >>> PurePosixPath('//some/path')
    PurePosixPath('//some/path')

Calling the constructor without any argument creates a path object
pointing to the logical "current directory" (without looking up its
absolute path, which is the job of the cwd() classmethod on concrete
paths):

    >>> PurePosixPath()
    PurePosixPath('.')

Representing

To represent a path (e.g. to pass it to third-party libraries), just
call str() on it:

    >>> p = PurePath('/home/antoine/pathlib/setup.py')
    >>> str(p)
    '/home/antoine/pathlib/setup.py'
    >>> p = PureWindowsPath('c:/windows')
    >>> str(p)
    'c:\\windows'

To force the string representation with forward slashes, use the
as_posix() method:

    >>> p.as_posix()
    'c:/windows'

To get the bytes representation (which might be useful under Unix
systems), call bytes() on it, which internally uses os.fsencode():

    >>> bytes(p)
    b'/home/antoine/pathlib/setup.py'

To represent the path as a file: URI, call the as_uri() method:

    >>> p = PurePosixPath('/etc/passwd')
    >>> p.as_uri()
    'file:///etc/passwd'
    >>> p = PureWindowsPath('c:/Windows')
    >>> p.as_uri()
    'file:///c:/Windows'

The repr() of a path always uses forward slashes, even under Windows,
for readability and to remind users that forward slashes are ok:

    >>> p = PureWindowsPath('c:/Windows')
    >>> p
    PureWindowsPath('c:/Windows')

Properties

Several simple properties are provided on every path (each can be
empty):

    >>> p = PureWindowsPath('c:/Downloads/pathlib.tar.gz')
    >>> p.drive
    'c:'
    >>> p.root
    '\\'
    >>> p.anchor
    'c:\\'
    >>> p.name
    'pathlib.tar.gz'
    >>> p.stem
    'pathlib.tar'
    >>> p.suffix
    '.gz'
    >>> p.suffixes
    ['.tar', '.gz']

Deriving new paths

Joining

A path can be joined with another using the / operator:

    >>> p = PurePosixPath('foo')
    >>> p / 'bar'
    PurePosixPath('foo/bar')
    >>> p / PurePosixPath('bar')
    PurePosixPath('foo/bar')
    >>> 'bar' / p
    PurePosixPath('bar/foo')

As with the constructor, multiple path components can be specified,
either collapsed or separately:

    >>> p / 'bar/xyzzy'
    PurePosixPath('foo/bar/xyzzy')
    >>> p / 'bar' / 'xyzzy'
    PurePosixPath('foo/bar/xyzzy')

A joinpath() method is also provided, with the same behaviour:

    >>> p.joinpath('Python')
    PurePosixPath('foo/Python')

Changing the path's final component

The with_name() method returns a new path, with the name changed:

    >>> p = PureWindowsPath('c:/Downloads/pathlib.tar.gz')
    >>> p.with_name('setup.py')
    PureWindowsPath('c:/Downloads/setup.py')

It fails with a ValueError if the path doesn't have an actual name:

    >>> p = PureWindowsPath('c:/')
    >>> p.with_name('setup.py')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "pathlib.py", line 875, in with_name
        raise ValueError("%r has an empty name" % (self,))
    ValueError: PureWindowsPath('c:/') has an empty name
    >>> p.name
    ''

The with_suffix() method returns a new path with the suffix changed.
However, if the path has no suffix, the new suffix is added:

    >>> p = PureWindowsPath('c:/Downloads/pathlib.tar.gz')
    >>> p.with_suffix('.bz2')
    PureWindowsPath('c:/Downloads/pathlib.tar.bz2')
    >>> p = PureWindowsPath('README')
    >>> p.with_suffix('.bz2')
    PureWindowsPath('README.bz2')

Making the path relative

The relative_to() method computes the relative difference of a path to
another:

    >>> PurePosixPath('/usr/bin/python').relative_to('/usr')
    PurePosixPath('bin/python')

ValueError is raised if the method cannot return a meaningful value:

    >>> PurePosixPath('/usr/bin/python').relative_to('/etc')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "pathlib.py", line 926, in relative_to
        .format(str(self), str(formatted)))
    ValueError: '/usr/bin/python' does not start with '/etc'

Sequence-like access

The parts property returns a tuple providing read-only sequence access
to a path's components:

    >>> p = PurePosixPath('/etc/init.d')
    >>> p.parts
    ('/', 'etc', 'init.d')

Windows paths handle the drive and the root as a single path component:

    >>> p = PureWindowsPath('c:/setup.py')
    >>> p.parts
    ('c:\\', 'setup.py')

(separating them would be wrong, since C: is not the parent of C:\\).

The parent property returns the logical parent of the path:

    >>> p = PureWindowsPath('c:/python33/bin/python.exe')
    >>> p.parent
    PureWindowsPath('c:/python33/bin')

The parents property returns an immutable sequence of the path's logical
ancestors:

    >>> p = PureWindowsPath('c:/python33/bin/python.exe')
    >>> len(p.parents)
    3
    >>> p.parents[0]
    PureWindowsPath('c:/python33/bin')
    >>> p.parents[1]
    PureWindowsPath('c:/python33')
    >>> p.parents[2]
    PureWindowsPath('c:/')

Querying

is_relative() returns True if the path is relative (see definition
above), False otherwise.

is_reserved() returns True if a Windows path is a reserved path such as
CON or NUL. It always returns False for POSIX paths.

match() matches the path against a glob pattern. It operates on
individual parts and matches from the right:

  >>> p = PurePosixPath('/usr/bin') >>> p.match('/usr/b*') True >>>
  p.match('usr/b*') True >>> p.match('b*') True >>> p.match('/u*') False

This behaviour respects the following expectations:

-   A simple pattern such as "*.py" matches arbitrarily long paths as
    long as the last part matches, e.g. "/usr/foo/bar.py".
-   Longer patterns can be used as well for more complex matching, e.g.
    "/usr/foo/*.py" matches "/usr/foo/bar.py".

Concrete paths API

In addition to the operations of the pure API, concrete paths provide
additional methods which actually access the filesystem to query or
mutate information.

Constructing

The classmethod cwd() creates a path object pointing to the current
working directory in absolute form:

    >>> Path.cwd()
    PosixPath('/home/antoine/pathlib')

File metadata

The stat() returns the file's stat() result; similarly, lstat() returns
the file's lstat() result (which is different iff the file is a symbolic
link):

    >>> p.stat()
    posix.stat_result(st_mode=33277, st_ino=7483155, st_dev=2053, st_nlink=1, st_uid=500, st_gid=500, st_size=928, st_atime=1343597970, st_mtime=1328287308, st_ctime=1343597964)

Higher-level methods help examine the kind of the file:

    >>> p.exists()
    True
    >>> p.is_file()
    True
    >>> p.is_dir()
    False
    >>> p.is_symlink()
    False
    >>> p.is_socket()
    False
    >>> p.is_fifo()
    False
    >>> p.is_block_device()
    False
    >>> p.is_char_device()
    False

The file owner and group names (rather than numeric ids) are queried
through corresponding methods:

    >>> p = Path('/etc/shadow')
    >>> p.owner()
    'root'
    >>> p.group()
    'shadow'

Path resolution

The resolve() method makes a path absolute, resolving any symlink on the
way (like the POSIX realpath() call). It is the only operation which
will remove ".." path components. On Windows, this method will also take
care to return the canonical path (with the right casing).

Directory walking

Simple (non-recursive) directory access is done by calling the iterdir()
method, which returns an iterator over the child paths:

    >>> p = Path('docs')
    >>> for child in p.iterdir(): child
    ...
    PosixPath('docs/conf.py')
    PosixPath('docs/_templates')
    PosixPath('docs/make.bat')
    PosixPath('docs/index.rst')
    PosixPath('docs/_build')
    PosixPath('docs/_static')
    PosixPath('docs/Makefile')

This allows simple filtering through list comprehensions:

    >>> p = Path('.')
    >>> [child for child in p.iterdir() if child.is_dir()]
    [PosixPath('.hg'), PosixPath('docs'), PosixPath('dist'), PosixPath('__pycache__'), PosixPath('build')]

Simple and recursive globbing is also provided:

    >>> for child in p.glob('**/*.py'): child
    ...
    PosixPath('test_pathlib.py')
    PosixPath('setup.py')
    PosixPath('pathlib.py')
    PosixPath('docs/conf.py')
    PosixPath('build/lib/pathlib.py')

File opening

The open() method provides a file opening API similar to the builtin
open() method:

    >>> p = Path('setup.py')
    >>> with p.open() as f: f.readline()
    ...
    '#!/usr/bin/env python3\n'

Filesystem modification

Several common filesystem operations are provided as methods: touch(),
mkdir(), rename(), replace(), unlink(), rmdir(), chmod(), lchmod(),
symlink_to(). More operations could be provided, for example some of the
functionality of the shutil module.

Detailed documentation of the proposed API can be found at the pathlib
docs.

Discussion

Division operator

The division operator came out first in a poll about the path joining
operator. Initial versions of pathlib used square brackets (i.e.
__getitem__) instead.

joinpath()

The joinpath() method was initially called join(), but several people
objected that it could be confused with str.join() which has different
semantics. Therefore, it was renamed to joinpath().

Case-sensitivity

Windows users consider filesystem paths to be case-insensitive and
expect path objects to observe that characteristic, even though in some
rare situations some foreign filesystem mounts may be case-sensitive
under Windows.

In the words of one commenter,

  "If glob("*.py") failed to find SETUP.PY on Windows, that would be a
  usability disaster".

  -- Paul Moore in
  https://mail.python.org/pipermail/python-dev/2013-April/125254.html

Copyright

This document has been placed into the public domain.


text:
PEP: 521 Title: Managing global context via 'with' blocks in generators
and coroutines Version: $Revision$ Last-Modified: $Date$ Author:
Nathaniel J. Smith<EMAIL_ADDRESS>Status: Withdrawn Type: Standards
Track Content-Type: text/x-rst Created: 27-Apr-2015 Python-Version: 3.6
Post-History: 29-Apr-2015

PEP Withdrawal

Withdrawn in favor of PEP 567.

Abstract

While we generally try to avoid global state when possible, there
nonetheless exist a number of situations where it is agreed to be the
best approach. In Python, a standard pattern for handling such cases is
to store the global state in global or thread-local storage, and then
use with blocks to limit modifications of this global state to a single
dynamic scope. Examples where this pattern is used include the standard
library's warnings.catch_warnings and decimal.localcontext, NumPy's
numpy.errstate (which exposes the error-handling settings provided by
the IEEE 754 floating point standard), and the handling of logging
context or HTTP request context in many server application frameworks.

However, there is currently no ergonomic way to manage such local
changes to global state when writing a generator or coroutine. For
example, this code:

    def f():
        with warnings.catch_warnings():
            for x in g():
                yield x

may or may not successfully catch warnings raised by g(), and may or may
not inadvertently swallow warnings triggered elsewhere in the code. The
context manager, which was intended to apply only to f and its callees,
ends up having a dynamic scope that encompasses arbitrary and
unpredictable parts of its call**ers**. This problem becomes
particularly acute when writing asynchronous code, where essentially all
functions become coroutines.

Here, we propose to solve this problem by notifying context managers
whenever execution is suspended or resumed within their scope, allowing
them to restrict their effects appropriately.

Specification

Two new, optional, methods are added to the context manager protocol:
__suspend__ and __resume__. If present, these methods will be called
whenever a frame's execution is suspended or resumed from within the
context of the with block.

More formally, consider the following code:

    with EXPR as VAR:
        PARTIAL-BLOCK-1
        f((yield foo))
        PARTIAL-BLOCK-2

Currently this is equivalent to the following code (copied from PEP
343):

    mgr = (EXPR)
    exit = type(mgr).__exit__  # Not calling it yet
    value = type(mgr).__enter__(mgr)
    exc = True
    try:
        try:
            VAR = value  # Only if "as VAR" is present
            PARTIAL-BLOCK-1
            f((yield foo))
            PARTIAL-BLOCK-2
        except:
            exc = False
            if not exit(mgr, *sys.exc_info()):
                raise
    finally:
        if exc:
            exit(mgr, None, None, None)

This PEP proposes to modify with block handling to instead become:

    mgr = (EXPR)
    exit = type(mgr).__exit__  # Not calling it yet
    ### --- NEW STUFF ---
    if the_block_contains_yield_points:  # known statically at compile time
        suspend = getattr(type(mgr), "__suspend__", lambda: None)
        resume = getattr(type(mgr), "__resume__", lambda: None)
    ### --- END OF NEW STUFF ---
    value = type(mgr).__enter__(mgr)
    exc = True
    try:
        try:
            VAR = value  # Only if "as VAR" is present
            PARTIAL-BLOCK-1
            ### --- NEW STUFF ---
            suspend(mgr)
            tmp = yield foo
            resume(mgr)
            f(tmp)
            ### --- END OF NEW STUFF ---
            PARTIAL-BLOCK-2
        except:
            exc = False
            if not exit(mgr, *sys.exc_info()):
                raise
    finally:
        if exc:
            exit(mgr, None, None, None)

Analogous suspend/resume calls are also wrapped around the yield points
embedded inside the yield from, await, async with, and async for
constructs.

Nested blocks

Given this code:

    def f():
        with OUTER:
            with INNER:
                yield VALUE

then we perform the following operations in the following sequence:

    INNER.__suspend__()
    OUTER.__suspend__()
    yield VALUE
    OUTER.__resume__()
    INNER.__resume__()

Note that this ensures that the following is a valid refactoring:

    def f():
        with OUTER:
            yield from g()

    def g():
        with INNER
            yield VALUE

Similarly, with statements with multiple context managers suspend from
right to left, and resume from left to right.

Other changes

Appropriate __suspend__ and __resume__ methods are added to
warnings.catch_warnings and decimal.localcontext.

Rationale

In the abstract, we gave an example of plausible but incorrect code:

    def f():
        with warnings.catch_warnings():
            for x in g():
                yield x

To make this correct in current Python, we need to instead write
something like:

    def f():
        with warnings.catch_warnings():
            it = iter(g())
        while True:
            with warnings.catch_warnings():
                try:
                    x = next(it)
                except StopIteration:
                    break
            yield x

OTOH, if this PEP is accepted then the original code will become correct
as-is. Or if this isn't convincing, then here's another example of
broken code; fixing it requires even greater gyrations, and these are
left as an exercise for the reader:

    async def test_foo_emits_warning():
        with warnings.catch_warnings(record=True) as w:
            await foo()
        assert len(w) == 1
        assert "xyzzy" in w[0].message

And notice that this last example isn't artificial at all -- this is
exactly how you write a test that an async/await-using coroutine
correctly raises a warning. Similar issues arise for pretty much any use
of warnings.catch_warnings, decimal.localcontext, or numpy.errstate in
async/await-using code. So there's clearly a real problem to solve here,
and the growing prominence of async code makes it increasingly urgent.

Alternative approaches

The main alternative that has been proposed is to create some kind of
"task-local storage", analogous to "thread-local storage" [1]. In
essence, the idea would be that the event loop would take care to
allocate a new "task namespace" for each task it schedules, and provide
an API to at any given time fetch the namespace corresponding to the
currently executing task. While there are many details to be worked
out[2], the basic idea seems doable, and it is an especially natural way
to handle the kind of global context that arises at the top-level of
async application frameworks (e.g., setting up context objects in a web
framework). But it also has a number of flaws:

-   It only solves the problem of managing global state for coroutines
    that yield back to an asynchronous event loop. But there actually
    isn't anything about this problem that's specific to asyncio -- as
    shown in the examples above, simple generators run into exactly the
    same issue.

-   It creates an unnecessary coupling between event loops and code that
    needs to manage global state. Obviously an async web framework needs
    to interact with some event loop API anyway, so it's not a big deal
    in that case. But it's weird that warnings or decimal or NumPy
    should have to call into an async library's API to access their
    internal state when they themselves involve no async code. Worse,
    since there are multiple event loop APIs in common use, it isn't
    clear how to choose which to integrate with. (This could be somewhat
    mitigated by CPython providing a standard API for creating and
    switching "task-local domains" that asyncio, Twisted, tornado, etc.
    could then work with.)

-   It's not at all clear that this can be made acceptably fast. NumPy
    has to check the floating point error settings on every single
    arithmetic operation. Checking a piece of data in thread-local
    storage is absurdly quick, because modern platforms have put massive
    resources into optimizing this case (e.g. dedicating a CPU register
    for this purpose); calling a method on an event loop to fetch a
    handle to a namespace and then doing lookup in that namespace is
    much slower.

    More importantly, this extra cost would be paid on every access to
    the global data, even for programs which are not otherwise using an
    event loop at all. This PEP's proposal, by contrast, only affects
    code that actually mixes with blocks and yield statements, meaning
    that the users who experience the costs are the same users who also
    reap the benefits.

On the other hand, such tight integration between task context and the
event loop does potentially allow other features that are beyond the
scope of the current proposal. For example, an event loop could note
which task namespace was in effect when a task called call_soon, and
arrange that the callback when run would have access to the same task
namespace. Whether this is useful, or even well-defined in the case of
cross-thread calls (what does it mean to have task-local storage
accessed from two threads simultaneously?), is left as a puzzle for
event loop implementors to ponder -- nothing in this proposal rules out
such enhancements as well. It does seem though that such features would
be useful primarily for state that already has a tight integration with
the event loop -- while we might want a request id to be preserved
across call_soon, most people would not expect:

    with warnings.catch_warnings():
        loop.call_soon(f)

to result in f being run with warnings disabled, which would be the
result if call_soon preserved global context in general. It's also
unclear how this would even work given that the warnings context manager
__exit__ would be called before f.

So this PEP takes the position that __suspend__/__resume__ and
"task-local storage" are two complementary tools that are both useful in
different circumstances.

Backwards compatibility

Because __suspend__ and __resume__ are optional and default to no-ops,
all existing context managers continue to work exactly as before.

Speed-wise, this proposal adds additional overhead when entering a with
block (where we must now check for the additional methods; failed
attribute lookup in CPython is rather slow, since it involves allocating
an AttributeError), and additional overhead at suspension points. Since
the position of with blocks and suspension points is known statically,
the compiler can straightforwardly optimize away this overhead in all
cases except where one actually has a yield inside a with. Furthermore,
because we only do attribute checks for __suspend__ and __resume__ once
at the start of a with block, when these attributes are undefined then
the per-yield overhead can be optimized down to a single C-level
if (frame->needs_suspend_resume_calls) { ... }. Therefore, we expect the
overall overhead to be negligible.

Interaction with PEP 492

PEP 492 added new asynchronous context managers, which are like regular
context managers, but instead of having regular methods __enter__ and
__exit__ they have coroutine methods __aenter__ and __aexit__.

Following this pattern, one might expect this proposal to add
__asuspend__ and __aresume__ coroutine methods. But this doesn't make
much sense, since the whole point is that __suspend__ should be called
before yielding our thread of execution and allowing other code to run.
The only thing we accomplish by making __asuspend__ a coroutine is to
make it possible for __asuspend__ itself to yield. So either we need to
recursively call __asuspend__ from inside __asuspend__, or else we need
to give up and allow these yields to happen without calling the suspend
callback; either way it defeats the whole point.

Well, with one exception: one possible pattern for coroutine code is to
call yield in order to communicate with the coroutine runner, but
without actually suspending their execution (i.e., the coroutine might
know that the coroutine runner will resume them immediately after
processing the yielded message). An example of this is the
curio.timeout_after async context manager, which yields a special
set_timeout message to the curio kernel, and then the kernel immediately
(synchronously) resumes the coroutine which sent the message. And from
the user point of view, this timeout value acts just like the kinds of
global variables that motivated this PEP. But, there is a crucal
difference: this kind of async context manager is, by definition,
tightly integrated with the coroutine runner. So, the coroutine runner
can take over responsibility for keeping track of which timeouts apply
to which coroutines without any need for this PEP at all (and this is
indeed how curio.timeout_after works).

That leaves two reasonable approaches to handling async context
managers:

1)  Add plain __suspend__ and __resume__ methods.
2)  Leave async context managers alone for now until we have more
    experience with them.

Either seems plausible, so out of laziness / YAGNI this PEP tentatively
proposes to stick with option (2).

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] https://groups.google.com/forum/#!topic/python-tulip/zix5HQxtElg
https://github.com/python/asyncio/issues/165

[2] For example, we would have to decide whether there is a single
task-local namespace shared by all users (in which case we need a way
for multiple third-party libraries to adjudicate access to this
namespace), or else if there are multiple task-local namespaces, then we
need some mechanism for each library to arrange for their task-local
namespaces to be created and destroyed at appropriate moments. The
preliminary patch linked from the github issue above doesn't seem to
provide any mechanism for such lifecycle management.

license: Public Domain


text:
PEP: 443 Title: Single-dispatch generic functions Version: $Revision$
Last-Modified: $Date$ Author: Łukasz Langa<EMAIL_ADDRESS>Discussions-To<EMAIL_ADDRESS>Status: Final Type: Standards
Track Content-Type: text/x-rst Created: 22-May-2013 Python-Version: 3.4
Post-History: 22-May-2013, 25-May-2013, 31-May-2013 Replaces: 245, 246,
3124

Abstract

This PEP proposes a new mechanism in the functools standard library
module that provides a simple form of generic programming known as
single-dispatch generic functions.

A generic function is composed of multiple functions implementing the
same operation for different types. Which implementation should be used
during a call is determined by the dispatch algorithm. When the
implementation is chosen based on the type of a single argument, this is
known as single dispatch.

Rationale and Goals

Python has always provided a variety of built-in and standard-library
generic functions, such as len(), iter(), pprint.pprint(), copy.copy(),
and most of the functions in the operator module. However, it currently:

1.  does not have a simple or straightforward way for developers to
    create new generic functions,
2.  does not have a standard way for methods to be added to existing
    generic functions (i.e., some are added using registration
    functions, others require defining __special__ methods, possibly by
    monkeypatching).

In addition, it is currently a common anti-pattern for Python code to
inspect the types of received arguments, in order to decide what to do
with the objects.

For example, code may wish to accept either an object of some type, or a
sequence of objects of that type. Currently, the "obvious way" to do
this is by type inspection, but this is brittle and closed to extension.

Abstract Base Classes make it easier to discover present behaviour, but
don't help adding new behaviour. A developer using an already-written
library may be unable to change how their objects are treated by such
code, especially if the objects they are using were created by a third
party.

Therefore, this PEP proposes a uniform API to address dynamic
overloading using decorators.

User API

To define a generic function, decorate it with the @singledispatch
decorator. Note that the dispatch happens on the type of the first
argument. Create your function accordingly:

    >>> from functools import singledispatch
    >>> @singledispatch
    ... def fun(arg, verbose=False):
    ...     if verbose:
    ...         print("Let me just say,", end=" ")
    ...     print(arg)

To add overloaded implementations to the function, use the register()
attribute of the generic function. This is a decorator, taking a type
parameter and decorating a function implementing the operation for that
type:

    >>> @fun.register(int)
    ... def _(arg, verbose=False):
    ...     if verbose:
    ...         print("Strength in numbers, eh?", end=" ")
    ...     print(arg)
    ...
    >>> @fun.register(list)
    ... def _(arg, verbose=False):
    ...     if verbose:
    ...         print("Enumerate this:")
    ...     for i, elem in enumerate(arg):
    ...         print(i, elem)

To enable registering lambdas and pre-existing functions, the register()
attribute can be used in a functional form:

    >>> def nothing(arg, verbose=False):
    ...     print("Nothing.")
    ...
    >>> fun.register(type(None), nothing)

The register() attribute returns the undecorated function. This enables
decorator stacking, pickling, as well as creating unit tests for each
variant independently:

    >>> @fun.register(float)
    ... @fun.register(Decimal)
    ... def fun_num(arg, verbose=False):
    ...     if verbose:
    ...         print("Half of your number:", end=" ")
    ...     print(arg / 2)
    ...
    >>> fun_num is fun
    False

When called, the generic function dispatches on the type of the first
argument:

    >>> fun("Hello, world.")
    Hello, world.
    >>> fun("test.", verbose=True)
    Let me just say, test.
    >>> fun(42, verbose=True)
    Strength in numbers, eh? 42
    >>> fun(['spam', 'spam', 'eggs', 'spam'], verbose=True)
    Enumerate this:
    0 spam
    1 spam
    2 eggs
    3 spam
    >>> fun(None)
    Nothing.
    >>> fun(1.23)
    0.615

Where there is no registered implementation for a specific type, its
method resolution order is used to find a more generic implementation.
The original function decorated with @singledispatch is registered for
the base object type, which means it is used if no better implementation
is found.

To check which implementation will the generic function choose for a
given type, use the dispatch() attribute:

    >>> fun.dispatch(float)
    <function fun_num at 0x104319058>
    >>> fun.dispatch(dict)    # note: default implementation
    <function fun at 0x103fe0000>

To access all registered implementations, use the read-only registry
attribute:

    >>> fun.registry.keys()
    dict_keys([<class 'NoneType'>, <class 'int'>, <class 'object'>,
              <class 'decimal.Decimal'>, <class 'list'>,
              <class 'float'>])
    >>> fun.registry[float]
    <function fun_num at 0x1035a2840>
    >>> fun.registry[object]
    <function fun at 0x103fe0000>

The proposed API is intentionally limited and opinionated, as to ensure
it is easy to explain and use, as well as to maintain consistency with
existing members in the functools module.

Implementation Notes

The functionality described in this PEP is already implemented in the
pkgutil standard library module as simplegeneric. Because this
implementation is mature, the goal is to move it largely as-is. The
reference implementation is available on hg.python.org[1].

The dispatch type is specified as a decorator argument. An alternative
form using function annotations was considered but its inclusion has
been rejected. As of May 2013, this usage pattern is out of scope for
the standard library[2], and the best practices for annotation usage are
still debated.

Based on the current pkgutil.simplegeneric implementation, and following
the convention on registering virtual subclasses on Abstract Base
Classes, the dispatch registry will not be thread-safe.

Abstract Base Classes

The pkgutil.simplegeneric implementation relied on several forms of
method resolution order (MRO). @singledispatch removes special handling
of old-style classes and Zope's ExtensionClasses. More importantly, it
introduces support for Abstract Base Classes (ABC).

When a generic function implementation is registered for an ABC, the
dispatch algorithm switches to an extended form of C3 linearization,
which includes the relevant ABCs in the MRO of the provided argument.
The algorithm inserts ABCs where their functionality is introduced, i.e.
issubclass(cls, abc) returns True for the class itself but returns False
for all its direct base classes. Implicit ABCs for a given class (either
registered or inferred from the presence of a special method like
__len__()) are inserted directly after the last ABC explicitly listed in
the MRO of said class.

In its most basic form, this linearization returns the MRO for the given
type:

    >>> _compose_mro(dict, [])
    [<class 'dict'>, <class 'object'>]

When the second argument contains ABCs that the specified type is a
subclass of, they are inserted in a predictable order:

    >>> _compose_mro(dict, [Sized, MutableMapping, str,
    ...                     Sequence, Iterable])
    [<class 'dict'>, <class 'collections.abc.MutableMapping'>,
     <class 'collections.abc.Mapping'>, <class 'collections.abc.Sized'>,
     <class 'collections.abc.Iterable'>, <class 'collections.abc.Container'>,
     <class 'object'>]

While this mode of operation is significantly slower, all dispatch
decisions are cached. The cache is invalidated on registering new
implementations on the generic function or when user code calls
register() on an ABC to implicitly subclass it. In the latter case, it
is possible to create a situation with ambiguous dispatch, for instance:

    >>> from collections.abc import Iterable, Container
    >>> class P:
    ...     pass
    >>> Iterable.register(P)
    <class '__main__.P'>
    >>> Container.register(P)
    <class '__main__.P'>

Faced with ambiguity, @singledispatch refuses the temptation to guess:

    >>> @singledispatch
    ... def g(arg):
    ...     return "base"
    ...
    >>> g.register(Iterable, lambda arg: "iterable")
    <function <lambda> at 0x108b49110>
    >>> g.register(Container, lambda arg: "container")
    <function <lambda> at 0x108b491c8>
    >>> g(P())
    Traceback (most recent call last):
    ...
    RuntimeError: Ambiguous dispatch: <class 'collections.abc.Container'>
    or <class 'collections.abc.Iterable'>

Note that this exception would not be raised if one or more ABCs had
been provided explicitly as base classes during class definition. In
this case dispatch happens in the MRO order:

    >>> class Ten(Iterable, Container):
    ...     def __iter__(self):
    ...         for i in range(10):
    ...             yield i
    ...     def __contains__(self, value):
    ...         return value in range(10)
    ...
    >>> g(Ten())
    'iterable'

A similar conflict arises when subclassing an ABC is inferred from the
presence of a special method like __len__() or __contains__():

    >>> class Q:
    ...   def __contains__(self, value):
    ...     return False
    ...
    >>> issubclass(Q, Container)
    True
    >>> Iterable.register(Q)
    >>> g(Q())
    Traceback (most recent call last):
    ...
    RuntimeError: Ambiguous dispatch: <class 'collections.abc.Container'>
    or <class 'collections.abc.Iterable'>

An early version of the PEP contained a custom approach that was simpler
but created a number of edge cases with surprising results[3].

Usage Patterns

This PEP proposes extending behaviour only of functions specifically
marked as generic. Just as a base class method may be overridden by a
subclass, so too a function may be overloaded to provide custom
functionality for a given type.

Universal overloading does not equal arbitrary overloading, in the sense
that we need not expect people to randomly redefine the behavior of
existing functions in unpredictable ways. To the contrary, generic
function usage in actual programs tends to follow very predictable
patterns and registered implementations are highly-discoverable in the
common case.

If a module is defining a new generic operation, it will usually also
define any required implementations for existing types in the same
place. Likewise, if a module is defining a new type, then it will
usually define implementations there for any generic functions that it
knows or cares about. As a result, the vast majority of registered
implementations can be found adjacent to either the function being
overloaded, or to a newly-defined type for which the implementation is
adding support.

It is only in rather infrequent cases that one will have implementations
registered in a module that contains neither the function nor the
type(s) for which the implementation is added. In the absence of
incompetence or deliberate intention to be obscure, the few
implementations that are not registered adjacent to the relevant type(s)
or function(s), will generally not need to be understood or known about
outside the scope where those implementations are defined. (Except in
the "support modules" case, where best practice suggests naming them
accordingly.)

As mentioned earlier, single-dispatch generics are already prolific
throughout the standard library. A clean, standard way of doing them
provides a way forward to refactor those custom implementations to use a
common one, opening them up for user extensibility at the same time.

Alternative approaches

In PEP 3124 Phillip J. Eby proposes a full-grown solution with
overloading based on arbitrary rule sets (with the default
implementation dispatching on argument types), as well as interfaces,
adaptation and method combining. PEAK-Rules[4] is a reference
implementation of the concepts described in PJE's PEP.

Such a broad approach is inherently complex, which makes reaching a
consensus hard. In contrast, this PEP focuses on a single piece of
functionality that is simple to reason about. It's important to note
this does not preclude the use of other approaches now or in the future.

In a 2005 article on Artima[5] Guido van Rossum presents a generic
function implementation that dispatches on types of all arguments on a
function. The same approach was chosen in Andrey Popp's generic package
available on PyPI[6], as well as David Mertz's
gnosis.magic.multimethods[7].

While this seems desirable at first, I agree with Fredrik Lundh's
comment that "if you design APIs with pages of logic just to sort out
what code a function should execute, you should probably hand over the
API design to someone else". In other words, the single argument
approach proposed in this PEP is not only easier to implement but also
clearly communicates that dispatching on a more complex state is an
anti-pattern. It also has the virtue of corresponding directly with the
familiar method dispatch mechanism in object oriented programming. The
only difference is whether the custom implementation is associated more
closely with the data (object-oriented methods) or the algorithm
(single-dispatch overloading).

PyPy's RPython offers extendabletype[8], a metaclass which enables
classes to be externally extended. In combination with pairtype() and
pair() factories, this offers a form of single-dispatch generics.

Acknowledgements

Apart from Phillip J. Eby's work on PEP 3124 and PEAK-Rules, influences
include Paul Moore's original issue [9] that proposed exposing
pkgutil.simplegeneric as part of the functools API, Guido van Rossum's
article on multimethods [10], and discussions with Raymond Hettinger on
a general pprint rewrite. Huge thanks to Alyssa Coghlan for encouraging
me to create this PEP and providing initial feedback.

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] http://hg.python.org/features/pep-443/file/tip/Lib/functools.py#l359

[2] PEP 8 states in the "Programming Recommendations" section that "the
Python standard library will not use function annotations as that would
result in a premature commitment to a particular annotation style".

[3] http://bugs.python.org/issue18244

[4] http://peak.telecommunity.com/DevCenter/PEAK_2dRules

[5] http://www.artima.com/weblogs/viewpost.jsp?thread=101605

[6] http://pypi.python.org/pypi/generic

[7] http://gnosis.cx/publish/programming/charming_python_b12.html

[8] https://bitbucket.org/pypy/pypy/raw/default/rpython/tool/pairtype.py

[9] http://bugs.python.org/issue5135

[10] http://www.artima.com/weblogs/viewpost.jsp?thread=101605


text:
PEP: 251 Title: Python 2.2 Release Schedule Author: Barry Warsaw
<EMAIL_ADDRESS>Guido van Rossum<EMAIL_ADDRESS>Status: Final
Type: Informational Topic: Release Content-Type: text/x-rst Created:
17-Apr-2001 Python-Version: 2.2 Post-History: 14-Aug-2001

Abstract

This document describes the Python 2.2 development and release schedule.
The schedule primarily concerns itself with PEP-sized items. Small bug
fixes and changes will occur up until the first beta release.

The schedule below represents the actual release dates of Python 2.2.
Note that any subsequent maintenance releases of Python 2.2 should be
covered by separate PEPs.

Release Schedule

Tentative future release dates. Note that we've slipped this compared to
the schedule posted around the release of 2.2a1.

-   21-Dec-2001: 2.2 [Released] (final release)
-   14-Dec-2001: 2.2c1 [Released]
-   14-Nov-2001: 2.2b2 [Released]
-   19-Oct-2001: 2.2b1 [Released]
-   28-Sep-2001: 2.2a4 [Released]
-   7-Sep-2001: 2.2a3 [Released]
-   22-Aug-2001: 2.2a2 [Released]
-   18-Jul-2001: 2.2a1 [Released]

Release Manager

Barry Warsaw was the Python 2.2 release manager.

Release Mechanics

We experimented with a new mechanism for releases: a week before every
alpha, beta or other release, we forked off a branch which became the
release. Changes to the branch are limited to the release manager and
his designated 'bots. This experiment was deemed a success and should be
observed for future releases. See PEP 101 for the actual release
mechanics.

New features for Python 2.2

The following new features are introduced in Python 2.2. For a more
detailed account, see Misc/NEWS[1] in the Python distribution, or Andrew
Kuchling's "What's New in Python 2.2" document[2].

-   iterators (PEP 234)
-   generators (PEP 255)
-   unifying long ints and plain ints (PEP 237)
-   division (PEP 238)
-   unification of types and classes (PEP 252, PEP 253)

References

Copyright

This document has been placed in the public domain.

[1] Misc/NEWS file from CVS
http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Misc/NEWS?rev=1.337.2.4&content-type=text/vnd.viewcvs-markup

[2] Andrew Kuchling, What's New in Python 2.2
http://www.python.org/doc/2.2.1/whatsnew/whatsnew22.html

authors:
Barry Warsaw
Guido van Rossum


text:
PEP: 3107 Title: Function Annotations Version: $Revision$ Last-Modified:
$Date$ Author: Collin Winter<EMAIL_ADDRESS>Tony Lownds
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 02-Dec-2006 Python-Version: 3.0 Post-History:

Abstract

This PEP introduces a syntax for adding arbitrary metadata annotations
to Python functions[1].

Rationale

Because Python's 2.x series lacks a standard way of annotating a
function's parameters and return values, a variety of tools and
libraries have appeared to fill this gap. Some utilise the decorators
introduced in PEP 318, while others parse a function's docstring,
looking for annotations there.

This PEP aims to provide a single, standard way of specifying this
information, reducing the confusion caused by the wide variation in
mechanism and syntax that has existed until this point.

Fundamentals of Function Annotations

Before launching into a discussion of the precise ins and outs of Python
3.0's function annotations, let's first talk broadly about what
annotations are and are not:

1.  Function annotations, both for parameters and return values, are
    completely optional.

2.  Function annotations are nothing more than a way of associating
    arbitrary Python expressions with various parts of a function at
    compile-time.

    By itself, Python does not attach any particular meaning or
    significance to annotations. Left to its own, Python simply makes
    these expressions available as described in Accessing Function
    Annotations below.

    The only way that annotations take on meaning is when they are
    interpreted by third-party libraries. These annotation consumers can
    do anything they want with a function's annotations. For example,
    one library might use string-based annotations to provide improved
    help messages, like so:

        def compile(source: "something compilable",
                    filename: "where the compilable thing comes from",
                    mode: "is this a single statement or a suite?"):
            ...

    Another library might be used to provide typechecking for Python
    functions and methods. This library could use annotations to
    indicate the function's expected input and return types, possibly
    something like:

        def haul(item: Haulable, *vargs: PackAnimal) -> Distance:
            ...

    However, neither the strings in the first example nor the type
    information in the second example have any meaning on their own;
    meaning comes from third-party libraries alone.

3.  Following from point 2, this PEP makes no attempt to introduce any
    kind of standard semantics, even for the built-in types. This work
    will be left to third-party libraries.

Syntax

Parameters

Annotations for parameters take the form of optional expressions that
follow the parameter name:

    def foo(a: expression, b: expression = 5):
        ...

In pseudo-grammar, parameters now look like
identifier [: expression] [= expression]. That is, annotations always
precede a parameter's default value and both annotations and default
values are optional. Just like how equal signs are used to indicate a
default value, colons are used to mark annotations. All annotation
expressions are evaluated when the function definition is executed, just
like default values.

Annotations for excess parameters (i.e., *args and **kwargs) are
indicated similarly:

    def foo(*args: expression, **kwargs: expression):
        ...

Annotations for nested parameters always follow the name of the
parameter, not the last parenthesis. Annotating all parameters of a
nested parameter is not required:

    def foo((x1, y1: expression),
            (x2: expression, y2: expression)=(None, None)):
        ...

Return Values

The examples thus far have omitted examples of how to annotate the type
of a function's return value. This is done like so:

    def sum() -> expression:
        ...

That is, the parameter list can now be followed by a literal -> and a
Python expression. Like the annotations for parameters, this expression
will be evaluated when the function definition is executed.

The grammar for function definitions[2] is now:

    decorator: '@' dotted_name [ '(' [arglist] ')' ] NEWLINE
    decorators: decorator+
    funcdef: [decorators] 'def' NAME parameters ['->' test] ':' suite
    parameters: '(' [typedargslist] ')'
    typedargslist: ((tfpdef ['=' test] ',')*
                    ('*' [tname] (',' tname ['=' test])* [',' '**' tname]
                     | '**' tname)
                    | tfpdef ['=' test] (',' tfpdef ['=' test])* [','])
    tname: NAME [':' test]
    tfpdef: tname | '(' tfplist ')'
    tfplist: tfpdef (',' tfpdef)* [',']

Lambda

lambda's syntax does not support annotations. The syntax of lambda could
be changed to support annotations, by requiring parentheses around the
parameter list. However it was decided [3] not to make this change
because:

1.  It would be an incompatible change.
2.  Lambdas are neutered anyway.
3.  The lambda can always be changed to a function.

Accessing Function Annotations

Once compiled, a function's annotations are available via the function's
__annotations__ attribute. This attribute is a mutable dictionary,
mapping parameter names to an object representing the evaluated
annotation expression

There is a special key in the __annotations__ mapping, "return". This
key is present only if an annotation was supplied for the function's
return value.

For example, the following annotation:

    def foo(a: 'x', b: 5 + 6, c: list) -> max(2, 9):
        ...

would result in an __annotations__ mapping of :

    {'a': 'x',
     'b': 11,
     'c': list,
     'return': 9}

The return key was chosen because it cannot conflict with the name of a
parameter; any attempt to use return as a parameter name would result in
a SyntaxError.

__annotations__ is an empty, mutable dictionary if there are no
annotations on the function or if the functions was created from a
lambda expression.

Use Cases

In the course of discussing annotations, a number of use-cases have been
raised. Some of these are presented here, grouped by what kind of
information they convey. Also included are examples of existing products
and packages that could make use of annotations.

-   Providing typing information
    -   Type checking ([4],[5])
    -   Let IDEs show what types a function expects and returns ([6])
    -   Function overloading / generic functions ([7])
    -   Foreign-language bridges ([8],[9])
    -   Adaptation ([10],[11])
    -   Predicate logic functions
    -   Database query mapping
    -   RPC parameter marshaling ([12])
-   Other information
    -   Documentation for parameters and return values ([13])

Standard Library

pydoc and inspect

The pydoc module should display the function annotations when displaying
help for a function. The inspect module should change to support
annotations.

Relation to Other PEPs

Function Signature Objects (PEP 362)

Function Signature Objects should expose the function's annotations. The
Parameter object may change or other changes may be warranted.

Implementation

A reference implementation has been checked into the py3k (formerly
"p3yk") branch as revision 53170[14].

Rejected Proposals

-   The BDFL rejected the author's idea for a special syntax for adding
    annotations to generators as being "too ugly"[15].
-   Though discussed early on ([16],[17]), including special objects in
    the stdlib for annotating generator functions and higher-order
    functions was ultimately rejected as being more appropriate for
    third-party libraries; including them in the standard library raised
    too many thorny issues.
-   Despite considerable discussion about a standard type
    parameterisation syntax, it was decided that this should also be
    left to third-party libraries. ([18], [19],[20]).
-   Despite yet more discussion, it was decided not to standardize a
    mechanism for annotation interoperability. Standardizing
    interoperability conventions at this point would be premature. We
    would rather let these conventions develop organically, based on
    real-world usage and necessity, than try to force all users into
    some contrived scheme. ([21],[22], [23]).

References and Footnotes

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] Unless specifically stated, "function" is generally used as a
synonym for "callable" throughout this document.

[2] http://docs.python.org/reference/compound_stmts.html#function-definitions

[3] https://mail.python.org/pipermail/python-3000/2006-May/001613.html

[4] http://web.archive.org/web/20070730120117/http://oakwinter.com/code/typecheck/

[5] http://web.archive.org/web/20070603221429/http://maxrepo.info/

[6] http://www.python.org/idle/doc/idle2.html#Tips

[7] http://www-128.ibm.com/developerworks/library/l-cppeak2/

[8] http://www.jython.org/Project/index.html

[9] http://www.codeplex.com/Wiki/View.aspx?ProjectName=IronPython

[10] http://www.artima.com/weblogs/viewpost.jsp?thread=155123

[11] http://peak.telecommunity.com/PyProtocols.html

[12] http://rpyc.wikispaces.com/

[13] http://docs.python.org/library/pydoc.html

[14] http://svn.python.org/view?rev=53170&view=rev

[15] https://mail.python.org/pipermail/python-3000/2006-May/002103.html

[16] https://mail.python.org/pipermail/python-3000/2006-May/002091.html

[17] https://mail.python.org/pipermail/python-3000/2006-May/001972.html

[18] https://mail.python.org/pipermail/python-3000/2006-May/002105.html

[19] https://mail.python.org/pipermail/python-3000/2006-May/002209.html

[20] https://mail.python.org/pipermail/python-3000/2006-June/002438.html

[21] https://mail.python.org/pipermail/python-3000/2006-August/002895.html

[22] https://mail.python.org/pipermail/python-ideas/2007-January/000032.html

[23] https://mail.python.org/pipermail/python-list/2006-December/420645.html