text:
PEP: 446 Title: Make newly created file descriptors non-inheritable
Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 05-Aug-2013 Python-Version: 3.4 Replaces: 433

Abstract

Leaking file descriptors in child processes causes various annoying
issues and is a known major security vulnerability. Using the subprocess
module with the close_fds parameter set to True is not possible in all
cases.

This PEP proposes to make all file descriptors created by Python
non-inheritable by default to reduce the risk of these issues. This PEP
fixes also a race condition in multi-threaded applications on operating
systems supporting atomic flags to create non-inheritable file
descriptors.

We are aware of the code breakage this is likely to cause, and doing it
anyway for the good of mankind. (Details in the section "Backward
Compatibility" below.)

Rationale

Inheritance of File Descriptors

Each operating system handles the inheritance of file descriptors
differently. Windows creates non-inheritable handles by default, whereas
UNIX and the POSIX API on Windows create inheritable file descriptors by
default. Python prefers the POSIX API over the native Windows API, to
have a single code base and to use the same type for file descriptors,
and so it creates inheritable file descriptors.

There is one exception: os.pipe() creates non-inheritable pipes on
Windows, whereas it creates inheritable pipes on UNIX. The reason is an
implementation artifact: os.pipe() calls CreatePipe() on Windows (native
API), whereas it calls pipe() on UNIX (POSIX API). The call to
CreatePipe() was added in Python in 1994, before the introduction of
pipe() in the POSIX API in Windows 98. The issue #4708 proposes to
change os.pipe() on Windows to create inheritable pipes.

Inheritance of File Descriptors on Windows

On Windows, the native type of file objects is handles (C type HANDLE).
These handles have a HANDLE_FLAG_INHERIT flag which defines if a handle
can be inherited in a child process or not. For the POSIX API, the C
runtime (CRT) also provides file descriptors (C type int). The handle of
a file descriptor can be retrieve using the function _get_osfhandle(fd).
A file descriptor can be created from a handle using the function
_open_osfhandle(handle).

Using CreateProcess(), handles are only inherited if their inheritable
flag (HANDLE_FLAG_INHERIT) is set and the bInheritHandles parameter of
CreateProcess() is TRUE; all file descriptors except standard streams
(0, 1, 2) are closed in the child process, even if bInheritHandles is
TRUE. Using the spawnv() function, all inheritable handles and all
inheritable file descriptors are inherited in the child process. This
function uses the undocumented fields cbReserved2 and lpReserved2 of the
STARTUPINFO structure to pass an array of file descriptors.

To replace standard streams (stdin, stdout, stderr) using
CreateProcess(), the STARTF_USESTDHANDLES flag must be set in the
dwFlags field of the STARTUPINFO structure and the bInheritHandles
parameter of CreateProcess() must be set to TRUE. So when at least one
standard stream is replaced, all inheritable handles are inherited by
the child process.

The default value of the close_fds parameter of subprocess process is
True (bInheritHandles=FALSE) if stdin, stdout and stderr parameters are
None, False (bInheritHandles=TRUE) otherwise.

See also:

-   Handle Inheritance
-   Stackoverflow: Can TCP SOCKET handles be set not inheritable?

Only Inherit Some Handles on Windows

Since Windows Vista, CreateProcess() supports an extension of the
STARTUPINFO structure: the STARTUPINFOEX structure. Using this new
structure, it is possible to specify a list of handles to inherit:
PROC_THREAD_ATTRIBUTE_HANDLE_LIST. Read Programmatically controlling
which handles are inherited by new processes in Win32 (Raymond Chen, Dec
2011) for more information.

Before Windows Vista, it is possible to make handles inheritable and
call CreateProcess() with bInheritHandles=TRUE. This option works if all
other handles are non-inheritable. There is a race condition: if another
thread calls CreateProcess() with bInheritHandles=TRUE, handles will
also be inherited in the second process.

Microsoft suggests to use a lock to avoid the race condition: read
Q315939: PRB: Child Inherits Unintended Handles During CreateProcess
Call (last review: November 2006). The Python issue #16500 "Add an
atfork module" proposes to add such lock, it can be used to make handles
non-inheritable without the race condition. Such lock only protects
against a race condition between Python threads; C threads are not
protected.

Another option is to duplicate handles that must be inherited, passing
the values of the duplicated handles to the child process, so the child
process can steal duplicated handles using DuplicateHandle() with
DUPLICATE_CLOSE_SOURCE. Handle values change between the parent and the
child process because the handles are duplicated (twice); the parent
and/or the child process must be adapted to handle this change. If the
child program cannot be modified, an intermediate program can be used to
steal handles from the parent process before spawning the final child
program. The intermediate program has to pass the handle from the child
process to the parent process. The parent may have to close duplicated
handles if all handles were not stolen, for example if the intermediate
process fails. If the command line is used to pass the handle values,
the command line must be modified when handles are duplicated, because
their values are modified.

This PEP does not include a solution to this problem because there is no
perfect solution working on all Windows versions. This point is deferred
until use cases relying on handle or file descriptor inheritance on
Windows are well known, so we can choose the best solution and carefully
test its implementation.

Inheritance of File Descriptors on UNIX

POSIX provides a close-on-exec flag on file descriptors to automatically
close a file descriptor when the C function execv() is called. File
descriptors with the close-on-exec flag cleared are inherited in the
child process, file descriptors with the flag set are closed in the
child process.

The flag can be set in two syscalls (one to get current flags, a second
to set new flags) using fcntl():

    int flags, res;
    flags = fcntl(fd, F_GETFD);
    if (flags == -1) { /* handle the error */ }
    flags |= FD_CLOEXEC;
    /* or "flags &= ~FD_CLOEXEC;" to clear the flag */
    res = fcntl(fd, F_SETFD, flags);
    if (res == -1) { /* handle the error */ }

FreeBSD, Linux, Mac OS X, NetBSD, OpenBSD and QNX also support setting
the flag in a single syscall using ioctl():

    int res;
    res = ioctl(fd, FIOCLEX, 0);
    if (!res) { /* handle the error */ }

NOTE: The close-on-exec flag has no effect on fork(): all file
descriptors are inherited by the child process. The Python issue #16500
"Add an atfork module" proposes to add a new atfork module to execute
code at fork, which may be used to automatically close file descriptors.

Issues with Inheritable File Descriptors

Most of the time, inheritable file descriptors "leaked" to child
processes are not noticed, because they don't cause major bugs. It does
not mean that these bugs must not be fixed.

Two common issues with inherited file descriptors:

-   On Windows, a directory cannot be removed before all file handles
    open in the directory are closed. The same issue can be seen with
    files, except if the file was created with the FILE_SHARE_DELETE
    flag (O_TEMPORARY mode for open()).
-   If a listening socket is leaked to a child process, the socket
    address cannot be reused before the parent and child processes
    terminated. For example, if a web server spawns a new program to
    handle a process, and the server restarts while the program is not
    done, the server cannot start because the TCP port is still in use.

Example of issues in open source projects:

-   Mozilla (Firefox): open since 2002-05
-   dbus library: fixed in 2008-05 (dbus commit), close file descriptors
    in the child process
-   autofs: fixed in 2009-02, set the CLOEXEC flag
-   qemu: fixed in 2009-12 (qemu commit), set CLOEXEC flag
-   Tor: fixed in 2010-12, set CLOEXEC flag
-   OCaml: open since 2011-04, "PR#5256: Processes opened using
    Unix.open_process* inherit all opened file descriptors (including
    sockets)"
-   ØMQ: open since 2012-08
-   Squid: open since 2012-07

See also: Excuse me son, but your code is leaking !!! (Dan Walsh, March
2012) for SELinux issues with leaked file descriptors.

Security Vulnerability

Leaking sensitive file handles and file descriptors can lead to security
vulnerabilities. An untrusted child process might read sensitive data
like passwords or take control of the parent process though a leaked
file descriptor. With a leaked listening socket, a child process can
accept new connections to read sensitive data.

Example of vulnerabilities:

-   Hijacking Apache https by mod_php (2003)
    -   Apache: Apr should set FD_CLOEXEC if APR_FOPEN_NOCLEANUP is not
        set: fixed in 2009
    -   PHP: system() (and similar) don't cleanup opened handles of
        Apache: open since 2006
-   CWE-403: Exposure of File Descriptor to Unintended Control Sphere
    (2008)
-   OpenSSH Security Advisory: portable-keysign-rand-helper.adv (2011)

Read also the CERT Secure Coding Standards: FIO42-C. Ensure files are
properly closed when they are no longer needed.

Issues fixed in the subprocess module

Inherited file descriptors caused 4 issues in the subprocess module:

-   Issue #2320: Race condition in subprocess using stdin (opened in
    2008)
-   Issue #3006: subprocess.Popen causes socket to remain open after
    close (opened in 2008)
-   Issue #7213: subprocess leaks open file descriptors between Popen
    instances causing hangs (opened in 2009)
-   Issue #12786: subprocess wait() hangs when stdin is closed (opened
    in 2011)

These issues were fixed in Python 3.2 by 4 different changes in the
subprocess module:

-   Pipes are now non-inheritable;
-   The default value of the close_fds parameter is now True, with one
    exception on Windows: the default value is False if at least one
    standard stream is replaced;
-   A new pass_fds parameter has been added;
-   Creation of a _posixsubprocess module implemented in C.

Atomic Creation of non-inheritable File Descriptors

In a multi-threaded application, an inheritable file descriptor may be
created just before a new program is spawned, before the file descriptor
is made non-inheritable. In this case, the file descriptor is leaked to
the child process. This race condition could be avoided if the file
descriptor is created directly non-inheritable.

FreeBSD, Linux, Mac OS X, Windows and many other operating systems
support creating non-inheritable file descriptors with the inheritable
flag cleared atomically at the creation of the file descriptor.

A new WSA_FLAG_NO_HANDLE_INHERIT flag for WSASocket() was added in
Windows 7 SP1 and Windows Server 2008 R2 SP1 to create non-inheritable
sockets. If this flag is used on an older Windows version (ex: Windows
XP SP3), WSASocket() fails with WSAEPROTOTYPE.

On UNIX, new flags were added for files and sockets:

-   O_CLOEXEC: available on Linux (2.6.23), FreeBSD (8.3), Mac OS 10.8,
    OpenBSD 5.0, Solaris 11, QNX, BeOS, next NetBSD release (6.1?). This
    flag is part of POSIX.1-2008.
-   SOCK_CLOEXEC flag for socket() and socketpair(), available on Linux
    2.6.27, OpenBSD 5.2, NetBSD 6.0.
-   fcntl(): F_DUPFD_CLOEXEC flag, available on Linux 2.6.24, OpenBSD
    5.0, FreeBSD 9.1, NetBSD 6.0, Solaris 11. This flag is part of
    POSIX.1-2008.
-   fcntl(): F_DUP2FD_CLOEXEC flag, available on FreeBSD 9.1 and Solaris
    11.
-   recvmsg(): MSG_CMSG_CLOEXEC, available on Linux 2.6.23, NetBSD 6.0.

On Linux older than 2.6.23, O_CLOEXEC flag is simply ignored. So fcntl()
must be called to check if the file descriptor is non-inheritable:
O_CLOEXEC is not supported if the FD_CLOEXEC flag is missing. On Linux
older than 2.6.27, socket() or socketpair() fail with errno set to
EINVAL if the SOCK_CLOEXEC flag is set in the socket type.

New functions:

-   dup3(): available on Linux 2.6.27 (and glibc 2.9)
-   pipe2(): available on Linux 2.6.27 (and glibc 2.9)
-   accept4(): available on Linux 2.6.28 (and glibc 2.10)

On Linux older than 2.6.28, accept4() fails with errno set to ENOSYS.

Summary:

  Operating System   Atomic File     Atomic Socket
  ------------------ --------------- --------------------------------------
  FreeBSD            8.3 (2012)      X
  Linux              2.6.23 (2007)   2.6.27 (2008)
  Mac OS X           10.8 (2012)     X
  NetBSD             6.1 (?)         6.0 (2012)
  OpenBSD            5.0 (2011)      5.2 (2012)
  Solaris            11 (2011)       X
  Windows            XP (2001)       Seven SP1 (2011), 2008 R2 SP1 (2011)

Legend:

-   "Atomic File": first version of the operating system supporting
    creating atomically a non-inheritable file descriptor using open()
-   "Atomic Socket": first version of the operating system supporting
    creating atomically a non-inheritable socket
-   "X": not supported yet

See also:

-   Secure File Descriptor Handling (Ulrich Drepper, 2008)
-   Ghosts of Unix past, part 2: Conflated designs (Neil Brown, 2010)
    explains the history of O_CLOEXEC and O_NONBLOCK flags
-   File descriptor handling changes in 2.6.27
-   FreeBSD: atomic close on exec

Status of Python 3.3

Python 3.3 creates inheritable file descriptors on all platforms, except
os.pipe() which creates non-inheritable file descriptors on Windows.

New constants and functions related to the atomic creation of
non-inheritable file descriptors were added to Python 3.3: os.O_CLOEXEC,
os.pipe2() and socket.SOCK_CLOEXEC.

On UNIX, the subprocess module closes all file descriptors in the child
process by default, except standard streams (0, 1, 2) and file
descriptors of the pass_fds parameter. If the close_fds parameter is set
to False, all inheritable file descriptors are inherited in the child
process.

On Windows, the subprocess closes all handles and file descriptors in
the child process by default. If at least one standard stream (stdin,
stdout or stderr) is replaced (ex: redirected into a pipe), all
inheritable handles and file descriptors 0, 1 and 2 are inherited in the
child process.

Using the functions of the os.execv*() and os.spawn*() families, all
inheritable handles and all inheritable file descriptors are inherited
by the child process.

On UNIX, the multiprocessing module uses os.fork() and so all file
descriptors are inherited by child processes.

On Windows, all inheritable handles and file descriptors 0, 1 and 2 are
inherited by the child process using the multiprocessing module, all
file descriptors except standard streams are closed.

Summary:

  Module                        FD on UNIX       Handles on Windows   FD on Windows
  ----------------------------- ---------------- -------------------- ---------------
  subprocess, default           STD, pass_fds    none                 STD
  subprocess, replace stdout    STD, pass_fds    all                  STD
  subprocess, close_fds=False   all              all                  STD
  multiprocessing               not applicable   all                  STD
  os.execv(), os.spawn()        all              all                  all

Legend:

-   "all": all inheritable file descriptors or handles are inherited in
    the child process
-   "none": all handles are closed in the child process
-   "STD": only file descriptors 0 (stdin), 1 (stdout) and 2 (stderr)
    are inherited in the child process
-   "pass_fds": file descriptors of the pass_fds parameter of the
    subprocess are inherited
-   "not applicable": on UNIX, the multiprocessing uses fork(), so this
    case is not affected by this PEP.

Closing All Open File Descriptors

On UNIX, the subprocess module closes almost all file descriptors in the
child process. This operation requires MAXFD system calls, where MAXFD
is the maximum number of file descriptors, even if there are only few
open file descriptors. This maximum can be read using:
os.sysconf("SC_OPEN_MAX").

The operation can be slow if MAXFD is large. For example, on a FreeBSD
buildbot with MAXFD=655,000, the operation took 300 ms: see issue
#11284: slow close file descriptors.

On Linux, Python 3.3 gets the list of all open file descriptors from
/proc/<PID>/fd/, and so performances depends on the number of open file
descriptors, not on MAXFD.

See also:

-   Python issue #1663329: subprocess close_fds perform poor if
    SC_OPEN_MAX is high
-   Squid Bug #837033: Squid should set CLOEXEC on opened FDs. "32k+
    close() calls in each child process take a long time ([12-56]
    seconds) in Xen PV guests."

Proposal

Non-inheritable File Descriptors

The following functions are modified to make newly created file
descriptors non-inheritable by default:

-   asyncore.dispatcher.create_socket()
-   io.FileIO
-   io.open()
-   open()
-   os.dup()
-   os.fdopen()
-   os.open()
-   os.openpty()
-   os.pipe()
-   select.devpoll()
-   select.epoll()
-   select.kqueue()
-   socket.socket()
-   socket.socket.accept()
-   socket.socket.dup()
-   socket.socket.fromfd()
-   socket.socketpair()

os.dup2() still creates inheritable by default, see below.

When available, atomic flags are used to make file descriptors
non-inheritable. The atomicity is not guaranteed because a fallback is
required when atomic flags are not available.

New Functions And Methods

New functions available on all platforms:

-   os.get_inheritable(fd: int): return True if the file descriptor can
    be inherited by child processes, False otherwise.
-   os.set_inheritable(fd: int, inheritable: bool): set the inheritable
    flag of the specified file descriptor.

New functions only available on Windows:

-   os.get_handle_inheritable(handle: int): return True if the handle
    can be inherited by child processes, False otherwise.
-   os.set_handle_inheritable(handle: int, inheritable: bool): set the
    inheritable flag of the specified handle.

New methods:

-   socket.socket.get_inheritable(): return True if the socket can be
    inherited by child processes, False otherwise.
-   socket.socket.set_inheritable(inheritable: bool): set the
    inheritable flag of the specified socket.

Other Changes

On UNIX, subprocess makes file descriptors of the pass_fds parameter
inheritable. The file descriptor is made inheritable in the child
process after the fork() and before execv(), so the inheritable flag of
file descriptors is unchanged in the parent process.

os.dup2() has a new optional inheritable parameter:
os.dup2(fd, fd2, inheritable=True). fd2 is created inheritable by
default, but non-inheritable if inheritable is False.

os.dup2() behaves differently than os.dup() because the most common use
case of os.dup2() is to replace the file descriptors of the standard
streams: stdin (0), stdout (1) and stderr (2). Standard streams are
expected to be inherited by child processes.

Backward Compatibility

This PEP break applications relying on inheritance of file descriptors.
Developers are encouraged to reuse the high-level Python module
subprocess which handles the inheritance of file descriptors in a
portable way.

Applications using the subprocess module with the pass_fds parameter or
using only os.dup2() to redirect standard streams should not be
affected.

Python no longer conform to POSIX, since file descriptors are now made
non-inheritable by default. Python was not designed to conform to POSIX,
but was designed to develop portable applications.

Related Work

The programming languages Go, Perl and Ruby make newly created file
descriptors non-inheritable by default: since Go 1.0 (2009), Perl 1.0
(1987) and Ruby 2.0 (2013).

The SCons project, written in Python, overrides builtin functions file()
and open() to make files non-inheritable on Windows: see win32.py.

Rejected Alternatives

Add a new open_noinherit() function

In June 2007, Henning von Bargen proposed on the python-dev mailing list
to add a new open_noinherit() function to fix issues of inherited file
descriptors in child processes. At this time, the default value of the
close_fds parameter of the subprocess module was False.

Read the mail thread: [Python-Dev] Proposal for a new function
"open_noinherit" to avoid problems with subprocesses and security risks.

PEP 433

PEP 433, "Easier suppression of file descriptor inheritance", was a
previous attempt proposing various other alternatives, but no consensus
could be reached.

Python Issues

-   #10115: Support accept4() for atomic setting of flags at socket
    creation
-   #12105: open() does not able to set flags, such as O_CLOEXEC
-   #12107: TCP listening sockets created without FD_CLOEXEC flag
-   #16850: Add "e" mode to open(): close-and-exec (O_CLOEXEC) /
    O_NOINHERIT
-   #16860: Use O_CLOEXEC in the tempfile module
-   #16946: subprocess: _close_open_fd_range_safe() does not set
    close-on-exec flag on Linux < 2.6.23 if O_CLOEXEC is defined
    <http://bugs.python.org/issue16946>
-   #17070: Use the new cloexec to improve security and avoid bugs
-   #18571: Implementation of the PEP 446: non-inheritable file
    descriptors

Copyright

This document has been placed into the public domain.


text:
PEP: 211 Title: Adding A New Outer Product Operator Author: Greg Wilson
<EMAIL_ADDRESS>Status: Rejected Type: Standards Track Created:
15-Jul-2000 Python-Version: 2.1 Post-History:

The approach in the later PEP 465 was eventually accepted in lieu of
this PEP. The Rejected Ideas
<465#rejected-alternatives-to-adding-a-new-operator> of that PEP
explains the rationale in more detail.

Introduction

This PEP describes a proposal to define @ (pronounced "across") as a new
outer product operator in Python 2.2. When applied to sequences (or
other iterable objects), this operator will combine their iterators, so
that:

    for (i, j) in S @ T:
        pass

will be equivalent to:

    for i in S:
        for j in T:
            pass

Classes will be able to overload this operator using the special methods
__across__, __racross__, and __iacross__. In particular, the new Numeric
module (PEP 209) will overload this operator for multi-dimensional
arrays to implement matrix multiplication.

Background

Number-crunching is now just a small part of computing, but many
programmers --- including many Python users --- still need to express
complex mathematical operations in code. Most numerical languages, such
as APL, Fortran-90, MATLAB, IDL, and Mathematica, therefore provide two
forms of the common arithmetic operators. One form works
element-by-element, e.g. multiplies corresponding elements of its matrix
arguments. The other implements the "mathematical" definition of that
operation, e.g. performs row-column matrix multiplication.

Zhu and Lielens have proposed <225> doubling up Python's operators in
this way. Their proposal would create six new binary infix operators,
and six new in-place operators.

The original version of this proposal was much more conservative. The
author consulted the developers of GNU Octave[1], an open source clone
of MATLAB. Its developers agreed that providing an infix operator for
matrix multiplication was important: numerical programmers really do
care whether they have to write mmul(A,B) instead of A op B.

On the other hand, when asked how important it was to have infix
operators for matrix solution and other operations, Prof. James Rawlings
replied[2]:

  I DON'T think it's a must have, and I do a lot of matrix inversion. I
  cannot remember if its A\b or b\A so I always write inv(A)*b instead.
  I recommend dropping \.

Based on this discussion, and feedback from students at the US national
laboratories and elsewhere, we recommended adding only one new operator,
for matrix multiplication, to Python.

Iterators

The planned addition of iterators to Python 2.2 opens up a broader scope
for this proposal. As part of the discussion of PEP 201, Lockstep
Iteration, the author of this proposal conducted an informal usability
experiment[3]. The results showed that users are psychologically
receptive to "cross-product" loop syntax. For example, most users
expected:

    S = [10, 20, 30]
    T = [1, 2, 3]
    for x in S; y in T:
        print x+y,

to print 11 12 13 21 22 23 31 32 33. We believe that users will have the
same reaction to:

    for (x, y) in S @ T:
        print x+y

i.e. that they will naturally interpret this as a tidy way to write loop
nests.

This is where iterators come in. Actually constructing the cross-product
of two (or more) sequences before executing the loop would be very
expensive. On the other hand, @ could be defined to get its arguments'
iterators, and then create an outer iterator which returns tuples of the
values returned by the inner iterators.

Discussion

1.  Adding a named function "across" would have less impact on Python
    than a new infix operator. However, this would not make Python more
    appealing to numerical programmers, who really do care whether they
    can write matrix multiplication using an operator, or whether they
    have to write it as a function call.

2.  @ would have be chainable in the same way as comparison operators,
    i.e.:

        (1, 2) @ (3, 4) @ (5, 6)

    would have to return (1, 3, 5) ... (2, 4, 6), and not
    ((1, 3), 5) ... ((2, 4), 6). This should not require special support
    from the parser, as the outer iterator created by the first @ could
    easily be taught how to combine itself with ordinary iterators.

3.  There would have to be some way to distinguish restartable iterators
    from ones that couldn't be restarted. For example, if S is an input
    stream (e.g. a file), and L is a list, then S @ L is
    straightforward, but L @ S is not, since iteration through the
    stream cannot be repeated. This could be treated as an error, or by
    having the outer iterator detect non-restartable inner iterators and
    cache their values.

4.  Whiteboard testing of this proposal in front of three novice Python
    users (all of them experienced programmers) indicates that users
    will expect:

        "ab" @ "cd"

    to return four strings, not four tuples of pairs of characters.
    Opinion was divided on what:

        ("a", "b") @ "cd"

    ought to return...

Alternatives

1.  Do nothing --- keep Python simple.

    This is always the default choice.

2.  Add a named function instead of an operator.

    Python is not primarily a numerical language; it may not be worth
    complexifying it for this special case. However, support for real
    matrix multiplication is frequently requested, and the proposed
    semantics for @ for built-in sequence types would simplify
    expression of a very common idiom (nested loops).

3.  Introduce prefixed forms of all existing operators, such as ~* and
    ~+, as proposed in PEP 225.

    Our objections to this are that there isn't enough demand to justify
    the additional complexity (see Rawlings' comments[4]), and that the
    proposed syntax fails the "low toner" readability test.

Acknowledgments

I am grateful to Huaiyu Zhu for initiating this discussion, and to James
Rawlings and students in various Python courses for their discussions of
what numerical programmers really care about.

References

[1] http://bevo.che.wisc.edu/octave/

[2] http://www.egroups.com/message/python-numeric/4

[3] https://mail.python.org/pipermail/python-dev/2000-July/006427.html

[4] http://www.egroups.com/message/python-numeric/4

license: Public Domain


text:
PEP: 250 Title: Using site-packages on Windows Author: Paul Moore
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 30-Mar-2001 Python-Version: 2.2 Post-History:
30-Mar-2001

Abstract

The standard Python distribution includes a directory Lib/site-packages,
which is used on Unix platforms to hold locally installed modules and
packages. The site.py module distributed with Python includes support
for locating other modules in the site-packages directory.

This PEP proposes that the site-packages directory should be used on the
Windows platform in a similar manner.

Motivation

On Windows platforms, the default setting for sys.path does not include
a directory suitable for users to install locally developed modules. The
"expected" location appears to be the directory containing the Python
executable itself. This is also the location where distutils (and
distutils-generated installers) installs packages. Including locally
developed code in the same directory as installed executables is not
good practice.

Clearly, users can manipulate sys.path, either in a locally modified
site.py, or in a suitable sitecustomize.py, or even via .pth files.
However, there should be a standard location for such files, rather than
relying on every individual site having to set their own policy.

In addition, with distutils becoming more prevalent as a means of
distributing modules, the need for a standard install location for
distributed modules will become more common. It would be better to
define such a standard now, rather than later when more distutils-based
packages exist which will need rebuilding.

It is relevant to note that prior to Python 2.1, the site-packages
directory was not included in sys.path for Macintosh platforms. This has
been changed in 2.1, and Macintosh includes sys.path now, leaving
Windows as the only major platform with no site-specific modules
directory.

Implementation

The implementation of this feature is fairly trivial. All that would be
required is a change to site.py, to change the section setting sitedirs.
The Python 2.1 version has:

    if os.sep == '/':
        sitedirs = [makepath(prefix,
                            "lib",
                            "python" + sys.version[:3],
                            "site-packages"),
                    makepath(prefix, "lib", "site-python")]
    elif os.sep == ':':
        sitedirs = [makepath(prefix, "lib", "site-packages")]
    else:
        sitedirs = [prefix]

A suitable change would be to simply replace the last 4 lines with:

    else:
        sitedirs == [prefix, makepath(prefix, "lib", "site-packages")]

Changes would also be required to distutils, to reflect this change in
policy. A patch is available on Sourceforge, patch ID 445744, which
implements this change. Note that the patch checks the Python version
and only invokes the new behaviour for Python versions from 2.2 onwards.
This is to ensure that distutils remains compatible with earlier
versions of Python.

Finally, the executable code which implements the Windows installer used
by the bdist_wininst command will need changing to use the new location.
A separate patch is available for this, currently maintained by Thomas
Heller.

Notes

-   This change does not preclude packages using the current location --
    the change only adds a directory to sys.path, it does not remove
    anything.
-   Both the current location (sys.prefix) and the new directory
    (site-packages) are included in sitedirs, so that .pth files will be
    recognised in either location.
-   This proposal adds a single additional site-packages directory to
    sitedirs. On Unix platforms, two directories are added, one for
    version-independent files (Python code) and one for
    version-dependent code (C extensions). This is necessary on Unix, as
    the sitedirs include a common (across Python versions) package
    location, in /usr/local by default. As there is no such common
    location available on Windows, there is also no need for having two
    separate package directories.
-   If users want to keep DLLs in a single location on Windows, rather
    than keeping them in the package directory, the DLLs subdirectory of
    the Python install directory is already available for that purpose.
    Adding an extra directory solely for DLLs should not be necessary.

Open Issues

-   Comments from Unix users indicate that there may be issues with the
    current setup on the Unix platform. Rather than become involved in
    cross-platform issues, this PEP specifically limits itself to the
    Windows platform, leaving changes for other platforms to be covered
    in other PEPs.
-   There could be issues with applications which embed Python. To the
    author's knowledge, there should be no problem as a result of this
    change. There have been no comments (supportive or otherwise) from
    users who embed Python.

Copyright

This document has been placed in the public domain.


text:
PEP: 736 Title: Shorthand syntax for keyword arguments at invocation
Author: Joshua Bambrick<EMAIL_ADDRESS>Chris Angelico
<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pep-736-shorthand-syntax-for-keyword-arguments-at-invocation/43432
Status: Draft Type: Standards Track Created: 28-Nov-2023 Python-Version:
3.14 Post-History: 14-Oct-2023, 17-Jan-2024,

Abstract

This PEP proposes introducing syntactic sugar f(x=) for the pattern
where a named argument is the same as the name of the variable
corresponding to its value f(x=x).

Motivation

Keyword argument syntax can become needlessly repetitive and verbose.

Consider the following call: :

    my_function(
      my_first_variable=my_first_variable,
      my_second_variable=my_second_variable,
      my_third_variable=my_third_variable,
    )

The case of a keyword argument name matching the variable name of its
value is prevalent among Python libraries. This verbosity and redundancy
discourages use of named arguments and reduces readability by increasing
visual noise.

Rationale

There are two ways to invoke a function with arguments: by position and
by keyword. Keyword arguments confer many benefits by being explicit,
thus increasing readability and minimising the risk of inadvertent
transposition. On the flipside, positional arguments are often used
simply to minimise verbosity and visual noise.

We contend that a simple syntactic sugar used to simplify this common
pattern which would confer numerous benefits:

Encourages use of named arguments

This syntax would encourage the use of named arguments, thereby
increasing readability and reducing bugs from argument transposition.

Reduces verbosity

By minimising visual noise and in some cases lines of code, we can
increase readability.

Encourages consistent variable names

A common problem is that semantically identical variables have different
names depending on their contexts. This syntax would encourage authors
to use the same variable name when calling a function as the argument
name, which would increase consistency of variable names used and hence
also readability.

Highlights arguments not following this pattern

With the current syntax, function calls where many arguments are
forwarded from the local context can make other argument values easy to
miss due to the visual noise. For example:

    add_middleware(
        excluded_urls=excluded_urls,
        server_request=server_request,
        client_request=client_request,
        client_response=client_response,
        span_details=_get_span_details(),
        tracer=tracer,
        meter=meter,
    )

With this syntax, the exceptional arguments become easier to identify:

    add_middleware(
        excluded_urls=,
        server_request=,
        client_request=,
        client_response=,
        span_details=_get_span_details(),
        tracer=,
        meter=,
    )

Applicability to dictionary construction

This syntax can be applied to dictionary construction where a similar
pattern frequently occurs (where dictionary keys are identical the names
of the variables assigned as their values), {"x": x, "y": y} or
dict(x=x, y=y). With this feature, this can now also be trivially
written as dict(x=, y=). Whether to further support similar syntax in
dictionary literals is an open question out of the scope of this PEP.

Specification

We propose to introduce syntactic sugar such that, if the value of a
keyword argument is omitted from a function invocation, the argument's
value is inferred to be the variable matching that name at the
invocation scope.

For example, the function invocation: :

    my_function(my_first_variable=, my_second_variable=, my_third_variable=)

Will be interpreted exactly equivalently to following in existing
syntax: :

    my_function(
      my_first_variable=my_first_variable,
      my_second_variable=my_second_variable,
      my_third_variable=my_third_variable,
    )

If no variable matches that name in the invocation scope, a NameError is
raised in an identical manner as would be with the established expanded
syntax.

This proposal only pertains to function invocations; function
definitions are unaffected by the syntax change. All existing valid
syntax is unchanged.

Backwards Compatibility

Only new syntax is added which was previously syntactically erroneous.
No existing valid syntax is modified. As such, the changes proposed are
fully backwards compatible.

Security Implications

There are no security implications for this change.

Prior Art

Python already possesses a very similar feature in f-string
interpolation where f'{x=}' is effectively expanded to f'x={x}' (see
related GitHub issue).

Several modern languages provide similar features during function
invocation, sometimes referred to as 'punning'. For example:

-   In Ruby, f(x:, y:) is syntactic sugar for f(x: x, y: y). See the
    Ruby 3.1.0 release notes (search for "keyword arguments").
-   In ReasonML, f(~x, ~y) is syntactic sugar for f(~x=x, ~y=y). See the
    ReasonML function documentation (search for "punning").
-   In SystemVerilog, (.mult, .mop1, .data); is syntactic sugar for
    (.mult(mult), .mop1(mop1),  .data(data));. See SystemVerilog
    Implicit Port Connections.
-   In Jakt, f(x, y) is syntactic sugar for f(x: x, y: y). See The Jakt
    programming language.

Beyond function invocation specifically, more languages offer similar
features:

-   In OCaml, let+ x in … is syntactic sugar for let+ x = x in …. See
    OCaml Short notation for variable bindings (let-punning).
-   In JavaScript, { x, y } is syntactic sugar for {x: x, y: y}. See
    JavaScript Object Initializer.
-   In Rust, User { x, y } is shorthand for User {x: x, y: y}. See Rust
    Using the Field Init Shorthand.

Applicability

We analysed popular Python libraries from the last few years using this
script to compute:

-   The number of keyword arguments were of the form f(x=x) at
    invocation.
-   The percentage of keyword arguments which had the form f(x=x) at
    invocation.
-   The number of lines of code which could be saved by using this
    syntactic sugar to reduce the need for line wraps.

The purpose of this exercise was to compute statistics about the
prevalence of this pattern and should not be interpreted as a
recommendation that the proposed syntactic sugar should be applied
universally.

  Statistic                                                          polars   fastapi   rich     httpx
  ------------------------------------------------------------------ -------- --------- -------- --------
  Number of keyword arguments of the form f(x=x) at invocation       1,654    1,408     566      759
  Percentage of keyword arguments of the form f(x=x) at invocation   15.83%   28.11%    15.74%   45.13%
  Lines saved                                                        170      35        62       117

Based on this, we note that the f(x=x) keyword argument pattern is
widespread, accounting for anywhere from 15% to just below half of all
keyword argument uses depending on the codebase.

Proposed Syntax

While this feature has been proposed on numerous occasions with several
different forms[1][2][3][4][5],[6] we have opted to advocate for the
f(x=) form for the following reasons:

-   This feature has been proposed frequently over a ten year period
    with the f(x=) or f(=x) being by far the most common
    syntax[7][8][9]. This is a strong indicator that it is the obvious
    notation.
-   The proposed syntax closely matches the f-string debug f'{var=}'
    syntax (established Pythonic style) and serves an almost identical
    purpose.
-   The proposed syntax is exactly analogous to the Ruby keyword
    argument syntactic sugar. See the Ruby 3.1.0 release notes (search
    for "keyword arguments").
-   The syntax is easy to implement as it is simple syntactic sugar.
-   When compared to the prefix form (see Rejected Ideas), this syntax
    communicates "here is a parameter, go find its argument" which is
    more appropriate given the semantics of named arguments.
-   A poll of Python developers indicates that this is the most popular
    syntax among those proposed.

How to Teach This

To ease the communication of and search for this feature, it may also be
valuable to provide this feature with a name, such as 'keyword argument
shorthand'.

Keen Python developers will likely hear about this feature through
typical information channels, such as newsboards, social media, mailing
lists, online forums, or word of mouth. Many more will encounter this
feature while reading code and noting the omission of the value in a
keyword argument at invocation, violating their expectations. We should
ensure such developers have easy access to documentation that explains
the semantics of this feature and that this documentation is easy to
find when searching. For example, the Python Glossary and Tutorial may
be updated accordingly and reasonable keywords may be used to help with
search discoverability. A StackOverflow question could be written to
help explain this feature to those searching for an explanation.

A teacher may explain this feature to new Python programmers as, "where
you see an argument followed by an equals sign, such as f(x=), this
represents a keyword argument where the name of the argument and its
value are the same. This can be written equivalently in the expanded
notation, f(x=x)." Depending on a student's background, a teacher might
further compare this to equivalent syntax in other languages or Python's
f-string syntax f"{x=}".

To understand this, a student of Python would need to be familiar with
the basics of functions in addition to the existing keyword argument
syntax. Given that this feature is a relatively straightforward
syntactic sugar, it is reasonable that a student who possesses a grasp
of keyword arguments will be able to absorb this concept quickly. This
is evidenced by the success of the f-string syntax as well as similar
features in other languages (see Prior Art).

Rejected Ideas

Many alternative syntaxes have been proposed however no syntax other
than f(=x) or f(x=) has garnered significant support. We here enumerate
some of the most popular proposed alternatives and why we ultimately
reject them.

f(a, b, *, x)

On a few occasions the idea has been floated to borrow the syntax from
keyword-only function definitions.

In favour of this proposal:

-   This syntax is familiar from its use to require keyword-only
    arguments in function definitions.
-   A poll of Python developers indicates that this is the second most
    popular syntax among those proposed.

However, we object that:

-   For any given argument, it is less clear from local context whether
    it is positional or named. The * could easily be missed in a long
    argument list and named arguments may be read as positional or vice
    versa.
-   It is unclear whether keyword arguments for which the value was not
    elided may follow the *. If so, then their relative position will be
    confusingly arbitrary, but if not, then an arbitrary grouping is
    enforced between different types of keyword arguments and reordering
    of arguments would be necessary if only one name (the argument or
    its value) was changed.
-   The use of * in function calls is established and this proposal
    would introduce a new effect which could cause confusion. For
    example, f(a, *x, y) would mean something different than
    f(a, *, x, y).

f(=x)

In favour of this form:

-   The prefix operator is more similar to the established *args and
    **kwargs syntax for function calls.
-   It draws more attention to itself when arguments are arranged
    vertically. In particular, if the arguments are of different lengths
    it is harder to find the equal sign at the end. Moreover, since
    Python is read left to right, the use of this feature is clearer to
    the reader earlier on.

On the contrary:

-   While the prefix version is visually louder, in practice, there is
    no need for this feature to shout its presence any more than a
    typical named argument. By the time we read to the = it is clear
    that the value is filled in automatically just as the value is clear
    in the typical keyword argument case.
-   Semantically, this form communicates 'here is a value, fill in the
    parameter' which is not what we want to convey.
-   It is less similar to f-string syntax.
-   It is less obvious that arbitrary expressions are invalid, e.g.
    f(=a + b).

f(%x) or f(:x) or f(.x)

Several flavours of this syntax have been proposed with the prefix form
substituting another character for =. However, no such form has gained
traction and the choice of symbol seems arbitrary compared to =.
Additionally, there is less precedent in terms of existing language
features (such as f-string) or other languages (such as Ruby).

Objections

There are only a few hard objections to the introduction of this
syntactic sugar. Most of those not in favour of this feature are in the
camp of 'I wouldn't use it'. However, over the extensive conversations
about this feature, the following objections were the most common:

The syntax is ugly

This objection is by far the most common. On the contrary, we argue
that:

-   This objection is subjective and many community members disagree.
-   A nearly-identical syntax is already established for f-strings.
-   Programmers will, as ever, adjust over time.

The feature is confusing

We argue that:

-   Introducing new features typically has this impact temporarily.
-   The syntax is very similar to the established f'{x=}' syntax.
-   The feature and syntax are familiar from other popular modern
    languages.
-   The expansion of x= to x=x is in fact a trivial feature and
    inherently significantly less complex than *arg and **kwarg
    expansion.
-   This particular syntactic form has been independently proposed on
    numerous occasions, indicating that it is the most
    obvious[10][11][12].

The feature is not explicit

We recognise that, in an obvious sense, the argument value is 'implicit'
in this proposed syntax. However, we do not think that this is what the
Zen of Python is aiming to discourage.

In the sense that we take the Zen to be referring to, keyword arguments
(for example) are more explicit than positional arguments where the
argument name is omitted and impossible to tell from the local context.
Conversely, the syntactic sugar for integers x += 1 is not more implicit
than x = x + 1 in this sense, even though the variable is omitted from
the right hand side, because it is immediately obvious from the local
context what it is.

The syntax proposed in this PEP is much more closely analogous to the
x += 1 example (although simpler since we do not propose to introduce a
new operation). Moreover, the introduction of this syntactic sugar
should encourage the use of keyword arguments over positional ones,
making typical Python codebases more explicit in general.

The feature adds another way of doing things

The same argument can be made against all syntax changes. This is a
simple syntactic sugar, much as x += 1 is sugar for x = x + 1 when x is
an integer. This isn't tantamount to a 'new way' of passing arguments
but a more readable notation for the same way.

Renaming the variable in the calling context will break the code

A NameError would make the mistake clear in most cases. There may be
confusion if a variable from a broader scope has the same name as the
original variable, so no NameError would be raised. However, this issue
can also occur with keyword arguments using the current syntax
(arguably, this syntactic sugar could make it harder to spot). Moreover,
having variables with the same name in different scopes is broadly
considered bad practice and discouraged by linters.

Code editors could highlight the issue based on static analysis - f(x=)
is exactly equivalent to writing f(x=x). If x does not exist, modern
editors have no problem highlighting the issue.

This syntax increases coupling

We recognise that, as ever, all syntax has the potential for misuse and
so should be applied judiciously to improve codebases. In this case, if
a parameter and its value have the same semantics in both contexts, that
may suggest that using this new syntax is appropriate and will help
ameliorate the risk of unintentional desynchronisation which harms
readability.

However, if the two variables have different semantics, that may suggest
that this feature should not be used to encourage consistency or even
that they should be renamed.

Recommendations for using this syntax

As with any other language feature, the programmer should exercise their
own judgement about whether it is prudent to use it in any given
context. We do not recommend enforcing a rule to use the feature in all
cases where it may be applicable.

As described above, we propose that a reasonable rule of thumb would be
to use this in cases where a parameter and its argument have the same
semantics in order to reduce unintentional desynchronisation without
causing inappropriate coupling.

Impact on editing

Using a plain text editor

Editing with a plain text editor should generally be unaffected.

When renaming a variable using a 'Find-Replace' method, where this
syntax is used the developer will come across the function argument at
invocation (as they would if this syntax was not used). At that point,
they can as usual decide whether to update the argument as well or
expand to the full f(x=x) syntax.

As with the current syntax, a 'Find-Replace All' method would fail since
the keyword argument would not exist at function definition, in the vast
majority of cases.

If the developer leaves the argument name unchanged and forgets to
update its value, a NameError will typically be raised as described
above.

Proposals for IDEs

In response to community feedback, we include some suggestions regarding
how IDEs could handle this syntax. However, we of course defer to the
domain experts developing IDEs to use their own discretion.

Most considerations are made simple by recognising that f(x=) is just
syntactic sugar for f(x=x) and should be treated the same as at present.

Highlighting NameErrors

IDEs typically offer a feature to highlight code that may cause a
NameError. We recommend that this syntax be treated similarly to the
expanded form f(x=x) to identify and highlight cases where the elided
value variable may not exist. What visual cue may be used to highlight
these cases may be the same or different from that which would be used
with the current syntax, depending on the IDE.

Jump to definition

There are a few possible ways that a 'jump to definition' feature could
be implemented depending on the caret/cursor position.

One option is to:

-   Jump to the argument in the function definition if the caret/cursor
    is on the argument
-   Jump to the definition of the elided variable if the caret/cursor is
    on the character following the = in our proposed syntax

Another, potentially complementary, option would be to expand the syntax
visually on mouseover and enable a Ctrl+Click (or Cmd+Click) to the
definition of the variable.

Highlighting other references

IDEs frequently highlight matching code references to the value at the
current caret/cursor position. With this shorthand syntax, when the
caret/cursor is on the argument name it may be valuable to either:

-   Highlight both references to the argument and its value reflecting
    the fact that this name now refers to both
-   Visually expand the syntax on mouseover (as above) and apply
    established highlighting logic according to the cursor

Rename symbol

There are a few ways that IDEs may wish to support a 'Rename symbol'
feature for this syntax. For example, if the argument is being renamed,
the IDE may:

-   Also rename the variable used as its value in each calling context
    where this syntax is used
-   Expand to use the full syntax to pass the variable used as its value
-   Prompt the developer to select between the two above options

The last option here seems most preferable in order to reduce
unintentional desynchronisation of names while highlighting the user to
the changes.

Reference Implementation

A proposed implementation for cpython has been provided by @Hels15. We
will extend this implementation to add an AST node attribute indicating
for keywords whether the value was elided.

References

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

[1] Short form for keyword arguments and dicts (2013)
<EMAIL_ADDRESS>
[2] Keyword arguments self-assignment (2020)
<EMAIL_ADDRESS>
[3] Shorthand notation of dict literal and function call (2020)
https://discuss.python.org/t/shorthand-notation-of-dict-literal-and-function-call/5697/1

[4] Allow identifiers as keyword arguments at function call site
(extension of PEP 3102?) (2023)
https://discuss.python.org/t/allow-identifiers-as-keyword-arguments-at-function-call-site-extension-of-pep-3102/31677

[5] Shorten Keyword Arguments with Implicit Notation: foo(a=a, b=b) to
foo(.a, .b) (2023)
https://discuss.python.org/t/shorten-keyword-arguments-with-implicit-notation-foo-a-a-b-b-to-foo-a-b/33080

[6] Syntactic sugar to encourage use of named arguments (2023)
https://discuss.python.org/t/syntactic-sugar-to-encourage-use-of-named-arguments/36217

[7] Short form for keyword arguments and dicts (2013)
<EMAIL_ADDRESS>
[8] Keyword arguments self-assignment (2020)
<EMAIL_ADDRESS>
[9] Syntactic sugar to encourage use of named arguments (2023)
https://discuss.python.org/t/syntactic-sugar-to-encourage-use-of-named-arguments/36217

[10] Short form for keyword arguments and dicts (2013)
<EMAIL_ADDRESS>
[11] Keyword arguments self-assignment (2020)
<EMAIL_ADDRESS>
[12] Syntactic sugar to encourage use of named arguments (2023)
https://discuss.python.org/t/syntactic-sugar-to-encourage-use-of-named-arguments/36217


text:
PEP: 399 Title: Pure Python/C Accelerator Module Compatibility
Requirements Version: $Revision: 88219 $ Last-Modified: $Date:
2011-01-27 13:47:00 -0800 (Thu, 27 Jan 2011) $ Author: Brett Cannon
<EMAIL_ADDRESS>Status: Final Type: Informational Content-Type:
text/x-rst Created: 04-Apr-2011 Python-Version: 3.3 Post-History:
04-Apr-2011, 12-Apr-2011, 17-Jul-2011, 15-Aug-2011, 01-Jan-2013

Abstract

The Python standard library under CPython contains various instances of
modules implemented in both pure Python and C (either entirely or
partially). This PEP requires that in these instances that the C code
must pass the test suite used for the pure Python code so as to act as
much as a drop-in replacement as reasonably possible (C- and VM-specific
tests are exempt). It is also required that new C-based modules lacking
a pure Python equivalent implementation get special permission to be
added to the standard library.

Rationale

Python has grown beyond the CPython virtual machine (VM). IronPython,
Jython, and PyPy are all currently viable alternatives to the CPython
VM. The VM ecosystem that has sprung up around the Python programming
language has led to Python being used in many different areas where
CPython cannot be used, e.g., Jython allowing Python to be used in Java
applications.

A problem all of the VMs other than CPython face is handling modules
from the standard library that are implemented (to some extent) in C.
Since other VMs do not typically support the entire C API of CPython
they are unable to use the code used to create the module. Oftentimes
this leads these other VMs to either re-implement the modules in pure
Python or in the programming language used to implement the VM itself
(e.g., in C# for IronPython). This duplication of effort between
CPython, PyPy, Jython, and IronPython is extremely unfortunate as
implementing a module at least in pure Python would help mitigate this
duplicate effort.

The purpose of this PEP is to minimize this duplicate effort by
mandating that all new modules added to Python's standard library must
have a pure Python implementation unless special dispensation is given.
This makes sure that a module in the stdlib is available to all VMs and
not just to CPython (pre-existing modules that do not meet this
requirement are exempt, although there is nothing preventing someone
from adding in a pure Python implementation retroactively).

Re-implementing parts (or all) of a module in C (in the case of CPython)
is still allowed for performance reasons, but any such accelerated code
must pass the same test suite (sans VM- or C-specific tests) to verify
semantics and prevent divergence. To accomplish this, the test suite for
the module must have comprehensive coverage of the pure Python
implementation before the acceleration code may be added.

Details

Starting in Python 3.3, any modules added to the standard library must
have a pure Python implementation. This rule can only be ignored if the
Python development team grants a special exemption for the module.
Typically the exemption will be granted only when a module wraps a
specific C-based library (e.g., sqlite3). In granting an exemption it
will be recognized that the module will be considered exclusive to
CPython and not part of Python's standard library that other VMs are
expected to support. Usage of ctypes to provide an API for a C library
will continue to be frowned upon as ctypes lacks compiler guarantees
that C code typically relies upon to prevent certain errors from
occurring (e.g., API changes).

Even though a pure Python implementation is mandated by this PEP, it
does not preclude the use of a companion acceleration module. If an
acceleration module is provided it is to be named the same as the module
it is accelerating with an underscore attached as a prefix, e.g.,
_warnings for warnings. The common pattern to access the accelerated
code from the pure Python implementation is to import it with an
import *, e.g., from _warnings import *. This is typically done at the
end of the module to allow it to overwrite specific Python objects with
their accelerated equivalents. This kind of import can also be done
before the end of the module when needed, e.g., an accelerated base
class is provided but is then subclassed by Python code. This PEP does
not mandate that pre-existing modules in the stdlib that lack a pure
Python equivalent gain such a module. But if people do volunteer to
provide and maintain a pure Python equivalent (e.g., the PyPy team
volunteering their pure Python implementation of the csv module and
maintaining it) then such code will be accepted. In those instances the
C version is considered the reference implementation in terms of
expected semantics.

Any new accelerated code must act as a drop-in replacement as close to
the pure Python implementation as reasonable. Technical details of the
VM providing the accelerated code are allowed to differ as necessary,
e.g., a class being a type when implemented in C. To verify that the
Python and equivalent C code operate as similarly as possible, both code
bases must be tested using the same tests which apply to the pure Python
code (tests specific to the C code or any VM do not follow under this
requirement). The test suite is expected to be extensive in order to
verify expected semantics.

Acting as a drop-in replacement also dictates that no public API be
provided in accelerated code that does not exist in the pure Python
code. Without this requirement people could accidentally come to rely on
a detail in the accelerated code which is not made available to other
VMs that use the pure Python implementation. To help verify that the
contract of semantic equivalence is being met, a module must be tested
both with and without its accelerated code as thoroughly as possible.

As an example, to write tests which exercise both the pure Python and C
accelerated versions of a module, a basic idiom can be followed:

    from test.support import import_fresh_module
    import unittest

    c_heapq = import_fresh_module('heapq', fresh=['_heapq'])
    py_heapq = import_fresh_module('heapq', blocked=['_heapq'])

    class ExampleTest:

        def test_example(self):
            self.assertTrue(hasattr(self.module, 'heapify'))

    class PyExampleTest(ExampleTest, unittest.TestCase):
        module = py_heapq

    @unittest.skipUnless(c_heapq, 'requires the C _heapq module')
    class CExampleTest(ExampleTest, unittest.TestCase):
        module = c_heapq

    if __name__ == '__main__':
        unittest.main()

The test module defines a base class (ExampleTest) with test methods
that access the heapq module through a self.heapq class attribute, and
two subclasses that set this attribute to either the Python or the C
version of the module. Note that only the two subclasses inherit from
unittest.TestCase -- this prevents the ExampleTest class from being
detected as a TestCase subclass by unittest test discovery. A skipUnless
decorator can be added to the class that tests the C code in order to
have these tests skipped when the C module is not available.

If this test were to provide extensive coverage for heapq.heappop() in
the pure Python implementation then the accelerated C code would be
allowed to be added to CPython's standard library. If it did not, then
the test suite would need to be updated until proper coverage was
provided before the accelerated C code could be added.

To also help with compatibility, C code should use abstract APIs on
objects to prevent accidental dependence on specific types. For
instance, if a function accepts a sequence then the C code should
default to using PyObject_GetItem() instead of something like
PyList_GetItem(). C code is allowed to have a fast path if the proper
PyList_CheckExact() is used, but otherwise APIs should work with any
object that duck types to the proper interface instead of a specific
type.

Copyright

This document has been placed in the public domain.


text:
PEP: 201 Title: Lockstep Iteration Author: Barry Warsaw
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 13-Jul-2000 Python-Version: 2.0 Post-History:
27-Jul-2000

Introduction

This PEP describes the 'lockstep iteration' proposal. This PEP tracks
the status and ownership of this feature, slated for introduction in
Python 2.0. It contains a description of the feature and outlines
changes necessary to support the feature. This PEP summarizes
discussions held in mailing list forums, and provides URLs for further
information, where appropriate. The CVS revision history of this file
contains the definitive historical record.

Motivation

Standard for-loops in Python iterate over every element in a sequence
until the sequence is exhausted[1]. However, for-loops iterate over only
a single sequence, and it is often desirable to loop over more than one
sequence in a lock-step fashion. In other words, in a way such that the
i-th iteration through the loop returns an object containing the i-th
element from each sequence.

The common idioms used to accomplish this are unintuitive. This PEP
proposes a standard way of performing such iterations by introducing a
new builtin function called zip.

While the primary motivation for zip() comes from lock-step iteration,
by implementing zip() as a built-in function, it has additional utility
in contexts other than for-loops.

Lockstep For-Loops

Lockstep for-loops are non-nested iterations over two or more sequences,
such that at each pass through the loop, one element from each sequence
is taken to compose the target. This behavior can already be
accomplished in Python through the use of the map() built-in function:

    >>> a = (1, 2, 3)
    >>> b = (4, 5, 6)
    >>> for i in map(None, a, b): print i
    ...
    (1, 4)
    (2, 5)
    (3, 6)
    >>> map(None, a, b)
    [(1, 4), (2, 5), (3, 6)]

The for-loop simply iterates over this list as normal.

While the map() idiom is a common one in Python, it has several
disadvantages:

-   It is non-obvious to programmers without a functional programming
    background.

-   The use of the magic None first argument is non-obvious.

-   It has arbitrary, often unintended, and inflexible semantics when
    the lists are not of the same length: the shorter sequences are
    padded with None:

        >>> c = (4, 5, 6, 7)
        >>> map(None, a, c)
        [(1, 4), (2, 5), (3, 6), (None, 7)]

For these reasons, several proposals were floated in the Python 2.0 beta
time frame for syntactic support of lockstep for-loops. Here are two
suggestions:

    for x in seq1, y in seq2:
      # stuff

    for x, y in seq1, seq2:
      # stuff

Neither of these forms would work, since they both already mean
something in Python and changing the meanings would break existing code.
All other suggestions for new syntax suffered the same problem, or were
in conflict with other another proposed feature called 'list
comprehensions' (see PEP 202).

The Proposed Solution

The proposed solution is to introduce a new built-in sequence generator
function, available in the __builtin__ module. This function is to be
called zip and has the following signature:

    zip(seqa, [seqb, [...]])

zip() takes one or more sequences and weaves their elements together,
just as map(None, ...) does with sequences of equal length. The weaving
stops when the shortest sequence is exhausted.

Return Value

zip() returns a real Python list, the same way map() does.

Examples

Here are some examples, based on the reference implementation below:

    >>> a = (1, 2, 3, 4)
    >>> b = (5, 6, 7, 8)
    >>> c = (9, 10, 11)
    >>> d = (12, 13)

    >>> zip(a, b)
    [(1, 5), (2, 6), (3, 7), (4, 8)]

    >>> zip(a, d)
    [(1, 12), (2, 13)]

    >>> zip(a, b, c, d)
    [(1, 5, 9, 12), (2, 6, 10, 13)]

Note that when the sequences are of the same length, zip() is
reversible:

    >>> a = (1, 2, 3)
    >>> b = (4, 5, 6)
    >>> x = zip(a, b)
    >>> y = zip(*x) # alternatively, apply(zip, x)
    >>> z = zip(*y) # alternatively, apply(zip, y)
    >>> x
    [(1, 4), (2, 5), (3, 6)]
    >>> y
    [(1, 2, 3), (4, 5, 6)]
    >>> z
    [(1, 4), (2, 5), (3, 6)]
    >>> x == z
    1

It is not possible to reverse zip this way when the sequences are not
all the same length.

Reference Implementation

Here is a reference implementation, in Python of the zip() built-in
function. This will be replaced with a C implementation after final
approval:

    def zip(*args):
        if not args:
            raise TypeError('zip() expects one or more sequence arguments')
        ret = []
        i = 0
        try:
            while 1:
                item = []
                for s in args:
                    item.append(s[i])
                ret.append(tuple(item))
                i = i + 1
        except IndexError:
            return ret

BDFL Pronouncements

Note: the BDFL refers to Guido van Rossum, Python's Benevolent Dictator
For Life.

-   The function's name. An earlier version of this PEP included an open
    issue listing 20+ proposed alternative names to zip(). In the face
    of no overwhelmingly better choice, the BDFL strongly prefers zip()
    due to its Haskell[2] heritage. See version 1.7 of this PEP for the
    list of alternatives.
-   zip() shall be a built-in function.
-   Optional padding. An earlier version of this PEP proposed an
    optional pad keyword argument, which would be used when the argument
    sequences were not the same length. This is similar behavior to the
    map(None, ...) semantics except that the user would be able to
    specify pad object. This has been rejected by the BDFL in favor of
    always truncating to the shortest sequence, because of the KISS
    principle. If there's a true need, it is easier to add later. If it
    is not needed, it would still be impossible to delete it in the
    future.
-   Lazy evaluation. An earlier version of this PEP proposed that zip()
    return a built-in object that performed lazy evaluation using
    __getitem__() protocol. This has been strongly rejected by the BDFL
    in favor of returning a real Python list. If lazy evaluation is
    desired in the future, the BDFL suggests an xzip() function be
    added.
-   zip() with no arguments. the BDFL strongly prefers this raise a
    TypeError exception.
-   zip() with one argument. the BDFL strongly prefers that this return
    a list of 1-tuples.
-   Inner and outer container control. An earlier version of this PEP
    contains a rather lengthy discussion on a feature that some people
    wanted, namely the ability to control what the inner and outer
    container types were (they are tuples and list respectively in this
    version of the PEP). Given the simplified API and implementation,
    this elaboration is rejected. For a more detailed analysis, see
    version 1.7 of this PEP.

Subsequent Change to zip()

In Python 2.4, zip() with no arguments was modified to return an empty
list rather than raising a TypeError exception. The rationale for the
original behavior was that the absence of arguments was thought to
indicate a programming error. However, that thinking did not anticipate
the use of zip() with the * operator for unpacking variable length
argument lists. For example, the inverse of zip could be defined as:
unzip = lambda s: zip(*s). That transformation also defines a matrix
transpose or an equivalent row/column swap for tables defined as lists
of tuples. The latter transformation is commonly used when reading data
files with records as rows and fields as columns. For example, the code:

    date, rain, high, low = zip(*csv.reader(file("weather.csv")))

rearranges columnar data so that each field is collected into individual
tuples for straightforward looping and summarization:

    print "Total rainfall", sum(rain)

Using zip(*args) is more easily coded if zip(*[]) is handled as an
allowable case rather than an exception. This is especially helpful when
data is either built up from or recursed down to a null case with no
records.

Seeing this possibility, the BDFL agreed (with some misgivings) to have
the behavior changed for Py2.4.

Other Changes

-   The xzip() function discussed above was implemented in Py2.3 in the
    itertools module as itertools.izip(). This function provides lazy
    behavior, consuming single elements and producing a single tuple on
    each pass. The "just-in-time" style saves memory and runs faster
    than its list based counterpart, zip().

-   The itertools module also added itertools.repeat() and
    itertools.chain(). These tools can be used together to pad sequences
    with None (to match the behavior of map(None, seqn)):

        zip(firstseq, chain(secondseq, repeat(None)))

References

Greg Wilson's questionnaire on proposed syntax to some CS grad students
http://www.python.org/pipermail/python-dev/2000-July/013139.html

Copyright

This document has been placed in the public domain.

[1] http://docs.python.org/reference/compound_stmts.html#for

[2] http://www.haskell.org/onlinereport/standard-prelude.html#$vzip