text:
PEP: 296 Title: Adding a bytes Object Type Author: Scott Gilbert
<xscottg at yahoo.com> Status: Withdrawn Type: Standards Track
Content-Type: text/x-rst Created: 12-Jul-2002 Python-Version: 2.3
Post-History:

Notice

This PEP is withdrawn by the author (in favor of PEP 358).

Abstract

This PEP proposes the creation of a new standard type and builtin
constructor called 'bytes'. The bytes object is an efficiently stored
array of bytes with some additional characteristics that set it apart
from several implementations that are similar.

Rationale

Python currently has many objects that implement something akin to the
bytes object of this proposal. For instance the standard string, buffer,
array, and mmap objects are all very similar in some regards to the
bytes object. Additionally, several significant third party extensions
have created similar objects to try and fill similar needs.
Frustratingly, each of these objects is too narrow in scope and is
missing critical features to make it applicable to a wider category of
problems.

Specification

The bytes object has the following important characteristics:

1.  Efficient underlying array storage via the standard C type "unsigned
    char". This allows fine grain control over how much memory is
    allocated. With the alignment restrictions designated in the next
    item, it is trivial for low level extensions to cast the pointer to
    a different type as needed.

    Also, since the object is implemented as an array of bytes, it is
    possible to pass the bytes object to the extensive library of
    routines already in the standard library that presently work with
    strings. For instance, the bytes object in conjunction with the
    struct module could be used to provide a complete replacement for
    the array module using only Python script.

    If an unusual platform comes to light, one where there isn't a
    native unsigned 8 bit type, the object will do its best to represent
    itself at the Python script level as though it were an array of 8
    bit unsigned values. It is doubtful whether many extensions would
    handle this correctly, but Python script could be portable in these
    cases.

2.  Alignment of the allocated byte array is whatever is promised by the
    platform implementation of malloc. A bytes object created from an
    extension can be supplied that provides any arbitrary alignment as
    the extension author sees fit.

    This alignment restriction should allow the bytes object to be used
    as storage for all standard C types - including PyComplex objects or
    other structs of standard C type types. Further alignment
    restrictions can be provided by extensions as necessary.

3.  The bytes object implements a subset of the sequence operations
    provided by string/array objects, but with slightly different
    semantics in some cases. In particular, a slice always returns a new
    bytes object, but the underlying memory is shared between the two
    objects. This type of slice behavior has been called creating a
    "view". Additionally, repetition and concatenation are undefined for
    bytes objects and will raise an exception.

    As these objects are likely to find use in high performance
    applications, one motivation for the decision to use view slicing is
    that copying between bytes objects should be very efficient and not
    require the creation of temporary objects. The following code
    illustrates this:

        # create two 10 Meg bytes objects
        b1 = bytes(10000000)
        b2 = bytes(10000000)

        # copy from part of one to another with out creating a 1 Meg temporary
        b1[2000000:3000000] = b2[4000000:5000000]

    Slice assignment where the rvalue is not the same length as the
    lvalue will raise an exception. However, slice assignment will work
    correctly with overlapping slices (typically implemented with
    memmove).

4.  The bytes object will be recognized as a native type by the pickle
    and cPickle modules for efficient serialization. (In truth, this is
    the only requirement that can't be implemented via a third party
    extension.)

    Partial solutions to address the need to serialize the data stored
    in a bytes-like object without creating a temporary copy of the data
    into a string have been implemented in the past. The tofile and
    fromfile methods of the array object are good examples of this. The
    bytes object will support these methods too. However, pickling is
    useful in other situations - such as in the shelve module, or
    implementing RPC of Python objects, and requiring the end user to
    use two different serialization mechanisms to get an efficient
    transfer of data is undesirable.

    XXX: Will try to implement pickling of the new bytes object in such
    a way that previous versions of Python will unpickle it as a string
    object.

    When unpickling, the bytes object will be created from memory
    allocated from Python (via malloc). As such, it will lose any
    additional properties that an extension supplied pointer might have
    provided (special alignment, or special types of memory).

    XXX: Will try to make it so that C subclasses of bytes type can
    supply the memory that will be unpickled into. For instance, a
    derived class called PageAlignedBytes would unpickle to memory that
    is also page aligned.

    On any platform where an int is 32 bits (most of them), it is
    currently impossible to create a string with a length larger than
    can be represented in 31 bits. As such, pickling to a string will
    raise an exception when the operation is not possible.

    At least on platforms supporting large files (many of them),
    pickling large bytes objects to files should be possible via
    repeated calls to the file.write() method.

5.  The bytes type supports the PyBufferProcs interface, but a bytes
    object provides the additional guarantee that the pointer will not
    be deallocated or reallocated as long as a reference to the bytes
    object is held. This implies that a bytes object is not resizable
    once it is created, but allows the global interpreter lock (GIL) to
    be released while a separate thread manipulates the memory pointed
    to if the PyBytes_Check(...) test passes.

    This characteristic of the bytes object allows it to be used in
    situations such as asynchronous file I/O or on multiprocessor
    machines where the pointer obtained by PyBufferProcs will be used
    independently of the global interpreter lock.

    Knowing that the pointer can not be reallocated or freed after the
    GIL is released gives extension authors the capability to get true
    concurrency and make use of additional processors for long running
    computations on the pointer.

6.  In C/C++ extensions, the bytes object can be created from a supplied
    pointer and destructor function to free the memory when the
    reference count goes to zero.

    The special implementation of slicing for the bytes object allows
    multiple bytes objects to refer to the same pointer/destructor. As
    such, a refcount will be kept on the actual pointer/destructor. This
    refcount is separate from the refcount typically associated with
    Python objects.

    XXX: It may be desirable to expose the inner refcounted object as an
    actual Python object. If a good use case arises, it should be
    possible for this to be implemented later with no loss to backwards
    compatibility.

7.  It is also possible to signify the bytes object as readonly, in this
    case it isn't actually mutable, but does provide the other features
    of a bytes object.

8.  The bytes object keeps track of the length of its data with a Python
    LONG_LONG type. Even though the current definition for PyBufferProcs
    restricts the length to be the size of an int, this PEP does not
    propose to make any changes there. Instead, extensions can work
    around this limit by making an explicit PyBytes_Check(...) call, and
    if that succeeds they can make a PyBytes_GetReadBuffer(...) or
    PyBytes_GetWriteBuffer call to get the pointer and full length of
    the object as a LONG_LONG.

    The bytes object will raise an exception if the standard
    PyBufferProcs mechanism is used and the size of the bytes object is
    greater than can be represented by an integer.

    From Python scripting, the bytes object will be subscriptable with
    longs so the 32 bit int limit can be avoided.

    There is still a problem with the len() function as it is
    PyObject_Size() and this returns an int as well. As a workaround,
    the bytes object will provide a .length() method that will return a
    long.

9.  The bytes object can be constructed at the Python scripting level by
    passing an int/long to the bytes constructor with the number of
    bytes to allocate. For example:

        b = bytes(100000) # alloc 100K bytes

    The constructor can also take another bytes object. This will be
    useful for the implementation of unpickling, and in converting a
    read-write bytes object into a read-only one. An optional second
    argument will be used to designate creation of a readonly bytes
    object.

10. From the C API, the bytes object can be allocated using any of the
    following signatures:

        PyObject* PyBytes_FromLength(LONG_LONG len, int readonly);
        PyObject* PyBytes_FromPointer(void* ptr, LONG_LONG len, int readonly
                 void (*dest)(void *ptr, void *user), void* user);

    In the PyBytes_FromPointer(...) function, if the dest function
    pointer is passed in as NULL, it will not be called. This should
    only be used for creating bytes objects from statically allocated
    space.

    The user pointer has been called a closure in other places. It is a
    pointer that the user can use for whatever purposes. It will be
    passed to the destructor function on cleanup and can be useful for a
    number of things. If the user pointer is not needed, NULL should be
    passed instead.

11. The bytes type will be a new style class as that seems to be where
    all standard Python types are headed.

Contrast to existing types

The most common way to work around the lack of a bytes object has been
to simply use a string object in its place. Binary files, the
struct/array modules, and several other examples exist of this. Putting
aside the style issue that these uses typically have nothing to do with
text strings, there is the real problem that strings are not mutable, so
direct manipulation of the data returned in these cases is not possible.
Also, numerous optimizations in the string module (such as caching the
hash value or interning the pointers) mean that extension authors are on
very thin ice if they try to break the rules with the string object.

The buffer object seems like it was intended to address the purpose that
the bytes object is trying fulfill, but several shortcomings in its
implementation[1] have made it less useful in many common cases. The
buffer object made a different choice for its slicing behavior (it
returns new strings instead of buffers for slicing and other
operations), and it doesn't make many of the promises on alignment or
being able to release the GIL that the bytes object does.

Also in regards to the buffer object, it is not possible to simply
replace the buffer object with the bytes object and maintain backwards
compatibility. The buffer object provides a mechanism to take the
PyBufferProcs supplied pointer of another object and present it as its
own. Since the behavior of the other object can not be guaranteed to
follow the same set of strict rules that a bytes object does, it can't
be used in places that a bytes object could.

The array module supports the creation of an array of bytes, but it does
not provide a C API for supplying pointers and destructors to extension
supplied memory. This makes it unusable for constructing objects out of
shared memory, or memory that has special alignment or locking for
things like DMA transfers. Also, the array object does not currently
pickle. Finally since the array object allows its contents to grow, via
the extend method, the pointer can be changed if the GIL is not held
while using it.

Creating a buffer object from an array object has the same problem of
leaving an invalid pointer when the array object is resized.

The mmap object caters to its particular niche, but does not attempt to
solve a wider class of problems.

Finally, any third party extension can not implement pickling without
creating a temporary object of a standard Python type. For example, in
the Numeric community, it is unpleasant that a large array can't pickle
without creating a large binary string to duplicate the array data.

Backward Compatibility

The only possibility for backwards compatibility problems that the
author is aware of are in previous versions of Python that try to
unpickle data containing the new bytes type.

Reference Implementation

XXX: Actual implementation is in progress, but changes are still
possible as this PEP gets further review.

The following new files will be added to the Python baseline:

    Include/bytesobject.h  # C interface
    Objects/bytesobject.c  # C implementation
    Lib/test/test_bytes.py # unit testing
    Doc/lib/libbytes.tex   # documentation

The following files will also be modified:

    Include/Python.h       # adding bytesmodule.h include file
    Python/bltinmodule.c   # adding the bytes type object
    Modules/cPickle.c      # adding bytes to the standard types
    Lib/pickle.py          # adding bytes to the standard types

It is possible that several other modules could be cleaned up and
implemented in terms of the bytes object. The mmap module comes to mind
first, but as noted above it would be possible to reimplement the array
module as a pure Python module. While it is attractive that this PEP
could actually reduce the amount of source code by some amount, the
author feels that this could cause unnecessary risk for breaking
existing applications and should be avoided at this time.

Additional Notes/Comments

-   Guido van Rossum wondered whether it would make sense to be able to
    create a bytes object from a mmap object. The mmap object appears to
    support the requirements necessary to provide memory for a bytes
    object. (It doesn't resize, and the pointer is valid for the
    lifetime of the object.) As such, a method could be added to the
    mmap module such that a bytes object could be created directly from
    a mmap object. An initial stab at how this would be implemented
    would be to use the PyBytes_FromPointer() function described above
    and pass the mmap_object as the user pointer. The destructor
    function would decref the mmap_object for cleanup.

-   Todd Miller notes that it may be useful to have two new functions:
    PyObject_AsLargeReadBuffer() and PyObject_AsLargeWriteBuffer that
    are similar to PyObject_AsReadBuffer() and PyObject_AsWriteBuffer(),
    but support getting a LONG_LONG length in addition to the void*
    pointer. These functions would allow extension authors to work
    transparently with bytes object (that support LONG_LONG lengths) and
    most other buffer like objects (which only support int lengths).
    These functions could be in lieu of, or in addition to, creating a
    specific PyByte_GetReadBuffer() and PyBytes_GetWriteBuffer()
    functions.

    XXX: The author thinks this is very a good idea as it paves the way
    for other objects to eventually support large (64 bit) pointers, and
    it should only affect abstract.c and abstract.h. Should this be
    added above?

-   It was generally agreed that abusing the segment count of the
    PyBufferProcs interface is not a good hack to work around the 31 bit
    limitation of the length. If you don't know what this means, then
    you're in good company. Most code in the Python baseline, and
    presumably in many third party extensions, punt when the segment
    count is not 1.

References

Copyright

This document has been placed in the public domain.

[1] The buffer interface
https://mail.python.org/pipermail/python-dev/2000-October/009974.html


text:
PEP: 3135 Title: New Super Author: Calvin Spealman
<EMAIL_ADDRESS>Tim Delaney<EMAIL_ADDRESS>Lie
Ryan<EMAIL_ADDRESS>Status: Final Type: Standards Track
Content-Type: text/x-rst Created: 28-Apr-2007 Python-Version: 3.0
Post-History: 28-Apr-2007, 29-Apr-2007, 29-Apr-2007, 14-May-2007,
12-Mar-2009

Numbering Note

This PEP started its life as PEP 367. Since it is now targeted for
Python 3000, it has been moved into the 3xxx space.

Abstract

This PEP proposes syntactic sugar for use of the super type to
automatically construct instances of the super type binding to the class
that a method was defined in, and the instance (or class object for
classmethods) that the method is currently acting upon.

The premise of the new super usage suggested is as follows:

    super().foo(1, 2)

to replace the old:

    super(Foo, self).foo(1, 2)

Rationale

The current usage of super requires an explicit passing of both the
class and instance it must operate from, requiring a breaking of the DRY
(Don't Repeat Yourself) rule. This hinders any change in class name, and
is often considered a wart by many.

Specification

Within the specification section, some special terminology will be used
to distinguish similar and closely related concepts. "super class" will
refer to the actual builtin class named "super". A "super instance" is
simply an instance of the super class, which is associated with another
class and possibly with an instance of that class.

The new super semantics are only available in Python 3.0.

Replacing the old usage of super, calls to the next class in the MRO
(method resolution order) can be made without explicitly passing the
class object (although doing so will still be supported). Every function
will have a cell named __class__ that contains the class object that the
function is defined in.

The new syntax:

    super()

is equivalent to:

    super(__class__, <firstarg>)

where __class__ is the class that the method was defined in, and
<firstarg> is the first parameter of the method (normally self for
instance methods, and cls for class methods). For functions defined
outside a class body, __class__ is not defined, and will result in
runtime SystemError.

While super is not a reserved word, the parser recognizes the use of
super in a method definition and only passes in the __class__ cell when
this is found. Thus, calling a global alias of super without arguments
will not necessarily work.

Closed Issues

Determining the class object to use

The class object is taken from a cell named __class__.

Should super actually become a keyword?

No. It is not necessary for super to become a keyword.

super used with __call__ attributes

It was considered that it might be a problem that instantiating super
instances the classic way, because calling it would lookup the __call__
attribute and thus try to perform an automatic super lookup to the next
class in the MRO. However, this was found to be false, because calling
an object only looks up the __call__ method directly on the object's
type. The following example shows this in action.

    class A(object):
        def __call__(self):
            return '__call__'
        def __getattribute__(self, attr):
            if attr == '__call__':
                return lambda: '__getattribute__'
    a = A()
    assert a() == '__call__'
    assert a.__call__() == '__getattribute__'

In any case, this issue goes away entirely because classic calls to
super(<class>, <instance>) are still supported with the same meaning.

Alternative Proposals

No Changes

Although its always attractive to just keep things how they are, people
have sought a change in the usage of super calling for some time, and
for good reason, all mentioned previously.

-   Decoupling from the class name (which might not even be bound to the
    right class anymore!)
-   Simpler looking, cleaner super calls would be better

Dynamic attribute on super type

The proposal adds a dynamic attribute lookup to the super type, which
will automatically determine the proper class and instance parameters.
Each super attribute lookup identifies these parameters and performs the
super lookup on the instance, as the current super implementation does
with the explicit invocation of a super instance upon a class and
instance.

This proposal relies on sys._getframe(), which is not appropriate for
anything except a prototype implementation.

self.__super__.foo(*args)

The __super__ attribute is mentioned in this PEP in several places, and
could be a candidate for the complete solution, actually using it
explicitly instead of any super usage directly. However,
double-underscore names are usually an internal detail, and attempted to
be kept out of everyday code.

super(self, *args) or __super__(self, *args)

This solution only solves the problem of the type indication, does not
handle differently named super methods, and is explicit about the name
of the instance. It is less flexible without being able to enacted on
other method names, in cases where that is needed. One use case this
fails is where a base-class has a factory classmethod and a subclass has
two factory classmethods,both of which needing to properly make super
calls to the one in the base-class.

super.foo(self, *args)

This variation actually eliminates the problems with locating the proper
instance, and if any of the alternatives were pushed into the spotlight,
I would want it to be this one.

super(*p, **kw)

There has been the proposal that directly calling super(*p, **kw) would
be equivalent to calling the method on the super object with the same
name as the method currently being executed i.e. the following two
methods would be equivalent:

    def f(self, *p, **kw):
        super.f(*p, **kw)

    def f(self, *p, **kw):
        super(*p, **kw)

There is strong sentiment for and against this, but implementation and
style concerns are obvious. Guido has suggested that this should be
excluded from this PEP on the principle of KISS (Keep It Simple Stupid).

History

29-Apr-2007

    -   Changed title from "Super As A Keyword" to "New Super"
    -   Updated much of the language and added a terminology section for
        clarification in confusing places.
    -   Added reference implementation and history sections.

06-May-2007

    -   Updated by Tim Delaney to reflect discussions on the python-3000
        and python-dev mailing lists.

12-Mar-2009

    -   Updated to reflect the current state of implementation.

References

[1] Fixing super anyone?
(https://mail.python.org/pipermail/python-3000/2007-April/006667.html)

[2] PEP 3130: Access to Module/Class/Function Currently Being Defined
(this)
(https://mail.python.org/pipermail/python-ideas/2007-April/000542.html)

Copyright

This document has been placed in the public domain.

license: Public Domain


text:
PEP: 475 Title: Retry system calls failing with EINTR Version:
$Revision$ Last-Modified: $Date$ Author: Charles-François Natali
<EMAIL_ADDRESS>Victor Stinner<EMAIL_ADDRESS>BDFL-Delegate: Antoine Pitrou<EMAIL_ADDRESS>Status: Final Type:
Standards Track Content-Type: text/x-rst Created: 29-Jul-2014
Python-Version: 3.5 Resolution:
https://mail.python.org/pipermail/python-dev/2015-February/138018.html

Abstract

System call wrappers provided in the standard library should be retried
automatically when they fail with EINTR, to relieve application code
from the burden of doing so.

By system calls, we mean the functions exposed by the standard C library
pertaining to I/O or handling of other system resources.

Rationale

Interrupted system calls

On POSIX systems, signals are common. Code calling system calls must be
prepared to handle them. Examples of signals:

-   The most common signal is SIGINT, the signal sent when CTRL+c is
    pressed. By default, Python raises a KeyboardInterrupt exception
    when this signal is received.
-   When running subprocesses, the SIGCHLD signal is sent when a child
    process exits.
-   Resizing the terminal sends the SIGWINCH signal to the applications
    running in the terminal.
-   Putting the application in background (ex: press CTRL-z and then
    type the bg command) sends the SIGCONT signal.

Writing a C signal handler is difficult: only "async-signal-safe"
functions can be called (for example, printf() and malloc() are not
async-signal safe), and there are issues with reentrancy. Therefore,
when a signal is received by a process during the execution of a system
call, the system call can fail with the EINTR error to give the program
an opportunity to handle the signal without the restriction on
signal-safe functions.

This behaviour is system-dependent: on certain systems, using the
SA_RESTART flag, some system calls are retried automatically instead of
failing with EINTR. Regardless, Python's signal.signal() function clears
the SA_RESTART flag when setting the signal handler: all system calls
will probably fail with EINTR in Python.

Since receiving a signal is a non-exceptional occurrence, robust POSIX
code must be prepared to handle EINTR (which, in most cases, means retry
in a loop in the hope that the call eventually succeeds). Without
special support from Python, this can make application code much more
verbose than it needs to be.

Status in Python 3.4

In Python 3.4, handling the InterruptedError exception (EINTR's
dedicated exception class) is duplicated at every call site on a
case-by-case basis. Only a few Python modules actually handle this
exception, and fixes usually took several years to cover a whole module.
Example of code retrying file.read() on InterruptedError:

    while True:
        try:
            data = file.read(size)
            break
        except InterruptedError:
            continue

List of Python modules in the standard library which handle
InterruptedError:

-   asyncio
-   asyncore
-   io, _pyio
-   multiprocessing
-   selectors
-   socket
-   socketserver
-   subprocess

Other programming languages like Perl, Java and Go retry system calls
failing with EINTR at a lower level, so that libraries and applications
needn't bother.

Use Case 1: Don't Bother With Signals

In most cases, you don't want to be interrupted by signals and you don't
expect to get InterruptedError exceptions. For example, do you really
want to write such complex code for a "Hello World" example?

    while True:
        try:
            print("Hello World")
            break
        except InterruptedError:
            continue

InterruptedError can happen in unexpected places. For example,
os.close() and FileIO.close() may raise InterruptedError: see the
article close() and EINTR.

The Python issues related to EINTR section below gives examples of bugs
caused by EINTR.

The expectation in this use case is that Python hides the
InterruptedError and retries system calls automatically.

Use Case 2: Be notified of signals as soon as possible

Sometimes yet, you expect some signals and you want to handle them as
soon as possible. For example, you may want to immediately quit a
program using the CTRL+c keyboard shortcut.

Besides, some signals are not interesting and should not disrupt the
application. There are two options to interrupt an application on only
some signals:

-   Set up a custom signal handler which raises an exception, such as
    KeyboardInterrupt for SIGINT.
-   Use a I/O multiplexing function like select() together with Python's
    signal wakeup file descriptor: see the function
    signal.set_wakeup_fd().

The expectation in this use case is for the Python signal handler to be
executed timely, and the system call to fail if the handler raised an
exception -- otherwise restart.

Proposal

This PEP proposes to handle EINTR and retries at the lowest level, i.e.
in the wrappers provided by the stdlib (as opposed to higher-level
libraries and applications).

Specifically, when a system call fails with EINTR, its Python wrapper
must call the given signal handler (using PyErr_CheckSignals()). If the
signal handler raises an exception, the Python wrapper bails out and
fails with the exception.

If the signal handler returns successfully, the Python wrapper retries
the system call automatically. If the system call involves a timeout
parameter, the timeout is recomputed.

Modified functions

Example of standard library functions that need to be modified to comply
with this PEP:

-   open(), os.open(), io.open()
-   functions of the faulthandler module
-   os functions:
    -   os.fchdir()
    -   os.fchmod()
    -   os.fchown()
    -   os.fdatasync()
    -   os.fstat()
    -   os.fstatvfs()
    -   os.fsync()
    -   os.ftruncate()
    -   os.mkfifo()
    -   os.mknod()
    -   os.posix_fadvise()
    -   os.posix_fallocate()
    -   os.pread()
    -   os.pwrite()
    -   os.read()
    -   os.readv()
    -   os.sendfile()
    -   os.wait3()
    -   os.wait4()
    -   os.wait()
    -   os.waitid()
    -   os.waitpid()
    -   os.write()
    -   os.writev()
    -   special cases: os.close() and os.dup2() now ignore EINTR error,
        the syscall is not retried
-   select.select(), select.poll.poll(), select.epoll.poll(),
    select.kqueue.control(), select.devpoll.poll()
-   socket.socket() methods:
    -   accept()
    -   connect() (except for non-blocking sockets)
    -   recv()
    -   recvfrom()
    -   recvmsg()
    -   send()
    -   sendall()
    -   sendmsg()
    -   sendto()
-   signal.sigtimedwait(), signal.sigwaitinfo()
-   time.sleep()

(Note: the selector module already retries on InterruptedError, but it
doesn't recompute the timeout yet)

os.close, close() methods and os.dup2() are a special case: they will
ignore EINTR instead of retrying. The reason is complex but involves
behaviour under Linux and the fact that the file descriptor may really
be closed even if EINTR is returned. See articles:

-   Returning EINTR from close()
-   (LKML) Re: [patch 7/7] uml: retry host close() on EINTR
-   close() and EINTR

The socket.socket.connect() method does not retry connect() for
non-blocking sockets if it is interrupted by a signal (fails with
EINTR). The connection runs asynchronously in background. The caller is
responsible to wait until the socket becomes writable (ex: using
select.select()) and then call
socket.socket.getsockopt(socket.SOL_SOCKET, socket.SO_ERROR) to check if
the connection succeeded (getsockopt() returns 0) or failed.

InterruptedError handling

Since interrupted system calls are automatically retried, the
InterruptedError exception should not occur anymore when calling those
system calls. Therefore, manual handling of InterruptedError as
described in Status in Python 3.4 can be removed, which will simplify
standard library code.

Backward compatibility

Applications relying on the fact that system calls are interrupted with
InterruptedError will hang. The authors of this PEP don't think that
such applications exist, since they would be exposed to other issues
such as race conditions (there is an opportunity for deadlock if the
signal comes before the system call). Besides, such code would be
non-portable.

In any case, those applications must be fixed to handle signals
differently, to have a reliable behaviour on all platforms and all
Python versions. A possible strategy is to set up a signal handler
raising a well-defined exception, or use a wakeup file descriptor.

For applications using event loops, signal.set_wakeup_fd() is the
recommended option to handle signals. Python's low-level signal handler
will write signal numbers into the file descriptor and the event loop
will be awaken to read them. The event loop can handle those signals
without the restriction of signal handlers (for example, the loop can be
woken up in any thread, not just the main thread).

Appendix

Wakeup file descriptor

Since Python 3.3, signal.set_wakeup_fd() writes the signal number into
the file descriptor, whereas it only wrote a null byte before. It
becomes possible to distinguish between signals using the wakeup file
descriptor.

Linux has a signalfd() system call which provides more information on
each signal. For example, it's possible to know the pid and uid who sent
the signal. This function is not exposed in Python yet (see issue
12304).

On Unix, the asyncio module uses the wakeup file descriptor to wake up
its event loop.

Multithreading

A C signal handler can be called from any thread, but Python signal
handlers will always be called in the main Python thread.

Python's C API provides the PyErr_SetInterrupt() function which calls
the SIGINT signal handler in order to interrupt the main Python thread.

Signals on Windows

Control events

Windows uses "control events":

-   CTRL_BREAK_EVENT: Break (SIGBREAK)
-   CTRL_CLOSE_EVENT: Close event
-   CTRL_C_EVENT: CTRL+C (SIGINT)
-   CTRL_LOGOFF_EVENT: Logoff
-   CTRL_SHUTDOWN_EVENT: Shutdown

The SetConsoleCtrlHandler() function can be used to install a control
handler.

The CTRL_C_EVENT and CTRL_BREAK_EVENT events can be sent to a process
using the GenerateConsoleCtrlEvent() function. This function is exposed
in Python as os.kill().

Signals

The following signals are supported on Windows:

-   SIGABRT
-   SIGBREAK (CTRL_BREAK_EVENT): signal only available on Windows
-   SIGFPE
-   SIGILL
-   SIGINT (CTRL_C_EVENT)
-   SIGSEGV
-   SIGTERM

SIGINT

The default Python signal handler for SIGINT sets a Windows event
object: sigint_event.

time.sleep() is implemented with WaitForSingleObjectEx(), it waits for
the sigint_event object using time.sleep() parameter as the timeout. So
the sleep can be interrupted by SIGINT.

_winapi.WaitForMultipleObjects() automatically adds sigint_event to the
list of watched handles, so it can also be interrupted.

PyOS_StdioReadline() also used sigint_event when fgets() failed to check
if Ctrl-C or Ctrl-Z was pressed.

Links

Misc

-   glibc manual: Primitives Interrupted by Signals
-   Bug #119097 for perl5: print returning EINTR in 5.14.

Python issues related to EINTR

The main issue is: handle EINTR in the stdlib.

Open issues:

-   Add a new signal.set_wakeup_socket() function
-   signal.set_wakeup_fd(fd): set the fd to non-blocking mode
-   Use a monotonic clock to compute timeouts
-   sys.stdout.write on OS X is not EINTR safe
-   platform.uname() not EINTR safe
-   asyncore does not handle EINTR in recv, send, connect, accept,
-   socket.create_connection() doesn't handle EINTR properly

Closed issues:

-   Interrupted system calls are not retried
-   Solaris: EINTR exception in select/socket calls in telnetlib
-   subprocess: Popen.communicate() doesn't handle EINTR in some cases
-   multiprocessing.util._eintr_retry doesn't recalculate timeouts
-   file readline, readlines & readall methods can lose data on EINTR
-   multiprocessing BaseManager serve_client() does not check EINTR on
    recv
-   selectors behaviour on EINTR undocumented
-   asyncio: limit EINTR occurrences with SA_RESTART
-   smtplib.py socket.create_connection() also doesn't handle EINTR
    properly
-   Faulty RESTART/EINTR handling in Parser/myreadline.c
-   test_httpservers intermittent failure, test_post and EINTR
-   os.spawnv(P_WAIT, ...) on Linux doesn't handle EINTR
-   asyncore fails when EINTR happens in pol
-   file.write and file.read don't handle EINTR
-   socket.readline() interface doesn't handle EINTR properly
-   subprocess is not EINTR-safe
-   SocketServer doesn't handle syscall interruption
-   subprocess deadlock when read() is interrupted
-   time.sleep(1): call PyErr_CheckSignals() if the sleep was
    interrupted
-   siginterrupt with flag=False is reset when signal received
-   need siginterrupt() on Linux - impossible to do timeouts
-   [Windows] Can not interrupt time.sleep()

Python issues related to signals

Open issues:

-   signal.default_int_handler should set signal number on the raised
    exception
-   expose signalfd(2) in the signal module
-   missing return in win32_kill?
-   Interrupts are lost during readline PyOS_InputHook processing
-   cannot catch KeyboardInterrupt when using curses getkey()
-   Deferred KeyboardInterrupt in interactive mode

Closed issues:

-   sys.interrupt_main()

Implementation

The implementation is tracked in issue 23285. It was committed on
February 07, 2015.

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

authors:
Charles-François Natali
Victor Stinner


text:
PEP: 209 Title: Multi-dimensional Arrays Author: Paul Barrett
<EMAIL_ADDRESS>Travis Oliphant<EMAIL_ADDRESS>Status:
Withdrawn Type: Standards Track Created: 03-Jan-2001 Python-Version: 2.2
Post-History:

Abstract

This PEP proposes a redesign and re-implementation of the
multi-dimensional array module, Numeric, to make it easier to add new
features and functionality to the module. Aspects of Numeric 2 that will
receive special attention are efficient access to arrays exceeding a
gigabyte in size and composed of inhomogeneous data structures or
records. The proposed design uses four Python classes: ArrayType, UFunc,
Array, and ArrayView; and a low-level C-extension module, _ufunc, to
handle the array operations efficiently. In addition, each array type
has its own C-extension module which defines the coercion rules,
operations, and methods for that type. This design enables new types,
features, and functionality to be added in a modular fashion. The new
version will introduce some incompatibilities with the current Numeric.

Motivation

Multi-dimensional arrays are commonly used to store and manipulate data
in science, engineering, and computing. Python currently has an
extension module, named Numeric (henceforth called Numeric 1), which
provides a satisfactory set of functionality for users manipulating
homogeneous arrays of data of moderate size (of order 10 MB). For access
to larger arrays (of order 100 MB or more) of possibly inhomogeneous
data, the implementation of Numeric 1 is inefficient and cumbersome. In
the future, requests by the Numerical Python community for additional
functionality is also likely as PEPs 211: Adding New Linear Operators to
Python, and 225: Elementwise/Objectwise Operators illustrate.

Proposal

This proposal recommends a re-design and re-implementation of Numeric 1,
henceforth called Numeric 2, which will enable new types, features, and
functionality to be added in an easy and modular manner. The initial
design of Numeric 2 should focus on providing a generic framework for
manipulating arrays of various types and should enable a straightforward
mechanism for adding new array types and UFuncs. Functional methods that
are more specific to various disciplines can then be layered on top of
this core. This new module will still be called Numeric and most of the
behavior found in Numeric 1 will be preserved.

The proposed design uses four Python classes: ArrayType, UFunc, Array,
and ArrayView; and a low-level C-extension module to handle the array
operations efficiently. In addition, each array type has its own
C-extension module which defines the coercion rules, operations, and
methods for that type. At a later date, when core functionality is
stable, some Python classes can be converted to C-extension types.

Some planned features are:

1.  Improved memory usage

    This feature is particularly important when handling large arrays
    and can produce significant improvements in performance as well as
    memory usage. We have identified several areas where memory usage
    can be improved:

    a.  Use a local coercion model

        Instead of using Python's global coercion model which creates
        temporary arrays, Numeric 2, like Numeric 1, will implement a
        local coercion model as described in PEP 208 which defers the
        responsibility of coercion to the operator. By using internal
        buffers, a coercion operation can be done for each array
        (including output arrays), if necessary, at the time of the
        operation. Benchmarks[1] have shown that performance is at most
        degraded only slightly and is improved in cases where the
        internal buffers are less than the L2 cache size and the
        processor is under load. To avoid array coercion altogether, C
        functions having arguments of mixed type are allowed in Numeric
        2.

    b.  Avoid creation of temporary arrays

        In complex array expressions (i.e. having more than one
        operation), each operation will create a temporary array which
        will be used and then deleted by the succeeding operation. A
        better approach would be to identify these temporary arrays and
        reuse their data buffers when possible, namely when the array
        shape and type are the same as the temporary array being
        created. This can be done by checking the temporary array's
        reference count. If it is 1, then it will be deleted once the
        operation is done and is a candidate for reuse.

    c.  Optional use of memory-mapped files

        Numeric users sometimes need to access data from very large
        files or to handle data that is greater than the available
        memory. Memory-mapped arrays provide a mechanism to do this by
        storing the data on disk while making it appear to be in memory.
        Memory- mapped arrays should improve access to all files by
        eliminating one of two copy steps during a file access. Numeric
        should be able to access in-memory and memory-mapped arrays
        transparently.

    d.  Record access

        In some fields of science, data is stored in files as binary
        records. For example, in astronomy, photon data is stored as a 1
        dimensional list of photons in order of arrival time. These
        records or C-like structures contain information about the
        detected photon, such as its arrival time, its position on the
        detector, and its energy. Each field may be of a different type,
        such as char, int, or float. Such arrays introduce new issues
        that must be dealt with, in particular byte alignment or byte
        swapping may need to be performed for the numeric values to be
        properly accessed (though byte swapping is also an issue for
        memory mapped data). Numeric 2 is designed to automatically
        handle alignment and representational issues when data is
        accessed or operated on. There are two approaches to
        implementing records; as either a derived array class or a
        special array type, depending on your point-of-view. We defer
        this discussion to the Open Issues section.

2.  Additional array types

    Numeric 1 has 11 defined types: char, ubyte, sbyte, short, int,
    long, float, double, cfloat, cdouble, and object. There are no
    ushort, uint, or ulong types, nor are there more complex types such
    as a bit type which is of use to some fields of science and possibly
    for implementing masked-arrays. The design of Numeric 1 makes the
    addition of these and other types a difficult and error-prone
    process. To enable the easy addition (and deletion) of new array
    types such as a bit type described below, a re-design of Numeric is
    necessary.

    a.  Bit type

        The result of a rich comparison between arrays is an array of
        boolean values. The result can be stored in an array of type
        char, but this is an unnecessary waste of memory. A better
        implementation would use a bit or boolean type, compressing the
        array size by a factor of eight. This is currently being
        implemented for Numeric 1 (by Travis Oliphant) and should be
        included in Numeric 2.

3.  Enhanced array indexing syntax

    The extended slicing syntax was added to Python to provide greater
    flexibility when manipulating Numeric arrays by allowing step-sizes
    greater than 1. This syntax works well as a shorthand for a list of
    regularly spaced indices. For those situations where a list of
    irregularly spaced indices are needed, an enhanced array indexing
    syntax would allow 1-D arrays to be arguments.

4.  Rich comparisons

    The implementation of PEP 207: Rich Comparisons in Python 2.1
    provides additional flexibility when manipulating arrays. We intend
    to implement this feature in Numeric 2.

5.  Array broadcasting rules

    When an operation between a scalar and an array is done, the implied
    behavior is to create a new array having the same shape as the array
    operand containing the scalar value. This is called array
    broadcasting. It also works with arrays of lesser rank, such as
    vectors. This implicit behavior is implemented in Numeric 1 and will
    also be implemented in Numeric 2.

Design and Implementation

The design of Numeric 2 has four primary classes:

1.  ArrayType:

    This is a simple class that describes the fundamental properties of
    an array-type, e.g. its name, its size in bytes, its coercion
    relations with respect to other types, etc., e.g.

        Int32 = ArrayType('Int32', 4, 'doc-string')

    Its relation to the other types is defined when the C-extension
    module for that type is imported. The corresponding Python code is:

        Int32.astype[Real64] = Real64

    This says that the Real64 array-type has higher priority than the
    Int32 array-type.

    The following attributes and methods are proposed for the core
    implementation. Additional attributes can be added on an individual
    basis, e.g. .bitsize or .bitstrides for the bit type.

    Attributes:

        .name:                  e.g. "Int32", "Float64", etc.
        .typecode:              e.g. 'i', 'f', etc.
                                (for backward compatibility)
        .size (in bytes):       e.g. 4, 8, etc.
        .array_rules (mapping): rules between array types
        .pyobj_rules (mapping): rules between array and python types
        .doc:                   documentation string

    Methods:

        __init__():             initialization
        __del__():              destruction
        __repr__():             representation

    C-API: This still needs to be fleshed-out.

2.  UFunc:

    This class is the heart of Numeric 2. Its design is similar to that
    of ArrayType in that the UFunc creates a singleton callable object
    whose attributes are name, total and input number of arguments, a
    document string, and an empty CFunc dictionary; e.g.

        add = UFunc('add', 3, 2, 'doc-string')

    When defined the add instance has no C functions associated with it
    and therefore can do no work. The CFunc dictionary is populated or
    registered later when the C-extension module for an array-type is
    imported. The arguments of the register method are: function name,
    function descriptor, and the CUFunc object. The corresponding Python
    code is

        add.register('add', (Int32, Int32, Int32), cfunc-add)

    In the initialization function of an array type module, e.g. Int32,
    there are two C API functions: one to initialize the coercion rules
    and the other to register the CFunc objects.

    When an operation is applied to some arrays, the __call__ method is
    invoked. It gets the type of each array (if the output array is not
    given, it is created from the coercion rules) and checks the CFunc
    dictionary for a key that matches the argument types. If it exists
    the operation is performed immediately, otherwise the coercion rules
    are used to search for a related operation and set of conversion
    functions. The __call__ method then invokes a compute method written
    in C to iterate over slices of each array, namely:

        _ufunc.compute(slice, data, func, swap, conv)

    The 'func' argument is a CFuncObject, while the 'swap' and 'conv'
    arguments are lists of CFuncObjects for those arrays needing pre- or
    post-processing, otherwise None is used. The data argument is a list
    of buffer objects, and the slice argument gives the number of
    iterations for each dimension along with the buffer offset and step
    size for each array and each dimension.

    We have predefined several UFuncs for use by the __call__ method:
    cast, swap, getobj, and setobj. The cast and swap functions do
    coercion and byte-swapping, respectively and the getobj and setobj
    functions do coercion between Numeric arrays and Python sequences.

    The following attributes and methods are proposed for the core
    implementation.

    Attributes:

        .name:                  e.g. "add", "subtract", etc.
        .nargs:                 number of total arguments
        .iargs:                 number of input arguments
        .cfuncs (mapping):      the set C functions
        .doc:                   documentation string

    Methods:

        __init__():             initialization
        __del__():              destruction
        __repr__():             representation
        __call__():             look-up and dispatch method
        initrule():             initialize coercion rule
        uninitrule():           uninitialize coercion rule
        register():             register a CUFunc
        unregister():           unregister a CUFunc

    C-API: This still needs to be fleshed-out.

3.  Array:

    This class contains information about the array, such as shape,
    type, endian-ness of the data, etc.. Its operators, '+', '-', etc.
    just invoke the corresponding UFunc function, e.g.

        def __add__(self, other):
            return ufunc.add(self, other)

    The following attributes, methods, and functions are proposed for
    the core implementation.

    Attributes:

        .shape:                 shape of the array
        .format:                type of the array
        .real (only complex):   real part of a complex array
        .imag (only complex):   imaginary part of a complex array

    Methods:

        __init__():             initialization
        __del__():              destruction
        __repr_():              representation
        __str__():              pretty representation
        __cmp__():              rich comparison
        __len__():
        __getitem__():
        __setitem__():
        __getslice__():
        __setslice__():
        numeric methods:
        copy():                 copy of array
        aslist():               create list from array
        asstring():             create string from array

    Functions:

        fromlist():             create array from sequence
        fromstring():           create array from string
        array():                create array with shape and value
        concat():               concatenate two arrays
        resize():               resize array

    C-API: This still needs to be fleshed-out.

4.  ArrayView

    This class is similar to the Array class except that the reshape and
    flat methods will raise exceptions, since non-contiguous arrays
    cannot be reshaped or flattened using just pointer and step-size
    information.

    C-API: This still needs to be fleshed-out.

5.  C-extension modules:

    Numeric2 will have several C-extension modules.

    a.  _ufunc:

        The primary module of this set is the _ufuncmodule.c. The
        intention of this module is to do the bare minimum, i.e. iterate
        over arrays using a specified C function. The interface of these
        functions is the same as Numeric 1, i.e.

            int (*CFunc)(char *data, int *steps, int repeat, void *func);

        and their functionality is expected to be the same, i.e. they
        iterate over the inner-most dimension.

        The following attributes and methods are proposed for the core
        implementation.

        Attributes:

        Methods:

            compute():

        C-API: This still needs to be fleshed-out.

    b.  _int32, _real64, etc.:

        There will also be C-extension modules for each array type, e.g.
        _int32module.c, _real64module.c, etc. As mentioned previously,
        when these modules are imported by the UFunc module, they will
        automatically register their functions and coercion rules. New
        or improved versions of these modules can be easily implemented
        and used without affecting the rest of Numeric 2.

Open Issues

1.  Does slicing syntax default to copy or view behavior?

    The default behavior of Python is to return a copy of a sub-list or
    tuple when slicing syntax is used, whereas Numeric 1 returns a view
    into the array. The choice made for Numeric 1 is apparently for
    reasons of performance: the developers wish to avoid the penalty of
    allocating and copying the data buffer during each array operation
    and feel that the need for a deep copy of an array to be rare. Yet,
    some have argued that Numeric's slice notation should also have copy
    behavior to be consistent with Python lists. In this case the
    performance penalty associated with copy behavior can be minimized
    by implementing copy-on-write. This scheme has both arrays sharing
    one data buffer (as in view behavior) until either array is assigned
    new data at which point a copy of the data buffer is made. View
    behavior would then be implemented by an ArrayView class, whose
    behavior be similar to Numeric 1 arrays, i.e. .shape is not settable
    for non-contiguous arrays. The use of an ArrayView class also makes
    explicit what type of data the array contains.

2.  Does item syntax default to copy or view behavior?

    A similar question arises with the item syntax. For example, if
    a = [[0,1,2], [3,4,5]] and b = a[0], then changing b[0] also changes
    a[0][0], because a[0] is a reference or view of the first row of a.
    Therefore, if c is a 2-d array, it would appear that c[i] should
    return a 1-d array which is a view into, instead of a copy of, c for
    consistency. Yet, c[i] can be considered just a shorthand for c[i,:]
    which would imply copy behavior assuming slicing syntax returns a
    copy. Should Numeric 2 behave the same way as lists and return a
    view or should it return a copy.

3.  How is scalar coercion implemented?

    Python has fewer numeric types than Numeric which can cause coercion
    problems. For example, when multiplying a Python scalar of type
    float and a Numeric array of type float, the Numeric array is
    converted to a double, since the Python float type is actually a
    double. This is often not the desired behavior, since the Numeric
    array will be doubled in size which is likely to be annoying,
    particularly for very large arrays. We prefer that the array type
    trumps the python type for the same type class, namely integer,
    float, and complex. Therefore, an operation between a Python integer
    and an Int16 (short) array will return an Int16 array. Whereas an
    operation between a Python float and an Int16 array would return a
    Float64 (double) array. Operations between two arrays use normal
    coercion rules.

4.  How is integer division handled?

    In a future version of Python, the behavior of integer division will
    change. The operands will be converted to floats, so the result will
    be a float. If we implement the proposed scalar coercion rules where
    arrays have precedence over Python scalars, then dividing an array
    by an integer will return an integer array and will not be
    consistent with a future version of Python which would return an
    array of type double. Scientific programmers are familiar with the
    distinction between integer and float-point division, so should
    Numeric 2 continue with this behavior?

5.  How should records be implemented?

    There are two approaches to implementing records depending on your
    point-of-view. The first is two divide arrays into separate classes
    depending on the behavior of their types. For example, numeric
    arrays are one class, strings a second, and records a third, because
    the range and type of operations of each class differ. As such, a
    record array is not a new type, but a mechanism for a more flexible
    form of array. To easily access and manipulate such complex data,
    the class is comprised of numeric arrays having different byte
    offsets into the data buffer. For example, one might have a table
    consisting of an array of Int16, Real32 values. Two numeric arrays,
    one with an offset of 0 bytes and a stride of 6 bytes to be
    interpreted as Int16, and one with an offset of 2 bytes and a stride
    of 6 bytes to be interpreted as Real32 would represent the record
    array. Both numeric arrays would refer to the same data buffer, but
    have different offset and stride attributes, and a different numeric
    type.

    The second approach is to consider a record as one of many array
    types, albeit with fewer, and possibly different, array operations
    than for numeric arrays. This approach considers an array type to be
    a mapping of a fixed-length string. The mapping can either be
    simple, like integer and floating-point numbers, or complex, like a
    complex number, a byte string, and a C-structure. The record type
    effectively merges the struct and Numeric modules into a
    multi-dimensional struct array. This approach implies certain
    changes to the array interface. For example, the 'typecode' keyword
    argument should probably be changed to the more descriptive 'format'
    keyword.

    a.  How are record semantics defined and implemented?

        Which ever implementation approach is taken for records, the
        syntax and semantics of how they are to be accessed and
        manipulated must be decided, if one wishes to have access to
        sub-fields of records. In this case, the record type can
        essentially be considered an inhomogeneous list, like a tuple
        returned by the unpack method of the struct module; and a 1-d
        array of records may be interpreted as a 2-d array with the
        second dimension being the index into the list of fields. This
        enhanced array semantics makes access to an array of one or more
        of the fields easy and straightforward. It also allows a user to
        do array operations on a field in a natural and intuitive way.
        If we assume that records are implemented as an array type, then
        last dimension defaults to 0 and can therefore be neglected for
        arrays comprised of simple types, like numeric.

6.  How are masked-arrays implemented?

    Masked-arrays in Numeric 1 are implemented as a separate array
    class. With the ability to add new array types to Numeric 2, it is
    possible that masked-arrays in Numeric 2 could be implemented as a
    new array type instead of an array class.

7.  How are numerical errors handled (IEEE floating-point errors in
    particular)?

    It is not clear to the proposers (Paul Barrett and Travis Oliphant)
    what is the best or preferred way of handling errors. Since most of
    the C functions that do the operation, iterate over the inner-most
    (last) dimension of the array. This dimension could contain a
    thousand or more items having one or more errors of differing type,
    such as divide-by-zero, underflow, and overflow. Additionally,
    keeping track of these errors may come at the expense of
    performance. Therefore, we suggest several options:

    a.  Print a message of the most severe error, leaving it to the user
        to locate the errors.
    b.  Print a message of all errors that occurred and the number of
        occurrences, leaving it to the user to locate the errors.
    c.  Print a message of all errors that occurred and a list of where
        they occurred.
    d.  Or use a hybrid approach, printing only the most severe error,
        yet keeping track of what and where the errors occurred. This
        would allow the user to locate the errors while keeping the
        error message brief.

8.  What features are needed to ease the integration of FORTRAN
    libraries and code?

It would be a good idea at this stage to consider how to ease the
integration of FORTRAN libraries and user code in Numeric 2.

Implementation Steps

1.  Implement basic UFunc capability
    a.  Minimal Array class:

        Necessary class attributes and methods, e.g. .shape, .data,
        .type, etc.

    b.  Minimal ArrayType class:

        Int32, Real64, Complex64, Char, Object

    c.  Minimal UFunc class:

        UFunc instantiation, CFunction registration, UFunc call for 1-D
        arrays including the rules for doing alignment, byte-swapping,
        and coercion.

    d.  Minimal C-extension module:

        _UFunc, which does the innermost array loop in C.

        This step implements whatever is needed to do: 'c = add(a, b)'
        where a, b, and c are 1-D arrays. It teaches us how to add new
        UFuncs, to coerce the arrays, to pass the necessary information
        to a C iterator method and to do the actually computation.
2.  Continue enhancing the UFunc iterator and Array class
    a.  Implement some access methods for the Array class: print, repr,
        getitem, setitem, etc.
    b.  Implement multidimensional arrays
    c.  Implement some of basic Array methods using UFuncs: +, -, *, /,
        etc.
    d.  Enable UFuncs to use Python sequences.
3.  Complete the standard UFunc and Array class behavior
    a.  Implement getslice and setslice behavior
    b.  Work on Array broadcasting rules
    c.  Implement Record type
4.  Add additional functionality
    a.  Add more UFuncs
    b.  Implement buffer or mmap access

Incompatibilities

The following is a list of incompatibilities in behavior between Numeric
1 and Numeric 2.

1.  Scalar coercion rules

    Numeric 1 has single set of coercion rules for array and Python
    numeric types. This can cause unexpected and annoying problems
    during the calculation of an array expression. Numeric 2 intends to
    overcome these problems by having two sets of coercion rules: one
    for arrays and Python numeric types, and another just for arrays.

2.  No savespace attribute

    The savespace attribute in Numeric 1 makes arrays with this
    attribute set take precedence over those that do not have it set.
    Numeric 2 will not have such an attribute and therefore normal array
    coercion rules will be in effect.

3.  Slicing syntax returns a copy

    The slicing syntax in Numeric 1 returns a view into the original
    array. The slicing behavior for Numeric 2 will be a copy. You should
    use the ArrayView class to get a view into an array.

4.  Boolean comparisons return a boolean array

    A comparison between arrays in Numeric 1 results in a Boolean
    scalar, because of current limitations in Python. The advent of Rich
    Comparisons in Python 2.1 will allow an array of Booleans to be
    returned.

5.  Type characters are deprecated

    Numeric 2 will have an ArrayType class composed of Type instances,
    for example Int8, Int16, Int32, and Int for signed integers. The
    typecode scheme in Numeric 1 will be available for backward
    compatibility, but will be deprecated.

Appendices

A.  Implicit sub-arrays iteration

    A computer animation is composed of a number of 2-D images or frames
    of identical shape. By stacking these images into a single block of
    memory, a 3-D array is created. Yet the operations to be performed
    are not meant for the entire 3-D array, but on the set of 2-D
    sub-arrays. In most array languages, each frame has to be extracted,
    operated on, and then reinserted into the output array using a
    for-like loop. The J language allows the programmer to perform such
    operations implicitly by having a rank for the frame and array. By
    default these ranks will be the same during the creation of the
    array. It was the intention of the Numeric 1 developers to implement
    this feature, since it is based on the language J. The Numeric 1
    code has the required variables for implementing this behavior, but
    was never implemented. We intend to implement implicit sub-array
    iteration in Numeric 2, if the array broadcasting rules found in
    Numeric 1 do not fully support this behavior.

Copyright

This document is placed in the public domain.

Related PEPs

-   PEP 207: Rich Comparisons by Guido van Rossum and David Ascher
-   PEP 208: Reworking the Coercion Model by Neil Schemenauer and
    Marc-Andre' Lemburg
-   PEP 211: Adding New Linear Algebra Operators to Python by Greg
    Wilson
-   PEP 225: Elementwise/Objectwise Operators by Huaiyu Zhu
-   PEP 228: Reworking Python's Numeric Model by Moshe Zadka

References

[1]
P.  Greenfield 2000. private communication.

authors:
Paul Barrett
Travis Oliphant


text:
PEP: 202 Title: List Comprehensions Author: Barry Warsaw
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 13-Jul-2000 Python-Version: 2.0 Post-History:

Introduction

This PEP describes a proposed syntactical extension to Python, list
comprehensions.

The Proposed Solution

It is proposed to allow conditional construction of list literals using
for and if clauses. They would nest in the same way for loops and if
statements nest now.

Rationale

List comprehensions provide a more concise way to create lists in
situations where map() and filter() and/or nested loops would currently
be used.

Examples

    >>> print [i for i in range(10)]
    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

    >>> print [i for i in range(20) if i%2 == 0]
    [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]

    >>> nums = [1, 2, 3, 4]
    >>> fruit = ["Apples", "Peaches", "Pears", "Bananas"]
    >>> print [(i, f) for i in nums for f in fruit]
    [(1, 'Apples'), (1, 'Peaches'), (1, 'Pears'), (1, 'Bananas'),
     (2, 'Apples'), (2, 'Peaches'), (2, 'Pears'), (2, 'Bananas'),
     (3, 'Apples'), (3, 'Peaches'), (3, 'Pears'), (3, 'Bananas'),
     (4, 'Apples'), (4, 'Peaches'), (4, 'Pears'), (4, 'Bananas')]
    >>> print [(i, f) for i in nums for f in fruit if f[0] == "P"]
    [(1, 'Peaches'), (1, 'Pears'),
     (2, 'Peaches'), (2, 'Pears'),
     (3, 'Peaches'), (3, 'Pears'),
     (4, 'Peaches'), (4, 'Pears')]
    >>> print [(i, f) for i in nums for f in fruit if f[0] == "P" if i%2 == 1]
    [(1, 'Peaches'), (1, 'Pears'), (3, 'Peaches'), (3, 'Pears')]
    >>> print [i for i in zip(nums, fruit) if i[0]%2==0]
    [(2, 'Peaches'), (4, 'Bananas')]

Reference Implementation

List comprehensions become part of the Python language with release 2.0,
documented in[1].

BDFL Pronouncements

-   The syntax proposed above is the Right One.
-   The form [x, y for ...] is disallowed; one is required to write
    [(x, y) for ...].
-   The form [... for x... for y...] nests, with the last index varying
    fastest, just like nested for loops.

References

[1] http://docs.python.org/reference/expressions.html#list-displays