text:
PEP: 454 Title: Add a new tracemalloc module to trace Python memory
allocations Version: $Revision$ Last-Modified: $Date$ Author: Victor
Stinner<EMAIL_ADDRESS>BDFL-Delegate: Charles-François Natali
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 03-Sep-2013 Python-Version: 3.4 Resolution:
https://mail.python.org/pipermail/python-dev/2013-November/130491.html

Abstract

This PEP proposes to add a new tracemalloc module to trace memory blocks
allocated by Python.

Rationale

Classic generic tools like Valgrind can get the C traceback where a
memory block was allocated. Using such tools to analyze Python memory
allocations does not help because most memory blocks are allocated in
the same C function, in PyMem_Malloc() for example. Moreover, Python has
an allocator for small objects called "pymalloc" which keeps free blocks
for efficiency. This is not well handled by these tools.

There are debug tools dedicated to the Python language like Heapy
Pympler and Meliae which lists all alive objects using the garbage
collector module (functions like gc.get_objects(), gc.get_referrers()
and gc.get_referents()), compute their size (ex: using sys.getsizeof())
and group objects by type. These tools provide a better estimation of
the memory usage of an application. They are useful when most memory
leaks are instances of the same type and this type is only instantiated
in a few functions. Problems arise when the object type is very common
like str or tuple, and it is hard to identify where these objects are
instantiated.

Finding reference cycles is also a difficult problem. There are
different tools to draw a diagram of all references. These tools cannot
be used on large applications with thousands of objects because the
diagram is too huge to be analyzed manually.

Proposal

Using the customized allocation API from PEP 445, it becomes easy to set
up a hook on Python memory allocators. A hook can inspect Python
internals to retrieve Python tracebacks. The idea of getting the current
traceback comes from the faulthandler module. The faulthandler dumps the
traceback of all Python threads on a crash, here is the idea is to get
the traceback of the current Python thread when a memory block is
allocated by Python.

This PEP proposes to add a new tracemalloc module, a debug tool to trace
memory blocks allocated by Python. The module provides the following
information:

-   Traceback where an object was allocated
-   Statistics on allocated memory blocks per filename and per line
    number: total size, number and average size of allocated memory
    blocks
-   Computed differences between two snapshots to detect memory leaks

The API of the tracemalloc module is similar to the API of the
faulthandler module: enable() / start(), disable() / stop() and
is_enabled() / is_tracing() functions, an environment variable
(PYTHONFAULTHANDLER and PYTHONTRACEMALLOC), and a -X command line option
(-X faulthandler and -X tracemalloc). See the documentation of the
faulthandler module.

The idea of tracing memory allocations is not new. It was first
implemented in the PySizer project in 2005. PySizer was implemented
differently: the traceback was stored in frame objects and some Python
types were linked the trace with the name of object type. PySizer patch
on CPython adds an overhead on performances and memory footprint, even
if the PySizer was not used. tracemalloc attaches a traceback to the
underlying layer, to memory blocks, and has no overhead when the module
is not tracing memory allocations.

The tracemalloc module has been written for CPython. Other
implementations of Python may not be able to provide it.

API

To trace most memory blocks allocated by Python, the module should be
started as early as possible by setting the PYTHONTRACEMALLOC
environment variable to 1, or by using -X tracemalloc command line
option. The tracemalloc.start() function can be called at runtime to
start tracing Python memory allocations.

By default, a trace of an allocated memory block only stores the most
recent frame (1 frame). To store 25 frames at startup: set the
PYTHONTRACEMALLOC environment variable to 25, or use the
-X tracemalloc=25 command line option. The set_traceback_limit()
function can be used at runtime to set the limit.

Functions

clear_traces() function:

  Clear traces of memory blocks allocated by Python.

  See also stop().

get_object_traceback(obj) function:

  Get the traceback where the Python object obj was allocated. Return a
  Traceback instance, or None if the tracemalloc module is not tracing
  memory allocations or did not trace the allocation of the object.

  See also gc.get_referrers() and sys.getsizeof() functions.

get_traceback_limit() function:

  Get the maximum number of frames stored in the traceback of a trace.

  The tracemalloc module must be tracing memory allocations to get the
  limit, otherwise an exception is raised.

  The limit is set by the start() function.

get_traced_memory() function:

  Get the current size and maximum size of memory blocks traced by the
  tracemalloc module as a tuple: (size: int, max_size: int).

get_tracemalloc_memory() function:

  Get the memory usage in bytes of the tracemalloc module used to store
  traces of memory blocks. Return an int.

is_tracing() function:

  True if the tracemalloc module is tracing Python memory allocations,
  False otherwise.

  See also start() and stop() functions.

start(nframe: int=1) function:

  Start tracing Python memory allocations: install hooks on Python
  memory allocators. Collected tracebacks of traces will be limited to
  nframe frames. By default, a trace of a memory block only stores the
  most recent frame: the limit is 1. nframe must be greater or equal to
  1.

  Storing more than 1 frame is only useful to compute statistics grouped
  by 'traceback' or to compute cumulative statistics: see the
  Snapshot.compare_to() and Snapshot.statistics() methods.

  Storing more frames increases the memory and CPU overhead of the
  tracemalloc module. Use the get_tracemalloc_memory() function to
  measure how much memory is used by the tracemalloc module.

  The PYTHONTRACEMALLOC environment variable (PYTHONTRACEMALLOC=NFRAME)
  and the -X tracemalloc=NFRAME command line option can be used to start
  tracing at startup.

  See also stop(), is_tracing() and get_traceback_limit() functions.

stop() function:

  Stop tracing Python memory allocations: uninstall hooks on Python
  memory allocators. Clear also traces of memory blocks allocated by
  Python

  Call take_snapshot() function to take a snapshot of traces before
  clearing them.

  See also start() and is_tracing() functions.

take_snapshot() function:

  Take a snapshot of traces of memory blocks allocated by Python. Return
  a new Snapshot instance.

  The snapshot does not include memory blocks allocated before the
  tracemalloc module started to trace memory allocations.

  Tracebacks of traces are limited to get_traceback_limit() frames. Use
  the nframe parameter of the start() function to store more frames.

  The tracemalloc module must be tracing memory allocations to take a
  snapshot, see the start() function.

  See also the get_object_traceback() function.

Filter

Filter(inclusive: bool, filename_pattern: str, lineno: int=None, all_frames: bool=False)
class:

  Filter on traces of memory blocks.

  See the fnmatch.fnmatch() function for the syntax of filename_pattern.
  The '.pyc' and '.pyo' file extensions are replaced with '.py'.

  Examples:

  -   Filter(True, subprocess.__file__) only includes traces of the
      subprocess module
  -   Filter(False, tracemalloc.__file__) excludes traces of the
      tracemalloc module
  -   Filter(False, "<unknown>") excludes empty tracebacks

inclusive attribute:

  If inclusive is True (include), only trace memory blocks allocated in
  a file with a name matching filename_pattern at line number lineno.

  If inclusive is False (exclude), ignore memory blocks allocated in a
  file with a name matching filename_pattern at line number lineno.

lineno attribute:

  Line number (int) of the filter. If lineno is None, the filter matches
  any line number.

filename_pattern attribute:

  Filename pattern of the filter (str).

all_frames attribute:

  If all_frames is True, all frames of the traceback are checked. If
  all_frames is False, only the most recent frame is checked.

  This attribute is ignored if the traceback limit is less than 2. See
  the get_traceback_limit() function and Snapshot.traceback_limit
  attribute.

Frame

Frame class:

  Frame of a traceback.

  The Traceback class is a sequence of Frame instances.

filename attribute:

  Filename (str).

lineno attribute:

  Line number (int).

Snapshot

Snapshot class:

  Snapshot of traces of memory blocks allocated by Python.

  The take_snapshot() function creates a snapshot instance.

compare_to(old_snapshot: Snapshot, group_by: str, cumulative: bool=False)
method:

  Compute the differences with an old snapshot. Get statistics as a
  sorted list of StatisticDiff instances grouped by group_by.

  See the statistics() method for group_by and cumulative parameters.

  The result is sorted from the biggest to the smallest by: absolute
  value of StatisticDiff.size_diff, StatisticDiff.size, absolute value
  of StatisticDiff.count_diff, Statistic.count and then by
  StatisticDiff.traceback.

dump(filename) method:

  Write the snapshot into a file.

  Use load() to reload the snapshot.

filter_traces(filters) method:

  Create a new Snapshot instance with a filtered traces sequence,
  filters is a list of Filter instances. If filters is an empty list,
  return a new Snapshot instance with a copy of the traces.

  All inclusive filters are applied at once, a trace is ignored if no
  inclusive filters match it. A trace is ignored if at least one
  exclusive filter matches it.

load(filename) classmethod:

  Load a snapshot from a file.

  See also dump().

statistics(group_by: str, cumulative: bool=False) method:

  Get statistics as a sorted list of Statistic instances grouped by
  group_by:

    group_by      description
    ------------- --------------------------
    'filename'    filename
    'lineno'      filename and line number
    'traceback'   traceback

  If cumulative is True, cumulate size and count of memory blocks of all
  frames of the traceback of a trace, not only the most recent frame.
  The cumulative mode can only be used with group_by equals to
  'filename' and 'lineno' and traceback_limit greater than 1.

  The result is sorted from the biggest to the smallest by:
  Statistic.size, Statistic.count and then by Statistic.traceback.

traceback_limit attribute:

  Maximum number of frames stored in the traceback of traces: result of
  the get_traceback_limit() when the snapshot was taken.

traces attribute:

  Traces of all memory blocks allocated by Python: sequence of Trace
  instances.

  The sequence has an undefined order. Use the Snapshot.statistics()
  method to get a sorted list of statistics.

Statistic

Statistic class:

  Statistic on memory allocations.

  Snapshot.statistics() returns a list of Statistic instances.

  See also the StatisticDiff class.

count attribute:

  Number of memory blocks (int).

size attribute:

  Total size of memory blocks in bytes (int).

traceback attribute:

  Traceback where the memory block was allocated, Traceback instance.

StatisticDiff

StatisticDiff class:

  Statistic difference on memory allocations between an old and a new
  Snapshot instance.

  Snapshot.compare_to() returns a list of StatisticDiff instances. See
  also the Statistic class.

count attribute:

  Number of memory blocks in the new snapshot (int): 0 if the memory
  blocks have been released in the new snapshot.

count_diff attribute:

  Difference of number of memory blocks between the old and the new
  snapshots (int): 0 if the memory blocks have been allocated in the new
  snapshot.

size attribute:

  Total size of memory blocks in bytes in the new snapshot (int): 0 if
  the memory blocks have been released in the new snapshot.

size_diff attribute:

  Difference of total size of memory blocks in bytes between the old and
  the new snapshots (int): 0 if the memory blocks have been allocated in
  the new snapshot.

traceback attribute:

  Traceback where the memory blocks were allocated, Traceback instance.

Trace

Trace class:

  Trace of a memory block.

  The Snapshot.traces attribute is a sequence of Trace instances.

size attribute:

  Size of the memory block in bytes (int).

traceback attribute:

  Traceback where the memory block was allocated, Traceback instance.

Traceback

Traceback class:

  Sequence of Frame instances sorted from the most recent frame to the
  oldest frame.

  A traceback contains at least 1 frame. If the tracemalloc module
  failed to get a frame, the filename "<unknown>" at line number 0 is
  used.

  When a snapshot is taken, tracebacks of traces are limited to
  get_traceback_limit() frames. See the take_snapshot() function.

  The Trace.traceback attribute is an instance of Traceback instance.

Rejected Alternatives

Log calls to the memory allocator

A different approach is to log calls to malloc(), realloc() and free()
functions. Calls can be logged into a file or send to another computer
through the network. Example of a log entry: name of the function, size
of the memory block, address of the memory block, Python traceback where
the allocation occurred, timestamp.

Logs cannot be used directly, getting the current status of the memory
requires to parse previous logs. For example, it is not possible to get
directly the traceback of a Python object, like
get_object_traceback(obj) does with traces.

Python uses objects with a very short lifetime and so makes an extensive
use of memory allocators. It has an allocator optimized for small
objects (less than 512 bytes) with a short lifetime. For example, the
Python test suites calls malloc(), realloc() or free() 270,000 times per
second in average. If the size of log entry is 32 bytes, logging
produces 8.2 MB per second or 29.0 GB per hour.

The alternative was rejected because it is less efficient and has less
features. Parsing logs in a different process or a different computer is
slower than maintaining traces on allocated memory blocks in the same
process.

Prior Work

-   Python Memory Validator (2005-2013): commercial Python memory
    validator developed by Software Verification. It uses the Python
    Reflection API.
-   PySizer: Google Summer of Code 2005 project by Nick Smallbone.
-   Heapy (2006-2013): part of the Guppy-PE project written by Sverker
    Nilsson.
-   Draft PEP: Support Tracking Low-Level Memory Usage in CPython (Brett
    Canon, 2006)
-   Muppy: project developed in 2008 by Robert Schuppenies.
-   asizeof: a pure Python module to estimate the size of objects by
    Jean Brouwers (2008).
-   Heapmonitor: It provides facilities to size individual objects and
    can track all objects of certain classes. It was developed in 2008
    by Ludwig Haehne.
-   Pympler (2008-2011): project based on asizeof, muppy and HeapMonitor
-   objgraph (2008-2012)
-   Dozer: WSGI Middleware version of the CherryPy memory leak debugger,
    written by Marius Gedminas (2008-2013)
-   Meliae: Python Memory Usage Analyzer developed by John A Meinel
    since 2009
-   gdb-heap: gdb script written in Python by Dave Malcolm (2010-2011)
    to analyze the usage of the heap memory
-   memory_profiler: written by Fabian Pedregosa (2011-2013)
-   caulk: written by Ben Timby in 2012

See also Pympler Related Work.

Links

tracemalloc:

-   #18874: Add a new tracemalloc module to trace Python memory
    allocations
-   pytracemalloc on PyPI

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:


text:
PEP: 312 Title: Simple Implicit Lambda Version: $Revision$
Last-Modified: $Date$ Author: Roman Suzi<EMAIL_ADDRESS>Alex Martelli
<EMAIL_ADDRESS>Status: Deferred Type: Standards Track Content-Type:
text/x-rst Created: 11-Feb-2003 Python-Version: 2.4 Post-History:

Abstract

This PEP proposes to make argumentless lambda keyword optional in some
cases where it is not grammatically ambiguous.

Deferral

The BDFL hates the unary colon syntax. This PEP needs to go back to the
drawing board and find a more Pythonic syntax (perhaps an alternative
unary operator). See python-dev discussion on 17 June 2005[1].

Also, it is probably a good idea to eliminate the alternative
propositions which have no chance at all. The examples section is good
and highlights the readability improvements. It would carry more weight
with additional examples and with real-world referents (instead of the
abstracted dummy calls to :A and :B).

Motivation

Lambdas are useful for defining anonymous functions, e.g. for use as
callbacks or (pseudo)-lazy evaluation schemes. Often, lambdas are not
used when they would be appropriate, just because the keyword "lambda"
makes code look complex. Omitting lambda in some special cases is
possible, with small and backwards compatible changes to the grammar,
and provides a cheap cure against such "lambdaphobia".

Rationale

Sometimes people do not use lambdas because they fear to introduce a
term with a theory behind it. This proposal makes introducing
argumentless lambdas easier, by omitting the "lambda" keyword. itself.
Implementation can be done simply changing grammar so it lets the
"lambda" keyword be implied in a few well-known cases. In particular,
adding surrounding brackets lets you specify nullary lambda anywhere.

Syntax

An argumentless "lambda" keyword can be omitted in the following cases:

-   immediately after "=" in named parameter assignment or default value
    assignment;
-   immediately after "(" in any expression;
-   immediately after a "," in a function argument list;
-   immediately after a ":" in a dictionary literal; (not implemented)
-   in an assignment statement; (not implemented)

Examples of Use

1)  Inline if:

        def ifelse(cond, true_part, false_part):
            if cond:
                return true_part()
            else:
                return false_part()

        # old syntax:
        print ifelse(a < b, lambda:A, lambda:B)

        # new syntax:
        print ifelse(a < b, :A, :B)

        # parts A and B may require extensive processing, as in:
        print ifelse(a < b, :ext_proc1(A), :ext_proc2(B))

2)  Locking:

        def with(alock, acallable):
            alock.acquire()
            try:
                acallable()
            finally:
                alock.release()

        with(mylock, :x(y(), 23, z(), 'foo'))

Implementation

Implementation requires some tweaking of the Grammar/Grammar file in the
Python sources, and some adjustment of Modules/parsermodule.c to make
syntactic and pragmatic changes.

(Some grammar/parser guru is needed to make a full implementation.)

Here are the changes needed to Grammar to allow implicit lambda:

    varargslist: (fpdef ['=' imptest] ',')* ('*' NAME [',' '**'
    NAME] | '**' NAME) | fpdef ['=' imptest] (',' fpdef ['='
    imptest])* [',']

    imptest: test | implambdef

    atom: '(' [imptestlist] ')' | '[' [listmaker] ']' |
    '{' [dictmaker] '}' | '`' testlist1 '`' | NAME | NUMBER | STRING+

    implambdef: ':' test

    imptestlist: imptest (',' imptest)* [',']

    argument: [test '='] imptest

Three new non-terminals are needed: imptest for the place where implicit
lambda may occur, implambdef for the implicit lambda definition itself,
imptestlist for a place where imptest's may occur.

This implementation is not complete. First, because some files in Parser
module need to be updated. Second, some additional places aren't
implemented, see Syntax section above.

Discussion

This feature is not a high-visibility one (the only novel part is the
absence of lambda). The feature is intended to make null-ary lambdas
more appealing syntactically, to provide lazy evaluation of expressions
in some simple cases. This proposal is not targeted at more advanced
cases (demanding arguments for the lambda).

There is an alternative proposition for implicit lambda: implicit lambda
with unused arguments. In this case the function defined by such lambda
can accept any parameters, i.e. be equivalent to: lambda *args: expr.
This form would be more powerful. Grep in the standard library revealed
that such lambdas are indeed in use.

One more extension can provide a way to have a list of parameters passed
to a function defined by implicit lambda. However, such parameters need
some special name to be accessed and are unlikely to be included in the
language. Possible local names for such parameters are: _, __args__, __.
For example:

    reduce(:_[0] + _[1], [1,2,3], 0)
    reduce(:__[0] + __[1], [1,2,3], 0)
    reduce(:__args__[0] + __args__[1], [1,2,3], 0)

These forms do not look very nice, and in the PEP author's opinion do
not justify the removal of the lambda keyword in such cases.

Credits

The idea of dropping lambda was first coined by Paul Rubin at 08 Feb
2003 16:39:30 -0800 in comp.lang.python while discussing the thread "For
review: PEP 308 - If-then-else expression"[2].

References

Copyright

This document has been placed in the public domain.

[1] Guido van Rossum, Recommend accepting PEP 312 -- Simple Implicit
Lambda
https://mail.python.org/pipermail/python-dev/2005-June/054304.html

[2] Guido van Rossum, For review: PEP 308 - If-then-else expression
https://mail.python.org/pipermail/python-dev/2003-February/033178.html

license: Public Domain

authors:
Alex Martelli
Roman Suzi


text:
PEP: 412 Title: Key-Sharing Dictionary Version: $Revision$
Last-Modified: $Date$ Author: Mark Shannon<EMAIL_ADDRESS>Status:
Final Type: Standards Track Content-Type: text/x-rst Created:
08-Feb-2012 Python-Version: 3.3 Post-History: 08-Feb-2012

Abstract

This PEP proposes a change in the implementation of the builtin
dictionary type dict. The new implementation allows dictionaries which
are used as attribute dictionaries (the __dict__ attribute of an object)
to share keys with other attribute dictionaries of instances of the same
class.

Motivation

The current dictionary implementation uses more memory than is necessary
when used as a container for object attributes as the keys are
replicated for each instance rather than being shared across many
instances of the same class. Despite this, the current dictionary
implementation is finely tuned and performs very well as a
general-purpose mapping object.

By separating the keys (and hashes) from the values it is possible to
share the keys between multiple dictionaries and improve memory use. By
ensuring that keys are separated from the values only when beneficial,
it is possible to retain the high-performance of the current dictionary
implementation when used as a general-purpose mapping object.

Behaviour

The new dictionary behaves in the same way as the old implementation. It
fully conforms to the Python API, the C API and the ABI.

Performance

Memory Usage

Reduction in memory use is directly related to the number of
dictionaries with shared keys in existence at any time. These
dictionaries are typically half the size of the current dictionary
implementation.

Benchmarking shows that memory use is reduced by 10% to 20% for
object-oriented programs with no significant change in memory use for
other programs.

Speed

The performance of the new implementation is dominated by memory
locality effects. When keys are not shared (for example in module
dictionaries and dictionary explicitly created by dict() or {}) then
performance is unchanged (within a percent or two) from the current
implementation.

For the shared keys case, the new implementation tends to separate keys
from values, but reduces total memory usage. This will improve
performance in many cases as the effects of reduced memory usage
outweigh the loss of locality, but some programs may show a small slow
down.

Benchmarking shows no significant change of speed for most benchmarks.
Object-oriented benchmarks show small speed ups when they create large
numbers of objects of the same class (the gcbench benchmark shows a 10%
speed up; this is likely to be an upper limit).

Implementation

Both the old and new dictionaries consist of a fixed-sized dict struct
and a re-sizeable table. In the new dictionary the table can be further
split into a keys table and values array. The keys table holds the keys
and hashes and (for non-split tables) the values as well. It differs
only from the original implementation in that it contains a number of
fields that were previously in the dict struct. If a table is split the
values in the keys table are ignored, instead the values are held in a
separate array.

Split-Table dictionaries

When dictionaries are created to fill the __dict__ slot of an object,
they are created in split form. The keys table is cached in the type,
potentially allowing all attribute dictionaries of instances of one
class to share keys. In the event of the keys of these dictionaries
starting to diverge, individual dictionaries will lazily convert to the
combined-table form. This ensures good memory use in the common case,
and correctness in all cases.

When resizing a split dictionary it is converted to a combined table. If
resizing is as a result of storing an instance attribute, and there is
only instance of a class, then the dictionary will be re-split
immediately. Since most OO code will set attributes in the __init__
method, all attributes will be set before a second instance is created
and no more resizing will be necessary as all further instance
dictionaries will have the correct size. For more complex use patterns,
it is impossible to know what is the best approach, so the
implementation allows extra insertions up to the point of a resize when
it reverts to the combined table (non-shared keys).

A deletion from a split dictionary does not change the keys table, it
simply removes the value from the values array.

Combined-Table dictionaries

Explicit dictionaries (dict() or {}), module dictionaries and most other
dictionaries are created as combined-table dictionaries. A
combined-table dictionary never becomes a split-table dictionary.
Combined tables are laid out in much the same way as the tables in the
old dictionary, resulting in very similar performance.

Implementation

The new dictionary implementation is available at[1].

Pros and Cons

Pros

Significant memory savings for object-oriented applications. Small
improvement to speed for programs which create lots of similar objects.

Cons

Change to data structures: Third party modules which meddle with the
internals of the dictionary implementation will break.

Changes to repr() output and iteration order: For most cases, this will
be unchanged. However, for some split-table dictionaries the iteration
order will change.

Neither of these cons should be a problem. Modules which meddle with the
internals of the dictionary implementation are already broken and should
be fixed to use the API. The iteration order of dictionaries was never
defined and has always been arbitrary; it is different for Jython and
PyPy.

Alternative Implementation

An alternative implementation for split tables, which could save even
more memory, is to store an index in the value field of the keys table
(instead of ignoring the value field). This index would explicitly state
where in the value array to look. The value array would then only
require 1 field for each usable slot in the key table, rather than each
slot in the key table.

This "indexed" version would reduce the size of value array by about one
third. The keys table would need an extra "values_size" field,
increasing the size of combined dicts by one word. The extra indirection
adds more complexity to the code, potentially reducing performance a
little.

The "indexed" version will not be included in this implementation, but
should be considered deferred rather than rejected, pending further
experimentation.

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] Reference Implementation:
https://bitbucket.org/markshannon/cpython_new_dict


text:
PEP: 268 Title: Extended HTTP functionality and WebDAV Author: Greg
Stein<EMAIL_ADDRESS>Status: Rejected Type: Standards Track
Content-Type: text/x-rst Created: 20-Aug-2001 Python-Version: 2.x
Post-History: 21-Aug-2001

Rejection Notice

This PEP has been rejected. It has failed to generate sufficient
community support in the six years since its proposal.

Abstract

This PEP discusses new modules and extended functionality for Python's
HTTP support. Notably, the addition of authenticated requests, proxy
support, authenticated proxy usage, and WebDAV capabilities.

Rationale

Python has been quite popular as a result of its "batteries included"
positioning. One of the most heavily used protocols, HTTP (see 2616),
has been included with Python for years (httplib). However, this support
has not kept up with the full needs and requirements of many HTTP-based
applications and systems. In addition, new protocols based on HTTP, such
as WebDAV and XML-RPC, are becoming useful and are seeing increasing
usage. Supplying this functionality meets Python's "batteries included"
role and also keeps Python at the leading edge of new technologies.

While authentication and proxy support are two very notable features
missing from Python's core HTTP processing, they are minimally handled
as part of Python's URL handling (urllib and urllib2). However,
applications that need fine-grained or sophisticated HTTP handling
cannot make use of the features while they reside in urllib. Refactoring
these features into a location where they can be directly associated
with an HTTP connection will improve their utility for both urllib and
for sophisticated applications.

The motivation for this PEP was from several people requesting these
features directly, and from a number of feature requests on SourceForge.
Since the exact form of the modules to be provided and the
classes/architecture used could be subject to debate, this PEP was
created to provide a focal point for those discussions.

Specification

Two modules will be added to the standard library: httpx (HTTP extended
functionality), and davlib (WebDAV library).

[ suggestions for module names are welcome; davlib has some precedence,
but something like webdav might be desirable ]

HTTP Authentication

The httpx module will provide a mixin for performing HTTP authentication
(for both proxy and origin server authentication). This mixin
(httpx.HandleAuthentication) can be combined with the HTTPConnection and
the HTTPSConnection classes (the mixin may possibly work with the HTTP
and HTTPS compatibility classes, but that is not a requirement).

The mixin will delegate the authentication process to one or more
"authenticator" objects, allowing multiple connections to share
authenticators. The use of a separate object allows for a long term
connection to an authentication system (e.g. LDAP). An authenticator for
the Basic and Digest mechanisms (see 2617) will be provided.
User-supplied authenticator subclasses can be registered and used by the
connections.

A "credentials" object (httpx.Credentials) is also associated with the
mixin, and stores the credentials (e.g. username and password) needed by
the authenticators. Subclasses of Credentials can be created to hold
additional information (e.g. NT domain).

The mixin overrides the getresponse() method to detect
401 (Unauthorized) and 407 (Proxy Authentication Required) responses.
When this is found, the response object, the connection, and the
credentials are passed to the authenticator corresponding with the
authentication scheme specified in the response (multiple authenticators
are tried in decreasing order of security if multiple schemes are in the
response). Each authenticator can examine the response headers and
decide whether and how to resend the request with the correct
authentication headers. If no authenticator can successfully handle the
authentication, then an exception is raised.

Resending a request, with the appropriate credentials, is one of the
more difficult portions of the authentication system. The difficulty
arises in recording what was sent originally: the request line, the
headers, and the body. By overriding putrequest, putheader, and
endheaders, we can capture all but the body. Once the endheaders method
is called, then we capture all calls to send() (until the next
putrequest method call) to hold the body content. The mixin will have a
configurable limit for the amount of data to hold in this fashion (e.g.
only hold up to 100k of body content). Assuming that the entire body has
been stored, then we can resend the request with the appropriate
authentication information.

If the body is too large to be stored, then the getresponse() simply
returns the response object, indicating the 401 or 407 error. Since the
authentication information has been computed and cached (into the
Credentials object; see below), the caller can simply regenerate the
request. The mixin will attach the appropriate credentials.

A "protection space" (see 2617, section 1.2) is defined as a tuple of
the host, port, and authentication realm. When a request is initially
sent to an HTTP server, we do not know the authentication realm (the
realm is only returned when authentication fails). However, we do have
the path from the URL, and that can be useful in determining the
credentials to send to the server. The Basic authentication scheme is
typically set up hierarchically: the credentials for /path can be tried
for /path/subpath. The Digest authentication scheme has explicit support
for the hierarchical setup. The httpx.Credentials object will store
credentials for multiple protection spaces, and can be looked up in two
different ways:

1.  looked up using (host, port, path) -- this lookup scheme is used
    when generating a request for a path where we don't know the
    authentication realm.
2.  looked up using (host, port, realm) -- this mechanism is used during
    the authentication process when the server has specified that the
    Request-URI resides within a specific realm.

The HandleAuthentication mixin will override putrequest() to
automatically insert credentials, if available. The URL from the
putrequest is used to determine the appropriate authentication
information to use.

It is also important to note that two sets of credentials are used, and
stored by the mixin. One set for any proxy that may be used, and one
used for the target origin server. Since proxies do not have paths, the
protection spaces in the proxy credentials will always use "/" for
storing and looking up via a path.

Proxy Handling

The httpx module will provide a mixin for using a proxy to perform
HTTP(S) operations. This mixin (httpx.UseProxy) can be combined with the
HTTPConnection and the HTTPSConnection classes (the mixin may possibly
work with the HTTP and HTTPS compatibility classes, but that is not a
requirement).

The mixin will record the (host, port) of the proxy to use. XXX will be
overridden to use this host/port combination for connections and to
rewrite request URLs into the absoluteURIs referring to the origin
server (these URIs are passed to the proxy server).

Proxy authentication is handled by the httpx.HandleAuthentication class
since a user may directly use HTTP(S)Connection to speak with proxies.

WebDAV Features

The davlib module will provide a mixin for sending WebDAV requests to a
WebDAV-enabled server. This mixin (davlib.DAVClient) can be combined
with the HTTPConnection and the HTTPSConnection classes (the mixin may
possibly work with the HTTP and HTTPS compatibility classes, but that is
not a requirement).

The mixin provides methods to perform the various HTTP methods defined
by HTTP in 2616, and by WebDAV in 2518.

A custom response object is used to decode 207 (Multi-Status) responses.
The response object will use the standard library's xml package to parse
the multistatus XML information, producing a simple structure of objects
to hold the multistatus data. Multiple parsing schemes will be
tried/used, in order of decreasing speed.

Reference Implementation

The actual (future/final) implementation is being developed in the
/nondist/sandbox/Lib directory, until it is accepted and moved into the
main Lib directory.

References

Copyright

This document has been placed in the public domain.


text:
PEP: 323 Title: Copyable Iterators Version: $Revision$ Last-Modified:
$Date$ Author: Alex Martelli<EMAIL_ADDRESS>Status: Deferred Type:
Standards Track Content-Type: text/x-rst Created: 25-Oct-2003
Python-Version: 2.5 Post-History: 29-Oct-2003

Deferral

This PEP has been deferred. Copyable iterators are a nice idea, but
after four years, no implementation or widespread interest has emerged.

Abstract

This PEP suggests that some iterator types should support shallow copies
of their instances by exposing a __copy__ method which meets some
specific requirements, and indicates how code using an iterator might
exploit such a __copy__ method when present.

Update and Comments

Support for __copy__ was included in Py2.4's itertools.tee().

Adding __copy__ methods to existing iterators will change the behavior
under tee(). Currently, the copied iterators remain tied to the original
iterator. If the original advances, then so do all of the copies. Good
practice is to overwrite the original so that anomalies don't result:
a,b=tee(a). Code that doesn't follow that practice may observe a
semantic change if a __copy__ method is added to an iterator.

Motivation

In Python up to 2.3, most built-in iterator types don't let the user
copy their instances. User-coded iterators that do let their clients
call copy.copy on their instances may, or may not, happen to return, as
a result of the copy, a separate iterator object that may be iterated
upon independently from the original.

Currently, "support" for copy.copy in a user-coded iterator type is
almost invariably "accidental" -- i.e., the standard machinery of the
copy method in Python's standard library's copy module does build and
return a copy. However, the copy will be independently iterable with
respect to the original only if calling .next() on an instance of that
class happens to change instance state solely by rebinding some
attributes to new values, and not by mutating some attributes' existing
values.

For example, an iterator whose "index" state is held as an integer
attribute will probably give usable copies, since (integers being
immutable) .next() presumably just rebinds that attribute. On the other
hand, another iterator whose "index" state is held as a list attribute
will probably mutate the same list object when .next() executes, and
therefore copies of such an iterator will not be iterable separately and
independently from the original.

Given this existing situation, copy.copy(it) on some iterator object
isn't very useful, nor, therefore, is it at all widely used. However,
there are many cases in which being able to get a "snapshot" of an
iterator, as a "bookmark", so as to be able to keep iterating along the
sequence but later iterate again on the same sequence from the bookmark
onwards, is useful. To support such "bookmarking", module itertools, in
2.4, has grown a 'tee' function, to be used as:

    it, bookmark = itertools.tee(it)

The previous value of 'it' must not be used again, which is why this
typical usage idiom rebinds the name. After this call, 'it' and
'bookmark' are independently-iterable iterators on the same underlying
sequence as the original value of 'it': this satisfies application needs
for "iterator copying".

However, when itertools.tee can make no hypotheses about the nature of
the iterator it is passed as an argument, it must save in memory all
items through which one of the two 'teed' iterators, but not yet both,
have stepped. This can be quite costly in terms of memory, if the two
iterators get very far from each other in their stepping; indeed, in
some cases it may be preferable to make a list from the iterator so as
to be able to step repeatedly through the subsequence, or, if that is
too costy in terms of memory, save items to disk, again in order to be
able to iterate through them repeatedly.

This PEP proposes another idea that will, in some important cases, allow
itertools.tee to do its job with minimal cost in terms of memory; user
code may also occasionally be able to exploit the idea in order to
decide whether to copy an iterator, make a list from it, or use an
auxiliary disk file.

The key consideration is that some important iterators, such as those
which built-in function iter builds over sequences, would be
intrinsically easy to copy: just get another reference to the same
sequence, and a copy of the integer index. However, in Python 2.3, those
iterators don't expose the state, and don't support copy.copy.

The purpose of this PEP, therefore, is to have those iterator types
expose a suitable __copy__ method. Similarly, user-coded iterator types
that can provide copies of their instances, suitable for separate and
independent iteration, with limited costs in time and space, should also
expose a suitable __copy__ method. While copy.copy also supports other
ways to let a type control the way its instances are copied, it is
suggested, for simplicity, that iterator types that support copying
always do so by exposing a __copy__ method, and not in the other ways
copy.copy supports.

Having iterators expose a suitable __copy__ when feasible will afford
easy optimization of itertools.tee and similar user code, as in:

    def tee(it):
        it = iter(it)
        try: copier = it.__copy__
        except AttributeError:
            # non-copyable iterator, do all the needed hard work
            # [snipped!]
        else:
            return it, copier()

Note that this function does NOT call "copy.copy(it)", which (even after
this PEP is implemented) might well still "just happen to succeed". for
some iterator type that is implemented as a user-coded class. without
really supplying an adequate "independently iterable" copy object as its
result.

Specification

Any iterator type X may expose a method __copy__ that is callable
without arguments on any instance x of X. The method should be exposed
if and only if the iterator type can provide copyability with reasonably
little computational and memory effort. Furthermore, the new object y
returned by method __copy__ should be a new instance of X that is
iterable independently and separately from x, stepping along the same
"underlying sequence" of items.

For example, suppose a class Iter essentially duplicated the
functionality of the iter builtin for iterating on a sequence:

    class Iter(object):

        def __init__(self, sequence):
            self.sequence = sequence
            self.index = 0

        def __iter__(self):
            return self

        def next(self):
            try: result = self.sequence[self.index]
            except IndexError: raise StopIteration
            self.index += 1
            return result

To make this Iter class compliant with this PEP, the following addition
to the body of class Iter would suffice:

    def __copy__(self):
        result = self.__class__(self.sequence)
        result.index = self.index
        return result

Note that __copy__, in this case, does not even try to copy the
sequence; if the sequence is altered while either or both of the
original and copied iterators are still stepping on it, the iteration
behavior is quite likely to go awry anyway -- it is not __copy__'s
responsibility to change this normal Python behavior for iterators which
iterate on mutable sequences (that might, perhaps, be the specification
for a __deepcopy__ method of iterators, which, however, this PEP does
not deal with).

Consider also a "random iterator", which provides a nonterminating
sequence of results from some method of a random instance, called with
given arguments:

    class RandomIterator(object):

        def __init__(self, bound_method, *args):
            self.call = bound_method
            self.args = args

        def __iter__(self):
            return self

        def next(self):
            return self.call(*self.args)

        def __copy__(self):
            import copy, new
            im_self = copy.copy(self.call.im_self)
            method = new.instancemethod(self.call.im_func, im_self)
            return self.__class__(method, *self.args)

This iterator type is slightly more general than its name implies, as it
supports calls to any bound method (or other callable, but if the
callable is not a bound method, then method __copy__ will fail). But the
use case is for the purpose of generating random streams, as in:

    import random

    def show5(it):
        for i, result in enumerate(it):
            print '%6.3f'%result,
            if i==4: break
        print

    normit = RandomIterator(random.Random().gauss, 0, 1)
    show5(normit)
    copit = normit.__copy__()
    show5(normit)
    show5(copit)

which will display some output such as:

    -0.536  1.936 -1.182 -1.690 -1.184
     0.666 -0.701  1.214  0.348  1.373
     0.666 -0.701  1.214  0.348  1.373

the key point being that the second and third lines are equal, because
the normit and copit iterators will step along the same "underlying
sequence". (As an aside, note that to get a copy of self.call.im_self we
must use copy.copy, NOT try getting at a __copy__ method directly,
because for example instances of random.Random support copying via
__getstate__ and __setstate__, NOT via __copy__; indeed, using copy.copy
is the normal way to get a shallow copy of any object --copyable
iterators are different because of the already-mentioned uncertainty
about the result of copy.copy supporting these "copyable iterator"
specs).

Details

Besides adding to the Python docs a recommendation that user-coded
iterator types support a __copy__ method (if and only if it can be
implemented with small costs in memory and runtime, and produce an
independently-iterable copy of an iterator object), this PEP's
implementation will specifically include the addition of copyability to
the iterators over sequences that built-in iter returns, and also to the
iterators over a dictionary returned by the methods __iter__, iterkeys,
itervalues, and iteritems of built-in type dict.

Iterators produced by generator functions will not be copyable. However,
iterators produced by the new "generator expressions" of Python 2.4 (PEP
289) should be copyable if their underlying iterator[s] are; the strict
limitations on what is possible in a generator expression, compared to
the much vaster generality of a generator, should make that feasible.
Similarly, the iterators produced by the built-in function enumerate,
and certain functions suppiled by module itertools, should be copyable
if the underlying iterators are.

The implementation of this PEP will also include the optimization of the
new itertools.tee function mentioned in the Motivation section.

Rationale

The main use case for (shallow) copying of an iterator is the same as
for the function itertools.tee (new in 2.4). User code will not directly
attempt to copy an iterator, because it would have to deal separately
with uncopyable cases; calling itertools.tee will internally perform the
copy when appropriate, and implicitly fallback to a maximally efficient
non-copying strategy for iterators that are not copyable. (Occasionally,
user code may want more direct control, specifically in order to deal
with non-copyable iterators by other strategies, such as making a list
or saving the sequence to disk).

A tee'd iterator may serve as a "reference point", allowing processing
of a sequence to continue or resume from a known point, while the other
independent iterator can be freely advanced to "explore" a further part
of the sequence as needed. A simple example: a generator function which,
given an iterator of numbers (assumed to be positive), returns a
corresponding iterator, each of whose items is the fraction of the total
corresponding to each corresponding item of the input iterator. The
caller may pass the total as a value, if known in advance; otherwise,
the iterator returned by calling this generator function will first
compute the total.

    def fractions(numbers, total=None):
        if total is None:
            numbers, aux = itertools.tee(numbers)
            total = sum(aux)
        total = float(total)
        for item in numbers:
            yield item / total

The ability to tee the numbers iterator allows this generator to
precompute the total, if needed, without necessarily requiring O(N)
auxiliary memory if the numbers iterator is copyable.

As another example of "iterator bookmarking", consider a stream of
numbers with an occasional string as a "postfix operator" now and then.
By far most frequent such operator is a '+', whereupon we must sum all
previous numbers (since the last previous operator if any, or else since
the start) and yield the result. Sometimes we find a '*' instead, which
is the same except that the previous numbers must instead be multiplied,
not summed.

    def filter_weird_stream(stream):
        it = iter(stream)
        while True:
            it, bookmark = itertools.tee(it)
            total = 0
            for item in it:
                if item=='+':
                    yield total
                    break
                elif item=='*':
                    product = 1
                    for item in bookmark:
                        if item=='*':
                            yield product
                            break
                        else:
                            product *= item
               else:
                   total += item

Similar use cases of itertools.tee can support such tasks as "undo" on a
stream of commands represented by an iterator, "backtracking" on the
parse of a stream of tokens, and so on. (Of course, in each case, one
should also consider simpler possibilities such as saving relevant
portions of the sequence into lists while stepping on the sequence with
just one iterator, depending on the details of one's task).

Here is an example, in pure Python, of how the 'enumerate' built-in
could be extended to support __copy__ if its underlying iterator also
supported __copy__:

    class enumerate(object):

        def __init__(self, it):
            self.it = iter(it)
            self.i = -1

        def __iter__(self):
            return self

        def next(self):
            self.i += 1
            return self.i, self.it.next()

        def __copy__(self):
            result = self.__class__.__new__()
            result.it = self.it.__copy__()
            result.i = self.i
            return result

Here is an example of the kind of "fragility" produced by "accidental
copyability" of an iterator -- the reason why one must NOT use copy.copy
expecting, if it succeeds, to receive as a result an iterator which is
iterable-on independently from the original. Here is an iterator class
that iterates (in preorder) on "trees" which, for simplicity, are just
nested lists -- any item that's a list is treated as a subtree, any
other item as a leaf.

    class ListreeIter(object):

        def __init__(self, tree):
            self.tree = [tree]
            self.indx = [-1]

        def __iter__(self):
            return self

        def next(self):
            if not self.indx:
                raise StopIteration
            self.indx[-1] += 1
            try:
                result = self.tree[-1][self.indx[-1]]
            except IndexError:
                self.tree.pop()
                self.indx.pop()
                return self.next()
            if type(result) is not list:
                return result
            self.tree.append(result)
            self.indx.append(-1)
            return self.next()

Now, for example, the following code:

    import copy
    x = [ [1,2,3], [4, 5, [6, 7, 8], 9], 10, 11, [12] ]

    print 'showing all items:',
    it = ListreeIter(x)
    for i in it:
        print i,
        if i==6: cop = copy.copy(it)
    print

    print 'showing items >6 again:'
    for i in cop: print i,
    print

does NOT work as intended -- the "cop" iterator gets consumed, and
exhausted, step by step as the original "it" iterator is, because the
accidental (rather than deliberate) copying performed by copy.copy
shares, rather than duplicating the "index" list, which is the mutable
attribute it.indx (a list of numerical indices). Thus, this "client
code" of the iterator, which attempts to iterate twice over a portion of
the sequence via a copy.copy on the iterator, is NOT correct.

Some correct solutions include using itertools.tee, i.e., changing the
first for loop into:

    for i in it:
        print i,
        if i==6:
            it, cop = itertools.tee(it)
            break
    for i in it: print i,

(note that we MUST break the loop in two, otherwise we'd still be
looping on the ORIGINAL value of it, which must NOT be used further
after the call to tee!!!); or making a list, i.e.

    for i in it:
        print i,
        if i==6:
            cop = lit = list(it)
            break
    for i in lit: print i,

(again, the loop must be broken in two, since iterator 'it' gets
exhausted by the call list(it)).

Finally, all of these solutions would work if Listiter supplied a
suitable __copy__ method, as this PEP recommends:

    def __copy__(self):
        result = self.__class__.new()
        result.tree = copy.copy(self.tree)
        result.indx = copy.copy(self.indx)
        return result

There is no need to get any "deeper" in the copy, but the two mutable
"index state" attributes must indeed be copied in order to achieve a
"proper" (independently iterable) iterator-copy.

The recommended solution is to have class Listiter supply this __copy__
method AND have client code use itertools.tee (with the
split-in-two-parts loop as shown above). This will make client code
maximally tolerant of different iterator types it might be using AND
achieve good performance for tee'ing of this specific iterator type at
the same time.

References

[1] Discussion on python-dev starting at post:
https://mail.python.org/pipermail/python-dev/2003-October/038969.html

[2] Online documentation for the copy module of the standard library:
https://docs.python.org/release/2.6/library/copy.html

Copyright

This document has been placed in the public domain.


text:
PEP: 309 Title: Partial Function Application Version: $Revision$
Last-Modified: $Date$ Author: Peter Harris<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type: text/x-rst Created:
08-Feb-2003 Python-Version: 2.5 Post-History: 10-Feb-2003, 27-Feb-2003,
22-Feb-2004, 28-Apr-2006

Note

Following the acceptance of this PEP, further discussion on python-dev
and comp.lang.python revealed a desire for several tools that operated
on function objects, but were not related to functional programming.
Rather than create a new module for these tools, it was agreed[1] that
the "functional" module be renamed to "functools" to reflect its
newly-widened focus.

References in this PEP to a "functional" module have been left in for
historical reasons.

Abstract

This proposal is for a function or callable class that allows a new
callable to be constructed from a callable and a partial argument list
(including positional and keyword arguments).

I propose a standard library module called "functional", to hold useful
higher-order functions, including the implementation of partial().

An implementation has been submitted to SourceForge[2].

Acceptance

Patch #941881 was accepted and applied in 2005 for Py2.5. It is
essentially as outlined here, a partial() type constructor binding
leftmost positional arguments and any keywords. The partial object has
three read-only attributes func, args, and keywords. Calls to the
partial object can specify keywords that override those in the object
itself.

There is a separate and continuing discussion of whether to modify the
partial implementation with a __get__ method to more closely emulate the
behavior of an equivalent function.

Motivation

In functional programming, function currying is a way of implementing
multi-argument functions in terms of single-argument functions. A
function with N arguments is really a function with 1 argument that
returns another function taking (N-1) arguments. Function application in
languages like Haskell and ML works such that a function call:

    f x y z

actually means:

    (((f x) y) z)

This would be only an obscure theoretical issue except that in actual
programming it turns out to be very useful. Expressing a function in
terms of partial application of arguments to another function can be
both elegant and powerful, and in functional languages it is heavily
used.

In some functional languages, (e.g. Miranda) you can use an expression
such as (+1) to mean the equivalent of Python's (lambda x: x + 1).

In general, languages like that are strongly typed, so the compiler
always knows the number of arguments expected and can do the right thing
when presented with a functor and less arguments than expected.

Python does not implement multi-argument functions by currying, so if
you want a function with partially-applied arguments you would probably
use a lambda as above, or define a named function for each instance.

However, lambda syntax is not to everyone's taste, so say the least.
Furthermore, Python's flexible parameter passing using both positional
and keyword presents an opportunity to generalise the idea of partial
application and do things that lambda cannot.

Example Implementation

Here is one way to do a create a callable with partially-applied
arguments in Python. The implementation below is based on improvements
provided by Scott David Daniels:

    class partial(object):

        def __init__(*args, **kw):
            self = args[0]
            self.fn, self.args, self.kw = (args[1], args[2:], kw)

        def __call__(self, *args, **kw):
            if kw and self.kw:
                d = self.kw.copy()
                d.update(kw)
            else:
                d = kw or self.kw
            return self.fn(*(self.args + args), **d)

(A recipe similar to this has been in the Python Cookbook for some
time[3].)

Note that when the object is called as though it were a function,
positional arguments are appended to those provided to the constructor,
and keyword arguments override and augment those provided to the
constructor.

Positional arguments, keyword arguments or both can be supplied at when
creating the object and when calling it.

Examples of Use

So partial(operator.add, 1) is a bit like (lambda x: 1 + x). Not an
example where you see the benefits, of course.

Note too, that you could wrap a class in the same way, since classes
themselves are callable factories for objects. So in some cases, rather
than defining a subclass, you can specialise classes by partial
application of the arguments to the constructor.

For example, partial(Tkinter.Label, fg='blue') makes Tkinter Labels that
have a blue foreground by default.

Here's a simple example that uses partial application to construct
callbacks for Tkinter widgets on the fly:

    from Tkinter import Tk, Canvas, Button
    import sys
    from functional import partial

    win = Tk()
    c = Canvas(win,width=200,height=50)
    c.pack()

    for colour in sys.argv[1:]:
        b = Button(win, text=colour,
                   command=partial(c.config, bg=colour))
        b.pack(side='left')

    win.mainloop()

Abandoned Syntax Proposal

I originally suggested the syntax fn@(*args, **kw), meaning the same as
partial(fn, *args, **kw).

The @ sign is used in some assembly languages to imply register
indirection, and the use here is also a kind of indirection. f@(x) is
not f(x), but a thing that becomes f(x) when you call it.

It was not well-received, so I have withdrawn this part of the proposal.
In any case, @ has been taken for the new decorator syntax.

Feedback from comp.lang.python and python-dev

Among the opinions voiced were the following (which I summarise):

-   Lambda is good enough.
-   The @ syntax is ugly (unanimous).
-   It's really a curry rather than a closure. There is an almost
    identical implementation of a curry class on ActiveState's Python
    Cookbook.
-   A curry class would indeed be a useful addition to the standard
    library.
-   It isn't function currying, but partial application. Hence the name
    is now proposed to be partial().
-   It maybe isn't useful enough to be in the built-ins.
-   The idea of a module called functional was well received, and there
    are other things that belong there (for example function
    composition).
-   For completeness, another object that appends partial arguments
    after those supplied in the function call (maybe called rightcurry)
    has been suggested.

I agree that lambda is usually good enough, just not always. And I want
the possibility of useful introspection and subclassing.

I disagree that @ is particularly ugly, but it may be that I'm just
weird. We have dictionary, list and tuple literals neatly differentiated
by special punctuation -- a way of directly expressing partially-applied
function literals is not such a stretch. However, not one single person
has said they like it, so as far as I'm concerned it's a dead parrot.

I concur with calling the class partial rather than curry or closure, so
I have amended the proposal in this PEP accordingly. But not throughout:
some incorrect references to 'curry' have been left in since that's
where the discussion was at the time.

Partially applying arguments from the right, or inserting arguments at
arbitrary positions creates its own problems, but pending discovery of a
good implementation and non-confusing semantics, I don't think it should
be ruled out.

Carl Banks posted an implementation as a real functional closure:

    def curry(fn, *cargs, **ckwargs):
        def call_fn(*fargs, **fkwargs):
            d = ckwargs.copy()
            d.update(fkwargs)
            return fn(*(cargs + fargs), **d)
        return call_fn

which he assures me is more efficient.

I also coded the class in Pyrex, to estimate how the performance might
be improved by coding it in C:

    cdef class curry:

        cdef object fn, args, kw

        def __init__(self, fn, *args, **kw):
            self.fn=fn
            self.args=args
            self.kw = kw

        def __call__(self, *args, **kw):
            if self.kw:        # from Python Cookbook version
                d = self.kw.copy()
                d.update(kw)
            else:
                d=kw
            return self.fn(*(self.args + args), **d)

The performance gain in Pyrex is less than 100% over the nested function
implementation, since to be fully general it has to operate by Python
API calls. For the same reason, a C implementation will be unlikely to
be much faster, so the case for a built-in coded in C is not very
strong.

Summary

I prefer that some means to partially-apply functions and other
callables should be present in the standard library.

A standard library module functional should contain an implementation of
partial, and any other higher-order functions the community want. Other
functions that might belong there fall outside the scope of this PEP
though.

Patches for the implementation, documentation and unit tests (SF patches
931005, 931007, and 931010 respectively) have been submitted but not yet
checked in.

A C implementation by Hye-Shik Chang has also been submitted, although
it is not expected to be included until after the Python implementation
has proven itself useful enough to be worth optimising.

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 End:

[1] https://mail.python.org/pipermail/python-dev/2006-March/062290.html

[2] Patches 931005, 931007, and 931010.

[3] http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52549