text:
PEP: 509 Title: Add a private version to dict Author: Victor Stinner
<EMAIL_ADDRESS>Status: Superseded Type: Standards Track
Content-Type: text/x-rst Created: 04-Jan-2016 Python-Version: 3.6
Post-History: 08-Jan-2016, 11-Jan-2016, 14-Apr-2016, 19-Apr-2016
Superseded-By: 699 Resolution:
<EMAIL_ADDRESS>
Abstract

Add a new private version to the builtin dict type, incremented at each
dictionary creation and at each dictionary change, to implement fast
guards on namespaces.

Rationale

In Python, the builtin dict type is used by many instructions. For
example, the LOAD_GLOBAL instruction looks up a variable in the global
namespace, or in the builtins namespace (two dict lookups). Python uses
dict for the builtins namespace, globals namespace, type namespaces,
instance namespaces, etc. The local namespace (function namespace) is
usually optimized to an array, but it can be a dict too.

Python is hard to optimize because almost everything is mutable: builtin
functions, function code, global variables, local variables, ... can be
modified at runtime. Implementing optimizations respecting the Python
semantics requires to detect when "something changes": we will call
these checks "guards".

The speedup of optimizations depends on the speed of guard checks. This
PEP proposes to add a private version to dictionaries to implement fast
guards on namespaces.

Dictionary lookups can be skipped if the version does not change, which
is the common case for most namespaces. The version is globally unique,
so checking the version is also enough to verify that the namespace
dictionary was not replaced with a new dictionary.

When the dictionary version does not change, the performance of a guard
does not depend on the number of watched dictionary entries: the
complexity is O(1).

Example of optimization: copy the value of a global variable to function
constants. This optimization requires a guard on the global variable to
check if it was modified after it was copied. If the global variable is
not modified, the function uses the cached copy. If the global variable
is modified, the function uses a regular lookup, and maybe also
deoptimizes the function (to remove the overhead of the guard check for
next function calls).

See the 510 -- Specialized functions with guards <510> for concrete
usage of guards to specialize functions and for a more general rationale
on Python static optimizers.

Guard example

Pseudo-code of a fast guard to check if a dictionary entry was modified
(created, updated or deleted) using a hypothetical
dict_get_version(dict) function:

    UNSET = object()

    class GuardDictKey:
        def __init__(self, dict, key):
            self.dict = dict
            self.key = key
            self.value = dict.get(key, UNSET)
            self.version = dict_get_version(dict)

        def check(self):
            """Return True if the dictionary entry did not change
            and the dictionary was not replaced."""

            # read the version of the dictionary
            version = dict_get_version(self.dict)
            if version == self.version:
                # Fast-path: dictionary lookup avoided
                return True

            # lookup in the dictionary
            value = self.dict.get(self.key, UNSET)
            if value is self.value:
                # another key was modified:
                # cache the new dictionary version
                self.version = version
                return True

            # the key was modified
            return False

Usage of the dict version

Speedup method calls

Yury Selivanov wrote a patch to optimize method calls. The patch depends
on the "implement per-opcode cache in ceval" patch which requires
dictionary versions to invalidate the cache if the globals dictionary or
the builtins dictionary has been modified.

The cache also requires that the dictionary version is globally unique.
It is possible to define a function in a namespace and call it in a
different namespace, using exec() with the globals parameter for
example. In this case, the globals dictionary was replaced and the cache
must also be invalidated.

Specialized functions using guards

PEP 510 proposes an API to support specialized functions with guards. It
allows to implement static optimizers for Python without breaking the
Python semantics.

The fatoptimizer of the FAT Python project is an example of a static
Python optimizer. It implements many optimizations which require guards
on namespaces:

-   Call pure builtins: to replace len("abc") with 3, guards on
    builtins.__dict__['len'] and globals()['len'] are required
-   Loop unrolling: to unroll the loop for i in range(...): ..., guards
    on builtins.__dict__['range'] and globals()['range'] are required
-   etc.

Pyjion

According of Brett Cannon, one of the two main developers of Pyjion,
Pyjion can benefit from dictionary version to implement optimizations.

Pyjion is a JIT compiler for Python based upon CoreCLR (Microsoft .NET
Core runtime).

Cython

Cython can benefit from dictionary version to implement optimizations.

Cython is an optimising static compiler for both the Python programming
language and the extended Cython programming language.

Unladen Swallow

Even if dictionary version was not explicitly mentioned, optimizing
globals and builtins lookup was part of the Unladen Swallow plan:
"Implement one of the several proposed schemes for speeding lookups of
globals and builtins." (source: Unladen Swallow ProjectPlan).

Unladen Swallow is a fork of CPython 2.6.1 adding a JIT compiler
implemented with LLVM. The project stopped in 2011: Unladen Swallow
Retrospective.

Changes

Add a ma_version_tag field to the PyDictObject structure with the C type
PY_UINT64_T, 64-bit unsigned integer. Add also a global dictionary
version.

Each time a dictionary is created, the global version is incremented and
the dictionary version is initialized to the global version.

Each time the dictionary content is modified, the global version must be
incremented and copied to the dictionary version. Dictionary methods
which can modify its content:

-   clear()
-   pop(key)
-   popitem()
-   setdefault(key, value)
-   __delitem__(key)
-   __setitem__(key, value)
-   update(...)

The choice of increasing or not the version when a dictionary method
does not change its content is left to the Python implementation. A
Python implementation can decide to not increase the version to avoid
dictionary lookups in guards. Examples of cases when dictionary methods
don't modify its content:

-   clear() if the dict is already empty
-   pop(key) if the key does not exist
-   popitem() if the dict is empty
-   setdefault(key, value) if the key already exists
-   __delitem__(key) if the key does not exist
-   __setitem__(key, value) if the new value is identical to the current
    value
-   update() if called without argument or if new values are identical
    to current values

Setting a key to a new value equals to the old value is also considered
as an operation modifying the dictionary content.

Two different empty dictionaries must have a different version to be
able to identify a dictionary just by its version. It allows to verify
in a guard that a namespace was not replaced without storing a strong
reference to the dictionary. Using a borrowed reference does not work:
if the old dictionary is destroyed, it is possible that a new dictionary
is allocated at the same memory address. By the way, dictionaries don't
support weak references.

The version increase must be atomic. In CPython, the Global Interpreter
Lock (GIL) already protects dict methods to make changes atomic.

Example using a hypothetical dict_get_version(dict) function:

    >>> d = {}
    >>> dict_get_version(d)
    100
    >>> d['key'] = 'value'
    >>> dict_get_version(d)
    101
    >>> d['key'] = 'new value'
    >>> dict_get_version(d)
    102
    >>> del d['key']
    >>> dict_get_version(d)
    103

The field is called ma_version_tag, rather than ma_version, to suggest
to compare it using version_tag == old_version_tag, rather than
version <= old_version which becomes wrong after an integer overflow.

Backwards Compatibility

Since the PyDictObject structure is not part of the stable ABI and the
new dictionary version not exposed at the Python scope, changes are
backward compatible.

Implementation and Performance

The issue #26058: PEP 509: Add ma_version_tag to PyDictObject contains a
patch implementing this PEP.

On pybench and timeit microbenchmarks, the patch does not seem to add
any overhead on dictionary operations. For example, the following timeit
micro-benchmarks takes 318 nanoseconds before and after the change:

    python3.6 -m timeit 'd={1: 0}; d[2]=0; d[3]=0; d[4]=0; del d[1]; del d[2]; d.clear()'

When the version does not change, PyDict_GetItem() takes 14.8 ns for a
dictionary lookup, whereas a guard check only takes 3.8 ns. Moreover, a
guard can watch for multiple keys. For example, for an optimization
using 10 global variables in a function, 10 dictionary lookups costs 148
ns, whereas the guard still only costs 3.8 ns when the version does not
change (39x as fast).

The fat module implements such guards: fat.GuardDict is based on the
dictionary version.

Integer overflow

The implementation uses the C type PY_UINT64_T to store the version: a
64 bits unsigned integer. The C code uses version++. On integer
overflow, the version is wrapped to 0 (and then continues to be
incremented) according to the C standard.

After an integer overflow, a guard can succeed whereas the watched
dictionary key was modified. The bug only occurs at a guard check if
there are exactly 2 ** 64 dictionary creations or modifications since
the previous guard check.

If a dictionary is modified every nanosecond, 2 ** 64 modifications
takes longer than 584 years. Using a 32-bit version, it only takes 4
seconds. That's why a 64-bit unsigned type is also used on 32-bit
systems. A dictionary lookup at the C level takes 14.8 ns.

A risk of a bug every 584 years is acceptable.

Alternatives

Expose the version at Python level as a read-only __version__ property

The first version of the PEP proposed to expose the dictionary version
as a read-only __version__ property at Python level, and also to add the
property to collections.UserDict (since this type must mimic the dict
API).

There are multiple issues:

-   To be consistent and avoid bad surprises, the version must be added
    to all mapping types. Implementing a new mapping type would require
    extra work for no benefit, since the version is only required on the
    dict type in practice.

-   All Python implementations would have to implement this new
    property, it gives more work to other implementations, whereas they
    may not use the dictionary version at all.

-   Exposing the dictionary version at the Python level can lead the
    false assumption on performances. Checking dict.__version__ at the
    Python level is not faster than a dictionary lookup. A dictionary
    lookup in Python has a cost of 48.7 ns and checking the version has
    a cost of 47.5 ns, the difference is only 1.2 ns (3%):

        $ python3.6 -m timeit -s 'd = {str(i):i for i in range(100)}' 'd["33"] == 33'
        10000000 loops, best of 3: 0.0487 usec per loop
        $ python3.6 -m timeit -s 'd = {str(i):i for i in range(100)}' 'd.__version__ == 100'
        10000000 loops, best of 3: 0.0475 usec per loop

-   The __version__ can be wrapped on integer overflow. It is error
    prone: using dict.__version__ <= guard_version is wrong,
    dict.__version__ == guard_version must be used instead to reduce the
    risk of bug on integer overflow (even if the integer overflow is
    unlikely in practice).

Mandatory bikeshedding on the property name:

-   __cache_token__: name proposed by Alyssa Coghlan, name coming from
    abc.get_cache_token().
-   __version__
-   __version_tag__
-   __timestamp__

Add a version to each dict entry

A single version per dictionary requires to keep a strong reference to
the value which can keep the value alive longer than expected. If we add
also a version per dictionary entry, the guard can only store the entry
version (a simple integer) to avoid the strong reference to the value:
only strong references to the dictionary and to the key are needed.

Changes: add a me_version_tag field to the PyDictKeyEntry structure, the
field has the C type PY_UINT64_T. When a key is created or modified, the
entry version is set to the dictionary version which is incremented at
any change (create, modify, delete).

Pseudo-code of a fast guard to check if a dictionary key was modified
using hypothetical dict_get_version(dict) and
dict_get_entry_version(dict) functions:

    UNSET = object()

    class GuardDictKey:
        def __init__(self, dict, key):
            self.dict = dict
            self.key = key
            self.dict_version = dict_get_version(dict)
            self.entry_version = dict_get_entry_version(dict, key)

        def check(self):
            """Return True if the dictionary entry did not change
            and the dictionary was not replaced."""

            # read the version of the dictionary
            dict_version = dict_get_version(self.dict)
            if dict_version == self.version:
                # Fast-path: dictionary lookup avoided
                return True

            # lookup in the dictionary to read the entry version
            entry_version = get_dict_key_version(dict, key)
            if entry_version == self.entry_version:
                # another key was modified:
                # cache the new dictionary version
                self.dict_version = dict_version
                self.entry_version = entry_version
                return True

            # the key was modified
            return False

The main drawback of this option is the impact on the memory footprint.
It increases the size of each dictionary entry, so the overhead depends
on the number of buckets (dictionary entries, used or not used). For
example, it increases the size of each dictionary entry by 8 bytes on
64-bit system.

In Python, the memory footprint matters and the trend is to reduce it.
Examples:

-   PEP 393 -- Flexible String Representation
-   PEP 412 -- Key-Sharing Dictionary

Add a new dict subtype

Add a new verdict type, subtype of dict. When guards are needed, use the
verdict for namespaces (module namespace, type namespace, instance
namespace, etc.) instead of dict.

Leave the dict type unchanged to not add any overhead (CPU, memory
footprint) when guards are not used.

Technical issue: a lot of C code in the wild, including CPython core,
expecting the exact dict type. Issues:

-   exec() requires a dict for globals and locals. A lot of code use
    globals={}. It is not possible to cast the dict to a dict subtype
    because the caller expects the globals parameter to be modified
    (dict is mutable).
-   C functions call directly PyDict_xxx() functions, instead of calling
    PyObject_xxx() if the object is a dict subtype
-   PyDict_CheckExact() check fails on dict subtype, whereas some
    functions require the exact dict type.
-   Python/ceval.c does not completely supports dict subtypes for
    namespaces

The exec() issue is a blocker issue.

Other issues:

-   The garbage collector has a special code to "untrack" dict
    instances. If a dict subtype is used for namespaces, the garbage
    collector can be unable to break some reference cycles.
-   Some functions have a fast-path for dict which would not be taken
    for dict subtypes, and so it would make Python a little bit slower.

Prior Art

Method cache and type version tag

In 2007, Armin Rigo wrote a patch to implement a cache of methods. It
was merged into Python 2.6. The patch adds a "type attribute cache
version tag" (tp_version_tag) and a "valid version tag" flag to types
(the PyTypeObject structure).

The type version tag is not exposed at the Python level.

The version tag has the C type unsigned int. The cache is a global hash
table of 4096 entries, shared by all types. The cache is global to "make
it fast, have a deterministic and low memory footprint, and be easy to
invalidate". Each cache entry has a version tag. A global version tag is
used to create the next version tag, it also has the C type
unsigned int.

By default, a type has its "valid version tag" flag cleared to indicate
that the version tag is invalid. When the first method of the type is
cached, the version tag and the "valid version tag" flag are set. When a
type is modified, the "valid version tag" flag of the type and its
subclasses is cleared. Later, when a cache entry of these types is used,
the entry is removed because its version tag is outdated.

On integer overflow, the whole cache is cleared and the global version
tag is reset to 0.

See Method cache (issue #1685986) and Armin's method cache optimization
updated for Python 2.6 (issue #1700288).

Globals / builtins cache

In 2010, Antoine Pitrou proposed a Globals / builtins cache (issue
#10401) which adds a private ma_version field to the PyDictObject
structure (dict type), the field has the C type Py_ssize_t.

The patch adds a "global and builtin cache" to functions and frames, and
changes LOAD_GLOBAL and STORE_GLOBAL instructions to use the cache.

The change on the PyDictObject structure is very similar to this PEP.

Cached globals+builtins lookup

In 2006, Andrea Griffini proposed a patch implementing a Cached
globals+builtins lookup optimization. The patch adds a private timestamp
field to the PyDictObject structure (dict type), the field has the C
type size_t.

Thread on python-dev: About dictionary lookup caching (December 2006).

Guard against changing dict during iteration

In 2013, Serhiy Storchaka proposed Guard against changing dict during
iteration (issue #19332) which adds a ma_count field to the PyDictObject
structure (dict type), the field has the C type size_t. This field is
incremented when the dictionary is modified.

PySizer

PySizer: a memory profiler for Python, Google Summer of Code 2005
project by Nick Smallbone.

This project has a patch for CPython 2.4 which adds key_time and
value_time fields to dictionary entries. It uses a global process-wide
counter for dictionaries, incremented each time that a dictionary is
modified. The times are used to decide when child objects first appeared
in their parent objects.

Discussion

Thread on the mailing lists:

-   python-dev: Updated PEP 509
-   python-dev: RFC: PEP 509: Add a private version to dict
-   python-dev: PEP 509: Add a private version to dict (January 2016)
-   python-ideas: RFC: PEP: Add dict.__version__ (January 2016)

Acceptance

The PEP was accepted on 2016-09-07 by Guido van Rossum. The PEP
implementation has since been committed to the repository.

Copyright

This document has been placed in the public domain.


text:
PEP: 228 Title: Reworking Python's Numeric Model Author: Moshe Zadka
<EMAIL_ADDRESS>Guido van Rossum<EMAIL_ADDRESS>Status:
Withdrawn Type: Standards Track Content-Type: text/x-rst Created:
04-Nov-2000 Post-History:

Withdrawal

This PEP has been withdrawn in favor of PEP 3141.

Abstract

Today, Python's numerical model is similar to the C numeric model: there
are several unrelated numerical types, and when operations between
numerical types are requested, coercions happen. While the C rationale
for the numerical model is that it is very similar to what happens at
the hardware level, that rationale does not apply to Python. So, while
it is acceptable to C programmers that 2/3 == 0, it is surprising to
many Python programmers.

NOTE: in the light of recent discussions in the newsgroup, the
motivation in this PEP (and details) need to be extended.

Rationale

In usability studies, one of the least usable aspect of Python was the
fact that integer division returns the floor of the division. This makes
it hard to program correctly, requiring casts to float() in various
parts through the code. Python's numerical model stems from C, while a
model that might be easier to work with can be based on the mathematical
understanding of numbers.

Other Numerical Models

Perl's numerical model is that there is one type of numbers --floating
point numbers. While it is consistent and superficially non-surprising,
it tends to have subtle gotchas. One of these is that printing numbers
is very tricky, and requires correct rounding. In Perl, there is also a
mode where all numbers are integers. This mode also has its share of
problems, which arise from the fact that there is not even an
approximate way of dividing numbers and getting meaningful answers.

Suggested Interface For Python's Numerical Model

While coercion rules will remain for add-on types and classes, the built
in type system will have exactly one Python type -- a number. There are
several things which can be considered "number methods":

1.  isnatural()
2.  isintegral()
3.  isrational()
4.  isreal()
5.  iscomplex()
6.  isexact()

Obviously, a number which answers true to a question from 1 to 5, will
also answer true to any following question. If isexact() is not true,
then any answer might be wrong. (But not horribly wrong: it's close to
the truth.)

Now, there is two thing the models promises for the field operations (+,
-, /, *):

-   If both operands satisfy isexact(), the result satisfies isexact().
-   All field rules are true, except that for not-isexact() numbers,
    they might be only approximately true.

One consequence of these two rules is that all exact calculations are
done as (complex) rationals: since the field laws must hold, then :

    (a/b)*b == a

must hold.

There is built-in function, inexact() which takes a number and returns
an inexact number which is a good approximation. Inexact numbers must be
as least as accurate as if they were using IEEE-754.

Several of the classical Python functions will return exact numbers even
when given inexact numbers: e.g, int().

Coercion

The number type does not define nb_coerce Any numeric operation slot,
when receiving something other then PyNumber, refuses to implement it.

Inexact Operations

The functions in the math module will be allowed to return inexact
results for exact values. However, they will never return a non-real
number. The functions in the cmath module are also allowed to return an
inexact result for an exact argument, and are furthermore allowed to
return a complex result for a real argument.

Numerical Python Issues

People who use Numerical Python do so for high-performance vector
operations. Therefore, NumPy should keep its hardware based numeric
model.

Unresolved Issues

Which number literals will be exact, and which inexact?

How do we deal with IEEE 754 operations? (probably, isnan/isinf should
be methods)

On 64-bit machines, comparisons between ints and floats may be broken
when the comparison involves conversion to float. Ditto for comparisons
between longs and floats. This can be dealt with by avoiding the
conversion to float. (Due to Andrew Koenig.)

Copyright

This document has been placed in the public domain.

license: Public Domain

authors:
Guido van Rossum
Moshe Zadka


text:
PEP: 3154 Title: Pickle protocol version 4 Version: $Revision$
Last-Modified: $Date$ Author: Antoine Pitrou<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type: text/x-rst Created:
11-Aug-2011 Python-Version: 3.4 Post-History: 12-Aug-2011 Resolution:
https://mail.python.org/pipermail/python-dev/2013-November/130439.html

Abstract

Data serialized using the pickle module must be portable across Python
versions. It should also support the latest language features as well as
implementation-specific features. For this reason, the pickle module
knows about several protocols (currently numbered from 0 to 3), each of
which appeared in a different Python version. Using a low-numbered
protocol version allows to exchange data with old Python versions, while
using a high-numbered protocol allows access to newer features and
sometimes more efficient resource use (both CPU time required for
(de)serializing, and disk size / network bandwidth required for data
transfer).

Rationale

The latest current protocol, coincidentally named protocol 3, appeared
with Python 3.0 and supports the new incompatible features in the
language (mainly, unicode strings by default and the new bytes object).
The opportunity was not taken at the time to improve the protocol in
other ways.

This PEP is an attempt to foster a number of incremental improvements in
a new pickle protocol version. The PEP process is used in order to
gather as many improvements as possible, because the introduction of a
new pickle protocol should be a rare occurrence.

Proposed changes

Framing

Traditionally, when unpickling an object from a stream (by calling
load() rather than loads()), many small read() calls can be issued on
the file-like object, with a potentially huge performance impact.

Protocol 4, by contrast, features binary framing. The general structure
of a pickle is thus the following:

    +------+------+
    | 0x80 | 0x04 |              protocol header (2 bytes)
    +------+------+
    |  OP  |                     FRAME opcode (1 byte)
    +------+------+-----------+
    | MM MM MM MM MM MM MM MM |  frame size (8 bytes, little-endian)
    +------+------------------+
    | .... |                     first frame contents (M bytes)
    +------+
    |  OP  |                     FRAME opcode (1 byte)
    +------+------+-----------+
    | NN NN NN NN NN NN NN NN |  frame size (8 bytes, little-endian)
    +------+------------------+
    | .... |                     second frame contents (N bytes)
    +------+
      etc.

To keep the implementation simple, it is forbidden for a pickle opcode
to straddle frame boundaries. The pickler takes care not to produce such
pickles, and the unpickler refuses them. Also, there is no "last frame"
marker. The last frame is simply the one which ends with a STOP opcode.

A well-written C implementation doesn't need additional memory copies
for the framing layer, preserving general (un)pickling efficiency.

Note

How the pickler decides to partition the pickle stream into frames is an
implementation detail. For example, "closing" a frame as soon as it
reaches ~64 KiB is a reasonable choice for both performance and pickle
size overhead.

Binary encoding for all opcodes

The GLOBAL opcode, which is still used in protocol 3, uses the so-called
"text" mode of the pickle protocol, which involves looking for newlines
in the pickle stream. It also complicates the implementation of binary
framing.

Protocol 4 forbids use of the GLOBAL opcode and replaces it with
STACK_GLOBAL, a new opcode which takes its operand from the stack.

Serializing more "lookupable" objects

By default, pickle is only able to serialize module-global functions and
classes. Supporting other kinds of objects, such as unbound methods[1],
is a common request. Actually, third-party support for some of them,
such as bound methods, is implemented in the multiprocessing module[2].

The __qualname__ attribute from PEP 3155 makes it possible to lookup
many more objects by name. Making the STACK_GLOBAL opcode accept
dot-separated names would allow the standard pickle implementation to
support all those kinds of objects.

64-bit opcodes for large objects

Current protocol versions export object sizes for various built-in types
(str, bytes) as 32-bit ints. This forbids serialization of large
data[3]. New opcodes are required to support very large bytes and str
objects.

Native opcodes for sets and frozensets

Many common built-in types (such as str, bytes, dict, list, tuple) have
dedicated opcodes to improve resource consumption when serializing and
deserializing them; however, sets and frozensets don't. Adding such
opcodes would be an obvious improvement. Also, dedicated set support
could help remove the current impossibility of pickling self-referential
sets[4].

Calling __new__ with keyword arguments

Currently, classes whose __new__ mandates the use of keyword-only
arguments can not be pickled (or, rather, unpickled)[5]. Both a new
special method (__getnewargs_ex__) and a new opcode (NEWOBJ_EX) are
needed. The __getnewargs_ex__ method, if it exists, must return a
two-tuple (args, kwargs) where the first item is the tuple of positional
arguments and the second item is the dict of keyword arguments for the
class's __new__ method.

Better string encoding

Short str objects currently have their length coded as a 4-bytes
integer, which is wasteful. A specific opcode with a 1-byte length would
make many pickles smaller.

Smaller memoization

The PUT opcodes all require an explicit index to select in which entry
of the memo dictionary the top-of-stack is memoized. However, in
practice those numbers are allocated in sequential order. A new opcode,
MEMOIZE, will instead store the top-of-stack in at the index equal to
the current size of the memo dictionary. This allows for shorter
pickles, since PUT opcodes are emitted for all non-atomic datatypes.

Summary of new opcodes

These reflect the state of the proposed implementation (thanks mostly to
Alexandre Vassalotti's work):

-   FRAME: introduce a new frame (followed by the 8-byte frame size and
    the frame contents).
-   SHORT_BINUNICODE: push a utf8-encoded str object with a one-byte
    size prefix (therefore less than 256 bytes long).
-   BINUNICODE8: push a utf8-encoded str object with an eight-byte size
    prefix (for strings longer than 2**32 bytes, which therefore cannot
    be serialized using BINUNICODE).
-   BINBYTES8: push a bytes object with an eight-byte size prefix (for
    bytes objects longer than 2**32 bytes, which therefore cannot be
    serialized using BINBYTES).
-   EMPTY_SET: push a new empty set object on the stack.
-   ADDITEMS: add the topmost stack items to the set (to be used with
    EMPTY_SET).
-   FROZENSET: create a frozenset object from the topmost stack items,
    and push it on the stack.
-   NEWOBJ_EX: take the three topmost stack items cls, args and kwargs,
    and push the result of calling cls.__new__(*args, **kwargs).
-   STACK_GLOBAL: take the two topmost stack items module_name and
    qualname, and push the result of looking up the dotted qualname in
    the module named module_name.
-   MEMOIZE: store the top-of-stack object in the memo dictionary with
    an index equal to the current size of the memo dictionary.

Alternative ideas

Prefetching

Serhiy Storchaka suggested to replace framing with a special PREFETCH
opcode (with a 2- or 4-bytes argument) to declare known pickle chunks
explicitly. Large data may be pickled outside such chunks. A naïve
unpickler should be able to skip the PREFETCH opcode and still decode
pickles properly, but good error handling would require checking that
the PREFETCH length falls on an opcode boundary.

Acknowledgments

In alphabetic order:

-   Alexandre Vassalotti, for starting the second PEP 3154
    implementation[6]
-   Serhiy Storchaka, for discussing the framing proposal[7]
-   Stefan Mihaila, for starting the first PEP 3154 implementation as a
    Google Summer of Code project mentored by Alexandre Vassalotti[8].

References

Copyright

This document has been placed in the public domain.

[1] "pickle should support methods": http://bugs.python.org/issue9276

[2] Lib/multiprocessing/forking.py:
http://hg.python.org/cpython/file/baea9f5f973c/Lib/multiprocessing/forking.py#l54

[3] "pickle not 64-bit ready": http://bugs.python.org/issue11564

[4] "Cannot pickle self-referencing sets":
http://bugs.python.org/issue9269

[5] "pickle/copyreg doesn't support keyword only arguments in __new__":
http://bugs.python.org/issue4727

[6] Implement PEP 3154, by Alexandre Vassalotti
http://bugs.python.org/issue17810

[7] Implement PEP 3154, by Alexandre Vassalotti
http://bugs.python.org/issue17810

[8] Implement PEP 3154, by Stefan Mihaila
http://bugs.python.org/issue15642


text:
PEP: 270 Title: uniq method for list objects Author: Jason Petrone
<EMAIL_ADDRESS>Status: Rejected Type: Standards Track Content-Type:
text/x-rst Created: 21-Aug-2001 Python-Version: 2.2 Post-History:

Notice

This PEP is withdrawn by the author. He writes:

  Removing duplicate elements from a list is a common task, but there
  are only two reasons I can see for making it a built-in. The first is
  if it could be done much faster, which isn't the case. The second is
  if it makes it significantly easier to write code. The introduction of
  sets.py eliminates this situation since creating a sequence without
  duplicates is just a matter of choosing a different data structure: a
  set instead of a list.

As described in PEP 218, sets are being added to the standard library
for Python 2.3.

Abstract

This PEP proposes adding a method for removing duplicate elements to the
list object.

Rationale

Removing duplicates from a list is a common task. I think it is useful
and general enough to belong as a method in list objects. It also has
potential for faster execution when implemented in C, especially if
optimization using hashing or sorted cannot be used.

On comp.lang.python there are many, many, posts[1] asking about the best
way to do this task. It's a little tricky to implement optimally and it
would be nice to save people the trouble of figuring it out themselves.

Considerations

Tim Peters suggests trying to use a hash table, then trying to sort, and
finally falling back on brute force[2]. Should uniq maintain list order
at the expense of speed?

Is it spelled 'uniq' or 'unique'?

Reference Implementation

I've written the brute force version. It's about 20 lines of code in
listobject.c. Adding support for hash table and sorted duplicate removal
would only take another hour or so.

References

Copyright

This document has been placed in the public domain.

[1] https://groups.google.com/forum/#!searchin/comp.lang.python/duplicates

[2] Tim Peters unique() entry in the Python cookbook:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/52560/index_txt


text:
PEP: 42 Title: Feature Requests Author: Jeremy Hylton
<EMAIL_ADDRESS>Status: Withdrawn Type: Process Created:
12-Sep-2000 Post-History:

It is obsolete. All new feature requests should either go to the Python
bug tracker for very simple requests or the Ideas Discourse category for
everything else. The rest of this document is retained for historical
purposes only.

Introduction

This PEP contains a list of feature requests that may be considered for
future versions of Python. Large feature requests should not be included
here, but should be described in separate PEPs; however a large feature
request that doesn't have its own PEP can be listed here until its own
PEP is created. See PEP 0 for details.

This PEP was created to allow us to close bug reports that are really
feature requests. Marked as Open, they distract from the list of real
bugs (which should ideally be less than a page). Marked as Closed, they
tend to be forgotten. The procedure now is: if a bug report is really a
feature request, add the feature request to this PEP; mark the bug as
"feature request", "later", and "closed"; and add a comment to the bug
saying that this is the case (mentioning the PEP explicitly). It is also
acceptable to move large feature requests directly from the bugs
database to a separate PEP.

This PEP should really be separated into four different categories
(categories due to Laura Creighton):

1.  BDFL rejects as a bad idea. Don't come back with it.

2.  BDFL will put in if somebody writes the code. (Or at any rate, BDFL
    will say 'change this and I will put it in' if you show up with
    code.)

    possibly divided into:

      a)  BDFL would really like to see some code!
      b)  BDFL is never going to be enthusiastic about this, but will
          work it in when it's easy.

3.  If you show up with code, BDFL will make a pronouncement. It might
    be ICK.

4.  This is too vague. This is rejected, but only on the grounds of
    vagueness. If you like this enhancement, make a new PEP.

Core Language / Builtins

-   The parser should handle more deeply nested parse trees.

    The following will fail -- eval("["*50 + "]"*50) -- because the
    parser has a hard-coded limit on stack size. This limit should be
    raised or removed. Removal would be hard because the current
    compiler can overflow the C stack if the nesting is too deep.

    https://bugs.python.org/issue215555

-   Non-accidental IEEE-754 support (Infs, NaNs, settable traps, etc).
    Big project.

-   Windows: Trying to create (or even access) files with certain magic
    names can hang or crash Windows systems. This is really a bug in the
    OSes, but some apps try to shield users from it. When it happens,
    the symptoms are very confusing.

    Hang using files named prn.txt, etc
    https://bugs.python.org/issue481171

-   eval and free variables: It might be useful if there was a way to
    pass bindings for free variables to eval when a code object with
    free variables is passed. https://bugs.python.org/issue443866

Standard Library

-   The urllib module should support proxies which require
    authentication. See SourceForge bug #210619 for information:

    https://bugs.python.org/issue210619

-   os.rename() should be modified to handle EXDEV errors on platforms
    that don't allow rename() to operate across filesystem boundaries by
    copying the file over and removing the original. Linux is one system
    that requires this treatment.

    https://bugs.python.org/issue212317

-   signal handling doesn't always work as expected. E.g. if
    sys.stdin.readline() is interrupted by a (returning) signal handler,
    it returns "". It would be better to make it raise an exception
    (corresponding to EINTR) or to restart. But these changes would have
    to applied to all places that can do blocking interruptible I/O. So
    it's a big project.

    https://bugs.python.org/issue210599

-   Extend Windows utime to accept directory paths.

    https://bugs.python.org/issue214245

-   Extend copy.py to module & function types.

    https://bugs.python.org/issue214553

-   Better checking for bad input to marshal.load*().

    https://bugs.python.org/issue214754

-   rfc822.py should be more lenient than the spec in the types of
    address fields it parses. Specifically, an invalid address of the
    form "From: Amazon.com<EMAIL_ADDRESS>should be parsed
    correctly.

    https://bugs.python.org/issue210678

-   cgi.py's FieldStorage class should be more conservative with memory
    in the face of large binary file uploads.

    https://bugs.python.org/issue210674

    There are two issues here: first, because
    read_lines_to_outerboundary() uses readline() it is possible that a
    large amount of data will be read into memory for a binary file
    upload. This should probably look at the Content-Type header of the
    section and do a chunked read if it's a binary type.

    The second issue was related to the self.lines attribute, which was
    removed in revision 1.56 of cgi.py (see also):

    https://bugs.python.org/issue219806

-   urllib should support proxy definitions that contain just the host
    and port

    https://bugs.python.org/issue210849

-   urlparse should be updated to comply with 2396, which defines
    optional parameters for each segment of the path.

    https://bugs.python.org/issue210834

-   The exceptions raised by pickle and cPickle are currently different;
    these should be unified (probably the exceptions should be defined
    in a helper module that's imported by both). [No bug report; I just
    thought of this.]

-   More standard library routines should support Unicode. For example,
    urllib.quote() could convert Unicode strings to UTF-8 and then do
    the usual %HH conversion. But this is not the only one!

    https://bugs.python.org/issue216716

-   There should be a way to say that you don't mind if str() or
    __str__() return a Unicode string object. Or a different function --
    ustr() has been proposed. Or something...

    http://sf.net/patch/?func=detailpatch&patch_id=101527&group_id=5470

-   Killing a thread from another thread. Or maybe sending a signal. Or
    maybe raising an asynchronous exception.

    https://bugs.python.org/issue221115

-   The debugger (pdb) should understand packages.

    https://bugs.python.org/issue210631

-   Jim Fulton suggested the following:

        I wonder if it would be a good idea to have a new kind of
        temporary file that stored data in memory unless:

        - The data exceeds some size, or

        - Somebody asks for a fileno.

        Then the cgi module (and other apps) could use this thing in a
        uniform way.

    https://bugs.python.org/issue415692

-   Jim Fulton pointed out that binascii's b2a_base64() function has
    situations where it makes sense not to append a newline, or to
    append something else than a newline.

    Proposal:

    -   add an optional argument giving the delimiter string to be
        appended, defaulting to "\n"
    -   possibly special-case None as the delimiter string to avoid
        adding the pad bytes too???

    https://bugs.python.org/issue415694

-   pydoc should be integrated with the HTML docs, or at least be able
    to link to them.

    https://bugs.python.org/issue405554

-   Distutils should deduce dependencies for .c and .h files.

    https://bugs.python.org/issue472881

-   asynchat is buggy in the face of multithreading.

    https://bugs.python.org/issue595217

-   It would be nice if the higher level modules (httplib, smtplib,
    nntplib, etc.) had options for setting socket timeouts.

    https://bugs.python.org/issue723287

-   The curses library is missing two important calls: newterm() and
    delscreen()

    https://bugs.python.org/issue665572, http://bugs.debian.org/175590

-   It would be nice if the built-in SSL socket type could be used for
    non-blocking SSL I/O. Currently packages such as Twisted which
    implement async servers using SSL have to require third-party
    packages such as pyopenssl.

-   reST as a standard library module

-   The import lock could use some redesign.

    https://bugs.python.org/issue683658

-   A nicer API to open text files, replacing the ugly (in some people's
    eyes) "U" mode flag. There's a proposal out there to have a new
    built-in type textfile(filename, mode, encoding). (Shouldn't it have
    a bufsize argument too?)

-   Support new widgets and/or parameters for Tkinter

-   For a class defined inside another class, the __name__ should be
    "outer.inner", and pickling should work. (GvR is no longer certain
    this is easy or even right.)

    https://bugs.python.org/issue633930

-   Decide on a clearer deprecation policy (especially for modules) and
    act on it.

    https://mail.python.org/pipermail/python-dev/2002-April/023165.html

-   Provide alternatives for common uses of the types module; Skip
    Montanaro has posted a proto-PEP for this idea:

    https://mail.python.org/pipermail/python-dev/2002-May/024346.html

-   Use pending deprecation for the types and string modules. This
    requires providing alternatives for the parts that aren't covered
    yet (e.g. string.whitespace and types.TracebackType). It seems we
    can't get consensus on this.

-   Lazily tracking tuples?

    https://mail.python.org/pipermail/python-dev/2002-May/023926.html
    https://bugs.python.org/issue558745

-   Make 'as' a keyword. It has been a pseudo-keyword long enough. (It's
    deprecated in 2.5, and will become a keyword in 2.6.)

C API wishes

-   Add C API functions to help Windows users who are building embedded
    applications where the FILE * structure does not match the FILE *
    the interpreter was compiled with.

    https://bugs.python.org/issue210821

    See this bug report for a specific suggestion that will allow a
    Borland C++ builder application to interact with a python.dll build
    with MSVC.

Tools

-   Python could use a GUI builder.

    https://bugs.python.org/issue210820

Building and Installing

-   Modules/makesetup should make sure the 'config.c' file it generates
    from the various Setup files, is valid C. It currently accepts
    module names with characters that are not allowable in Python or C
    identifiers.

    https://bugs.python.org/issue216326

-   Building from source should not attempt to overwrite the
    Include/graminit.h and Parser/graminit.c files, at least for people
    downloading a source release rather than working from Subversion or
    snapshots. Some people find this a problem in unusual build
    environments.

    https://bugs.python.org/issue219221

-   The configure script has probably grown a bit crufty with age and
    may not track autoconf's more recent features very well. It should
    be looked at and possibly cleaned up.

    https://mail.python.org/pipermail/python-dev/2004-January/041790.html

-   Make Python compliant to the FHS (the Filesystem Hierarchy Standard)

    http://bugs.python.org/issue588756


text:
PEP: 745 Title: Python 3.14 Release Schedule Author: Hugo van Kemenade
Status: Active Type: Informational Topic: Release Created: 24-Apr-2024
Python-Version: 3.14

Abstract

This document describes the development and release schedule for Python
3.14.

Release Manager and Crew

-   3.14 Release Manager: Hugo van Kemenade
-   Windows installers: Steve Dower
-   Mac installers: Ned Deily
-   Documentation: Julien Palard

Release Schedule

3.14.0 schedule

The dates below use a 17-month development period that results in a
12-month release cadence between feature versions, as defined by PEP
602.

Actual:

-   3.14 development begins: Wednesday, 2024-05-08
-   3.14.0 alpha 1: Tuesday, 2024-10-15

Expected:

-   3.14.0 alpha 2: Tuesday, 2024-11-19
-   3.14.0 alpha 3: Tuesday, 2024-12-17
-   3.14.0 alpha 4: Tuesday, 2025-01-14
-   3.14.0 alpha 5: Tuesday, 2025-02-11
-   3.14.0 alpha 6: Friday, 2025-03-14
-   3.14.0 alpha 7: Tuesday, 2025-04-08
-   3.14.0 beta 1: Tuesday, 2025-05-06 (No new features beyond this
    point.)
-   3.14.0 beta 2: Tuesday, 2025-05-27
-   3.14.0 beta 3: Tuesday, 2025-06-17
-   3.14.0 beta 4: Tuesday, 2025-07-08
-   3.14.0 candidate 1: Tuesday, 2025-07-22
-   3.14.0 candidate 2: Tuesday, 2025-08-26
-   3.14.0 final: Wednesday, 2025-10-01

Subsequent bugfix releases every two months.

3.14 lifespan

Python 3.14 will receive bugfix updates approximately every two months
for approximately 24 months. Around the time of the release of 3.16.0
final, the final 3.14 bugfix update will be released. After that, it is
expected that security updates (source only) will be released until five
years after the release of 3.14.0 final, so until approximately October
2030.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 3102 Title: Keyword-Only Arguments Version: $Revision$
Last-Modified: $Date$ Author: Talin<EMAIL_ADDRESS>Status: Final
Type: Standards Track Content-Type: text/x-rst Created: 22-Apr-2006
Python-Version: 3.0 Post-History: 28-Apr-2006, 19-May-2006

Abstract

This PEP proposes a change to the way that function arguments are
assigned to named parameter slots. In particular, it enables the
declaration of "keyword-only" arguments: arguments that can only be
supplied by keyword and which will never be automatically filled in by a
positional argument.

Rationale

The current Python function-calling paradigm allows arguments to be
specified either by position or by keyword. An argument can be filled in
either explicitly by name, or implicitly by position.

There are often cases where it is desirable for a function to take a
variable number of arguments. The Python language supports this using
the 'varargs' syntax (*name), which specifies that any 'left over'
arguments be passed into the varargs parameter as a tuple.

One limitation on this is that currently, all of the regular argument
slots must be filled before the vararg slot can be.

This is not always desirable. One can easily envision a function which
takes a variable number of arguments, but also takes one or more
'options' in the form of keyword arguments. Currently, the only way to
do this is to define both a varargs argument, and a 'keywords' argument
(**kwargs), and then manually extract the desired keywords from the
dictionary.

Specification

Syntactically, the proposed changes are fairly simple. The first change
is to allow regular arguments to appear after a varargs argument:

    def sortwords(*wordlist, case_sensitive=False):
        ...

This function accepts any number of positional arguments, and it also
accepts a keyword option called 'case_sensitive'. This option will never
be filled in by a positional argument, but must be explicitly specified
by name.

Keyword-only arguments are not required to have a default value. Since
Python requires that all arguments be bound to a value, and since the
only way to bind a value to a keyword-only argument is via keyword, such
arguments are therefore 'required keyword' arguments. Such arguments
must be supplied by the caller, and they must be supplied via keyword.

The second syntactical change is to allow the argument name to be
omitted for a varargs argument. The meaning of this is to allow for
keyword-only arguments for functions that would not otherwise take a
varargs argument:

    def compare(a, b, *, key=None):
        ...

The reasoning behind this change is as follows. Imagine for a moment a
function which takes several positional arguments, as well as a keyword
argument:

    def compare(a, b, key=None):
        ...

Now, suppose you wanted to have 'key' be a keyword-only argument. Under
the above syntax, you could accomplish this by adding a varargs argument
immediately before the keyword argument:

    def compare(a, b, *ignore, key=None):
        ...

Unfortunately, the 'ignore' argument will also suck up any erroneous
positional arguments that may have been supplied by the caller. Given
that we'd prefer any unwanted arguments to raise an error, we could do
this:

    def compare(a, b, *ignore, key=None):
        if ignore:  # If ignore is not empty
            raise TypeError

As a convenient shortcut, we can simply omit the 'ignore' name, meaning
'don't allow any positional arguments beyond this point'.

(Note: After much discussion of alternative syntax proposals, the BDFL
has pronounced in favor of this 'single star' syntax for indicating the
end of positional parameters.)

Function Calling Behavior

The previous section describes the difference between the old behavior
and the new. However, it is also useful to have a description of the new
behavior that stands by itself, without reference to the previous model.
So this next section will attempt to provide such a description.

When a function is called, the input arguments are assigned to formal
parameters as follows:

-   For each formal parameter, there is a slot which will be used to
    contain the value of the argument assigned to that parameter.
-   Slots which have had values assigned to them are marked as 'filled'.
    Slots which have no value assigned to them yet are considered
    'empty'.
-   Initially, all slots are marked as empty.
-   Positional arguments are assigned first, followed by keyword
    arguments.
-   For each positional argument:
    -   Attempt to bind the argument to the first unfilled parameter
        slot. If the slot is not a vararg slot, then mark the slot as
        'filled'.
    -   If the next unfilled slot is a vararg slot, and it does not have
        a name, then it is an error.
    -   Otherwise, if the next unfilled slot is a vararg slot then all
        remaining non-keyword arguments are placed into the vararg slot.
-   For each keyword argument:
    -   If there is a parameter with the same name as the keyword, then
        the argument value is assigned to that parameter slot. However,
        if the parameter slot is already filled, then that is an error.
    -   Otherwise, if there is a 'keyword dictionary' argument, the
        argument is added to the dictionary using the keyword name as
        the dictionary key, unless there is already an entry with that
        key, in which case it is an error.
    -   Otherwise, if there is no keyword dictionary, and no matching
        named parameter, then it is an error.
-   Finally:
    -   If the vararg slot is not yet filled, assign an empty tuple as
        its value.
    -   For each remaining empty slot: if there is a default value for
        that slot, then fill the slot with the default value. If there
        is no default value, then it is an error.

In accordance with the current Python implementation, any errors
encountered will be signaled by raising TypeError. (If you want
something different, that's a subject for a different PEP.)

Backwards Compatibility

The function calling behavior specified in this PEP is a superset of the
existing behavior - that is, it is expected that any existing programs
will continue to work.

Copyright

This document has been placed in the public domain.


text:
PEP: 753 Title: Uniform project URLs in core metadata Author: William
Woodruff<EMAIL_ADDRESS>Facundo Tuesca
<EMAIL_ADDRESS>Sponsor: Barry Warsaw <barry at
python.org> PEP-Delegate: Paul Moore <p.f.moore at gmail.com>
Discussions-To:
https://discuss.python.org/t/pep-753-uniform-urls-in-core-metadata/62792
Status: Accepted Type: Standards Track Topic: Packaging Created:
29-Aug-2024 Post-History: 26-Aug-2024, 03-Sep-2024 Resolution:
10-Oct-2024

Abstract

This PEP recommends two discrete changes to the processing of core
metadata by indices (such as PyPI) and other core metadata consumers:

-   Deprecation of the Home-page and Download-URL fields in favor of
    their Project-URL equivalents;
-   A set of conventions <conventions-for-labels> for normalizing and
    assigning semantics to Project-URL labels during consumer-side
    metadata processing.

Rationale and Motivation

Python's standard core metadata <packaging:core-metadata> has gone
through many years of revisions, with various standardized milestone
versions.

These revisions of the core metadata have introduced various mechanisms
for expressing a package's relationship to external resources, via URLs:

1.  Metadata 1.0 introduced Home-page, a single-use field containing a
    URL to the distribution's home page.

        Home-page: https://example.com/sampleproject

2.  Metadata 1.1 introduced Download-URL, a complementary single-use
    field containing a URL suitable for downloading the current
    distribution.

        Download-URL: https://example.com/sampleproject/sampleproject-1.2.3.tar.gz

3.  Metadata 1.2 introduced Project-URL, a multiple-use field containing
    a label-and-URL pair. Each label is free text conveying the URL's
    semantics.

        Project-URL: Homepage, https://example.com/sampleproject
        Project-URL: Download, https://example.com/sampleproject/sampleproject-1.2.3.tar.gz
        Project-URL: Documentation, https://example.com/sampleproject/docs

Metadata 2.1, 2.2, and 2.3 leave the behavior of these fields as
originally specified.

Because Project-URL allows free text labels and is multiple-use,
informal conventions have arisen for representing the values of
Home-page and Download-URL within Project-URL instead.

These conventions have seen significant adoption, with PEP 621
explicitly choosing to provide only a project.urls table rather than a
project.home-page field. From PEP 621's rejected ideas:

  While the core metadata supports it, having a single field for a
  project's URL while also supporting a full table seemed redundant and
  confusing.

This PEP exists to formalize the informal conventions that have arisen,
as well as explicitly document Home-page and Download-URL as deprecated
in favor of equivalent Project-URL representations.

Specification

This PEP proposes that Home-page and Download-URL be considered
deprecated. This deprecation has implications for both package metadata
producers (e.g. build backends and packaging tools) and package indices
(e.g. PyPI).

Metadata producers

This PEP stipulates the following for metadata producers:

-   When generating metadata 1.2 or later, producers SHOULD emit only
    Project-URL, and SHOULD NOT emit Home-page or Download-URL fields.

These stipulations do not change the optionality of URL fields in core
metadata. In other words, producers MAY choose to omit Project-URL
entirely at their discretion.

This PEP does not propose the outright removal of support for Home-page
or Download-URL. However, see future-considerations for thoughts on how
a new (as of yet unspecified) major core metadata version could complete
the deprecation cycle via removal of these deprecated fields.

Similarly, this PEP does not propose that metadata producers emit
normalized labels <label-normalization>. Label normalization is
performed solely on the processing and consumption side (i.e. within
indices and other consumers of distribution metadata).

Package indices

This PEP stipulates the following for package indices:

-   When interpreting a distribution's metadata of version 1.2 or later
    (e.g. for rendering on a web page), the index MUST prefer
    Project-URL fields as a source of URLs over Home-page and
    Download-URL, even if the latter are explicitly provided.
-   If a distribution's metadata contains only the Home-page and
    Download-URL fields, the index MAY choose to ignore those fields and
    behave as though no URLs were provided in the metadata. In this
    case, the index SHOULD present an appropriate warning or
    notification to the uploading party.
    -   The mechanism for presenting this warning or notification is not
        specified, since it will vary by index. By way of example, an
        index may choose to present a warning in the HTTP response to an
        upload request, or send an email or other notification to the
        maintainer(s) of the project.
-   If a distribution's metadata contains both sets of fields, the index
    MAY choose to reject the distribution outright. However, this is NOT
    RECOMMENDED until a future unspecified major metadata version
    formally removes support for Home-page and Download-URL.
-   Any changes to the interpretation of metadata of version 1.2 or
    later that result in previously recognized URLs no longer being
    recognized SHOULD NOT be retroactively applied to previously
    uploaded packages.

These stipulations do not change the optionality of URL processing by
indices. In other words, an index that does not process URLs within
uploaded distributions may continue to ignore all URL fields entirely.

Similarly, these stipulations do not imply that the index should persist
any normalizations that it performs to a distribution's metadata. In
other words, this PEP stipulates label normalization for the purpose of
user presentation, not for the purpose of modifying an uploaded
distribution or its "sidecar" PEP 658 metadata file.

Conventions for Project-URL labels

The deprecations proposed above require a formalization of the currently
informal relationship between Home-page, Download-URL, and their
Project-URL equivalents.

This formalization has two parts:

1.  A set of rules for normalizing Project-URL labels during index-side
    processing;
2.  A set of "well-known" normalized label values that indices may
    specialize URL presentation for.

Label normalization

The core metadata specification stipulates that Project-URL labels are
free text, limited to 32 characters.

This PEP proposes adding the concept of a "normalized" label to the core
metadata specification. Label normalization is defined via the following
Python function:

    import string
    def normalize_label(label: str) -> str:
        chars_to_remove = string.punctuation + string.whitespace
        removal_map = str.maketrans("", "", chars_to_remove)
        return label.translate(removal_map).lower()

In plain language: a label is normalized by deleting all ASCII
punctuation and whitespace, and then converting the result to lowercase.

The following table shows examples of labels before (raw) and after
normalization:

  Raw           Normalized
  ------------- ------------
  Homepage      homepage
  Home-page     homepage
  Home page     homepage
  Change_Log    changelog
  What's New?   whatsnew

When processing distribution metadata, package indices SHOULD perform
label normalization to determine if a given label is
well known <well-known> for subsequent special processing. Labels that
are not well-known MUST be processed in their un-normalized form.

Normalization does not change pre-existing semantics around duplicated
Project-URL labels. In other words, normalization may result in
duplicate labels in the project's metadata, but only in the same manner
that was already permitted (since the
core metadata specification <packaging:core-metadata> does not stipulate
that labels are unique).

Excerpted examples of normalized metadata fields are provided in
Appendix A <appendix-a>.

Well-known labels

In addition to the normalization rules above, this PEP proposes a fixed
(but extensible) set of "well-known" Project-URL labels, as well as
aliases and human-readable equivalents.

The following table lists these labels, in normalized form:

  Label (Human-readable equivalent)   Description                                                               Aliases
  ----------------------------------- ------------------------------------------------------------------------- ------------------------------------------------
  homepage (Homepage)                 The project's home page                                                   (none)
  source (Source Code)                The project's hosted source code or repository                            repository, sourcecode, github
  download (Download)                 A download URL for the current distribution, equivalent to Download-URL   (none)
  changelog (Changelog)               The project's comprehensive changelog                                     changes, whatsnew, history
  releasenotes (Release Notes)        The project's curated release notes                                       (none)
  documentation (Documentation)       The project's online documentation                                        docs
  issues (Issue Tracker)              The project's bug tracker                                                 bugs, issue, tracker, issuetracker, bugtracker
  funding (Funding)                   Funding Information                                                       sponsor, donate, donation

Indices MAY choose to use the human-readable equivalents suggested above
in their UI elements, if appropriate. Alternatively, indices MAY choose
their own appropriate human-readable equivalents for UI elements.

Packagers and metadata producers MAY choose to use any label that
normalizes to these labels (or their aliases) to communicate specific
URL intents to package indices and downstreams.

Similarly, indices MAY choose to specialize their rendering or
presentation of URLs with these labels, e.g. by presenting an
appropriate icon or tooltip for each label.

Indices MAY also specialize the rendering or presentation of additional
labels or URLs, including (but not limited to), labels that start with a
well-known label, and URLs that refer to a known service provider domain
(e.g. for documentation hosting or issue tracking).

This PEP recognizes that the list of well-known labels is unlikely to
remain static, and that subsequent additions to it should not require
the overhead associated with a formal PEP process or new metadata
version. As such, this PEP proposes that the list above become a
"living" list within the
PyPA specifications <packaging:packaging-specifications>.

Backwards Compatibility

Limited Impact

This PEP is expected to have little to no impact on existing packaging
tooling or package indices:

-   Packaging tooling: no changes to the correctness or well-formedness
    of the core metadata. This PEP proposes deprecations as well as
    behavioral refinements, but all currently (and historically)
    produced metadata will continue to be valid per the rules of its
    respective version.
-   Package indices: indices will continue to expect well-formed core
    metadata, with no behavioral changes. Indices MAY choose to emit
    warnings or notifications on the presence of now-deprecated fields,
    per above <package-indices>.

Future Considerations

This PEP does not stipulate or require any future metadata changes.

However, per metadata-producers and conventions-for-labels, we identify
the following potential future goals for a new major release of the core
metadata standards:

-   Outright removal of support for Home-page and Download-URL in the
    next major core metadata version. If removed, package indices and
    consumers MUST reject metadata containing these fields when said
    metadata is of the new major version.
-   Enforcement of label normalization. If enforced, package producers
    MUST emit only normalized Project-URL labels when generating
    distribution metadata, and package indices and consumers MUST reject
    distributions containing non-normalized labels. Note: requiring
    normalization merely restricts labels to lowercase text, and
    excludes whitespace and punctuation. It does NOT restrict project
    URLs solely to the use of "well-known" labels.

These potential changes would be backwards incompatible, hence their
inclusion only in this section. Acceptance of this PEP does NOT commit
any future metadata revision to actually making these changes.

Security Implications

This PEP does not identify any positive or negative security
implications associated with deprecating Home-page and Download-URL or
with label normalization.

How To Teach This

The changes in this PEP should be transparent to the majority of the
packaging ecosystem's userbase; the primary beneficiaries of this PEP's
changes are packaging tooling authors and index maintainers, who will be
able to reduce the number of unique URL fields produced and checked.

A small number of package maintainers may observe new warnings or
notifications from their index of choice, should the index choose to
ignore Home-page and Download-URL as suggested. Similarly, a small
number of package maintainers may observe that their index of choice no
longer renders their URLs, if only present in the deprecated fields.
However, no package maintainers should observe rejected package uploads
or other breaking changes to packaging workflows due to this PEP's
proposed changes.

Anybody who observes warnings or changes to the presentation of URLs on
indices can be taught about this PEP's behavior via official packaging
resources, such as the Python Packaging User Guide <packaging> and
PyPI's user documentation, the latter of which already contains an
informal description of PyPI's URL handling behavior.

If this PEP is accepted, the authors of this PEP will coordinate to
update and cross-link the resources mentioned above.

Appendix A: Label normalization examples

This appendix provides an illustrative excerpt of a distribution's
metadata, before and after index-side processing:

Before:

    Project-URL: Home-page, https://example.com
    Project-URL: Homepage, https://another.example.com
    Project-URL: Source, https://github.com/example/example
    Project-URL: GitHub, https://github.com/example/example
    Project-URL: Another Service, https://custom.example.com

After:

    Project-URL: homepage, https://example.com
    Project-URL: homepage, https://another.example.com
    Project-URL: source, https://github.com/example/example
    Project-URL: github, https://github.com/example/example
    Project-URL: Another Service, https://custom.example.com

In particular, observe:

-   Normalized duplicates are preserved (both Home-page and Homepage
    become homepage);
-   Source and GitHub are both normalized into their respective forms,
    but github is not transformed into source.
-   Another Service is not normalized, since its normal form
    (anotherservice) is not a well-known label <well-known>.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 20 Title: The Zen of Python Author: Tim Peters
<EMAIL_ADDRESS>Status: Active Type: Informational Content-Type:
text/x-rst Created: 19-Aug-2004 Post-History: 22-Aug-2004

Abstract

Long time Pythoneer Tim Peters succinctly channels the BDFL's guiding
principles for Python's design into 20 aphorisms, only 19 of which have
been written down.

The Zen of Python

    Beautiful is better than ugly.
    Explicit is better than implicit.
    Simple is better than complex.
    Complex is better than complicated.
    Flat is better than nested.
    Sparse is better than dense.
    Readability counts.
    Special cases aren't special enough to break the rules.
    Although practicality beats purity.
    Errors should never pass silently.
    Unless explicitly silenced.
    In the face of ambiguity, refuse the temptation to guess.
    There should be one-- and preferably only one --obvious way to do it.
    Although that way may not be obvious at first unless you're Dutch.
    Now is better than never.
    Although never is often better than *right* now.
    If the implementation is hard to explain, it's a bad idea.
    If the implementation is easy to explain, it may be a good idea.
    Namespaces are one honking great idea -- let's do more of those!

Easter Egg

    >>> import this

References

Originally posted to<EMAIL_ADDRESS>under a
thread called "The Way of Python"

Copyright

This document has been placed in the public domain.


text:
PEP: 235 Title: Import on Case-Insensitive Platforms Version: $Revision$
Last-Modified: $Date$ Author: Tim Peters<EMAIL_ADDRESS>Status:
Final Type: Standards Track Content-Type: text/x-rst Created:
21-Feb-2001 Python-Version: 2.1 Post-History: 16-Feb-2001

Note

This is essentially a retroactive PEP: the issue came up too late in the
2.1 release process to solicit wide opinion before deciding what to do,
and can't be put off until 2.2 without also delaying the Cygwin and
MacOS X ports.

Motivation

File systems vary across platforms in whether or not they preserve the
case of filenames, and in whether or not the platform C library
file-opening functions do or don't insist on case-sensitive matches:

    case-preserving     case-destroying
    +-------------------+------------------+
    case-sensitive   | most Unix flavors | brrrrrrrrrr      |
    +-------------------+------------------+
    case-insensitive | Windows           | some unfortunate |
    | MacOSX HFS+       | network schemes  |
    | Cygwin            |                  |
    |                   | OpenVMS          |
    +-------------------+------------------+

In the upper left box, if you create "fiLe" it's stored as "fiLe", and
only open("fiLe") will open it (open("file") will not, nor will the 14
other variations on that theme).

In the lower right box, if you create "fiLe", there's no telling what
it's stored as -- but most likely as "FILE" -- and any of the 16 obvious
variations on open("FilE") will open it.

The lower left box is a mix: creating "fiLe" stores "fiLe" in the
platform directory, but you don't have to match case when opening it;
any of the 16 obvious variations on open("FILe") work.

NONE OF THAT IS CHANGING! Python will continue to follow platform
conventions w.r.t. whether case is preserved when creating a file, and
w.r.t. whether open() requires a case-sensitive match. In practice, you
should always code as if matches were case-sensitive, else your program
won't be portable.

What's proposed is to change the semantics of Python "import"
statements, and there only in the lower left box.

Current Lower-Left Semantics

Support for MacOSX HFS+, and for Cygwin, is new in 2.1, so nothing is
changing there. What's changing is Windows behavior. Here are the
current rules for import on Windows:

1.  Despite that the filesystem is case-insensitive, Python insists on a
    case-sensitive match. But not in the way the upper left box works:
    if you have two files, FiLe.py and file.py on sys.path, and do :

        import file

    then if Python finds FiLe.py first, it raises a NameError. It does
    not go on to find file.py; indeed, it's impossible to import any but
    the first case-insensitive match on sys.path, and then only if case
    matches exactly in the first case-insensitive match.

2.  An ugly exception: if the first case-insensitive match on sys.path
    is for a file whose name is entirely in upper case (FILE.PY or
    FILE.PYC or FILE.PYO), then the import silently grabs that, no
    matter what mixture of case was used in the import statement. This
    is apparently to cater to miserable old filesystems that really fit
    in the lower right box. But this exception is unique to Windows, for
    reasons that may or may not exist.

3.  And another exception: if the environment variable PYTHONCASEOK
    exists, Python silently grabs the first case-insensitive match of
    any kind.

So these Windows rules are pretty complicated, and neither match the
Unix rules nor provide semantics natural for the native filesystem. That
makes them hard to explain to Unix or Windows users. Nevertheless,
they've worked fine for years, and in isolation there's no compelling
reason to change them.

However, that was before the MacOSX HFS+ and Cygwin ports arrived. They
also have case-preserving case-insensitive filesystems, but the people
doing the ports despised the Windows rules. Indeed, a patch to make HFS+
act like Unix for imports got past a reviewer and into the code base,
which incidentally made Cygwin also act like Unix (but this met the
unbounded approval of the Cygwin folks, so they sure didn't complain --
they had patches of their own pending to do this, but the reviewer for
those balked).

At a higher level, we want to keep Python consistent, by following the
same rules on all platforms with case-preserving case-insensitive
filesystems.

Proposed Semantics

The proposed new semantics for the lower left box:

A.  If the PYTHONCASEOK environment variable exists, same as before:
    silently accept the first case-insensitive match of any kind; raise
    ImportError if none found.
B.  Else search sys.path for the first case-sensitive match; raise
    ImportError if none found.

#B is the same rule as is used on Unix, so this will improve
cross-platform portability. That's good. #B is also the rule the Mac and
Cygwin folks want (and wanted enough to implement themselves, multiple
times, which is a powerful argument in PythonLand). It can't cause any
existing non-exceptional Windows import to fail, because any existing
non-exceptional Windows import finds a case-sensitive match first in the
path -- and it still will. An exceptional Windows import currently blows
up with a NameError or ImportError, in which latter case it still will,
or in which former case will continue searching, and either succeed or
blow up with an ImportError.

#A is needed to cater to case-destroying filesystems mounted on Windows,
and may also be used by people so enamored of "natural" Windows behavior
that they're willing to set an environment variable to get it. I don't
intend to implement #A for Unix too, but that's just because I'm not
clear on how I could do so efficiently (I'm not going to slow imports
under Unix just for theoretical purity).

The potential damage is here: #2 (matching on ALLCAPS.PY) is proposed to
be dropped. Case-destroying filesystems are a vanishing breed, and
support for them is ugly. We're already supporting (and will continue to
support) PYTHONCASEOK for their benefit, but they don't deserve multiple
hacks in 2001.


text:
PEP: 330 Title: Python Bytecode Verification Version: $Revision$
Last-Modified: $Date$ Author: Michel Pelletier
<EMAIL_ADDRESS>Status: Rejected Type: Standards Track
Content-Type: text/x-rst Created: 17-Jun-2004 Python-Version: 2.6
Post-History:

Abstract

If Python Virtual Machine (PVM) bytecode is not "well-formed" it is
possible to crash or exploit the PVM by causing various errors such as
under/overflowing the value stack or reading/writing into arbitrary
areas of the PVM program space. Most of these kinds of errors can be
eliminated by verifying that PVM bytecode does not violate a set of
simple constraints before execution.

This PEP proposes a set of constraints on the format and structure of
Python Virtual Machine (PVM) bytecode and provides an implementation in
Python of this verification process.

Pronouncement

Guido believes that a verification tool has some value. If someone wants
to add it to Tools/scripts, no PEP is required.

Such a tool may have value for validating the output from
"bytecodehacks" or from direct edits of PYC files. As security measure,
its value is somewhat limited because perfectly valid bytecode can still
do horrible things. That situation could change if the concept of
restricted execution were to be successfully resurrected.

Motivation

The Python Virtual Machine executes Python programs that have been
compiled from the Python language into a bytecode representation. The
PVM assumes that any bytecode being executed is "well-formed" with
regard to a number implicit constraints. Some of these constraints are
checked at run-time, but most of them are not due to the overhead they
would create.

When running in debug mode the PVM does do several run-time checks to
ensure that any particular bytecode cannot violate these constraints
that, to a degree, prevent bytecode from crashing or exploiting the
interpreter. These checks add a measurable overhead to the interpreter,
and are typically turned off in common use.

Bytecode that is not well-formed and executed by a PVM not running in
debug mode may create a variety of fatal and non-fatal errors.
Typically, ill-formed code will cause the PVM to seg-fault and cause the
OS to immediately and abruptly terminate the interpreter.

Conceivably, ill-formed bytecode could exploit the interpreter and allow
Python bytecode to execute arbitrary C-level machine instructions or to
modify private, internal data structures in the interpreter. If used
cleverly this could subvert any form of security policy an application
may want to apply to its objects.

Practically, it would be difficult for a malicious user to "inject"
invalid bytecode into a PVM for the purposes of exploitation, but not
impossible. Buffer overflow and memory overwrite attacks are commonly
understood, particularly when the exploit payload is transmitted
unencrypted over a network or when a file or network security permission
weakness is used as a foothold for further attacks.

Ideally, no bytecode should ever be allowed to read or write underlying
C-level data structures to subvert the operation of the PVM, whether the
bytecode was maliciously crafted or not. A simple pre-execution
verification step could ensure that bytecode cannot over/underflow the
value stack or access other sensitive areas of PVM program space at
run-time.

This PEP proposes several validation steps that should be taken on
Python bytecode before it is executed by the PVM so that it compiles
with static and structure constraints on its instructions and their
operands. These steps are simple and catch a large class of invalid
bytecode that can cause crashes. There is also some possibility that
some run-time checks can be eliminated up front by a verification pass.

There is, of course, no way to verify that bytecode is "completely
safe", for every definition of complete and safe. Even with bytecode
verification, Python programs can and most likely in the future will
seg-fault for a variety of reasons and continue to cause many different
classes of run-time errors, fatal or not. The verification step proposed
here simply plugs an easy hole that can cause a large class of fatal and
subtle errors at the bytecode level.

Currently, the Java Virtual Machine (JVM) verifies Java bytecode in a
way very similar to what is proposed here. The JVM Specification version
2[1], Sections 4.8 and 4.9 were therefore used as a basis for some of
the constraints explained below. Any Python bytecode verification
implementation at a minimum must enforce these constraints, but may not
be limited to them.

Static Constraints on Bytecode Instructions

1.  The bytecode string must not be empty. (len(co_code) > 0).
2.  The bytecode string cannot exceed a maximum size
    (len(co_code) < sizeof(unsigned char) - 1).
3.  The first instruction in the bytecode string begins at index 0.
4.  Only valid byte-codes with the correct number of operands can be in
    the bytecode string.

Static Constraints on Bytecode Instruction Operands

1.  The target of a jump instruction must be within the code boundaries
    and must fall on an instruction, never between an instruction and
    its operands.
2.  The operand of a LOAD_* instruction must be a valid index into its
    corresponding data structure.
3.  The operand of a STORE_* instruction must be a valid index into its
    corresponding data structure.

Structural Constraints between Bytecode Instructions

1.  Each instruction must only be executed with the appropriate number
    of arguments in the value stack, regardless of the execution path
    that leads to its invocation.
2.  If an instruction can be executed along several different execution
    paths, the value stack must have the same depth prior to the
    execution of the instruction, regardless of the path taken.
3.  At no point during execution can the value stack grow to a depth
    greater than that implied by co_stacksize.
4.  Execution never falls off the bottom of co_code.

Implementation

This PEP is the working document for a Python bytecode verification
implementation written in Python. This implementation is not used
implicitly by the PVM before executing any bytecode, but is to be used
explicitly by users concerned about possibly invalid bytecode with the
following snippet:

    import verify
    verify.verify(object)

The verify module provides a verify function which accepts the same kind
of arguments as dis.dis: classes, methods, functions, or code objects.
It verifies that the object's bytecode is well-formed according to the
specifications of this PEP.

If the code is well-formed the call to verify returns silently without
error. If an error is encountered, it throws a VerificationError whose
argument indicates the cause of the failure. It is up to the programmer
whether or not to handle the error in some way or execute the invalid
code regardless.

Phillip Eby has proposed a pseudo-code algorithm for bytecode stack
depth verification used by the reference implementation.

Verification Issues

This PEP describes only a small number of verifications. While
discussion and analysis will lead to many more, it is highly possible
that future verification may need to be done or custom, project-specific
verifications. For this reason, it might be desirable to add a
verification registration interface to the test implementation to
register future verifiers. The need for this is minimal since custom
verifiers can subclass and extend the current implementation for added
behavior.

Required Changes

Armin Rigo noted that several byte-codes will need modification in order
for their stack effect to be statically analyzed. These are END_FINALLY,
POP_BLOCK, and MAKE_CLOSURE. Armin and Guido have already agreed on how
to correct the instructions. Currently the Python implementation punts
on these instructions.

This PEP does not propose to add the verification step to the
interpreter, but only to provide the Python implementation in the
standard library for optional use. Whether or not this verification
procedure is translated into C, included with the PVM or enforced in any
way is left for future discussion.

References

Copyright

This document has been placed in the public domain.

[1] The Java Virtual Machine Specification 2nd Edition
http://java.sun.com/docs/books/vmspec/2nd-edition/html/ClassFile.doc.html


text:
PEP: 416 Title: Add a frozendict builtin type Version: $Revision$
Last-Modified: $Date$ Author: Victor Stinner<EMAIL_ADDRESS>Status: Rejected Type: Standards Track Content-Type: text/x-rst Created:
29-Feb-2012 Python-Version: 3.3

Rejection Notice

I'm rejecting this PEP. A number of reasons (not exhaustive):

-   According to Raymond Hettinger, use of frozendict is low. Those that
    do use it tend to use it as a hint only, such as declaring global or
    class-level "constants": they aren't really immutable, since anyone
    can still assign to the name.
-   There are existing idioms for avoiding mutable default values.
-   The potential of optimizing code using frozendict in PyPy is unsure;
    a lot of other things would have to change first. The same holds for
    compile-time lookups in general.
-   Multiple threads can agree by convention not to mutate a shared
    dict, there's no great need for enforcement. Multiple processes
    can't share dicts.
-   Adding a security sandbox written in Python, even with a limited
    scope, is frowned upon by many, due to the inherent difficulty with
    ever proving that the sandbox is actually secure. Because of this we
    won't be adding one to the stdlib any time soon, so this use case
    falls outside the scope of a PEP.

On the other hand, exposing the existing read-only dict proxy as a
built-in type sounds good to me. (It would need to be changed to allow
calling the constructor.) GvR.

Update (2012-04-15): A new MappingProxyType type was added to the types
module of Python 3.3.

Abstract

Add a new frozendict builtin type.

Rationale

A frozendict is a read-only mapping: a key cannot be added nor removed,
and a key is always mapped to the same value. However, frozendict values
can be not hashable. A frozendict is hashable if and only if all values
are hashable.

Use cases:

-   Immutable global variable like a default configuration.
-   Default value of a function parameter. Avoid the issue of mutable
    default arguments.
-   Implement a cache: frozendict can be used to store function
    keywords. frozendict can be used as a key of a mapping or as a
    member of set.
-   frozendict avoids the need of a lock when the frozendict is shared
    by multiple threads or processes, especially hashable frozendict. It
    would also help to prohibe coroutines (generators + greenlets) to
    modify the global state.
-   frozendict lookup can be done at compile time instead of runtime
    because the mapping is read-only. frozendict can be used instead of
    a preprocessor to remove conditional code at compilation, like code
    specific to a debug build.
-   frozendict helps to implement read-only object proxies for security
    modules. For example, it would be possible to use frozendict type
    for __builtins__ mapping or type.__dict__. This is possible because
    frozendict is compatible with the PyDict C API.
-   frozendict avoids the need of a read-only proxy in some cases.
    frozendict is faster than a proxy because getting an item in a
    frozendict is a fast lookup whereas a proxy requires a function
    call.

Constraints

-   frozendict has to implement the Mapping abstract base class
-   frozendict keys and values can be unorderable
-   a frozendict is hashable if all keys and values are hashable
-   frozendict hash does not depend on the items creation order

Implementation

-   Add a PyFrozenDictObject structure based on PyDictObject with an
    extra "Py_hash_t hash;" field
-   frozendict.__hash__() is implemented using
    hash(frozenset(self.items())) and caches the result in its private
    hash attribute
-   Register frozendict as a collections.abc.Mapping
-   frozendict can be used with PyDict_GetItem(), but PyDict_SetItem()
    and PyDict_DelItem() raise a TypeError

Recipe: hashable dict

To ensure that a frozendict is hashable, values can be checked before
creating the frozendict:

    import itertools

    def hashabledict(*args, **kw):
        # ensure that all values are hashable
        for key, value in itertools.chain(args, kw.items()):
            if isinstance(value, (int, str, bytes, float, frozenset, complex)):
                # avoid the compute the hash (which may be slow) for builtin
                # types known to be hashable for any value
                continue
            hash(value)
            # don't check the key: frozendict already checks the key
        return frozendict.__new__(cls, *args, **kw)

Objections

namedtuple may fit the requirements of a frozendict.

A namedtuple is not a mapping, it does not implement the Mapping
abstract base class.

frozendict can be implemented in Python using descriptors" and
"frozendict just need to be practically constant.

If frozendict is used to harden Python (security purpose), it must be
implemented in C. A type implemented in C is also faster.

The PEP 351 was rejected.

The PEP 351 tries to freeze an object and so may convert a mutable
object to an immutable object (using a different type). frozendict
doesn't convert anything: hash(frozendict) raises a TypeError if a value
is not hashable. Freezing an object is not the purpose of this PEP.

Alternative: dictproxy

Python has a builtin dictproxy type used by type.__dict__ getter
descriptor. This type is not public. dictproxy is a read-only view of a
dictionary, but it is not read-only mapping. If a dictionary is
modified, the dictproxy is also modified.

dictproxy can be used using ctypes and the Python C API, see for example
the make dictproxy object via ctypes.pythonapi and type() (Python recipe
576540) by Ikkei Shimomura. The recipe contains a test checking that a
dictproxy is "mutable" (modify the dictionary linked to the dictproxy).

However dictproxy can be useful in some cases, where its mutable
property is not an issue, to avoid a copy of the dictionary.

Existing implementations

Whitelist approach.

-   Implementing an Immutable Dictionary (Python recipe 498072) by
    Aristotelis Mikropoulos. Similar to frozendict except that it is not
    truly read-only: it is possible to access to this private internal
    dict. It does not implement __hash__ and has an implementation
    issue: it is possible to call again __init__() to modify the
    mapping.
-   PyWebmail contains an ImmutableDict type:
    webmail.utils.ImmutableDict. It is hashable if keys and values are
    hashable. It is not truly read-only: its internal dict is a public
    attribute.
-   remember project: remember.dicts.FrozenDict. It is used to implement
    a cache: FrozenDict is used to store function callbacks. FrozenDict
    may be hashable. It has an extra supply_dict() class method to
    create a FrozenDict from a dict without copying the dict: store the
    dict as the internal dict. Implementation issue: __init__() can be
    called to modify the mapping and the hash may differ depending on
    item creation order. The mapping is not truly read-only: the
    internal dict is accessible in Python.

Blacklist approach: inherit from dict and override write methods to
raise an exception. It is not truly read-only: it is still possible to
call dict methods on such "frozen dictionary" to modify it.

-   brownie: brownie.datastructures.ImmutableDict. It is hashable if
    keys and values are hashable. werkzeug project has the same code:
    werkzeug.datastructures.ImmutableDict. ImmutableDict is used for
    global constant (configuration options). The Flask project uses
    ImmutableDict of werkzeug for its default configuration.
-   SQLAlchemy project: sqlalchemy.util.immutabledict. It is not
    hashable and has an extra method: union(). immutabledict is used for
    the default value of parameter of some functions expecting a
    mapping. Example: mapper_args=immutabledict() in SqlSoup.map().
-   Frozen dictionaries (Python recipe 414283) by Oren Tirosh. It is
    hashable if keys and values are hashable. Included in the following
    projects:
    -   lingospot: frozendict/frozendict.py
    -   factor-graphics: frozendict type in
        python/fglib/util_ext_frozendict.py
-   The gsakkis-utils project written by George Sakkis includes a
    frozendict type: datastructs.frozendict
-   characters: scripts/python/frozendict.py. It is hashable. __init__()
    sets __init__ to None.
-   Old NLTK (1.x): nltk.util.frozendict. Keys and values must be
    hashable. __init__() can be called twice to modify the mapping.
    frozendict is used to "freeze" an object.

Hashable dict: inherit from dict and just add an __hash__ method.

-   pypy.rpython.lltypesystem.lltype.frozendict. It is hashable but
    don't deny modification of the mapping.
-   factor-graphics: hashabledict type in
    python/fglib/util_ext_frozendict.py

Links

-   Issue #14162: PEP 416: Add a builtin frozendict type
-   PEP 412: Key-Sharing Dictionary (issue #13903)
-   PEP 351: The freeze protocol
-   The case for immutable dictionaries; and the central
    misunderstanding of PEP 351
-   make dictproxy object via ctypes.pythonapi and type() (Python
    recipe 576540) <http://code.activestate.com/recipes/576540/> by
    Ikkei Shimomura.
-   Python security modules implementing read-only object proxies using
    a C extension:
    -   pysandbox
    -   mxProxy
    -   zope.proxy
    -   zope.security

Copyright

This document has been placed in the public domain.