text:
PEP: 442 Title: Safe object finalization Version: $Revision$
Last-Modified: $Date$ Author: Antoine Pitrou<EMAIL_ADDRESS>BDFL-Delegate: Benjamin Peterson<EMAIL_ADDRESS>Status: Final
Type: Standards Track Content-Type: text/x-rst Created: 18-May-2013
Python-Version: 3.4 Post-History: 18-May-2013 Resolution:
https://mail.python.org/pipermail/python-dev/2013-June/126746.html

Abstract

This PEP proposes to deal with the current limitations of object
finalization. The goal is to be able to define and run finalizers for
any object, regardless of their position in the object graph.

This PEP doesn't call for any change in Python code. Objects with
existing finalizers will benefit automatically.

Definitions

Reference

    A directional link from an object to another. The target of the
    reference is kept alive by the reference, as long as the source is
    itself alive and the reference isn't cleared.

Weak reference

    A directional link from an object to another, which doesn't keep
    alive its target. This PEP focuses on non-weak references.

Reference cycle

    A cyclic subgraph of directional links between objects, which keeps
    those objects from being collected in a pure reference-counting
    scheme.

Cyclic isolate (CI)

    A standalone subgraph of objects in which no object is referenced
    from the outside, containing one or several reference cycles, and
    whose objects are still in a usable, non-broken state: they can
    access each other from their respective finalizers.

Cyclic garbage collector (GC)

    A device able to detect cyclic isolates and turn them into cyclic
    trash. Objects in cyclic trash are eventually disposed of by the
    natural effect of the references being cleared and their reference
    counts dropping to zero.

Cyclic trash (CT)

    A former cyclic isolate whose objects have started being cleared by
    the GC. Objects in cyclic trash are potential zombies; if they are
    accessed by Python code, the symptoms can vary from weird
    AttributeErrors to crashes.

Zombie / broken object

    An object part of cyclic trash. The term stresses that the object is
    not safe: its outgoing references may have been cleared, or one of
    the objects it references may be zombie. Therefore, it should not be
    accessed by arbitrary code (such as finalizers).

Finalizer

    A function or method called when an object is intended to be
    disposed of. The finalizer can access the object and release any
    resource held by the object (for example mutexes or file
    descriptors). An example is a __del__ method.

Resurrection

    The process by which a finalizer creates a new reference to an
    object in a CI. This can happen as a quirky but supported
    side-effect of __del__ methods.

Impact

While this PEP discusses CPython-specific implementation details, the
change in finalization semantics is expected to affect the Python
ecosystem as a whole. In particular, this PEP obsoletes the current
guideline that "objects with a __del__ method should not be part of a
reference cycle".

Benefits

The primary benefits of this PEP regard objects with finalizers, such as
objects with a __del__ method and generators with a finally block. Those
objects can now be reclaimed when they are part of a reference cycle.

The PEP also paves the way for further benefits:

-   The module shutdown procedure may not need to set global variables
    to None anymore. This could solve a well-known class of irritating
    issues.

The PEP doesn't change the semantics of:

-   Weak references caught in reference cycles.
-   C extension types with a custom tp_dealloc function.

Description

Reference-counted disposal

In normal reference-counted disposal, an object's finalizer is called
just before the object is deallocated. If the finalizer resurrects the
object, deallocation is aborted.

However, if the object was already finalized, then the finalizer isn't
called. This prevents us from finalizing zombies (see below).

Disposal of cyclic isolates

Cyclic isolates are first detected by the garbage collector, and then
disposed of. The detection phase doesn't change and won't be described
here. Disposal of a CI traditionally works in the following order:

1.  Weakrefs to CI objects are cleared, and their callbacks called. At
    this point, the objects are still safe to use.
2.  The CI becomes a CT as the GC systematically breaks all known
    references inside it (using the tp_clear function).
3.  Nothing. All CT objects should have been disposed of in step 2 (as a
    side-effect of clearing references); this collection is finished.

This PEP proposes to turn CI disposal into the following sequence (new
steps are in bold):

1.  Weakrefs to CI objects are cleared, and their callbacks called. At
    this point, the objects are still safe to use.
2.  The finalizers of all CI objects are called.
3.  The CI is traversed again to determine if it is still isolated. If
    it is determined that at least one object in CI is now reachable
    from outside the CI, this collection is aborted and the whole CI is
    resurrected. Otherwise, proceed.
4.  The CI becomes a CT as the GC systematically breaks all known
    references inside it (using the tp_clear function).
5.  Nothing. All CT objects should have been disposed of in step 4 (as a
    side-effect of clearing references); this collection is finished.

Note

The GC doesn't recalculate the CI after step 2 above, hence the need for
step 3 to check that the whole subgraph is still isolated.

C-level changes

Type objects get a new tp_finalize slot to which __del__ methods are
mapped (and reciprocally). Generators are modified to use this slot,
rather than tp_del. A tp_finalize function is a normal C function which
will be called with a valid and alive PyObject as its only argument. It
doesn't need to manipulate the object's reference count, as this will be
done by the caller. However, it must ensure that the original exception
state is restored before returning to the caller.

For compatibility, tp_del is kept in the type structure. Handling of
objects with a non-NULL tp_del is unchanged: when part of a CI, they are
not finalized and end up in gc.garbage. However, a non-NULL tp_del is
not encountered anymore in the CPython source tree (except for testing
purposes).

Two new C API functions are provided to ease calling of tp_finalize,
especially from custom deallocators.

On the internal side, a bit is reserved in the GC header for GC-managed
objects to signal that they were finalized. This helps avoid finalizing
an object twice (and, especially, finalizing a CT object after it was
broken by the GC).

Note

Objects which are not GC-enabled can also have a tp_finalize slot. They
don't need the additional bit since their tp_finalize function can only
be called from the deallocator: it therefore cannot be called twice,
except when resurrected.

Discussion

Predictability

Following this scheme, an object's finalizer is always called exactly
once, even if it was resurrected afterwards.

For CI objects, the order in which finalizers are called (step 2 above)
is undefined.

Safety

It is important to explain why the proposed change is safe. There are
two aspects to be discussed:

-   Can a finalizer access zombie objects (including the object being
    finalized)?
-   What happens if a finalizer mutates the object graph so as to impact
    the CI?

Let's discuss the first issue. We will divide possible cases in two
categories:

-   If the object being finalized is part of the CI: by construction, no
    objects in CI are zombies yet, since CI finalizers are called before
    any reference breaking is done. Therefore, the finalizer cannot
    access zombie objects, which don't exist.
-   If the object being finalized is not part of the CI/CT: by
    definition, objects in the CI/CT don't have any references pointing
    to them from outside the CI/CT. Therefore, the finalizer cannot
    reach any zombie object (that is, even if the object being finalized
    was itself referenced from a zombie object).

Now for the second issue. There are three potential cases:

-   The finalizer clears an existing reference to a CI object. The CI
    object may be disposed of before the GC tries to break it, which is
    fine (the GC simply has to be aware of this possibility).
-   The finalizer creates a new reference to a CI object. This can only
    happen from a CI object's finalizer (see above why). Therefore, the
    new reference will be detected by the GC after all CI finalizers are
    called (step 3 above), and collection will be aborted without any
    objects being broken.
-   The finalizer clears or creates a reference to a non-CI object. By
    construction, this is not a problem.

Implementation

An implementation is available in branch finalize of the repository at
http://hg.python.org/features/finalize/.

Validation

Besides running the normal Python test suite, the implementation adds
test cases for various finalization possibilities including reference
cycles, object resurrection and legacy tp_del slots.

The implementation has also been checked to not produce any regressions
on the following test suites:

-   Tulip, which makes an extensive use of generators
-   Tornado
-   SQLAlchemy
-   Django
-   zope.interface

References

Notes about reference cycle collection and weak reference callbacks:
http://hg.python.org/cpython/file/4e687d53b645/Modules/gc_weakref.txt

Generator memory leak: http://bugs.python.org/issue17468

Allow objects to decide if they can be collected by GC:
http://bugs.python.org/issue9141

Module shutdown procedure based on GC http://bugs.python.org/issue812369

Copyright

This document has been placed in the public domain.


text:
PEP: 664 Title: Python 3.11 Release Schedule Author: Pablo Galindo
Salgado<EMAIL_ADDRESS>Status: Active Type: Informational Topic:
Release Created: 12-Jul-2021 Python-Version: 3.11

Abstract

This document describes the development and release schedule for Python
3.11. The schedule primarily concerns itself with PEP-sized items.

Release Manager and Crew

-   3.11 Release Manager: Pablo Galindo Salgado
-   Windows installers: Steve Dower
-   Mac installers: Ned Deily
-   Documentation: Julien Palard

Release Schedule

3.11.0 schedule

Note: the dates below use a 17-month development period that results in
a 12-month release cadence between feature versions, as defined by PEP
602.

Actual:

-   3.11 development begins: Monday, 2021-05-03
-   3.11.0 alpha 1: Tuesday, 2021-10-05
-   3.11.0 alpha 2: Tuesday, 2021-11-02
-   3.11.0 alpha 3: Wednesday, 2021-12-08
-   3.11.0 alpha 4: Friday, 2022-01-14
-   3.11.0 alpha 5: Thursday, 2022-02-03
-   3.11.0 alpha 6: Monday, 2022-03-07
-   3.11.0 alpha 7: Tuesday, 2022-04-05
-   3.11.0 beta 1: Sunday, 2022-05-08 (No new features beyond this
    point.)
-   3.11.0 beta 2: Tuesday, 2022-05-31
-   3.11.0 beta 3: Wednesday, 2022-06-01
-   3.11.0 beta 4: Monday, 2022-07-11
-   3.11.0 beta 5: Tuesday, 2022-07-26
-   3.11.0 candidate 1: Monday, 2022-08-08
-   3.11.0 candidate 2: Monday, 2022-09-12
-   3.11.0 final: Monday, 2022-10-24

Bugfix releases

Actual:

-   3.11.1: Tuesday, 2022-12-06
-   3.11.2: Wednesday, 2023-02-08
-   3.11.3: Wednesday, 2023-04-05
-   3.11.4: Tuesday, 2023-06-06
-   3.11.5: Thursday, 2023-08-24
-   3.11.6: Monday, 2023-10-02
-   3.11.7: Monday, 2023-12-04
-   3.11.8: Tuesday, 2024-02-06
-   3.11.9: Tuesday, 2024-04-02 (final regular bugfix release with
    binary installers)

Source-only security fix releases

Provided irregularly on an "as-needed" basis until October 2027.

-   3.11.10: Saturday, 2024-09-07

3.11 Lifespan

3.11 will receive bugfix updates approximately every 2 months for
approximately 18 months. Some time after the release of 3.12.0 final,
the ninth and final 3.11 bugfix update will be released. After that, it
is expected that security updates (source only) will be released until 5
years after the release of 3.11.0 final, so until approximately October
2027.

Features for 3.11

Some of the notable features of Python 3.11 include:

-   PEP 654, Exception Groups and except*.
-   PEP 657, Enhanced error locations in tracebacks.
-   PEP 680, Support for parsing TOML in the standard library
-   Python 3.11 is up to 10-60% faster than Python 3.10. On average, we
    measured a 1.25x speedup on the standard benchmark suite. See Faster
    CPython for details.

Typing features:

-   PEP 646, Variadic generics.
-   PEP 655, Marking individual TypedDict items as required or
    potentially-missing.
-   PEP 673, Self type.
-   PEP 675, Arbitrary literal string type.
-   PEP 681, Dataclass transforms

Copyright

This document has been placed in the public domain.

license: Public Domain


text:
PEP: 715 Title: Disabling bdist_egg distribution uploads on PyPI Author:
William Woodruff<EMAIL_ADDRESS>Sponsor: Donald Stufft
<EMAIL_ADDRESS>PEP-Delegate: Donald Stufft<EMAIL_ADDRESS>Discussions-To: https://discuss.python.org/t/27610 Status: Final Type:
Standards Track Topic: Packaging Content-Type: text/x-rst Created:
06-Jun-2023 Post-History: 09-Jun-2023 Resolution: 24-Jun-2023

Abstract

This PEP recommends deprecating and then disabling new uploads of the
bdist_egg distribution type on PyPI. In a parallel move, this PEP
recommends deprecating and then disabling new uploads of distribution
filenames that have the .egg suffix.

After this PEP, PyPI will only accept new uploads of the sdist and
bdist_wheel types, corresponding to files with .tar.gz/.zip and .whl
suffixes respectively.

This PEP does not recommend removing or otherwise affecting any
previously uploaded bdist_egg distributions or files with the .egg
suffix.

Rationale

Previous Work

The groundwork for this proposal was established with PEP 527, which
proposed deprecating and eventually removing upload support for a
handful of un(der)used file extensions and distribution types.

In particular, PEP 527 proposed the removal of the bdist_dumb,
bdist_rpm, bdist_dmg, bdist_msi, and bdist_wininst distribution types,
as well as the .tar, .tar.bz2, .tar.xz, .tar.Z, .tgz, and .tbz file
extensions on distribution filenames.

PEP 527 was fully enacted with PR #7529 to Warehouse, which was merged
on 13 April 2020.

The bdist_egg format

The bdist_egg filetype identifies distributions in the
egg format <packaging:Egg>. The egg format was introduced by setuptools
in 2004 and is roughly equivalent in functionality to the
wheel format <packaging:Wheel> first introduced by PEP 427 in 2012 as
the standardized format <packaging:binary-distribution-format> for
built distributions <packaging:Built Distribution>.

Despite its longevity, the egg format has had limited adoption on PyPI.
Some observations from that issue:

-   In the month of May 2023, bdist_egg uploads accounted for 0.2% of
    all distribution uploads to PyPI;
-   pip deprecated its --egg option in 2016;
-   setuptools has considered egg support deprecated since 2019;
-   build only supports the sdist and bdist_wheel filetypes.

Given the above, this PEP proposes the removal of the bdist_egg format
under the same justifications presented in PEP 527, namely:

-   Egg distributions are of limited use to the broader ecosystem and
    therefore represent a non-reciprocal maintenance burden;
-   Having an additional built distribution format is confusing to end
    users, who may mistakenly pick it over the wheel format;

This PEP additionally offers an argument for removal rooted in
standardization and duplication: the egg format is not standardized by
any PEP or other community standard, and overlaps heavily with its
standardized and well-supported alternative (wheel).

The .egg file extension

The .egg file extension is used exclusively for distributions of the
bdist_egg format. As such, it serves no purpose in a scenario where PyPI
disables new distribution uploads for bdist_egg distributions.

Removal Process

This PEP does NOT propose removing any existing files from PyPI, only
disallowing new ones from being uploaded.

PyPI will provide a deprecation period of one month. At the beginning of
the deprecation period, maintainers of projects that have uploaded one
or more egg distributions since 1 Jan 2023 will receive a one-time email
informing them of the upcoming end of support for egg distribution
uploads.

During the deprecation period, users will continue to be allowed to
upload egg distributions to new and existing projects. Uploading an egg
distribution during this period will also send all maintainers of the
project a similar email as above, reminding them of the upcoming end of
support.

After the deprecation period, support for uploading egg distributions
will cease to exist on PyPI.

Prior Art

The removal process above was based on that of PEP 527, with the
following changes:

-   All projects will be continue to be allowed to upload eggs during
    the deprecation period, not just those that have done so previously.
-   Uploading an egg during the deprecation period will also trigger an
    email to maintainers, in addition to the one-time email at the
    beginning.

Backwards Compatibility

Limited Impact

As noted in the rationale section, this PEP is expected to have no
impact on the overwhelming majority of PyPI users and projects, and
there has been substantial community coordination over the past 1 1/2
years to minimize the impact on the few last use cases.

Hosted Files

This PEP does NOT propose the removal of any egg distributions that have
already been uploaded to PyPI. All previously uploaded egg distributions
will remain downloadable, ensuring that existing users will continue to
be able to download them.

Deprecation Period

This PEP uses the removal process documented above, which specifies a
deprecation period of 1 month for projects that have previously uploaded
egg distributions to PyPI.

After the end of the deprecation period, support for uploading new egg
distributions will cease to exist on PyPI.

Security Implications

This PEP does not identify any positive or negative security
implications associated with removing upload support for egg
distributions.

How To Teach This

As part of the removal process, PyPI will send emails to all maintainers
of projects that have previously uploaded egg distributions in 2023.

Additionally, PyPI will write a post on the PyPI blog that publicly
announces the deprecation period's start and end.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 288 Title: Generators Attributes and Exceptions Author: Raymond
Hettinger<EMAIL_ADDRESS>Status: Withdrawn Type: Standards Track
Content-Type: text/x-rst Created: 21-Mar-2002 Python-Version: 2.5
Post-History:

Abstract

This PEP proposes to enhance generators by providing mechanisms for
raising exceptions and sharing data with running generators.

Status

This PEP is withdrawn. The exception raising mechanism was extended and
subsumed into PEP 343. The attribute passing capability never built a
following, did not have a clear implementation, and did not have a clean
way for the running generator to access its own namespace.

Rationale

Currently, only class based iterators can provide attributes and
exception handling. However, class based iterators are harder to write,
less compact, less readable, and slower. A better solution is to enable
these capabilities for generators.

Enabling attribute assignments allows data to be passed to and from
running generators. The approach of sharing data using attributes
pervades Python. Other approaches exist but are somewhat hackish in
comparison.

Another evolutionary step is to add a generator method to allow
exceptions to be passed to a generator. Currently, there is no clean
method for triggering exceptions from outside the generator. Also,
generator exception passing helps mitigate the try/finally prohibition
for generators. The need is especially acute for generators needing to
flush buffers or close resources upon termination.

The two proposals are backwards compatible and require no new keywords.
They are being recommended for Python version 2.5.

Specification for Generator Attributes

Essentially, the proposal is to emulate attribute writing for classes.
The only wrinkle is that generators lack a way to refer to instances of
themselves. So, the proposal is to provide a function for discovering
the reference. For example:

    def mygen(filename):
        self = sys.get_generator()
        myfile = open(filename)
        for line in myfile:
            if len(line) < 10:
                continue
            self.pos = myfile.tell()
            yield line.upper()

    g = mygen('sample.txt')
    line1 = g.next()
    print 'Position', g.pos

Uses for generator attributes include:

1.  Providing generator clients with extra information (as shown above).
2.  Externally setting control flags governing generator operation
    (possibly telling a generator when to step in or step over data
    groups).
3.  Writing lazy consumers with complex execution states (an arithmetic
    encoder output stream for example).
4.  Writing co-routines (as demonstrated in Dr. Mertz's articles[1]).

The control flow of 'yield' and 'next' is unchanged by this proposal.
The only change is that data can passed to and from the generator. Most
of the underlying machinery is already in place, only the access
function needs to be added.

Specification for Generator Exception Passing

Add a .throw(exception) method to the generator interface:

    def logger():
        start = time.time()
        log = []
        try:
            while True:
                log.append(time.time() - start)
                yield log[-1]
        except WriteLog:
            writelog(log)

    g = logger()
    for i in [10,20,40,80,160]:
        testsuite(i)
        g.next()
    g.throw(WriteLog)

There is no existing work-around for triggering an exception inside a
generator. It is the only case in Python where active code cannot be
excepted to or through.

Generator exception passing also helps address an intrinsic limitation
on generators, the prohibition against their using try/finally to
trigger clean-up code (PEP 255).

Note A: The name of the throw method was selected for several reasons.
Raise is a keyword and so cannot be used as a method name. Unlike raise
which immediately raises an exception from the current execution point,
throw will first return to the generator and then raise the exception.
The word throw is suggestive of putting the exception in another
location. The word throw is already associated with exceptions in other
languages.

Alternative method names were considered: resolve(), signal(),
genraise(), raiseinto(), and flush(). None of these fit as well as
throw().

Note B: To keep the throw() syntax simple only the instance version of
the raise syntax would be supported (no variants for "raise string" or
"raise class, instance").

Calling g.throw(instance) would correspond to writing raise instance
immediately after the most recent yield.

References

Copyright

This document has been placed in the public domain.

[1] Dr. David Mertz's draft columns for Charming Python
http://gnosis.cx/publish/programming/charming_python_b5.txt
http://gnosis.cx/publish/programming/charming_python_b7.txt


text:
PEP: 403 Title: General purpose decorator clause (aka "@in" clause)
Version: $Revision$ Last-Modified: $Date$ Author: Alyssa Coghlan
<EMAIL_ADDRESS>Status: Deferred Type: Standards Track
Content-Type: text/x-rst Created: 13-Oct-2011 Python-Version: 3.4
Post-History: 13-Oct-2011

Abstract

This PEP proposes the addition of a new @in decorator clause that makes
it possible to override the name binding step of a function or class
definition.

The new clause accepts a single simple statement that can make a forward
reference to decorated function or class definition.

This new clause is designed to be used whenever a "one-shot" function or
class is needed, and placing the function or class definition before the
statement that uses it actually makes the code harder to read. It also
avoids any name shadowing concerns by making sure the new name is
visible only to the statement in the @in clause.

This PEP is based heavily on many of the ideas in PEP 3150 (Statement
Local Namespaces) so some elements of the rationale will be familiar to
readers of that PEP. Both PEPs remain deferred for the time being,
primarily due to the lack of compelling real world use cases in either
PEP.

Basic Examples

Before diving into the long history of this problem and the detailed
rationale for this specific proposed solution, here are a few simple
examples of the kind of code it is designed to simplify.

As a trivial example, a weakref callback could be defined as follows:

    @in x = weakref.ref(target, report_destruction)
    def report_destruction(obj):
        print("{} is being destroyed".format(obj))

This contrasts with the current (conceptually) "out of order" syntax for
this operation:

    def report_destruction(obj):
        print("{} is being destroyed".format(obj))

    x = weakref.ref(target, report_destruction)

That structure is OK when you're using the callable multiple times, but
it's irritating to be forced into it for one-off operations.

If the repetition of the name seems especially annoying, then a
throwaway name like f can be used instead:

    @in x = weakref.ref(target, f)
    def f(obj):
        print("{} is being destroyed".format(obj))

Similarly, a sorted operation on a particularly poorly defined type
could now be defined as:

    @in sorted_list = sorted(original, key=f)
    def f(item):
        try:
            return item.calc_sort_order()
        except NotSortableError:
            return float('inf')

Rather than:

    def force_sort(item):
        try:
            return item.calc_sort_order()
        except NotSortableError:
            return float('inf')

    sorted_list = sorted(original, key=force_sort)

And early binding semantics in a list comprehension could be attained
via:

    @in funcs = [adder(i) for i in range(10)]
    def adder(i):
        return lambda x: x + i

Proposal

This PEP proposes the addition of a new @in clause that is a variant of
the existing class and function decorator syntax.

The new @in clause precedes the decorator lines, and allows forward
references to the trailing function or class definition.

The trailing function or class definition is always named - the name of
the trailing definition is then used to make the forward reference from
the @in clause.

The @in clause is allowed to contain any simple statement (including
those that don't make any sense in that context, such as pass - while
such code would be legal, there wouldn't be any point in writing it).
This permissive structure is easier to define and easier to explain, but
a more restrictive approach that only permits operations that "make
sense" would also be possible (see PEP 3150 for a list of possible
candidates).

The @in clause will not create a new scope - all name binding operations
aside from the trailing function or class definition will affect the
containing scope.

The name used in the trailing function or class definition is only
visible from the associated @in clause, and behaves as if it was an
ordinary variable defined in that scope. If any nested scopes are
created in either the @in clause or the trailing function or class
definition, those scopes will see the trailing function or class
definition rather than any other bindings for that name in the
containing scope.

In a very real sense, this proposal is about making it possible to
override the implicit "name = <defined function or class>" name binding
operation that is part of every function or class definition,
specifically in those cases where the local name binding isn't actually
needed.

Under this PEP, an ordinary class or function definition:

    @deco2
    @deco1
    def name():
        ...

can be explained as being roughly equivalent to:

    @in name = deco2(deco1(name))
    def name():
        ...

Syntax Change

Syntactically, only one new grammar rule is needed:

    in_stmt: '@in' simple_stmt decorated

Grammar: http://hg.python.org/cpython/file/default/Grammar/Grammar

Design Discussion

Background

The question of "multi-line lambdas" has been a vexing one for many
Python users for a very long time, and it took an exploration of Ruby's
block functionality for me to finally understand why this bugs people so
much: Python's demand that the function be named and introduced before
the operation that needs it breaks the developer's flow of thought. They
get to a point where they go "I need a one-shot operation that does
<X>", and instead of being able to just say that directly, they instead
have to back up, name a function to do <X>, then call that function from
the operation they actually wanted to do in the first place. Lambda
expressions can help sometimes, but they're no substitute for being able
to use a full suite.

Ruby's block syntax also heavily inspired the style of the solution in
this PEP, by making it clear that even when limited to one anonymous
function per statement, anonymous functions could still be incredibly
useful. Consider how many constructs Python has where one expression is
responsible for the bulk of the heavy lifting:

-   comprehensions, generator expressions, map(), filter()
-   key arguments to sorted(), min(), max()
-   partial function application
-   provision of callbacks (e.g. for weak references or asynchronous IO)
-   array broadcast operations in NumPy

However, adopting Ruby's block syntax directly won't work for Python,
since the effectiveness of Ruby's blocks relies heavily on various
conventions in the way functions are defined (specifically, using Ruby's
yield syntax to call blocks directly and the &arg mechanism to accept a
block as a function's final argument).

Since Python has relied on named functions for so long, the signatures
of APIs that accept callbacks are far more diverse, thus requiring a
solution that allows one-shot functions to be slotted in at the
appropriate location.

The approach taken in this PEP is to retain the requirement to name the
function explicitly, but allow the relative order of the definition and
the statement that references it to be changed to match the developer's
flow of thought. The rationale is essentially the same as that used when
introducing decorators, but covering a broader set of applications.

Relation to PEP 3150

PEP 3150 (Statement Local Namespaces) describes its primary motivation
as being to elevate ordinary assignment statements to be on par with
class and def statements where the name of the item to be defined is
presented to the reader in advance of the details of how the value of
that item is calculated. This PEP achieves the same goal in a different
way, by allowing the simple name binding of a standard function
definition to be replaced with something else (like assigning the result
of the function to a value).

Despite having the same author, the two PEPs are in direct competition
with each other. PEP 403 represents a minimalist approach that attempts
to achieve useful functionality with a minimum of change from the status
quo. This PEP instead aims for a more flexible standalone statement
design, which requires a larger degree of change to the language.

Note that where PEP 403 is better suited to explaining the behaviour of
generator expressions correctly, this PEP is better able to explain the
behaviour of decorator clauses in general. Both PEPs support adequate
explanations for the semantics of container comprehensions.

Keyword Choice

The proposal definitely requires some kind of prefix to avoid parsing
ambiguity and backwards compatibility problems with existing constructs.
It also needs to be clearly highlighted to readers, since it declares
that the following piece of code is going to be executed only after the
trailing function or class definition has been executed.

The in keyword was chosen as an existing keyword that can be used to
denote the concept of a forward reference.

The @ prefix was included in order to exploit the fact that Python
programmers are already used to decorator syntax as an indication of out
of order execution, where the function or class is actually defined
first and then decorators are applied in reverse order.

For functions, the construct is intended to be read as "in <this
statement that references NAME> define NAME as a function that does
<operation>".

The mapping to English prose isn't as obvious for the class definition
case, but the concept remains the same.

Better Debugging Support for Functions and Classes with Short Names

One of the objections to widespread use of lambda expressions is that
they have a negative effect on traceback intelligibility and other
aspects of introspection. Similar objections are raised regarding
constructs that promote short, cryptic function names (including this
one, which requires that the name of the trailing definition be supplied
at least twice, encouraging the use of shorthand placeholder names like
f).

However, the introduction of qualified names in PEP 3155 means that even
anonymous classes and functions will now have different representations
if they occur in different scopes. For example:

    >>> def f():
    ...     return lambda: y
    ...
    >>> f()
    <function f.<locals>.<lambda> at 0x7f6f46faeae0>

Anonymous functions (or functions that share a name) within the same
scope will still share representations (aside from the object ID), but
this is still a major improvement over the historical situation where
everything except the object ID was identical.

Possible Implementation Strategy

This proposal has at least one titanic advantage over PEP 3150:
implementation should be relatively straightforward.

The @in clause will be included in the AST for the associated function
or class definition and the statement that references it. When the @in
clause is present, it will be emitted in place of the local name binding
operation normally implied by a function or class definition.

The one potentially tricky part is changing the meaning of the
references to the statement local function or namespace while within the
scope of the in statement, but that shouldn't be too hard to address by
maintaining some additional state within the compiler (it's much easier
to handle this for a single name than it is for an unknown number of
names in a full nested suite).

Explaining Container Comprehensions and Generator Expressions

One interesting feature of the proposed construct is that it can be used
as a primitive to explain the scoping and execution order semantics of
both generator expressions and container comprehensions:

    seq2 = [x for x in y if q(x) for y in seq if p(y)]

    # would be equivalent to

    @in seq2 = f(seq):
    def f(seq)
        result = []
        for y in seq:
            if p(y):
                for x in y:
                    if q(x):
                        result.append(x)
        return result

The important point in this expansion is that it explains why
comprehensions appear to misbehave at class scope: only the outermost
iterator is evaluated at class scope, while all predicates, nested
iterators and value expressions are evaluated inside a nested scope.

An equivalent expansion is possible for generator expressions:

    gen = (x for x in y if q(x) for y in seq if p(y))

    # would be equivalent to

    @in gen = g(seq):
    def g(seq)
        for y in seq:
            if p(y):
                for x in y:
                    if q(x):
                        yield x

More Examples

Calculating attributes without polluting the local namespace (from
os.py):

    # Current Python (manual namespace cleanup)
    def _createenviron():
        ... # 27 line function

    environ = _createenviron()
    del _createenviron

    # Becomes:
    @in environ = _createenviron()
    def _createenviron():
        ... # 27 line function

Loop early binding:

    # Current Python (default argument hack)
    funcs = [(lambda x, i=i: x + i) for i in range(10)]

    # Becomes:
    @in funcs = [adder(i) for i in range(10)]
    def adder(i):
        return lambda x: x + i

    # Or even:
    @in funcs = [adder(i) for i in range(10)]
    def adder(i):
        @in return incr
        def incr(x):
            return x + i

A trailing class can be used as a statement local namespace:

    # Evaluate subexpressions only once
    @in c = math.sqrt(x.a*x.a + x.b*x.b)
    class x:
        a = calculate_a()
        b = calculate_b()

A function can be bound directly to a location which isn't a valid
identifier:

    @in dispatch[MyClass] = f
    def f():
        ...

Constructs that verge on decorator abuse can be eliminated:

    # Current Python
    @call
    def f():
        ...

    # Becomes:
    @in f()
    def f():
        ...

Reference Implementation

None as yet.

Acknowledgements

Huge thanks to Gary Bernhardt for being blunt in pointing out that I had
no idea what I was talking about in criticising Ruby's blocks, kicking
off a rather enlightening process of investigation.

Rejected Concepts

To avoid retreading previously covered ground, some rejected
alternatives are documented in this section.

Omitting the decorator prefix character

Earlier versions of this proposal omitted the @ prefix. However, without
that prefix, the bare in keyword didn't associate the clause strongly
enough with the subsequent function or class definition. Reusing the
decorator prefix and explicitly characterising the new construct as a
kind of decorator clause is intended to help users link the two concepts
and see them as two variants of the same idea.

Anonymous Forward References

A previous incarnation of this PEP (see[1]) proposed a syntax where the
new clause was introduced with : and the forward reference was written
using @. Feedback on this variant was almost universally negative, as it
was considered both ugly and excessively magical:

    :x = weakref.ref(target, @)
    def report_destruction(obj):
        print("{} is being destroyed".format(obj))

A more recent variant always used ... for forward references, along with
genuinely anonymous function and class definitions. However, this
degenerated quickly into a mass of unintelligible dots in more complex
cases:

    in funcs = [...(i) for i in range(10)]
    def ...(i):
      in return ...
      def ...(x):
          return x + i

    in c = math.sqrt(....a*....a + ....b*....b)
    class ...:
      a = calculate_a()
      b = calculate_b()

Using a nested suite

The problems with using a full nested suite are best described in PEP
3150. It's comparatively difficult to implement properly, the scoping
semantics are harder to explain and it creates quite a few situations
where there are two ways to do it without clear guidelines for choosing
between them (as almost any construct that can be expressed with
ordinary imperative code could instead be expressed using a given
statement). While the PEP does propose some new PEP 8 guidelines to help
address that last problem, the difficulties in implementation are not so
easily dealt with.

By contrast, the decorator inspired syntax in this PEP explicitly limits
the new feature to cases where it should actually improve readability,
rather than harming it. As in the case of the original introduction of
decorators, the idea of this new syntax is that if it can be used (i.e.
the local name binding of the function is completely unnecessary) then
it probably should be used.

Another possible variant of this idea is to keep the decorator based
semantics of this PEP, while adopting the prettier syntax from PEP 3150:

    x = weakref.ref(target, report_destruction) given:
        def report_destruction(obj):
            print("{} is being destroyed".format(obj))

There are a couple of problems with this approach. The main issue is
that this syntax variant uses something that looks like a suite, but
really isn't one. A secondary concern is that it's not clear how the
compiler will know which name(s) in the leading expression are forward
references (although that could potentially be addressed through a
suitable definition of the suite-that-is-not-a-suite in the language
grammar).

However, a nested suite has not yet been ruled out completely. The
latest version of PEP 3150 uses explicit forward reference and name
binding schemes that greatly simplify the semantics of the statement,
and it does offer the advantage of allowing the definition of arbitrary
subexpressions rather than being restricted to a single function or
class definition.

References

Copyright

This document has been placed in the public domain.

[1] Start of python-ideas thread:
https://mail.python.org/pipermail/python-ideas/2011-October/012276.html


text:
PEP: 724 Title: Stricter Type Guards Author: Rich Chiodo <rchiodo at
microsoft.com>, Eric Traut <erictr at microsoft.com>, Erik De Bonte
<erikd at microsoft.com>, Sponsor: Jelle Zijlstra
<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pep-724-stricter-type-guards/34124 Status:
Withdrawn Type: Standards Track Topic: Typing Content-Type: text/x-rst
Created: 28-Jul-2023 Python-Version: 3.13 Post-History: 30-Dec-2021,
19-Sep-2023,

Status

This PEP is withdrawn. The Typing Council was unable to reach consensus
on this proposal, and the authors decided to withdraw it.

Abstract

PEP 647 introduced the concept of a user-defined type guard function
which returns True if the type of the expression passed to its first
parameter matches its return TypeGuard type. For example, a function
that has a return type of TypeGuard[str] is assumed to return True if
and only if the type of the expression passed to its first input
parameter is a str. This allows type checkers to narrow types when a
user-defined type guard function returns True.

This PEP refines the TypeGuard mechanism introduced in PEP 647. It
allows type checkers to narrow types when a user-defined type guard
function returns False. It also allows type checkers to apply additional
(more precise) type narrowing under certain circumstances when the type
guard function returns True.

Motivation

User-defined type guard functions enable a type checker to narrow the
type of an expression when it is passed as an argument to the type guard
function. The TypeGuard mechanism introduced in PEP 647 is flexible, but
this flexibility imposes some limitations that developers have found
inconvenient for some uses.

Limitation 1: Type checkers are not allowed to narrow a type in the case
where the type guard function returns False. This means the type is not
narrowed in the negative ("else") clause.

Limitation 2: Type checkers must use the TypeGuard return type if the
type guard function returns True regardless of whether additional
narrowing can be applied based on knowledge of the pre-narrowed type.

The following code sample demonstrates both of these limitations.

    def is_iterable(val: object) -> TypeGuard[Iterable[Any]]:
        return isinstance(val, Iterable)

    def func(val: int | list[int]):
        if is_iterable(val):
            # The type is narrowed to 'Iterable[Any]' as dictated by
            # the TypeGuard return type
            reveal_type(val)  # Iterable[Any]
        else:
            # The type is not narrowed in the "False" case
            reveal_type(val)  # int | list[int]

        # If "isinstance" is used in place of the user-defined type guard
        # function, the results differ because type checkers apply additional
        # logic for "isinstance"

        if isinstance(val, Iterable):
            # Type is narrowed to "list[int]" because this is
            # a narrower (more precise) type than "Iterable[Any]"
            reveal_type(val)  # list[int]
        else:
            # Type is narrowed to "int" because the logic eliminates
            # "list[int]" from the original union
            reveal_type(val)  # int

PEP 647 imposed these limitations so it could support use cases where
the return TypeGuard type was not a subtype of the input type. Refer to
PEP 647 for examples.

Rationale

There are a number of issues where a stricter TypeGuard would have been
a solution:

-   Python typing issue - TypeGuard doesn't intersect like isinstance
-   Pyright issue - TypeGuard not eliminating possibility on branch
-   Pyright issue - Type narrowing for Literal doesn't work
-   Mypy issue - TypeGuard is incompatible with exhaustive check
-   Mypy issue - Incorrect type narrowing for inspect.isawaitable
-   Typeshed issue - asyncio.iscoroutinefunction is not a TypeGuard

Specification

The use of a user-defined type guard function involves five types:

-   I = TypeGuard input type
-   R = TypeGuard return type
-   A = Type of argument passed to type guard function (pre-narrowed)
-   NP = Narrowed type (positive)
-   NN = Narrowed type (negative)

    def guard(x: I) -> TypeGuard[R]: ...

    def func1(val: A):
        if guard(val):
            reveal_type(val)  # NP
        else:
            reveal_type(val)  # NN

This PEP proposes some modifications to PEP 647 to address the
limitations discussed above. These limitations are safe to eliminate
only when a specific condition is met. In particular, when the output
type R of a user-defined type guard function is consistent[1] with the
type of its first input parameter (I), type checkers should apply
stricter type guard semantics.

  # Stricter type guard semantics are used in this case because
      # "Kangaroo | Koala" is consistent with "Animal"
      def is_marsupial(val: Animal) -> TypeGuard[Kangaroo | Koala]:
          return isinstance(val, Kangaroo | Koala)

      # Stricter type guard semantics are not used in this case because
      # "list[T]"" is not consistent with "list[T | None]"
      def has_no_nones(val: list[T | None]) -> TypeGuard[list[T]]:
          return None not in val

When stricter type guard semantics are applied, the application of a
user-defined type guard function changes in two ways.

-   Type narrowing is applied in the negative ("else") case.

    def is_str(val: str | int) -> TypeGuard[str]:
        return isinstance(val, str)

    def func(val: str | int):
        if not is_str(val):
            reveal_type(val)  # int

-   Additional type narrowing is applied in the positive "if" case if
    applicable.

    def is_cardinal_direction(val: str) -> TypeGuard[Literal["N", "S", "E", "W"]]:
        return val in ("N", "S", "E", "W")

    def func(direction: Literal["NW", "E"]):
        if is_cardinal_direction(direction):
            reveal_type(direction)  # "Literal[E]"
        else:
            reveal_type(direction)  # "Literal[NW]"

The type-theoretic rules for type narrowing are specified in the
following table.

  \              Non-strict type guard     Strict type guard
  -------------- ------------------------- ---------------------
  Applies when   R not consistent with I   R consistent with I
  NP is ..       R                         A ∧ R
  NN is ..       A                         A ∧ ¬R

In practice, the theoretic types for strict type guards cannot be
expressed precisely in the Python type system. Type checkers should fall
back on practical approximations of these types. As a rule of thumb, a
type checker should use the same type narrowing logic -- and get results
that are consistent with -- its handling of "isinstance". This guidance
allows for changes and improvements if the type system is extended in
the future.

Additional Examples

Any is consistent[2] with any other type, which means stricter semantics
can be applied.

    # Stricter type guard semantics are used in this case because
    # "str" is consistent with "Any"
    def is_str(x: Any) -> TypeGuard[str]:
       return isinstance(x, str)

    def test(x: float | str):
       if is_str(x):
           reveal_type(x)  # str
       else:
           reveal_type(x)  # float

Backwards Compatibility

This PEP proposes to change the existing behavior of TypeGuard. This has
no effect at runtime, but it does change the types evaluated by a type
checker.

    def is_int(val: int | str) -> TypeGuard[int]:
        return isinstance(val, int)

    def func(val: int | str):
        if is_int(val):
            reveal_type(val)  # "int"
        else:
            reveal_type(val)  # Previously "int | str", now "str"

This behavioral change results in different types evaluated by a type
checker. It could therefore produce new (or mask existing) type errors.

Type checkers often improve narrowing logic or fix existing bugs in such
logic, so users of static typing will be used to this type of behavioral
change.

We also hypothesize that it is unlikely that existing typed Python code
relies on the current behavior of TypeGuard. To validate our hypothesis,
we implemented the proposed change in pyright and ran this modified
version on roughly 25 typed code bases using mypy primer to see if there
were any differences in the output. As predicted, the behavioral change
had minimal impact. The only noteworthy change was that some
# type: ignore comments were no longer necessary, indicating that these
code bases were already working around the existing limitations of
TypeGuard.

Breaking change

It is possible for a user-defined type guard function to rely on the old
behavior. Such type guard functions could break with the new behavior.

    def is_positive_int(val: int | str) -> TypeGuard[int]:
        return isinstance(val, int) and val > 0

    def func(val: int | str):
        if is_positive_int(val):
            reveal_type(val)  # "int"
        else:
            # With the older behavior, the type of "val" is evaluated as
            # "int | str"; with the new behavior, the type is narrowed to
            # "str", which is perhaps not what was intended.
            reveal_type(val)

We think it is unlikely that such user-defined type guards exist in
real-world code. The mypy primer results didn't uncover any such cases.

How to Teach This

Users unfamiliar with TypeGuard are likely to expect the behavior
outlined in this PEP, therefore making TypeGuard easier to teach and
explain.

Reference Implementation

A reference implementation of this idea exists in pyright.

To enable the modified behavior, the configuration flag
enableExperimentalFeatures must be set to true. This can be done on a
per-file basis by adding a comment:

    # pyright: enableExperimentalFeatures=true

Rejected Ideas

StrictTypeGuard

A new StrictTypeGuard construct was proposed. This alternative form
would be similar to a TypeGuard except it would apply stricter type
guard semantics. It would also enforce that the return type was
consistent [3] with the input type. See this thread for details:
StrictTypeGuard proposal

This idea was rejected because it is unnecessary in most cases and added
unnecessary complexity. It would require the introduction of a new
special form, and developers would need to be educated about the subtle
difference between the two forms.

TypeGuard with a second output type

Another idea was proposed where TypeGuard could support a second
optional type argument that indicates the type that should be used for
narrowing in the negative ("else") case.

    def is_int(val: int | str) -> TypeGuard[int, str]:
        return isinstance(val, int)

This idea was proposed here.

It was rejected because it was considered too complicated and addressed
only one of the two main limitations of TypeGuard. Refer to this thread
for the full discussion.

Footnotes

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

[1] PEP 483's discussion of is-consistent <483#summary-of-gradual-typing>

[2] PEP 483's discussion of is-consistent <483#summary-of-gradual-typing>

[3] PEP 483's discussion of is-consistent <483#summary-of-gradual-typing>


text:
PEP: 467 Title: Minor API improvements for binary sequences Version:
$Revision$ Last-Modified: $Date$ Author: Alyssa Coghlan
<EMAIL_ADDRESS>Ethan Furman<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/42001 Status: Draft Type: Standards Track
Content-Type: text/x-rst Created: 30-Mar-2014 Python-Version: 3.13
Post-History: 30-Mar-2014, 15-Aug-2014, 16-Aug-2014, 07-Jun-2016,
01-Sep-2016, 13-Apr-2021, 03-Nov-2021, 27-Dec-2023

Abstract

This PEP proposes small adjustments to the APIs of the bytes and
bytearray types to make it easier to operate entirely in the binary
domain:

-   Add fromsize alternative constructor
-   Add fromint alternative constructor
-   Add getbyte byte retrieval method
-   Add iterbytes alternative iterator

Rationale

During the initial development of the Python 3 language specification,
the core bytes type for arbitrary binary data started as the mutable
type that is now referred to as bytearray. Other aspects of operating in
the binary domain in Python have also evolved over the course of the
Python 3 series, for example with PEP 461.

Motivation

With Python 3 and the split between str and bytes, one small but
important area of programming became slightly more difficult, and much
more painful -- wire format protocols.

This area of programming is characterized by a mixture of binary data
and ASCII compatible segments of text (aka ASCII-encoded text). The
addition of the new constructors, methods, and iterators will aid both
in writing new wire format code, and in porting any remaining Python 2
wire format code.

Common use-cases include dbf and pdf file formats, email formats, and
FTP and HTTP communications, among many others.

Proposals

Addition of explicit "count and byte initialised sequence" constructors

To replace the discouraged behavior of creating zero-filled bytes-like
objects from the basic constructors (i.e. bytes(1) --> b'\x00'), this
PEP proposes the addition of an explicit fromsize alternative
constructor as a class method on both bytes and bytearray whose first
argument is the count, and whose second argument is the fill byte to use
(defaults to \x00):

    >>> bytes.fromsize(3)
    b'\x00\x00\x00'
    >>> bytearray.fromsize(3)
    bytearray(b'\x00\x00\x00')
    >>> bytes.fromsize(5, b'\x0a')
    b'\x0a\x0a\x0a\x0a\x0a'
    >>> bytearray.fromsize(5, fill=b'\x0a')
    bytearray(b'\x0a\x0a\x0a\x0a\x0a')

fromsize will behave just as the current constructors behave when passed
a single integer, while allowing for non-zero fill values when needed.

Addition of explicit "single byte" constructors

As binary counterparts to the text chr function, this PEP proposes the
addition of an explicit fromint alternative constructor as a class
method on both bytes and bytearray:

    >>> bytes.fromint(65)
    b'A'
    >>> bytearray.fromint(65)
    bytearray(b'A')

These methods will only accept integers in the range 0 to 255
(inclusive):

    >>> bytes.fromint(512)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ValueError: integer must be in range(0, 256)

    >>> bytes.fromint(1.0)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: 'float' object cannot be interpreted as an integer

The documentation of the ord builtin will be updated to explicitly note
that bytes.fromint is the primary inverse operation for binary data,
while chr is the inverse operation for text data, and that
bytearray.fromint also exists.

Behaviorally, bytes.fromint(x) will be equivalent to the current
bytes([x]) (and similarly for bytearray). The new spelling is expected
to be easier to discover and easier to read (especially when used in
conjunction with indexing operations on binary sequence types).

As a separate method, the new spelling will also work better with higher
order functions like map.

These new methods intentionally do NOT offer the same level of general
integer support as the existing int.to_bytes conversion method, which
allows arbitrarily large integers to be converted to arbitrarily long
bytes objects. The restriction to only accept positive integers that fit
in a single byte means that no byte order information is needed, and
there is no need to handle negative numbers. The documentation of the
new methods will refer readers to int.to_bytes for use cases where
handling of arbitrary integers is needed.

Addition of "getbyte" method to retrieve a single byte

This PEP proposes that bytes and bytearray gain the method getbyte which
will always return bytes:

    >>> b'abc'.getbyte(0)
    b'a'

If an index is asked for that doesn't exist, IndexError is raised:

    >>> b'abc'.getbyte(9)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    IndexError: index out of range

Addition of optimised iterator methods that produce bytes objects

This PEP proposes that bytes and bytearray gain an optimised iterbytes
method that produces length 1 bytes objects rather than integers:

    for x in data.iterbytes():
        # x is a length 1 ``bytes`` object, rather than an integer

For example:

    >>> tuple(b"ABC".iterbytes())
    (b'A', b'B', b'C')

Design discussion

Why not rely on sequence repetition to create zero-initialised sequences?

Zero-initialised sequences can be created via sequence repetition:

    >>> b'\x00' * 3
    b'\x00\x00\x00'
    >>> bytearray(b'\x00') * 3
    bytearray(b'\x00\x00\x00')

However, this was also the case when the bytearray type was originally
designed, and the decision was made to add explicit support for it in
the type constructor. The immutable bytes type then inherited that
feature when it was introduced in PEP 3137.

This PEP isn't revisiting that original design decision, just changing
the spelling as users sometimes find the current behavior of the binary
sequence constructors surprising. In particular, there's a reasonable
case to be made that bytes(x) (where x is an integer) should behave like
the bytes.fromint(x) proposal in this PEP. Providing both behaviors as
separate class methods avoids that ambiguity.

Current Workarounds

After nearly a decade, there's seems to be no consensus on the best
workarounds for byte iteration, as demonstrated by Get single-byte bytes
objects from bytes objects.

Omitting the originally proposed builtin function

When submitted to the Steering Council, this PEP proposed the
introduction of a bchr builtin (with the same behaviour as
bytes.fromint), recreating the ord/chr/unichr trio from Python 2 under a
different naming scheme (ord/bchr/chr).

The SC indicated they didn't think this functionality was needed often
enough to justify offering two ways of doing the same thing, especially
when one of those ways was a new builtin function. That part of the
proposal was therefore dropped as being redundant with the bytes.fromint
alternate constructor.

Developers that use this method frequently will instead have the option
to define their own bchr = bytes.fromint aliases.

Scope limitation: memoryview

Updating memoryview with the new item retrieval methods is outside the
scope of this PEP.

References

-   Initial March 2014 discussion thread on python-ideas
-   Guido's initial feedback in that thread
-   Issue proposing moving zero-initialised sequences to a dedicated API
-   Issue proposing to use calloc() for zero-initialised binary
    sequences
-   August 2014 discussion thread on python-dev
-   June 2016 discussion thread on python-dev
-   Get single-byte bytes objects from bytes objects

Copyright

This document has been placed in the public domain.

authors:
Alyssa Coghlan
Ethan Furman


text:
PEP: 386 Title: Changing the version comparison module in Distutils
Version: $Revision$ Last-Modified: $Date$ Author: Tarek Ziadé
<EMAIL_ADDRESS>Status: Superseded Type: Standards Track Topic:
Packaging Content-Type: text/x-rst Created: 04-Jun-2009 Superseded-By:
440

Abstract

Note: This PEP has been superseded by the version identification and
dependency specification scheme defined in PEP 440.

This PEP proposed a new version comparison schema system in Distutils.

Motivation

In Python there are no real restrictions yet on how a project should
manage its versions, and how they should be incremented.

Distutils provides a version distribution meta-data field but it is
freeform and current users, such as PyPI usually consider the latest
version pushed as the latest one, regardless of the expected semantics.

Distutils will soon extend its capabilities to allow distributions to
express a dependency on other distributions through the Requires-Dist
metadata field (see PEP 345) and it will optionally allow use of that
field to restrict the dependency to a set of compatible versions. Notice
that this field is replacing Requires that was expressing dependencies
on modules and packages.

The Requires-Dist field will allow a distribution to define a dependency
on another package and optionally restrict this dependency to a set of
compatible versions, so one may write:

    Requires-Dist: zope.interface (>3.5.0)

This means that the distribution requires zope.interface with a version
greater than 3.5.0.

This also means that Python projects will need to follow the same
convention as the tool that will be used to install them, so they are
able to compare versions.

That is why this PEP proposes, for the sake of interoperability, a
standard schema to express version information and its comparison
semantics.

Furthermore, this will make OS packagers' work easier when repackaging
standards compliant distributions, because as of now it can be difficult
to decide how two distribution versions compare.

Requisites and current status

It is not in the scope of this PEP to provide a universal versioning
schema intended to support all or even most of existing versioning
schemas. There will always be competing grammars, either mandated by
distro or project policies or by historical reasons that we cannot
expect to change.

The proposed schema should be able to express the usual versioning
semantics, so it's possible to parse any alternative versioning schema
and transform it into a compliant one. This is how OS packagers usually
deal with the existing version schemas and is a preferable alternative
than supporting an arbitrary set of versioning schemas.

Conformance to usual practice and conventions, as well as a simplicity
are a plus, to ease frictionless adoption and painless transition.
Practicality beats purity, sometimes.

Projects have very different versioning needs, but the following are
widely considered important semantics:

1.  it should be possible to express more than one versioning level
    (usually this is expressed as major and minor revision and,
    sometimes, also a micro revision).
2.  a significant number of projects need special meaning versions for
    "pre-releases" (such as "alpha", "beta", "rc"), and these have
    widely used aliases ("a" stands for "alpha", "b" for "beta" and "c"
    for "rc"). And these pre-release versions make it impossible to use
    a simple alphanumerical ordering of the version string components.
    (Example: 3.1a1 < 3.1)
3.  some projects also need "post-releases" of regular versions, mainly
    for installer work which can't be clearly expressed otherwise.
4.  development versions allow packagers of unreleased work to avoid
    version clash with later regular releases.

For people that want to go further and use a tool to manage their
version numbers, the two major ones are:

-   The current Distutils system[1]
-   Setuptools[2]

Distutils

Distutils currently provides a StrictVersion and a LooseVersion class
that can be used to manage versions.

The LooseVersion class is quite lax. From Distutils doc:

    Version numbering for anarchists and software realists.
    Implements the standard interface for version number classes as
    described above.  A version number consists of a series of numbers,
    separated by either periods or strings of letters.  When comparing
    version numbers, the numeric components will be compared
    numerically, and the alphabetic components lexically.  The following
    are all valid version numbers, in no particular order:

        1.5.1
        1.5.2b2
        161
        3.10a
        8.02
        3.4j
        1996.07.12
        3.2.pl0
        <IP_ADDRESS>
        2g6
        11g
        0.960923
        2.2beta29
        1.13++
        5.5.kw
        2.0b1pl0

    In fact, there is no such thing as an invalid version number under
    this scheme; the rules for comparison are simple and predictable,
    but may not always give the results you want (for some definition
    of "want").

This class makes any version string valid, and provides an algorithm to
sort them numerically then lexically. It means that anything can be used
to version your project:

    >>> from distutils.version import LooseVersion as V
    >>> v1 = V('FunkyVersion')
    >>> v2 = V('GroovieVersion')
    >>> v1 > v2
    False

The problem with this is that while it allows expressing any nesting
level it doesn't allow giving special meaning to versions (pre and
post-releases as well as development versions), as expressed in
requisites 2, 3 and 4.

The StrictVersion class is more strict. From the doc:

    Version numbering for meticulous retentive and software idealists.
    Implements the standard interface for version number classes as
    described above.  A version number consists of two or three
    dot-separated numeric components, with an optional "pre-release" tag
    on the end.  The pre-release tag consists of the letter 'a' or 'b'
    followed by a number.  If the numeric components of two version
    numbers are equal, then one with a pre-release tag will always
    be deemed earlier (lesser) than one without.

    The following are valid version numbers (shown in the order that
    would be obtained by sorting according to the supplied cmp function):

        0.4       0.4.0  (these two are equivalent)
        0.4.1
        0.5a1
        0.5b3
        0.5
        0.9.6
        1.0
        1.0.4a3
        1.0.4b1
        1.0.4

    The following are examples of invalid version numbers:

        1
        <IP_ADDRESS>
        1.3.a4
        1.3pl1
        1.3c4

This class enforces a few rules, and makes a decent tool to work with
version numbers:

    >>> from distutils.version import StrictVersion as V
    >>> v2 = V('GroovieVersion')
    Traceback (most recent call last):
    ...
    ValueError: invalid version number 'GroovieVersion'
    >>> v2 = V('1.1')
    >>> v3 = V('1.3')
    >>> v2 < v3
    True

It adds pre-release versions, and some structure, but lacks a few
semantic elements to make it usable, such as development releases or
post-release tags, as expressed in requisites 3 and 4.

Also, note that Distutils version classes have been present for years
but are not really used in the community.

Setuptools

Setuptools provides another version comparison tool[3] which does not
enforce any rules for the version, but tries to provide a better
algorithm to convert the strings to sortable keys, with a parse_version
function.

From the doc:

    Convert a version string to a chronologically-sortable key

    This is a rough cross between Distutils' StrictVersion and LooseVersion;
    if you give it versions that would work with StrictVersion, then it behaves
    the same; otherwise it acts like a slightly-smarter LooseVersion. It is
    *possible* to create pathological version coding schemes that will fool
    this parser, but they should be very rare in practice.

    The returned value will be a tuple of strings.  Numeric portions of the
    version are padded to 8 digits so they will compare numerically, but
    without relying on how numbers compare relative to strings.  Dots are
    dropped, but dashes are retained.  Trailing zeros between alpha segments
    or dashes are suppressed, so that e.g. "2.4.0" is considered the same as
    "2.4". Alphanumeric parts are lower-cased.

    The algorithm assumes that strings like "-" and any alpha string that
    alphabetically follows "final"  represents a "patch level".  So, "2.4-1"
    is assumed to be a branch or patch of "2.4", and therefore "2.4.1" is
    considered newer than "2.4-1", which in turn is newer than "2.4".

    Strings like "a", "b", "c", "alpha", "beta", "candidate" and so on (that
    come before "final" alphabetically) are assumed to be pre-release versions,
    so that the version "2.4" is considered newer than "2.4a1".

    Finally, to handle miscellaneous cases, the strings "pre", "preview", and
    "rc" are treated as if they were "c", i.e. as though they were release
    candidates, and therefore are not as new as a version string that does not
    contain them, and "dev" is replaced with an '@' so that it sorts lower
    than any other pre-release tag.

In other words, parse_version will return a tuple for each version
string, that is compatible with StrictVersion but also accept arbitrary
version and deal with them so they can be compared:

    >>> from pkg_resources import parse_version as V
    >>> V('1.2')
    ('00000001', '00000002', '*final')
    >>> V('1.2b2')
    ('00000001', '00000002', '*b', '00000002', '*final')
    >>> V('FunkyVersion')
    ('*funkyversion', '*final')

In this schema practicality takes priority over purity, but as a result
it doesn't enforce any policy and leads to very complex semantics due to
the lack of a clear standard. It just tries to adapt to widely used
conventions.

Caveats of existing systems

The major problem with the described version comparison tools is that
they are too permissive and, at the same time, aren't capable of
expressing some of the required semantics. Many of the versions on
PyPI[4] are obviously not useful versions, which makes it difficult for
users to grok the versioning that a particular package was using and to
provide tools on top of PyPI.

Distutils classes are not really used in Python projects, but the
Setuptools function is quite widespread because it's used by tools like
easy_install[5], pip[6] or zc.buildout [7] to install dependencies of a
given project.

While Setuptools does provide a mechanism for comparing/sorting
versions, it is much preferable if the versioning spec is such that a
human can make a reasonable attempt at that sorting without having to
run it against some code.

Also there's a problem with the use of dates at the "major" version
number (e.g. a version string "20090421") with RPMs: it means that any
attempt to switch to a more typical "major.minor..." version scheme is
problematic because it will always sort less than "20090421".

Last, the meaning of - is specific to Setuptools, while it is avoided in
some packaging systems like the one used by Debian or Ubuntu.

The new versioning algorithm

During Pycon, members of the Python, Ubuntu and Fedora community worked
on a version standard that would be acceptable for everyone.

It's currently called verlib and a prototype lives at[8].

The pseudo-format supported is:

    N.N[.N]+[{a|b|c|rc}N[.N]+][.postN][.devN]

The real regular expression is:

    expr = r"""^
    (?P<version>\d+\.\d+)         # minimum 'N.N'
    (?P<extraversion>(?:\.\d+)*)  # any number of extra '.N' segments
    (?:
        (?P<prerel>[abc]|rc)         # 'a' = alpha, 'b' = beta
                                     # 'c' or 'rc' = release candidate
        (?P<prerelversion>\d+(?:\.\d+)*)
    )?
    (?P<postdev>(\.post(?P<post>\d+))?(\.dev(?P<dev>\d+))?)?
    $"""

Some examples probably make it clearer:

    >>> from verlib import NormalizedVersion as V
    >>> (V('1.0a1')
    ...  < V('1.0a2.dev456')
    ...  < V('1.0a2')
    ...  < V('1.0a2.1.dev456')
    ...  < V('1.0a2.1')
    ...  < V('1.0b1.dev456')
    ...  < V('1.0b2')
    ...  < V('1.0b2.post345')
    ...  < V('1.0c1.dev456')
    ...  < V('1.0c1')
    ...  < V('1.0.dev456')
    ...  < V('1.0')
    ...  < V('1.0.post456.dev34')
    ...  < V('1.0.post456'))
    True

The trailing .dev123 is for pre-releases. The .post123 is for
post-releases -- which apparently are used by a number of projects out
there (e.g. Twisted[9]). For example, after a 1.2.0 release there might
be a 1.2.0-r678 release. We used post instead of r because the r is
ambiguous as to whether it indicates a pre- or post-release.

.post456.dev34 indicates a dev marker for a post release, that sorts
before a .post456 marker. This can be used to do development versions of
post releases.

Pre-releases can use a for "alpha", b for "beta" and c for "release
candidate". rc is an alternative notation for "release candidate" that
is added to make the version scheme compatible with Python's own version
scheme. rc sorts after c:

    >>> from verlib import NormalizedVersion as V
    >>> (V('1.0a1')
    ...  < V('1.0a2')
    ...  < V('1.0b3')
    ...  < V('1.0c1')
    ...  < V('1.0rc2')
    ...  < V('1.0'))
    True

Note that c is the preferred marker for third party projects.

verlib provides a NormalizedVersion class and a
suggest_normalized_version function.

NormalizedVersion

The NormalizedVersion class is used to hold a version and to compare it
with others. It takes a string as an argument, that contains the
representation of the version:

    >>> from verlib import NormalizedVersion
    >>> version = NormalizedVersion('1.0')

The version can be represented as a string:

    >>> str(version)
    '1.0'

Or compared with others:

    >>> NormalizedVersion('1.0') > NormalizedVersion('0.9')
    True
    >>> NormalizedVersion('1.0') < NormalizedVersion('1.1')
    True

A class method called from_parts is available if you want to create an
instance by providing the parts that composes the version.

Examples :

    >>> version = NormalizedVersion.from_parts((1, 0))
    >>> str(version)
    '1.0'

    >>> version = NormalizedVersion.from_parts((1, 0), ('c', 4))
    >>> str(version)
    '1.0c4'

    >>> version = NormalizedVersion.from_parts((1, 0), ('c', 4), ('dev', 34))
    >>> str(version)
    '1.0c4.dev34'

suggest_normalized_version

suggest_normalized_version is a function that suggests a normalized
version close to the given version string. If you have a version string
that isn't normalized (i.e. NormalizedVersion doesn't like it) then you
might be able to get an equivalent (or close) normalized version from
this function.

This does a number of simple normalizations to the given string, based
on an observation of versions currently in use on PyPI.

Given a dump of those versions on January 6th 2010, the function has
given those results out of the 8821 distributions PyPI had:

-   7822 (88.67%) already match NormalizedVersion without any change
-   717 (8.13%) match when using this suggestion method
-   282 (3.20%) don't match at all.

The 3.20% of projects that are incompatible with NormalizedVersion and
cannot be transformed into a compatible form, are for most of them
date-based version schemes, versions with custom markers, or dummy
versions. Examples:

-   working proof of concept
-   1 (first draft)
-   unreleased.unofficialdev
-   0.1.alphadev
-   2008-03-29_r219
-   etc.

When a tool needs to work with versions, a strategy is to use
suggest_normalized_version on the versions string. If this function
returns None, it means that the provided version is not close enough to
the standard scheme. If it returns a version that slightly differs from
the original version, it's a suggested normalized version. Last, if it
returns the same string, it means that the version matches the scheme.

Here's an example of usage:

    >>> from verlib import suggest_normalized_version, NormalizedVersion
    >>> import warnings
    >>> def validate_version(version):
    ...     rversion = suggest_normalized_version(version)
    ...     if rversion is None:
    ...         raise ValueError('Cannot work with "%s"' % version)
    ...     if rversion != version:
    ...         warnings.warn('"%s" is not a normalized version.\n'
    ...                       'It has been transformed into "%s" '
    ...                       'for interoperability.' % (version, rversion))
    ...     return NormalizedVersion(rversion)
    ...

    >>> validate_version('2.4-rc1')
    __main__:8: UserWarning: "2.4-rc1" is not a normalized version.
    It has been transformed into "2.4c1" for interoperability.
    NormalizedVersion('2.4c1')

    >>> validate_version('2.4c1')
    NormalizedVersion('2.4c1')

    >>> validate_version('foo')
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "<stdin>", line 4, in validate_version
    ValueError: Cannot work with "foo"

Roadmap

Distutils will deprecate its existing versions class in favor of
NormalizedVersion. The verlib module presented in this PEP will be
renamed to version and placed into the distutils package.

References

Acknowledgments

Trent Mick, Matthias Klose, Phillip Eby, David Lyon, and many people at
Pycon and Distutils-SIG.

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] http://docs.python.org/distutils

[2] http://peak.telecommunity.com/DevCenter/setuptools

[3] http://peak.telecommunity.com/DevCenter/setuptools#specifying-your-project-s-version

[4] http://pypi.python.org/pypi

[5] http://peak.telecommunity.com/DevCenter/EasyInstall

[6] http://pypi.python.org/pypi/pip

[7] http://pypi.python.org/pypi/zc.buildout

[8] http://bitbucket.org/tarek/distutilsversion/

[9] http://twistedmatrix.com/trac/


text:
PEP: 4 Title: Deprecation of Standard Modules Version: $Revision$
Last-Modified: $Date$ Author: Brett Cannon<EMAIL_ADDRESS>Martin
von Löwis<EMAIL_ADDRESS>Status: Active Type: Process
Content-Type: text/x-rst Created: 01-Oct-2000 Post-History:

Introduction

When new modules were added to the standard Python library in the past,
it was not possible to foresee whether they would still be useful in the
future. Even though Python "Comes With Batteries Included", batteries
may discharge over time. Carrying old modules around is a burden on the
maintainer, especially when there is no interest in the module anymore.

At the same time, removing a module from the distribution is difficult,
as it is not known in general whether anybody is still using it. This
PEP defines a procedure for removing modules from the standard Python
library. Usage of a module may be 'deprecated', which means that it may
be removed from a future Python release.

Procedure for declaring a module deprecated

To remove a top-level module/package from the standard library, a PEP is
required. The deprecation process is outlined in PEP 387.

For removing a submodule of a package in the standard library, PEP 387
must be followed, but a PEP is not required.

Copyright

This document has been placed in the public domain.

authors:
Brett Cannon
Martin von Löwis