text:
PEP: 313 Title: Adding Roman Numeral Literals to Python Version:
$Revision$ Last-Modified: $Date$ Author: Mike Meyer<EMAIL_ADDRESS>Status: Rejected Type: Standards Track Content-Type: text/x-rst Created:
01-Apr-2003 Python-Version: 2.4 Post-History:

Abstract

This PEP (also known as PEP CCCXIII) proposes adding Roman numerals as a
literal type. It also proposes the new built-in function "roman", which
converts an object to an integer, then converts the integer to a string
that is the Roman numeral literal equivalent to the integer.

BDFL Pronouncement

This PEP is rejected. While the majority of Python users deemed this to
be a nice-to-have feature, the community was unable to reach a consensus
on whether nine should be represented as IX, the modern form, or VIIII,
the classic form. Likewise, no agreement was reached on whether MXM or
MCMXC would be considered a well-formed representation of 1990. A vocal
minority of users has also requested support for lower-cased numerals
for use in (i) powerpoint slides, (ii) academic work, and (iii) Perl
documentation.

Rationale

Roman numerals are used in a number of areas, and adding them to Python
as literals would make computations in those areas easier. For instance,
Super Bowls are counted with Roman numerals, and many older movies have
copyright dates in Roman numerals. Further, LISP provides a Roman
numerals literal package, so adding Roman numerals to Python will help
ease the LISP-envy sometimes seen in comp.lang.python. Besides, the
author thinks this is the easiest way to get his name on a PEP.

Syntax for Roman literals

Roman numeral literals will consist of the characters M, D, C, L, X, V
and I, and only those characters. They must be in upper case, and
represent an integer with the following rules:

1.  Except as noted below, they must appear in the order M, D, C, L, X,
    V then I. Each occurrence of each character adds 1000, 500, 100, 50,
    10, 5 and 1 to the value of the literal, respectively.
2.  Only one D, V or L may appear in any given literal.
3.  At most three each of Is, Xs and Cs may appear consecutively in any
    given literal.
4.  A single I may appear immediately to the left of the single V,
    followed by no Is, and adds 4 to the value of the literal.
5.  A single I may likewise appear before the last X, followed by no Is
    or Vs, and adds 9 to the value.
6.  X is to L and C as I is to V and X, except the values are 40 and 90,
    respectively.
7.  C is to D and M as I is to V and X, except the values are 400 and
    900, respectively.

Any literal composed entirely of M, D, C, L, X, V and I characters that
does not follow this format will raise a syntax error, because explicit
is better than implicit.

Built-In "roman" Function

The new built-in function "roman" will aide the translation from
integers to Roman numeral literals. It will accept a single object as an
argument, and return a string containing the literal of the same value.
If the argument is not an integer or a rational (see PEP 239) it will
passed through the existing built-in "int" to obtain the value. This may
cause a loss of information if the object was a float. If the object is
a rational, then the result will be formatted as a rational literal (see
PEP 240) with the integers in the string being Roman numeral literals.

Compatibility Issues

No new keywords are introduced by this proposal. Programs that use
variable names that are all upper case and contain only the characters
M, D, C, L, X, V and I will be affected by the new literals. These
programs will now have syntax errors when those variables are assigned,
and either syntax errors or subtle bugs when those variables are
referenced in expressions. Since such variable names violate PEP 8, the
code is already broken, it just wasn't generating exceptions. This
proposal corrects that oversight in the language.

Copyright

This document has been placed in the public domain.


text:
PEP: 744 Title: JIT Compilation Author: Brandt Bucher
<EMAIL_ADDRESS>Savannah Ostrowski<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pep-744-jit-compilation/50756 Status: Draft
Type: Informational Created: 11-Apr-2024 Python-Version: 3.13
Post-History: 11-Apr-2024

Abstract

Earlier this year, an experimental "just-in-time" compiler was merged
into CPython's main development branch. While recent CPython releases
have included other substantial internal changes, this addition
represents a particularly significant departure from the way CPython has
traditionally executed Python code. As such, it deserves wider
discussion.

This PEP aims to summarize the design decisions behind this addition,
the current state of the implementation, and future plans for making the
JIT a permanent, non-experimental part of CPython. It does not seek to
provide a comprehensive overview of how the JIT works, instead focusing
on the particular advantages and disadvantages of the chosen approach,
as well as answering many questions that have been asked about the JIT
since its introduction.

Readers interested in learning more about the new JIT are encouraged to
consult the following resources:

-   The presentation which first introduced the JIT at the 2023 CPython
    Core Developer Sprint. It includes relevant background, a light
    technical introduction to the "copy-and-patch" technique used, and
    an open discussion of its design amongst the core developers
    present. Slides for this talk can be found on GitHub.
-   The open access paper originally describing copy-and-patch.
-   The blog post by the paper's author detailing the implementation of
    a copy-and-patch JIT compiler for Lua. While this is a great
    low-level explanation of the approach, note that it also
    incorporates other techniques and makes implementation decisions
    that are not particularly relevant to CPython's JIT.
-   The implementation itself.

Motivation

Until this point, CPython has always executed Python code by compiling
it to bytecode, which is interpreted at runtime. This bytecode is a
more-or-less direct translation of the source code: it is untyped, and
largely unoptimized.

Since the Python 3.11 release, CPython has used a "specializing adaptive
interpreter" (PEP 659), which rewrites these bytecode instructions
in-place with type-specialized versions as they run. This new
interpreter delivers significant performance improvements, despite the
fact that its optimization potential is limited by the boundaries of
individual bytecode instructions. It also collects a wealth of new
profiling information: the types flowing though a program, the memory
layout of particular objects, and what paths through the program are
being executed the most. In other words, what to optimize, and how to
optimize it.

Since the Python 3.12 release, CPython has generated this interpreter
from a C-like domain-specific language (DSL). In addition to taming some
of the complexity of the new adaptive interpreter, the DSL also allows
CPython's maintainers to avoid hand-writing tedious boilerplate code in
many parts of the interpreter, compiler, and standard library that must
be kept in sync with the instruction definitions. This ability to
generate large amounts of runtime infrastructure from a single source of
truth is not only convenient for maintenance; it also unlocks many
possibilities for expanding CPython's execution in new ways. For
instance, it makes it feasible to automatically generate tables for
translating a sequence of instructions into an equivalent sequence of
smaller "micro-ops", generate an optimizer for sequences of these
micro-ops, and even generate an entire second interpreter for executing
them.

In fact, since early in the Python 3.13 release cycle, all CPython
builds have included this exact micro-op translation, optimization, and
execution machinery. However, it is disabled by default; the overhead of
interpreting even optimized traces of micro-ops is just too large for
most code. Heavier optimization probably won't improve the situation
much either, since any efficiency gains made by new optimizations will
likely be offset by the interpretive overhead of even smaller, more
complex micro-ops.

The most obvious strategy to overcome this new bottleneck is to
statically compile these optimized traces. This presents opportunities
to avoid several sources of indirection and overhead introduced by
interpretation. In particular, it allows the removal of dispatch
overhead between micro-ops (by replacing a generic interpreter with a
straight-line sequence of hot code), instruction decoding overhead for
individual micro-ops (by "burning" the values or addresses of arguments,
constants, and cached values directly into machine instructions), and
memory traffic (by moving data off of heap-allocated Python frames and
into physical hardware registers).

Since much of this data varies even between identical runs of a program
and the existing optimization pipeline makes heavy use of runtime
profiling information, it doesn't make much sense to compile these
traces ahead of time and would be a substantial redesign of the existing
specification and micro-op tracing infrastructure that has already been
implemented. As has been demonstrated for many other dynamic languages
(and even Python itself), the most promising approach is to compile the
optimized micro-ops "just in time" for execution.

Rationale

Despite their reputation, JIT compilers are not magic "go faster"
machines. Developing and maintaining any sort of optimizing compiler for
even a single platform, let alone all of CPython's most popular
supported platforms, is an incredibly complicated, expensive task. Using
an existing compiler framework like LLVM can make this task simpler, but
only at the cost of introducing heavy runtime dependencies and
significantly higher JIT compilation overhead.

It's clear that successfully compiling Python code at runtime requires
not only high-quality Python-specific optimizations for the code being
run, but also quick generation of efficient machine code for the
optimized program. The Python core development team has the necessary
skills and experience for the former (a middle-end tightly coupled to
the interpreter), and copy-and-patch compilation provides an attractive
solution for the latter.

In a nutshell, copy-and-patch allows a high-quality template JIT
compiler to be generated from the same DSL used to generate the rest of
the interpreter. For a widely-used, volunteer-driven project like
CPython, this benefit cannot be overstated: CPython's maintainers, by
merely editing the bytecode definitions, will also get the JIT backend
updated "for free", for all JIT-supported platforms, at once. This is
equally true whether instructions are being added, modified, or removed.

Like the rest of the interpreter, the JIT compiler is generated at build
time, and has no runtime dependencies. It supports a wide range of
platforms (see the Support section below), and has comparatively low
maintenance burden. In all, the current implementation is made up of
about 900 lines of build-time Python code and 500 lines of runtime C
code.

Specification

The JIT is currently not part of the default build configuration, and it
is likely to remain that way for the foreseeable future (though official
binaries may include it). That said, the JIT will become
non-experimental once all of the following conditions are met:

1.  It provides a meaningful performance improvement for at least one
    popular platform (realistically, on the order of 5%).
2.  It can be built, distributed, and deployed with minimal disruption.
3.  The Steering Council, upon request, has determined that it would
    provide more value to the community if enabled than if disabled
    (considering tradeoffs such as maintenance burden, memory usage, or
    the feasibility of alternate designs).

These criteria should be considered a starting point, and may be
expanded over time. For example, discussion of this PEP may reveal that
additional requirements (such as multiple committed maintainers, a
security audit, documentation in the devguide, support for
out-of-process debugging, or a runtime option to disable the JIT) should
be added to this list.

Until the JIT is non-experimental, it should not be used in production,
and may be broken or removed at any time without warning.

Once the JIT is no longer experimental, it should be treated in much the
same way as other build options such as --enable-optimizations or
--with-lto. It may be a recommended (or even default) option for some
platforms, and release managers may choose to enable it in official
releases.

Support

The JIT has been developed for all of PEP 11's current tier one
platforms, most of its tier two platforms, and one of its tier three
platforms. Specifically, CPython's main branch has CI building and
testing the JIT for both release and debug builds on:

-   aarch64-apple-darwin/clang
-   aarch64-pc-windows/msvc[1]
-   aarch64-unknown-linux-gnu/clang[2]
-   aarch64-unknown-linux-gnu/gcc[3]
-   i686-pc-windows-msvc/msvc
-   x86_64-apple-darwin/clang
-   x86_64-pc-windows-msvc/msvc
-   x86_64-unknown-linux-gnu/clang
-   x86_64-unknown-linux-gnu/gcc

It's worth noting that some platforms, even future tier one platforms,
may never gain JIT support. This can be for a variety of reasons,
including insufficient LLVM support (powerpc64le-unknown-linux-gnu/gcc),
inherent limitations of the platform (wasm32-unknown-wasi/clang), or
lack of developer interest (x86_64-unknown-freebsd/clang).

Once JIT support for a platform is added (meaning, the JIT builds
successfully without displaying warnings to the user), it should be
treated in much the same way as PEP 11 prescribes: it should have
reliable CI/buildbots, and JIT failures on tier one and tier two
platforms should block releases. Though it's not necessary to update PEP
11 to specify JIT support, it may be helpful to do so anyway. Otherwise,
a list of supported platforms should be maintained in the JIT's README.

Since it should always be possible to build CPython without the JIT,
removing JIT support for a platform should not be considered a
backwards-incompatible change. However, if it is reasonable to do so,
the normal deprecation process should be followed as outlined in PEP
387.

The JIT's build-time dependencies may be changed between releases,
within reason.

Backwards Compatibility

Due to the fact that the current interpreter and the JIT backend are
both generated from the same specification, the behavior of Python code
should be completely unchanged. In practice, observable differences that
have been found and fixed during testing have tended to be bugs in the
existing micro-op translation and optimization stages, rather than bugs
in the copy-and-patch step.

Debugging

Tools that profile and debug Python code will continue to work fine.
This includes in-process tools that use Python-provided functionality
(like sys.monitoring, sys.settrace, or sys.setprofile), as well as
out-of-process tools that walk Python frames from the interpreter state.

However, it appears that profilers and debuggers for C code are
currently unable to trace back through JIT frames. Working with leaf
frames is possible (this is how the JIT itself is debugged), though it
is of limited utility due to the absence of proper debugging information
for JIT frames.

Since the code templates emitted by the JIT are compiled by Clang, it
may be possible to allow JIT frames to be traced through by simply
modifying the compiler flags to use frame pointers more carefully. It
may also be possible to harvest and emit the debugging information
produced by Clang. Neither of these ideas have been explored very
deeply.

While this is an issue that should be fixed, fixing it is not a
particularly high priority at this time. This is probably a problem best
explored by somebody with more domain expertise in collaboration with
those maintaining the JIT, who have little experience with the inner
workings of these tools.

Security Implications

This JIT, like any JIT, produces large amounts of executable data at
runtime. This introduces a potential new attack surface to CPython,
since a malicious actor capable of influencing the contents of this data
is therefore capable of executing arbitrary code. This is a well-known
vulnerability of JIT compilers.

In order to mitigate this risk, the JIT has been written with best
practices in mind. In particular, the data in question is not exposed by
the JIT compiler to other parts of the program while it remains
writable, and at no point is the data both writable and executable_.

The nature of template-based JITs also seriously limits the kinds of
code that can be generated, further reducing the likelihood of a
successful exploit. As an additional precaution, the templates
themselves are stored in static, read-only memory.

However, it would be naive to assume that no possible vulnerabilities
exist in the JIT, especially at this early stage. The author is not a
security expert, but is available to join or work closely with the
Python Security Response Team to triage and fix security issues as they
arise.

Apple Silicon

Though difficult to test without actually signing and packaging a macOS
release, it appears that macOS releases should enable the JIT
Entitlement for the Hardened Runtime.

This shouldn't make installing Python any harder, but may add additional
steps for release managers to perform.

How to Teach This

Choose the sections that best describe you:

-   If you are a Python programmer or end user...
    -   ...nothing changes for you. Nobody should be distributing
        JIT-enabled CPython interpreters to you while it is still an
        experimental feature. Once it is non-experimental, you will
        probably notice slightly better performance and slightly higher
        memory usage. You shouldn't be able to observe any other
        changes.
-   If you maintain third-party packages...
    -   ...nothing changes for you. There are no API or ABI changes, and
        the JIT is not exposed to third-party code. You shouldn't need
        to change your CI matrix, and you shouldn't be able to observe
        differences in the way your packages work when the JIT is
        enabled.
-   If you profile or debug Python code...
    -   ...nothing changes for you. All Python profiling and tracing
        functionality remains.
-   If you profile or debug C code...
    -   ...currently, the ability to trace through JIT frames is
        limited. This may cause issues if you need to observe the entire
        C call stack, rather than just "leaf" frames. See the Debugging
        section above for more information.
-   If you compile your own Python interpreter....
    -   ...if you don't wish to build the JIT, you can simply ignore it.
        Otherwise, you will need to install a compatible version of
        LLVM, and pass the appropriate flag to the build scripts. Your
        build may take up to a minute longer. Note that the JIT should
        not be distributed to end users or used in production while it
        is still in the experimental phase.
-   If you're a maintainer of CPython (or a fork of CPython)...
    -   ...and you change the bytecode definitions or the main
        interpreter loop...
        -   ...in general, the JIT shouldn't be much of an inconvenience
            to you (depending on what you're trying to do). The micro-op
            interpreter isn't going anywhere, and still offers a
            debugging experience similar to what the main bytecode
            interpreter provides today. There is moderate likelihood
            that larger changes to the interpreter (such as adding new
            local variables, changing error handling and deoptimization
            logic, or changing the micro-op format) will require changes
            to the C template used to generate the JIT, which is meant
            to mimic the main interpreter loop. You may also
            occasionally just get unlucky and break JIT code generation,
            which will require you to either modify the Python build
            scripts yourself, or solicit the help of somebody more
            familiar with them (see below).
    -   ...and you work on the JIT itself...
        -   ...you hopefully already have a decent idea of what you're
            getting yourself into. You will be regularly modifying the
            Python build scripts, the C template used to generate the
            JIT, and the C code that actually makes up the runtime
            portion of the JIT. You will also be dealing with all sorts
            of crashes, stepping over machine code in a debugger,
            staring at COFF/ELF/Mach-O dumps, developing on a wide range
            of platforms, and generally being the point of contact for
            the people changing the bytecode when CI starts failing on
            their PRs (see above). Ideally, you're at least familiar
            with assembly, have taken a couple of courses with
            "compilers" in their name, and have read a blog post or two
            about linkers.
    -   ...and you maintain other parts of CPython...
        -   ...nothing changes for you. You shouldn't need to develop
            locally with JIT builds. If you choose to do so (for
            example, to help reproduce and triage JIT issues), your
            builds may take up to a minute longer each time the relevant
            files are modified.

Reference Implementation

Key parts of the implementation include:

-   Tools/jit/README.md_: Instructions for how to build the JIT.
-   Python/jit.c_: The entire runtime portion of the JIT compiler.
-   jit_stencils.h_: An example of the JIT's generated templates.
-   Tools/jit/template.c_: The code which is compiled to produce the
    JIT's templates.
-   Tools/jit/_targets.py_: The code to compile and parse the templates
    at build time.

Rejected Ideas

Maintain it outside of CPython

While it is probably possible to maintain the JIT outside of CPython,
its implementation is tied tightly enough to the rest of the interpreter
that keeping it up-to-date would probably be more difficult than
actually developing the JIT itself. Additionally, contributors working
on the existing micro-op definitions and optimizations would need to
modify and build two separate projects to measure the effects of their
changes under the JIT (whereas today, infrastructure exists to do this
automatically for any proposed change).

Releases of the separate "JIT" project would probably also need to
correspond to specific CPython pre-releases and patch releases,
depending on exactly what changes are present. Individual CPython
commits between releases likely wouldn't have corresponding JIT releases
at all, further complicating debugging efforts (such as bisection to
find breaking changes upstream).

Since the JIT is already quite stable, and the ultimate goal is for it
to be a non-experimental part of CPython, keeping it in main seems to be
the best path forward. With that said, the relevant code is organized in
such a way that the JIT can be easily "deleted" if it does not end up
meeting its goals.

Turn it on by default

On the other hand, some have suggested that the JIT should be enabled by
default in its current form.

Again, it is important to remember that a JIT is not a magic "go faster"
machine; currently, the JIT is about as fast as the existing
specializing interpreter. This may sound underwhelming, but it is
actually a fairly significant achievement, and it's the main reason why
this approach was considered viable enough to be merged into main for
further development.

While the JIT provides significant gains over the existing micro-op
interpreter, it isn't yet a clear win when always enabled (especially
considering its increased memory consumption and additional build-time
dependencies). That's the purpose of this PEP: to clarify expectations
about the objective criteria that should be met in order to "flip the
switch".

At least for now, having this in main, but off by default, seems to be a
good compromise between always turning it on and not having it available
at all.

Support multiple compiler toolchains

Clang is specifically needed because it's the only C compiler with
support for guaranteed tail calls (musttail_), which are required by
CPython's continuation-passing-style approach to JIT compilation.
Without it, the tail-recursive calls between templates could result in
unbounded C stack growth (and eventual overflow).

Since LLVM also includes other functionalities required by the JIT build
process (namely, utilities for object file parsing and disassembly), and
additional toolchains introduce additional testing and maintenance
burden, it's convenient to only support one major version of one
toolchain at this time.

Compile the base interpreter's bytecode

Most of the prior art for copy-and-patch uses it as a fast baseline JIT,
whereas CPython's JIT is using the technique to compile optimized
micro-op traces.

In practice, the new JIT currently sits somewhere between the "baseline"
and "optimizing" compiler tiers of other dynamic language runtimes. This
is because CPython uses its specializing adaptive interpreter to collect
runtime profiling information, which is used to detect and optimize
"hot" paths through the code. This step is carried out using
self-modifying code, a technique which is much more difficult to
implement with a JIT compiler.

While it's possible to compile normal bytecode using copy-and-patch (in
fact, early prototypes predated the micro-op interpreter and did exactly
this), it just doesn't seem to provide enough optimization potential as
the more granular micro-op format.

Add GPU support

The JIT is currently CPU-only. It does not, for example, offload NumPy
array computations to CUDA GPUs, as JITs like Numba do.

There is already a rich ecosystem of tools for accelerating these sorts
of specialized tasks, and CPython's JIT is not intended to replace them.
Instead, it is meant to improve the performance of general-purpose
Python code, which is less likely to benefit from deeper GPU
integration.

Open Issues

Speed

Currently, the JIT is about as fast as the existing specializing
interpreter on most platforms. Improving this is obviously a top
priority at this point, since providing a significant performance gain
is the entire motivation for having a JIT at all. A number of proposed
improvements are already underway, and this ongoing work is being
tracked in GH-115802.

Memory

Because it allocates additional memory for executable machine code, the
JIT does use more memory than the existing interpreter at runtime.
According to the official benchmarks, the JIT currently uses about
10-20% more memory than the base interpreter. The upper end of this
range is due to aarch64-apple-darwin, which has larger page sizes (and
thus, a larger minimum allocation granularity).

However, these numbers should be taken with a grain of salt, as the
benchmarks themselves don't actually have a very high baseline of memory
usage. Since they have a higher ratio of code to data, the JIT's memory
overhead is more pronounced than it would be in a typical workload where
memory pressure is more likely to be a real concern.

Not much effort has been put into optimizing the JIT's memory usage yet,
so these numbers likely represent a maximum that will be reduced over
time. Improving this is a medium priority, and is being tracked in
GH-116017. We may consider exposing configurable parameters for limiting
memory consumption in the future, but no official APIs will be exposed
until the JIT meets the requirements to be considered non-experimental.

Earlier versions of the JIT had a more complicated memory allocation
scheme which imposed a number of fragile limitations on the size and
layout of the emitted code, and significantly bloated the memory
footprint of Python executable. These issues are no longer present in
the current design.

Dependencies

At the time of writing, the JIT has a build-time dependency on LLVM.
LLVM is used to compile individual micro-op instructions into blobs of
machine code, which are then linked together to form the JIT's
templates. These templates are used to build CPython itself. The JIT has
no runtime dependency on LLVM and is therefore not at all exposed as a
dependency to end users.

Building the JIT adds between 3 and 60 seconds to the build process,
depending on platform. It is only rebuilt whenever the generated files
become out-of-date, so only those who are actively developing the main
interpreter loop will be rebuilding it with any frequency.

Unlike many other generated files in CPython, the JIT's generated files
are not tracked by Git. This is because they contain compiled binary
code templates specific to not only the host platform, but also the
current build configuration for that platform. As such, hosting them
would require a significant engineering effort in order to build and
host dozens of large binary files for each commit that changes the
generated code. While perhaps feasible, this is not a priority, since
installing the required tools is not prohibitively difficult for most
people building CPython, and the build step is not particularly
time-consuming.

Since some still remain interested in this possibility, discussion is
being tracked in GH-115869.

Footnotes

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

[1] Due to lack of available hardware, the JIT is built, but not tested,
for this platform.

[2] Due to lack of available hardware, the JIT is built using
cross-compilation and tested using hardware emulation for this platform.
Some tests are skipped because emulation causes them to fail. However,
the JIT has been successfully built and tested for this platform on
non-emulated hardware.

[3] Due to lack of available hardware, the JIT is built using
cross-compilation and tested using hardware emulation for this platform.
Some tests are skipped because emulation causes them to fail. However,
the JIT has been successfully built and tested for this platform on
non-emulated hardware.

license: Public Domain


text:
PEP: 3133 Title: Introducing Roles Version: $Revision$ Last-Modified:
$Date$ Author: Collin Winter<EMAIL_ADDRESS>Status: Rejected
Type: Standards Track Content-Type: text/x-rst Requires: 3115, 3129
Created: 01-May-2007 Python-Version: 3.0 Post-History: 13-May-2007

Rejection Notice

This PEP has helped push PEP 3119 towards a saner, more minimalistic
approach. But given the latest version of PEP 3119 I much prefer that.
GvR.

Abstract

Python's existing object model organizes objects according to their
implementation. It is often desirable -- especially in duck typing-based
language like Python -- to organize objects by the part they play in a
larger system (their intent), rather than by how they fulfill that part
(their implementation). This PEP introduces the concept of roles, a
mechanism for organizing objects according to their intent rather than
their implementation.

Rationale

In the beginning were objects. They allowed programmers to marry
function and state, and to increase code reusability through concepts
like polymorphism and inheritance, and lo, it was good. There came a
time, however, when inheritance and polymorphism weren't enough. With
the invention of both dogs and trees, we were no longer able to be
content with knowing merely, "Does it understand 'bark'?" We now needed
to know what a given object thought that "bark" meant.

One solution, the one detailed here, is that of roles, a mechanism
orthogonal and complementary to the traditional class/instance system.
Whereas classes concern themselves with state and implementation, the
roles mechanism deals exclusively with the behaviours embodied in a
given class.

This system was originally called "traits" and implemented for Squeak
Smalltalk[1]. It has since been adapted for use in Perl 6[2] where it is
called "roles", and it is primarily from there that the concept is now
being interpreted for Python 3. Python 3 will preserve the name "roles".

In a nutshell: roles tell you what an object does, classes tell you how
an object does it.

In this PEP, I will outline a system for Python 3 that will make it
possible to easily determine whether a given object's understanding of
"bark" is tree-like or dog-like. (There might also be more serious
examples.)

A Note on Syntax

A syntax proposals in this PEP are tentative and should be considered to
be strawmen. The necessary bits that this PEP depends on -- namely PEP
3115's class definition syntax and PEP 3129's class decorators -- are
still being formalized and may change. Function names will, of course,
be subject to lengthy bikeshedding debates.

Performing Your Role

Static Role Assignment

Let's start out by defining Tree and Dog classes :

    class Tree(Vegetable):

      def bark(self):
        return self.is_rough()

    class Dog(Animal):

      def bark(self):
        return self.goes_ruff()

While both implement a bark() method with the same signature, they do
wildly different things. We need some way of differentiating what we're
expecting. Relying on inheritance and a simple isinstance() test will
limit code reuse and/or force any dog-like classes to inherit from Dog,
whether or not that makes sense. Let's see if roles can help. :

    @perform_role(Doglike)
    class Dog(Animal):
      ...

    @perform_role(Treelike)
    class Tree(Vegetable):
      ...

    @perform_role(SitThere)
    class Rock(Mineral):
      ...

We use class decorators from PEP 3129 to associate a particular role or
roles with a class. Client code can now verify that an incoming object
performs the Doglike role, allowing it to handle Wolf, LaughingHyena and
Aibo[3] instances, too.

Roles can be composed via normal inheritance: :

    @perform_role(Guard, MummysLittleDarling)
    class GermanShepherd(Dog):

      def guard(self, the_precious):
        while True:
          if intruder_near(the_precious):
            self.growl()

      def get_petted(self):
        self.swallow_pride()

Here, GermanShepherd instances perform three roles: Guard and
MummysLittleDarling are applied directly, whereas Doglike is inherited
from Dog.

Assigning Roles at Runtime

Roles can be assigned at runtime, too, by unpacking the syntactic sugar
provided by decorators.

Say we import a Robot class from another module, and since we know that
Robot already implements our Guard interface, we'd like it to play
nicely with guard-related code, too. :

    >>> perform(Guard)(Robot)

This takes effect immediately and impacts all instances of Robot.

Asking Questions About Roles

Just because we've told our robot army that they're guards, we'd like to
check in on them occasionally and make sure they're still at their task.
:

    >>> performs(our_robot, Guard)
    True

What about that one robot over there? :

    >>> performs(that_robot_over_there, Guard)
    True

The performs() function is used to ask if a given object fulfills a
given role. It cannot be used, however, to ask a class if its instances
fulfill a role: :

    >>> performs(Robot, Guard)
    False

This is because the Robot class is not interchangeable with a Robot
instance.

Defining New Roles

Empty Roles

Roles are defined like a normal class, but use the Role metaclass. :

    class Doglike(metaclass=Role):
      ...

Metaclasses are used to indicate that Doglike is a Role in the same way
5 is an int and tuple is a type.

Composing Roles via Inheritance

Roles may inherit from other roles; this has the effect of composing
them. Here, instances of Dog will perform both the Doglike and FourLegs
roles. :

    class FourLegs(metaclass=Role):
      pass

    class Doglike(FourLegs, Carnivor):
      pass

    @perform_role(Doglike)
    class Dog(Mammal):
      pass

Requiring Concrete Methods

So far we've only defined empty roles -- not very useful things. Let's
now require that all classes that claim to fulfill the Doglike role
define a bark() method: :

    class Doglike(FourLegs):

      def bark(self):
        pass

No decorators are required to flag the method as "abstract", and the
method will never be called, meaning whatever code it contains (if any)
is irrelevant. Roles provide only abstract methods; concrete default
implementations are left to other, better-suited mechanisms like mixins.

Once you have defined a role, and a class has claimed to perform that
role, it is essential that that claim be verified. Here, the programmer
has misspelled one of the methods required by the role. :

    @perform_role(FourLegs)
    class Horse(Mammal):

      def run_like_teh_wind(self)
        ...

This will cause the role system to raise an exception, complaining that
you're missing a run_like_the_wind() method. The role system carries out
these checks as soon as a class is flagged as performing a given role.

Concrete methods are required to match exactly the signature demanded by
the role. Here, we've attempted to fulfill our role by defining a
concrete version of bark(), but we've missed the mark a bit. :

    @perform_role(Doglike)
    class Coyote(Mammal):

      def bark(self, target=moon):
        pass

This method's signature doesn't match exactly with what the Doglike role
was expecting, so the role system will throw a bit of a tantrum.

Mechanism

The following are strawman proposals for how roles might be expressed in
Python. The examples here are phrased in a way that the roles mechanism
may be implemented without changing the Python interpreter. (Examples
adapted from an article on Perl 6 roles by Curtis Poe [4].)

1.  Static class role assignment :

        @perform_role(Thieving)
        class Elf(Character):
          ...

    perform_role() accepts multiple arguments, such that this is also
    legal: :

        @perform_role(Thieving, Spying, Archer)
        class Elf(Character):
          ...

    The Elf class now performs both the Thieving, Spying, and Archer
    roles.

2.  Querying instances :

        if performs(my_elf, Thieving):
          ...

    The second argument to performs() may also be anything with a
    __contains__() method, meaning the following is legal: :

        if performs(my_elf, set([Thieving, Spying, BoyScout])):
          ...

    Like isinstance(), the object needs only to perform a single role
    out of the set in order for the expression to be true.

Relationship to Abstract Base Classes

Early drafts of this PEP[5] envisioned roles as competing with the
abstract base classes proposed in PEP 3119. After further discussion and
deliberation, a compromise and a delegation of responsibilities and
use-cases has been worked out as follows:

-   Roles provide a way of indicating an object's semantics and abstract
    capabilities. A role may define abstract methods, but only as a way
    of delineating an interface through which a particular set of
    semantics are accessed. An Ordering role might require that some set
    of ordering operators be defined. :

        class Ordering(metaclass=Role):
          def __ge__(self, other):
            pass

          def __le__(self, other):
            pass

          def __ne__(self, other):
            pass

          # ...and so on

    In this way, we're able to indicate an object's role or function
    within a larger system without constraining or concerning ourselves
    with a particular implementation.

-   Abstract base classes, by contrast, are a way of reusing common,
    discrete units of implementation. For example, one might define an
    OrderingMixin that implements several ordering operators in terms of
    other operators. :

        class OrderingMixin:
          def __ge__(self, other):
            return self > other or self == other

          def __le__(self, other):
            return self < other or self == other

          def __ne__(self, other):
            return not self == other

          # ...and so on

    Using this abstract base class - more properly, a concrete mixin -
    allows a programmer to define a limited set of operators and let the
    mixin in effect "derive" the others.

By combining these two orthogonal systems, we're able to both a) provide
functionality, and b) alert consumer systems to the presence and
availability of this functionality. For example, since the OrderingMixin
class above satisfies the interface and semantics expressed in the
Ordering role, we say the mixin performs the role: :

    @perform_role(Ordering)
    class OrderingMixin:
      def __ge__(self, other):
        return self > other or self == other

      def __le__(self, other):
        return self < other or self == other

      def __ne__(self, other):
        return not self == other

      # ...and so on

Now, any class that uses the mixin will automatically -- that is,
without further programmer effort -- be tagged as performing the
Ordering role.

The separation of concerns into two distinct, orthogonal systems is
desirable because it allows us to use each one separately. Take, for
example, a third-party package providing a RecursiveHash role that
indicates a container takes its contents into account when determining
its hash value. Since Python's built-in tuple and frozenset classes
follow this semantic, the RecursiveHash role can be applied to them. :

    >>> perform_role(RecursiveHash)(tuple)
    >>> perform_role(RecursiveHash)(frozenset)

Now, any code that consumes RecursiveHash objects will now be able to
consume tuples and frozensets.

Open Issues

Allowing Instances to Perform Different Roles Than Their Class

Perl 6 allows instances to perform different roles than their class.
These changes are local to the single instance and do not affect other
instances of the class. For example: :

    my_elf = Elf()
    my_elf.goes_on_quest()
    my_elf.becomes_evil()
    now_performs(my_elf, Thieving) # Only this one elf is a thief
    my_elf.steals(["purses", "candy", "kisses"])

In Perl 6, this is done by creating an anonymous class that inherits
from the instance's original parent and performs the additional role(s).
This is possible in Python 3, though whether it is desirable is still is
another matter.

Inclusion of this feature would, of course, make it much easier to
express the works of Charles Dickens in Python: :

    >>> from literature import role, BildungsRoman
    >>> from dickens import Urchin, Gentleman
    >>>
    >>> with BildungsRoman() as OliverTwist:
    ...   mr_brownlow = Gentleman()
    ...   oliver, artful_dodger = Urchin(), Urchin()
    ...   now_performs(artful_dodger, [role.Thief, role.Scoundrel])
    ...
    ...   oliver.has_adventures_with(ArtfulDodger)
    ...   mr_brownlow.adopt_orphan(oliver)
    ...   now_performs(oliver, role.RichWard)

Requiring Attributes

Neal Norwitz has requested the ability to make assertions about the
presence of attributes using the same mechanism used to require methods.
Since roles take effect at class definition-time, and since the vast
majority of attributes are defined at runtime by a class's __init__()
method, there doesn't seem to be a good way to check for attributes at
the same time as methods.

It may still be desirable to include non-enforced attributes in the role
definition, if only for documentation purposes.

Roles of Roles

Under the proposed semantics, it is possible for roles to have roles of
their own. :

    @perform_role(Y)
    class X(metaclass=Role):
      ...

While this is possible, it is meaningless, since roles are generally not
instantiated. There has been some off-line discussion about giving
meaning to this expression, but so far no good ideas have emerged.

class_performs()

It is currently not possible to ask a class if its instances perform a
given role. It may be desirable to provide an analogue to performs()
such that :

    >>> isinstance(my_dwarf, Dwarf)
    True
    >>> performs(my_dwarf, Surly)
    True
    >>> performs(Dwarf, Surly)
    False
    >>> class_performs(Dwarf, Surly)
    True

Prettier Dynamic Role Assignment

An early draft of this PEP included a separate mechanism for dynamically
assigning a role to a class. This was spelled :

    >>> now_perform(Dwarf, GoldMiner)

This same functionality already exists by unpacking the syntactic sugar
provided by decorators: :

    >>> perform_role(GoldMiner)(Dwarf)

At issue is whether dynamic role assignment is sufficiently important to
warrant a dedicated spelling.

Syntax Support

Though the phrasings laid out in this PEP are designed so that the roles
system could be shipped as a stand-alone package, it may be desirable to
add special syntax for defining, assigning and querying roles. One
example might be a role keyword, which would translate :

    class MyRole(metaclass=Role):
      ...

into :

    role MyRole:
      ...

Assigning a role could take advantage of the class definition arguments
proposed in PEP 3115: :

    class MyClass(performs=MyRole):
      ...

Implementation

A reference implementation is forthcoming.

Acknowledgements

Thanks to Jeffery Yasskin, Talin and Guido van Rossum for several hours
of in-person discussion to iron out the differences, overlap and finer
points of roles and abstract base classes.

References

Copyright

This document has been placed in the public domain.

[1] http://www.iam.unibe.ch/~scg/Archive/Papers/Scha03aTraits.pdf

[2] http://dev.perl.org/perl6/doc/design/syn/S12.html

[3] http://en.wikipedia.org/wiki/AIBO

[4] http://www.perlmonks.org/?node_id=384858

[5] https://mail.python.org/pipermail/python-3000/2007-April/007026.html


text:
PEP: 8100 Title: January 2019 Steering Council election Author:
Nathaniel J. Smith<EMAIL_ADDRESS>Ee Durbin<EMAIL_ADDRESS>Status:
Final Type: Informational Topic: Governance Content-Type: text/x-rst
Created: 03-Jan-2019

Abstract

This document describes the schedule and other details of the January
2019 election for the Python steering council, as specified in PEP 13.
This is the first steering council election.

Returns officer

In future elections, the returns officer will be appointed by the
outgoing steering council. Since this is the first election, we have no
outgoing steering council, and PEP 13 says that the returns officer is
instead appointed by the PSF Executive Director, Ewa Jodlowska. She
appointed Ee Durbin.

Schedule

There will be a two-week nomination period, followed by a two-week vote.

The nomination period is: January 7, 2019 through January 20, 2019

The voting period is: January 21, 2019 12:00 UTC through February 4,
2019 12:00 UTC (The end of February 3, 2019 Anywhere on Earth)

Candidates

Candidates must be nominated by a core team member. If the candidate is
a core team member, they may nominate themselves.

Once the nomination period opens, candidates will be listed here:

1.  Brett Cannon
2.  Alyssa (Nick) Coghlan
3.  Barry Warsaw
4.  Guido van Rossum
5.  Victor Stinner
6.  Yury Selivanov
7.  David Mertz
8.  Łukasz Langa
9.  Benjamin Peterson
10. Mariatta
11. Carol Willing
12. Emily Morehouse
13. Peter Wang
14. Donald Stufft
15. Travis Oliphant
16. Kushal Das
17. Gregory P. Smith

Voter Roll

All active Python core team members are eligible to vote.

Ballots will be distributed based on the The Python Voter Roll for this
election [1].

While this file is not public as it contains private email addresses,
the Complete Voter Roll by name is available.

Election Implementation

The election will be conducted using the Helios Voting Service.

Configuration

Short name: 2019-python-steering-committee

Name: 2019 Python Steering Committee Election

Description:
Election for the Python steering council, as specified in PEP 13. This is the first steering council election.

type: Election

Use voter aliases: [X]

Randomize answer order: [X]

Private: [X]

Help Email Address<EMAIL_ADDRESS>
Voting starts at: January 21, 2019 12:00 UTC

Voting ends at: February 4, 2019 12:00 UTC

This will create an election in which:

-   Voting is not open to the public, only those on the Voter Roll may
    participate. Ballots will be emailed when voting starts.
-   Candidates are presented in random order, to help avoid bias.
-   Voter identities and ballots are protected against cryptographic
    advances.

Questions

Question 1

Select between 0 and 5 answers. Result Type: absolute

Question: Select candidates for the Python Steering Council

Answer #1 - #N: Candidates from Candidates_ Section

Results

Of the 96 eligible voters, 69 cast ballots.

The top five vote-getters are:

-   Barry Warsaw
-   Brett Cannon
-   Carol Willing
-   Guido van Rossum
-   Alyssa (Nick) Coghlan

No conflict of interest as defined in PEP 13 were observed.

The full vote counts are as follows:

  ----------------------------------------
  Candidate               Votes Received
  ----------------------- ----------------
  Guido van Rossum        45

  Brett Cannon            44

  Carol Willing           33

  Barry Warsaw            31

  Alyssa (Nick) Coghlan   25

  Benjamin Peterson       22

  Łukasz Langa            21

  Victor Stinner          21

  Mariatta                20

  Emily Morehouse         18

  Yury Selivanov          15

  Donald Stufft           11

  Peter Wang              10

  Travis Oliphant         8

  Kushal Das              7

  Gregory P. Smith        6

  David Mertz             3
  ----------------------------------------

Copyright

This document has been placed in the public domain.

Complete Voter Roll

Active Python core developers

    Alex Gaynor
    Alex Martelli
    Alexander Belopolsky
    Alexandre Vassalotti
    Amaury Forgeot d'Arc
    Andrew Kuchling
    Andrew Svetlov
    Antoine Pitrou
    Armin Ronacher
    Barry Warsaw
    Benjamin Peterson
    Berker Peksag
    Brett Cannon
    Brian Curtin
    Carol Willing
    Chris Jerdonek
    Chris Withers
    Christian Heimes
    David Malcolm
    David Wolever
    Davin Potts
    Dino Viehland
    Donald Stufft
    Doug Hellmann
    Eli Bendersky
    Emily Morehouse
    Éric Araujo
    Eric Snow
    Eric V. Smith
    Ethan Furman
    Ezio Melotti
    Facundo Batista
    Fred Drake
    Georg Brandl
    Giampaolo Rodola'
    Gregory P. Smith
    Guido van Rossum
    Hyeshik Chang
    Hynek Schlawack
    INADA Naoki
    Ivan Levkivskyi
    Jack Diederich
    Jack Jansen
    Jason R. Coombs
    Jeff Hardy
    Jeremy Hylton
    Jesús Cea
    Julien Palard
    Kurt B. Kaiser
    Kushal Das
    Larry Hastings
    Lars Gustäbel
    Lisa Roach
    Łukasz Langa
    Marc-Andre Lemburg
    Mariatta
    Mark Dickinson
    Mark Hammond
    Mark Shannon
    Martin Panter
    Matthias Klose
    Meador Inge
    Michael Hudson-Doyle
    Nathaniel J. Smith
    Ned Deily
    Neil Schemenauer
    Alyssa Coghlan
    Pablo Galindo
    Paul Moore
    Petr Viktorin
    Petri Lehtinen
    Philip Jenvey
    R. David Murray
    Raymond Hettinger
    Robert Collins
    Ronald Oussoren
    Sandro Tosi
    Senthil Kumaran
    Serhiy Storchaka
    Sjoerd Mullender
    Stefan Krah
    Steve Dower
    Steven Daprano
    T. Wouters
    Tal Einat
    Terry Jan Reedy
    Thomas Heller
    Tim Golden
    Tim Peters
    Trent Nelson
    Victor Stinner
    Vinay Sajip
    Walter Dörwald
    Xiang Zhang
    Yury Selivanov
    Zachary Ware

[1] This repository is private and accessible only to Python Core
Developers, administrators, and Python Software Foundation Staff as it
contains personal email addresses.

authors:
Ee Durbin
Nathaniel J. Smith


text:
PEP: 478 Title: Python 3.5 Release Schedule Version: $Revision$
Last-Modified: $Date$ Author: Larry Hastings<EMAIL_ADDRESS>Status: Final Type: Informational Topic: Release Content-Type:
text/x-rst Created: 22-Sep-2014 Python-Version: 3.5

Abstract

This document describes the development and release schedule for Python
3.5. The schedule primarily concerns itself with PEP-sized items.

Release Manager and Crew

-   3.5 Release Manager: Larry Hastings
-   Windows installers: Steve Dower
-   Mac installers: Ned Deily
-   Documentation: Georg Brandl

Release Schedule

Python 3.5 has now reached its end-of-life and has been retired. No more
releases will be made.

These are all the historical releases of Python 3.5, including their
release dates.

-   3.5.0 alpha 1: February 8, 2015
-   3.5.0 alpha 2: March 9, 2015
-   3.5.0 alpha 3: March 29, 2015
-   3.5.0 alpha 4: April 19, 2015
-   3.5.0 beta 1: May 24, 2015 (Beta 1 is also "feature freeze"--no new
    features beyond this point.)
-   3.5.0 beta 2: May 31, 2015
-   3.5.0 beta 3: July 5, 2015
-   3.5.0 beta 4: July 26, 2015
-   3.5.0 release candidate 1: August 10, 2015
-   3.5.0 release candidate 2: August 25, 2015
-   3.5.0 release candidate 3: September 7, 2015
-   3.5.0 final: September 13, 2015
-   3.5.1 release candidate 1: November 22, 2015
-   3.5.1 final: December 6, 2015
-   3.5.2 release candidate 1: Sunday, June 12, 2016
-   3.5.2 final: Sunday, June 26, 2016
-   3.5.3 candidate 1: January 2, 2017
-   3.5.3 final: January 17, 2017
-   3.5.4 candidate 1: July 25, 2017
-   3.5.4 final: August 8, 2017
-   3.5.5 candidate 1: January 23, 2018
-   3.5.5 final: February 4, 2018
-   3.5.6 candidate 1: July 19, 2018
-   3.5.6 final: August 2, 2018
-   3.5.7 candidate 1: March 4, 2019
-   3.5.7 final: March 18, 2019
-   3.5.8 candidate 1: September 9, 2019
-   3.5.8 candidate 2: October 12, 2019
-   3.5.8 final: October 29, 2019
-   3.5.9 final: November 1, 2019
-   3.5.10 rc1: August 21, 2020
-   3.5.10 final: September 5, 2020

Features for 3.5

-   PEP 441, improved Python zip application support
-   PEP 448, additional unpacking generalizations
-   PEP 461, "%-formatting" for bytes and bytearray objects
-   PEP 465, a new operator ("@") for matrix multiplication
-   PEP 471, os.scandir(), a fast new directory traversal function
-   PEP 475, adding support for automatic retries of interrupted system
    calls
-   PEP 479, change StopIteration handling inside generators
-   PEP 484, the typing module, a new standard for type annotations
-   PEP 485, math.isclose(), a function for testing approximate equality
-   PEP 486, making the Windows Python launcher aware of virtual
    environments
-   PEP 488, eliminating .pyo files
-   PEP 489, a new and improved mechanism for loading extension modules
-   PEP 492, coroutines with async and await syntax

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:


text:
PEP: 371 Title: Addition of the multiprocessing package to the standard
library Version: $Revision$ Last-Modified: $Date$ Author: Jesse Noller
<EMAIL_ADDRESS>Richard Oudkerk<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type: text/x-rst Created:
06-May-2008 Python-Version: 2.6, 3.0 Post-History: 03-Jun-2008

Abstract

This PEP proposes the inclusion of the pyProcessing[1] package into the
Python standard library, renamed to "multiprocessing".

The processing package mimics the standard library threading module
functionality to provide a process-based approach to threaded
programming allowing end-users to dispatch multiple tasks that
effectively side-step the global interpreter lock.

The package also provides server and client functionality
(processing.Manager) to provide remote sharing and management of objects
and tasks so that applications may not only leverage multiple cores on
the local machine, but also distribute objects and tasks across a
cluster of networked machines.

While the distributed capabilities of the package are beneficial, the
primary focus of this PEP is the core threading-like API and
capabilities of the package.

Rationale

The current CPython interpreter implements the Global Interpreter Lock
(GIL) and barring work in Python 3000 or other versions currently
planned[2], the GIL will remain as-is within the CPython interpreter for
the foreseeable future. While the GIL itself enables clean and easy to
maintain C code for the interpreter and extensions base, it is
frequently an issue for those Python programmers who are leveraging
multi-core machines.

The GIL itself prevents more than a single thread from running within
the interpreter at any given point in time, effectively removing
Python's ability to take advantage of multi-processor systems.

The pyprocessing package offers a method to side-step the GIL allowing
applications within CPython to take advantage of multi-core
architectures without asking users to completely change their
programming paradigm (i.e.: dropping threaded programming for another
"concurrent" approach - Twisted, Actors, etc).

The Processing package offers CPython a "known API" which mirrors albeit
in a PEP 8 compliant manner, that of the threading API, with known
semantics and easy scalability.

In the future, the package might not be as relevant should the CPython
interpreter enable "true" threading, however for some applications,
forking an OS process may sometimes be more desirable than using
lightweight threads, especially on those platforms where process
creation is fast and optimized.

For example, a simple threaded application:

    from threading import Thread as worker

    def afunc(number):
        print number * 3

    t = worker(target=afunc, args=(4,))
    t.start()
    t.join()

The pyprocessing package mirrored the API so well, that with a simple
change of the import to:

    from processing import process as worker

The code would now execute through the processing.process class.
Obviously, with the renaming of the API to PEP 8 compliance there would
be additional renaming which would need to occur within user
applications, however minor.

This type of compatibility means that, with a minor (in most cases)
change in code, users' applications will be able to leverage all cores
and processors on a given machine for parallel execution. In many cases
the pyprocessing package is even faster than the normal threading
approach for I/O bound programs. This of course, takes into account that
the pyprocessing package is in optimized C code, while the threading
module is not.

The "Distributed" Problem

In the discussion on Python-Dev about the inclusion of this package[3]
there was confusion about the intentions this PEP with an attempt to
solve the "Distributed" problem - frequently comparing the functionality
of this package with other solutions like MPI-based communication[4],
CORBA, or other distributed object approaches[5].

The "distributed" problem is large and varied. Each programmer working
within this domain has either very strong opinions about their favorite
module/method or a highly customized problem for which no existing
solution works.

The acceptance of this package does not preclude or recommend that
programmers working on the "distributed" problem not examine other
solutions for their problem domain. The intent of including this package
is to provide entry-level capabilities for local concurrency and the
basic support to spread that concurrency across a network of machines -
although the two are not tightly coupled, the pyprocessing package could
in fact, be used in conjunction with any of the other solutions
including MPI/etc.

If necessary - it is possible to completely decouple the local
concurrency abilities of the package from the network-capable/shared
aspects of the package. Without serious concerns or cause however, the
author of this PEP does not recommend that approach.

Performance Comparison

As we all know - there are "lies, damned lies, and benchmarks". These
speed comparisons, while aimed at showcasing the performance of the
pyprocessing package, are by no means comprehensive or applicable to all
possible use cases or environments. Especially for those platforms with
sluggish process forking timing.

All benchmarks were run using the following:

-   4 Core Intel Xeon CPU @ 3.00GHz
-   16 GB of RAM
-   Python 2.5.2 compiled on Gentoo Linux (kernel <IP_ADDRESS>)
-   pyProcessing 0.52

All of the code for this can be downloaded from
http://jessenoller.com/code/bench-src.tgz

The basic method of execution for these benchmarks is in the
run_benchmarks.py[6] script, which is simply a wrapper to execute a
target function through a single threaded (linear), multi-threaded (via
threading), and multi-process (via pyprocessing) function for a static
number of iterations with increasing numbers of execution loops and/or
threads.

The run_benchmarks.py script executes each function 100 times, picking
the best run of that 100 iterations via the timeit module.

First, to identify the overhead of the spawning of the workers, we
execute a function which is simply a pass statement (empty):

    cmd: python run_benchmarks.py empty_func.py
    Importing empty_func
    Starting tests ...
    non_threaded (1 iters)  0.000001 seconds
    threaded (1 threads)    0.000796 seconds
    processes (1 procs)     0.000714 seconds

    non_threaded (2 iters)  0.000002 seconds
    threaded (2 threads)    0.001963 seconds
    processes (2 procs)     0.001466 seconds

    non_threaded (4 iters)  0.000002 seconds
    threaded (4 threads)    0.003986 seconds
    processes (4 procs)     0.002701 seconds

    non_threaded (8 iters)  0.000003 seconds
    threaded (8 threads)    0.007990 seconds
    processes (8 procs)     0.005512 seconds

As you can see, process forking via the pyprocessing package is faster
than the speed of building and then executing the threaded version of
the code.

The second test calculates 50000 Fibonacci numbers inside of each thread
(isolated and shared nothing):

    cmd: python run_benchmarks.py fibonacci.py
    Importing fibonacci
    Starting tests ...
    non_threaded (1 iters)  0.195548 seconds
    threaded (1 threads)    0.197909 seconds
    processes (1 procs)     0.201175 seconds

    non_threaded (2 iters)  0.397540 seconds
    threaded (2 threads)    0.397637 seconds
    processes (2 procs)     0.204265 seconds

    non_threaded (4 iters)  0.795333 seconds
    threaded (4 threads)    0.797262 seconds
    processes (4 procs)     0.206990 seconds

    non_threaded (8 iters)  1.591680 seconds
    threaded (8 threads)    1.596824 seconds
    processes (8 procs)     0.417899 seconds

The third test calculates the sum of all primes below 100000, again
sharing nothing:

    cmd: run_benchmarks.py crunch_primes.py
    Importing crunch_primes
    Starting tests ...
    non_threaded (1 iters)  0.495157 seconds
    threaded (1 threads)    0.522320 seconds
    processes (1 procs)     0.523757 seconds

    non_threaded (2 iters)  1.052048 seconds
    threaded (2 threads)    1.154726 seconds
    processes (2 procs)     0.524603 seconds

    non_threaded (4 iters)  2.104733 seconds
    threaded (4 threads)    2.455215 seconds
    processes (4 procs)     0.530688 seconds

    non_threaded (8 iters)  4.217455 seconds
    threaded (8 threads)    5.109192 seconds
    processes (8 procs)     1.077939 seconds

The reason why tests two and three focused on pure numeric crunching is
to showcase how the current threading implementation does hinder non-I/O
applications. Obviously, these tests could be improved to use a queue
for coordination of results and chunks of work but that is not required
to show the performance of the package and core processing.process
module.

The next test is an I/O bound test. This is normally where we see a
steep improvement in the threading module approach versus a
single-threaded approach. In this case, each worker is opening a
descriptor to lorem.txt, randomly seeking within it and writing lines to
/dev/null:

    cmd: python run_benchmarks.py file_io.py
    Importing file_io
    Starting tests ...
    non_threaded (1 iters)  0.057750 seconds
    threaded (1 threads)    0.089992 seconds
    processes (1 procs)     0.090817 seconds

    non_threaded (2 iters)  0.180256 seconds
    threaded (2 threads)    0.329961 seconds
    processes (2 procs)     0.096683 seconds

    non_threaded (4 iters)  0.370841 seconds
    threaded (4 threads)    1.103678 seconds
    processes (4 procs)     0.101535 seconds

    non_threaded (8 iters)  0.749571 seconds
    threaded (8 threads)    2.437204 seconds
    processes (8 procs)     0.203438 seconds

As you can see, pyprocessing is still faster on this I/O operation than
using multiple threads. And using multiple threads is slower than the
single threaded execution itself.

Finally, we will run a socket-based test to show network I/O
performance. This function grabs a URL from a server on the LAN that is
a simple error page from tomcat. It gets the page 100 times. The network
is silent, and a 10G connection:

    cmd: python run_benchmarks.py url_get.py
    Importing url_get
    Starting tests ...
    non_threaded (1 iters)  0.124774 seconds
    threaded (1 threads)    0.120478 seconds
    processes (1 procs)     0.121404 seconds

    non_threaded (2 iters)  0.239574 seconds
    threaded (2 threads)    0.146138 seconds
    processes (2 procs)     0.138366 seconds

    non_threaded (4 iters)  0.479159 seconds
    threaded (4 threads)    0.200985 seconds
    processes (4 procs)     0.188847 seconds

    non_threaded (8 iters)  0.960621 seconds
    threaded (8 threads)    0.659298 seconds
    processes (8 procs)     0.298625 seconds

We finally see threaded performance surpass that of single-threaded
execution, but the pyprocessing package is still faster when increasing
the number of workers. If you stay with one or two threads/workers, then
the timing between threads and pyprocessing is fairly close.

One item of note however, is that there is an implicit overhead within
the pyprocessing package's Queue implementation due to the object
serialization.

Alec Thomas provided a short example based on the run_benchmarks.py
script to demonstrate this overhead versus the default Queue
implementation:

    cmd: run_bench_queue.py
    non_threaded (1 iters)  0.010546 seconds
    threaded (1 threads)    0.015164 seconds
    processes (1 procs)     0.066167 seconds

    non_threaded (2 iters)  0.020768 seconds
    threaded (2 threads)    0.041635 seconds
    processes (2 procs)     0.084270 seconds

    non_threaded (4 iters)  0.041718 seconds
    threaded (4 threads)    0.086394 seconds
    processes (4 procs)     0.144176 seconds

    non_threaded (8 iters)  0.083488 seconds
    threaded (8 threads)    0.184254 seconds
    processes (8 procs)     0.302999 seconds

Additional benchmarks can be found in the pyprocessing package's source
distribution's examples/ directory. The examples will be included in the
package's documentation.

Maintenance

Richard M. Oudkerk - the author of the pyprocessing package has agreed
to maintain the package within Python SVN. Jesse Noller has volunteered
to also help maintain/document and test the package.

API Naming

While the aim of the package's API is designed to closely mimic that of
the threading and Queue modules as of python 2.x, those modules are not
PEP 8 compliant. It has been decided that instead of adding the package
"as is" and therefore perpetuating the non-PEP 8 compliant naming, we
will rename all APIs, classes, etc to be fully PEP 8 compliant.

This change does affect the ease-of-drop in replacement for those using
the threading module, but that is an acceptable side-effect in the view
of the authors, especially given that the threading module's own API
will change.

Issue 3042 in the tracker proposes that for Python 2.6 there will be two
APIs for the threading module - the current one, and the PEP 8 compliant
one. Warnings about the upcoming removal of the original java-style API
will be issued when -3 is invoked.

In Python 3000, the threading API will become PEP 8 compliant, which
means that the multiprocessing module and the threading module will
again have matching APIs.

Timing/Schedule

Some concerns have been raised about the timing/lateness of this PEP for
the 2.6 and 3.0 releases this year, however it is felt by both the
authors and others that the functionality this package offers surpasses
the risk of inclusion.

However, taking into account the desire not to destabilize Python-core,
some refactoring of pyprocessing's code "into" Python-core can be
withheld until the next 2.x/3.x releases. This means that the actual
risk to Python-core is minimal, and largely constrained to the actual
package itself.

Open Issues

-   Confirm no "default" remote connection capabilities, if needed
    enable the remote security mechanisms by default for those classes
    which offer remote capabilities.
-   Some of the API (Queue methods qsize(), task_done() and join())
    either need to be added, or the reason for their exclusion needs to
    be identified and documented clearly.

Closed Issues

-   The PyGILState bug patch submitted in issue 1683 by roudkerk must be
    applied for the package unit tests to work.
-   Existing documentation has to be moved to ReST formatting.
-   Reliance on ctypes: The pyprocessing package's reliance on ctypes
    prevents the package from functioning on platforms where ctypes is
    not supported. This is not a restriction of this package, but rather
    of ctypes.
-   DONE: Rename top-level package from "pyprocessing" to
    "multiprocessing".
-   DONE: Also note that the default behavior of process spawning does
    not make it compatible with use within IDLE as-is, this will be
    examined as a bug-fix or "setExecutable" enhancement.
-   DONE: Add in "multiprocessing.setExecutable()" method to override
    the default behavior of the package to spawn processes using the
    current executable name rather than the Python interpreter. Note
    that Mark Hammond has suggested a factory-style interface for
    this[7].

References

Copyright

This document has been placed in the public domain.

[1] The 2008 era PyProcessing project (the pyprocessing name was since
repurposed)
https://web.archive.org/web/20080914113946/https://pyprocessing.berlios.de/

[2] See Adam Olsen's "safe threading" project
https://code.google.com/archive/p/python-safethread/

[3] See: Addition of "pyprocessing" module to standard lib.
https://mail.python.org/pipermail/python-dev/2008-May/079417.html

[4] https://mpi4py.readthedocs.io/

[5] See "Cluster Computing"
https://wiki.python.org/moin/ParallelProcessing#Cluster_Computing

[6] The original run_benchmark.py code was published in Python Magazine
in December 2007: "Python Threads and the Global Interpreter Lock" by
Jesse Noller. It has been modified for this PEP.

[7] http://groups.google.com/group/python-dev2/msg/54cf06d15cbcbc34


text:
PEP: 719 Title: Python 3.13 Release Schedule Author: Thomas Wouters
<EMAIL_ADDRESS>Status: Active Type: Informational Topic: Release
Created: 26-May-2023 Python-Version: 3.13

Abstract

This document describes the development and release schedule for Python
3.13.

Release Manager and Crew

-   3.13 Release Manager: Thomas Wouters
-   Windows installers: Steve Dower
-   Mac installers: Ned Deily
-   Documentation: Julien Palard

Release Schedule

3.13.0 schedule

Note: the dates below use a 17-month development period that results in
a 12-month release cadence between feature versions, as defined by PEP
602.

Actual:

-   3.13 development begins: Monday, 2023-05-22
-   3.13.0 alpha 1: Friday, 2023-10-13
-   3.13.0 alpha 2: Wednesday, 2023-11-22
-   3.13.0 alpha 3: Wednesday, 2024-01-17
-   3.13.0 alpha 4: Thursday, 2024-02-15
-   3.13.0 alpha 5: Tuesday, 2024-03-12
-   3.13.0 alpha 6: Tuesday, 2024-04-09
-   3.13.0 beta 1: Wednesday, 2024-05-08 (No new features beyond this
    point.)
-   3.13.0 beta 2: Wednesday, 2024-06-05
-   3.13.0 beta 3: Thursday, 2024-06-27
-   3.13.0 beta 4: Thursday, 2024-07-18
-   3.13.0 candidate 1: Thursday, 2024-08-01
-   3.13.0 candidate 2: Friday, 2024-09-06
-   3.13.0 candidate 3: Tuesday, 2024-10-01
-   3.13.0 final: Monday, 2024-10-07

Expected:

Subsequent bugfix releases every two months.

3.13 Lifespan

3.13 will receive bugfix updates approximately every 2 months for
approximately 24 months. Around the time of the release of 3.15.0 final,
the final 3.13 bugfix update will be released. After that, it is
expected that security updates (source only) will be released until 5
years after the release of 3.13.0 final, so until approximately October
2029.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 473 Title: Adding structured data to built-in exceptions Version:
$Revision$ Last-Modified: $Date$ Author: Sebastian Kreft
<EMAIL_ADDRESS>Status: Rejected Type: Standards Track Content-Type:
text/x-rst Created: 29-Mar-2014 Post-History: Resolution:
https://mail.python.org/pipermail/python-dev/2019-March/156692.html

Abstract

Exceptions like AttributeError, IndexError, KeyError, LookupError,
NameError, TypeError, and ValueError do not provide all information
required by programmers to debug and better understand what caused them.
Furthermore, in some cases the messages even have slightly different
formats, which makes it really difficult for tools to automatically
provide additional information to diagnose the problem. To tackle the
former and to lay ground for the latter, it is proposed to expand these
exceptions so to hold both the offending and affected entities.

Rationale

The main issue this PEP aims to solve is the fact that currently error
messages are not that expressive and lack some key information to
resolve the exceptions. Additionally, the information present on the
error message is not always in the same format, which makes it very
difficult for third-party libraries to provide automated diagnosis of
the error.

These automated tools could, for example, detect typos or display or log
extra debug information. These could be particularly useful when running
tests or in a long running application.

Although it is in theory possible to have such libraries, they need to
resort to hacks in order to achieve the goal. One such example is
python-improved-exceptions[1], which modifies the byte-code to keep
references to the possibly interesting objects and also parses the error
messages to extract information like types or names. Unfortunately, such
approach is extremely fragile and not portable.

A similar proposal[2] has been implemented for ImportError and in the
same fashion this idea has received support[3]. Additionally, almost 10
years ago Guido asked in[4] to have a clean API to access the affected
objects in Exceptions like KeyError, AttributeError, NameError, and
IndexError. Similar issues and proposals ideas have been written in the
last year. Some other issues have been created, but despite receiving
support they finally get abandoned. References to the created issues are
listed below:

-   AttributeError:[5],[6],[7],[8],[9]
-   IndexError:[10],[11],[12]
-   KeyError:[13],[14],[15]
-   LookupError:[16]
-   NameError:[17],[18],[19]
-   TypeError:[20]
-   ValueError:[21]

To move forward with the development and to centralize the information
and discussion, this PEP aims to be a meta-issue summarizing all the
above discussions and ideas.

Examples

IndexError

The error message does not reference the list's length nor the index
used.

    a = [1, 2, 3, 4, 5]
    a[5]
    IndexError: list index out of range

KeyError

By convention the key is the first element of the error's argument, but
there's no other information regarding the affected dictionary (keys
types, size, etc.)

    b = {'foo': 1}
    b['fo']
    KeyError: 'fo'

AttributeError

The object's type and the offending attribute are part of the error
message. However, there are some different formats and the information
is not always available. Furthermore, although the object type is useful
in some cases, given the dynamic nature of Python, it would be much more
useful to have a reference to the object itself. Additionally the
reference to the type is not fully qualified and in some cases the type
is just too generic to provide useful information, for example in case
of accessing a module's attribute.

    c = object()
    c.foo
    AttributeError: 'object' object has no attribute 'foo'

    import string
    string.foo
    AttributeError: 'module' object has no attribute 'foo'

    a = string.Formatter()
    a.foo
    AttributeError: 'Formatter' object has no attribute 'foo'

NameError

The error message provides typically the name.

    foo = 1
    fo
    NameError: global name 'fo' is not defined

Other Cases

Issues are even harder to debug when the target object is the result of
another expression, for example:

    a[b[c[0]]]

This issue is also related to the fact that opcodes only have line
number information and not the offset. This proposal would help in this
case but not as much as having offsets.

Proposal

Extend the exceptions AttributeError, IndexError, KeyError, LookupError,
NameError, TypeError, and ValueError with the following:

-   AttributeError: target ^(w), attribute
-   IndexError: target ^(w), key ^(w), index (just an alias to key)
-   KeyError: target ^(w), key ^(w)
-   LookupError: target ^(w), key ^(w)
-   NameError: name, scope?
-   TypeError: unexpected_type
-   ValueError: unexpected_value ^(w)

Attributes with the superscript ^(w) may need to be weak references[22]
to prevent any memory cycles. However, this may add an unnecessary extra
complexity as noted by R. David Murray[23]. This is specially true given
that builtin types do not support being weak referenced.

TODO(skreft): expand this with examples of corner cases.

To remain backwards compatible these new attributes will be optional and
keyword only.

It is proposed to add this information, rather than just improve the
error, as the former would allow new debugging frameworks and tools and
also in the future to switch to a lazy generated message. Generated
messages are discussed in[24], although they are not implemented at the
moment. They would not only save some resources, but also uniform the
messages.

The stdlib will be then gradually changed so to start using these new
attributes.

Potential Uses

An automated tool could for example search for similar keys within the
object, allowing to display the following::

    a = {'foo': 1}
    a['fo']
    KeyError: 'fo'. Did you mean 'foo'?

    foo = 1
    fo
    NameError: global name 'fo' is not defined. Did you mean 'foo'?

See[25] for the output a TestRunner could display.

Performance

Filling these new attributes would only require two extra parameters
with data already available so the impact should be marginal. However,
it may need special care for KeyError as the following pattern is
already widespread.

    try:
      a[foo] = a[foo] + 1
    except:
      a[foo] = 0

Note as well that storing these objects into the error itself would
allow the lazy generation of the error message, as discussed in[26].

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] Python Exceptions Improved
(https://www.github.com/sk-/python-exceptions-improved)

[2] ImportError needs attributes for module and file name
(http://bugs.python.org/issue1559549)

[3] Enhance exceptions by attaching some more information to them
(https://mail.python.org/pipermail/python-ideas/2014-February/025601.html)

[4] LookupError etc. need API to get the key
(http://bugs.python.org/issue614557)

[5] LookupError etc. need API to get the key
(http://bugs.python.org/issue614557)

[6] making builtin exceptions more informative
(http://bugs.python.org/issue1182143)

[7] Add an 'attr' attribute to AttributeError
(http://bugs.python.org/issue18156)

[8] Specificity in AttributeError
(https://mail.python.org/pipermail/python-ideas/2013-April/020308.html)

[9] Enhance exceptions by attaching some more information to them
(https://mail.python.org/pipermail/python-ideas/2014-February/025601.html)

[10] LookupError etc. need API to get the key
(http://bugs.python.org/issue614557)

[11] Add index attribute to IndexError
(http://bugs.python.org/issue18162)

[12] Enhance exceptions by attaching some more information to them
(https://mail.python.org/pipermail/python-ideas/2014-February/025601.html)

[13] LookupError etc. need API to get the key
(http://bugs.python.org/issue614557)

[14] Add a 'key' attribute to KeyError
(http://bugs.python.org/issue18163)

[15] Enhance exceptions by attaching some more information to them
(https://mail.python.org/pipermail/python-ideas/2014-February/025601.html)

[16] LookupError etc. need API to get the key
(http://bugs.python.org/issue614557)

[17] LookupError etc. need API to get the key
(http://bugs.python.org/issue614557)

[18] making builtin exceptions more informative
(http://bugs.python.org/issue1182143)

[19] Enhance exceptions by attaching some more information to them
(https://mail.python.org/pipermail/python-ideas/2014-February/025601.html)

[20] Add 'unexpected_type' to TypeError
(http://bugs.python.org/issue18165)

[21] 'value' attribute for ValueError
(http://bugs.python.org/issue18166)

[22] weakref - Weak References
(https://docs.python.org/3/library/weakref.html)

[23] Message by R. David Murray: Weak refs on exceptions?
(http://bugs.python.org/issue18163#msg190791)

[24] ImportError needs attributes for module and file name
(http://bugs.python.org/issue1559549)

[25] Enhance exceptions by attaching some more information to them
(https://mail.python.org/pipermail/python-ideas/2014-February/025601.html)

[26] ImportError needs attributes for module and file name
(http://bugs.python.org/issue1559549)