text:
PEP: 512 Title: Migrating from hg.python.org to GitHub Version:
$Revision$ Last-Modified: $Date$ Author: Brett Cannon<EMAIL_ADDRESS>Discussions-To<EMAIL_ADDRESS>Status: Final Type: Process
Content-Type: text/x-rst Created: 17-Jan-2015 Post-History: 17-Jan-2016,
19-Jan-2016, 23-Jan-2016

Note

CPython's development process moved to https://github.com/python/cpython
on 2017-02-10.

Abstract

This PEP outlines the steps required to migrate Python's development
process from Mercurial[1] as hosted at hg.python.org[2] to Git[3] on
GitHub[4]. Meeting the minimum goals of this PEP should allow for the
development process of Python to be as productive as it currently is,
and meeting its extended goals should improve the development process
from its status quo.

Rationale

In 2014, it became obvious that Python's custom development process was
becoming a hindrance. As an example, for an external contributor to
submit a fix for a bug that eventually was committed, the basic steps
were:

1.  Open an issue for the bug at bugs.python.org[5].
2.  Checkout out the CPython source code from hg.python.org[6].
3.  Make the fix.
4.  Upload a patch.
5.  Have a core developer review the patch using our fork of the
    Rietveld code review tool[7].
6.  Download the patch to make sure it still applies cleanly.
7.  Run the test suite manually.
8.  Update the NEWS, ACKS, and "What's New" document as necessary
9.  Pull changes to avoid a merge race.
10. Commit the change manually.
11. If the change was for a bugfix release, merge into the
    in-development branch.
12. Run the test suite manually again.
13. Commit the merge.
14. Push the changes.

This is a very heavy, manual process for core developers. Even in the
simple case, you could only possibly skip the code review step, as you
would still need to build the documentation. This led to patches
languishing on the issue tracker due to core developers not being able
to work through the backlog fast enough to keep up with submissions. In
turn, that led to a side-effect issue of discouraging outside
contribution due to frustration from lack of attention, which is a
dangerous problem for an open source project with no corporate backing
as it runs counter to having a viable future for the project. While
allowing patches to be uploaded to bugs.python.org[8] is potentially
simple for an external contributor, it is as slow and burdensome as it
gets for a core developer to work with.

Hence the decision was made in late 2014 that a move to a new
development process was needed. A request for PEPs proposing new
workflows was made, in the end leading to two: PEP 481 and PEP 507
proposing GitHub and GitLab[9], respectively.

The year 2015 was spent off-and-on working on those proposals and trying
to tease out details of what made them different from each other on the
core-workflow mailing list[10]. PyCon US 2015 also showed that the
community was a bit frustrated with our process due to both cognitive
overhead for new contributors and how long it was taking for core
developers to look at a patch (see the end of Guido van Rossum's keynote
at PyCon US 2015[11] as an example of the frustration).

On January 1, 2016, the decision was made by Brett Cannon to move the
development process to GitHub. The key reasons for choosing GitHub
were[12]:

-   Maintaining custom infrastructure has been a burden on volunteers
    (e.g., an unmaintained, custom fork of Rietveld[13] is currently
    being used).
-   The custom workflow is very time-consuming for core developers (not
    enough automated tooling built to help support it).
-   The custom workflow is a hindrance to external contributors (acts as
    a barrier of entry due to time required to ramp up on development
    process unique to CPython itself).
-   There is no feature differentiating GitLab from GitHub beyond GitLab
    being open source.
-   Familiarity with GitHub is far higher among core developers and
    external contributors than with GitLab.
-   Our BDFL prefers GitHub (who would be the first person to tell you
    that his opinion shouldn't matter, but the person making the
    decision felt it was important that the BDFL feel comfortable with
    the workflow of his own programming language to encourage his
    continued participation).

There's even already an unofficial logo to represent the migration to
GitHub[14].

The overarching goal of this migration is to improve the development
process to the extent that a core developer can go from external
contribution submission through all the steps leading to committing said
contribution from within a browser on a tablet with WiFi using some
development process (this does not inherently mean GitHub's default
workflow). The final solution will also allow an external contributor to
contribute even if they chose not to use GitHub (although there is not
guarantee in feature parity).

Repositories to Migrate

While hg.python.org[15] hosts many repositories, there are only five key
repositories that need to move:

1.  devinabox[16] (done)
2.  benchmarks[17] (skipped)
3.  peps[18] (done)
4.  devguide[19] (done)
5.  cpython[20]

The devinabox repository is code-only. The peps and devguide
repositories involve the generation of webpages. And the cpython
repository has special requirements for integration with
bugs.python.org[21].

Migration Plan

The migration plan is separated into sections based on what is required
to migrate the repositories listed in the Repositories to Migrate
section. Completion of requirements outlined in each section should
unblock the migration of the related repositories. The sections are
expected to be completed in order, but not necessarily the requirements
within a section.

Requirements for Code-Only Repositories

Completion of the requirements in this section will allow the devinabox
repository to move to GitHub.

Create a 'Python core' team

To manage permissions, a 'Python core' team will be created as part of
the python organization[22]. Any repository that is moved will have the
'Python core' team added to it with write permissions[23]. Anyone who
previously had rights to manage SSH keys on hg.python.org will become a
team maintainer for the 'Python core' team.

Define commands to move a Mercurial repository to Git

Since moving to GitHub also entails moving to Git[24], we must decide
what tools and commands we will run to translate a Mercurial repository
to Git. The tools developed specifically for this migration are hosted
at https://github.com/orsenthil/cpython-hg-to-git .

CLA enforcement

A key part of any open source project is making sure that its source
code can be properly licensed. This requires making sure all people
making contributions have signed a contributor license agreement
(CLA)[25]. Up until now, enforcement of CLA signing of contributed code
has been enforced by core developers checking whether someone had an *
by their username on bugs.python.org[26]. With this migration, the plan
is to start off with automated checking and enforcement of contributors
signing the CLA.

Adding GitHub username support to bugs.python.org

To keep tracking of CLA signing under the direct control of the PSF,
tracking who has signed the PSF CLA will be continued by marking that
fact as part of someone's bugs.python.org user profile. What this means
is that an association will be needed between a person's
bugs.python.org[27] account and their GitHub account, which will be done
through a new field in a user's profile. This does implicitly require
that contributors will need both a GitHub and bugs.python.org account in
order to sign the CLA and contribute through GitHub.

An API is provided to query bugs.python.org to see if a GitHub username
corresponds to someone who has signed the CLA. Making a GET request to
e.g.
http://bugs.python.org/user?@template=clacheck&github_names=brettcannon,notanuser
returns a JSON dictionary with the keys of the usernames requested and a
true value if they have signed the CLA, false if they have not, and null
if no corresponding GitHub username was found.

A bot to enforce CLA signing

With an association between someone's GitHub account and their
bugs.python.org[28] account, which has the data as to whether someone
has signed the CLA, a bot can monitor pull requests on GitHub and denote
whether the contributor has signed the CLA.

If the user has signed the CLA, the bot will add a positive label to the
issue to denote the pull request has no CLA issues (e.g., a green label
stating, "CLA signed"). If the contributor has not signed a CLA, a
negative label will be added to the pull request will be blocked using
GitHub's status API (e.g., a red label stating, "CLA not signed"). If a
contributor lacks a bugs.python.org account, that will lead to the
negative label being used as well. Using a label for both positive and
negative cases provides a fallback signal if the bot happens to fail,
preventing potential false-positives or false-negatives. It also allows
for an easy way to trigger the bot again by simply removing a
CLA-related label (this is in contrast to using a GitHub status
check[29] which is only triggered on code changes).

As no pre-existing bot exists to meet our needs, it will be hosted on
Heroku[30] and written to target Python 3.5 to act as a showcase for
asynchronous programming. The code for the bot is hosted in the Knights
Who Say Ni project[31].

Make old repository read-only

Updating .hg/hgrc in the now-old Mercurial repository in the [hooks]
section with:

    pretxnchangegroup.reject = echo " * This repo has been migrated to github.com/python/peps and does not accept new commits in Mercurial!" 2>&1; exit 1

will make the repository read-only.

Requirements for Web-Related Repositories

Due to their use for generating webpages, the devguide[32] and peps[33]
repositories need their respective processes updated to pull from their
new Git repositories.

Requirements for the cpython Repository

Obviously the most active and important repository currently hosted at
hg.python.org[34] is the cpython repository[35]. Because of its
importance and high-frequency use, it requires more tooling before being
moved to GitHub compared to the other repositories mentioned in this
PEP.

Document steps to commit a pull request

During the process of choosing a new development workflow, it was
decided that a linear history is desired. People preferred having a
single commit representing a single change instead of having a set of
unrelated commits lead to a merge commit that represented a single
change. This means that the convenient "Merge" button in GitHub pull
requests will be set to only do squash commits and not merge commits.

A second set of recommended commands will also be written for committing
a contribution from a patch file uploaded to bugs.python.org[36]. This
will obviously help keep the linear history, but it will need to be made
to have attribution to the patch author.

The exact sequence of commands that will be given as guidelines to core
developers is an open issue: Git CLI commands for committing a pull
request to cpython.

Linking pull requests to issues

Historically, external contributions were attached to an issue on
bugs.python.org[37] thanks to the fact that all external contributions
were uploaded as a file. For changes committed by a core developer who
committed a change directly, the specifying of an issue number in the
commit message of the format Issue # at the start of the message led to
a comment being posted to the issue linking to the commit.

Linking a pull request to an issue

An association between a pull request and an issue is needed to track
when a fix has been proposed. The association needs to be many-to-one as
there can take multiple pull requests to solve a single issue
(technically it should be a many-to-many association for when a single
fix solves multiple issues, but this is fairly rare and issues can be
merged into one using the Superseder field on the issue tracker).

The association between a pull request and an issue will be done based
on detecting an issue number. If the issue is specified in either the
title or in the body of a message on a pull request then a connection
will be made on bugs.python.org[38]. Some visible notification -- e.g.
label or message -- will be made to the pull request to notify that the
association was successfully made.

Notify the issue if a commit is made

Once a commit is made, the corresponding issue should be updated to
reflect this fact. This should work regardless of whether the commit
came from a pull request or a direct commit.

Update the linking service for mapping commit IDs to URLs

Currently you can use https://hg.python.org/lookup/ with a revision ID
from either the Subversion or Mercurial copies of the cpython repo[39]
to get redirected to the URL for that revision in the Mercurial
repository. The URL rewriter will need to be updated to redirect to the
Git repository and to support the new revision IDs created for the Git
repository.

The most likely design is to statically know all the Mercurial changeset
numbers once the migration has occurred. The lookup code will then be
updated to accept hashes from 7 to 40 hexadecimal digits. Any
hexadecimal of length 12 or 40 will be compared against the Mercurial
changeset numbers. If the number doesn't match or is of some other
length between 7 and 40 then it will be assumed to be a Git hash.

The bugs.python.org commit number rewriter will also need to be updated
to accept hashes as short as 7 digits as Git will match on hashes that
short or longer.

Deprecate sys._mercurial

Once Python is no longer kept in Mercurial, the sys._mercurial attribute
will need to be changed to return ('CPython', '', ''). An equivalent
sys._git attribute will be added which fulfills the same use-cases.

Update the devguide

The devguide will need to be updated with details of the new workflow.
Mostly likely work will take place in a separate branch until the
migration actually occurs.

Update PEP 101

The release process will need to be updated as necessary.

Optional, Planned Features

Once the cpython repository[40] is migrated, all repositories will have
been moved to GitHub and the development process should be on equal
footing as before the move. But a key reason for this migration is to
improve the development process, making it better than it has ever been.
This section outlines some plans on how to improve things.

It should be mentioned that overall feature planning for
bugs.python.org[41] -- which includes plans independent of this
migration -- are tracked on their own wiki page[42].

Handling Misc/NEWS

Traditionally the Misc/NEWS file[43] has been problematic for changes
which spanned Python releases. Oftentimes there will be merge conflicts
when committing a change between e.g., 3.5 and 3.6 only in the Misc/NEWS
file. It's so common, in fact, that the example instructions in the
devguide explicitly mention how to resolve conflicts in the Misc/NEWS
file [44]. As part of our tool modernization, working with the Misc/NEWS
file will be simplified.

The planned approach is to use an individual file per news entry,
containing the text for the entry. In this scenario, each feature
release would have its own directory for news entries and a separate
file would be created in that directory that was either named after the
issue it closed or a timestamp value (which prevents collisions). Merges
across branches would have no issue as the news entry file would still
be uniquely named and in the directory of the latest version that
contained the fix. A script would collect all news entry files no matter
what directory they reside in and create an appropriate news file (the
release directory can be ignored as the mere fact that the file exists
is enough to represent that the entry belongs to the release).
Classification can either be done by keyword in the new entry file
itself or by using subdirectories representing each news entry
classification in each release directory (or classification of news
entries could be dropped since critical information is captured by the
"What's New" documents which are organized). The benefit of this
approach is that it keeps the changes with the code that was actually
changed. It also ties the message to being part of the commit which
introduced the change. For a commit made through the CLI, a script could
be provided to help generate the file. In a bot-driven scenario, the
merge bot could have a way to specify a specific news entry and create
the file as part of its flattened commit (while most likely also
supporting using the first line of the commit message if no specific
news entry was specified). If a web-based workflow is used then a status
check could be used to verify that a new entry file is in the pull
request to act as a reminder that the file is missing. Code for this
approach has been written previously for the Mercurial workflow at
http://bugs.python.org/issue18967. There is also tools from the
community like https://pypi.python.org/pypi/towncrier,
https://github.com/twisted/newsbuilder, and
http://docs.openstack.org/developer/reno/.

Discussions at the Sep 2016 Python core-dev sprints led to this decision
compared to the rejected approaches outlined in the Rejected Ideas
section of this PEP. The separate files approach seems to have the right
balance of flexibility and potential tooling out of the various options
while solving the motivating problem.

Work for this is being tracked at
https://github.com/python/core-workflow/issues/6.

Handling Misc/ACKS

Traditionally the Misc/ACKS file[45] has been managed by hand. But
thanks to Git supporting an author value as well as a committer value
per commit, authorship of a commit can be part of the history of the
code itself.

As such, manual management of Misc/ACKS will become optional. A script
will be written that will collect all author and committer names and
merge them into Misc/ACKS with all of the names listed prior to the move
to Git. Running this script will become part of the release process.

The script should also generate a list of all people who contributed
since the last execution. This will allow having a list of those who
contributed to a specific release so they can be explicitly thanked.

Work for this is being tracked at
https://github.com/python/core-workflow/issues/7.

Create https://git.python.org

Just as hg.python.org[46] currently points to the Mercurial repository
for Python, git.python.org should do the equivalent for the Git
repository.

Backup of pull request data

Since GitHub is going to be used for code hosting and code review, those
two things need to be backed up. In the case of code hosting, the backup
is implicit as all non-shallow Git[47] clones contain the full history
of the repository, hence there will be many backups of the repository.

The code review history does not have the same implicit backup mechanism
as the repository itself. That means a daily backup of code review
history should be done so that it is not lost in case of any issues with
GitHub. It also helps guarantee that a migration from GitHub to some
other code review system is feasible were GitHub to disappear overnight.

Bot to generate cherry-pick pull requests

Since the decision has been made to work with cherry-picks instead of
forward merging of branches, it would be convenient to have a bot that
would generate pull requests based on cherry-picking for any pull
requests that affect multiple branches. The most likely design is a bot
that monitors merged pull requests with key labels applied that
delineate what branches the pull request should be cherry-picked into.
The bot would then generate cherry-pick pull requests for each label and
remove the labels as the pull requests are created (this allows for easy
detection when automatic cherry-picking failed).

Work for this is being tracked at
https://github.com/python/core-workflow/issues/8.

Pull request commit queue

This would linearly apply accepted pull requests and verify that the
commits did not interfere with each other by running the test suite and
backing out commits if the test run failed. To help facilitate the speed
of testing, all patches committed since the last test run can be applied
at once under a single test run as the optimistic assumption is that the
patches will work in tandem. Some mechanism to re-run the tests in case
of test flakiness will be needed, whether it is from removing a "test
failed" label, web interface for core developers to trigger another
testing event, etc.

Inspiration or basis of the bot could be taken from pre-existing bots
such as Homu[48] or Zuul[49].

The name given to this bot in order to give it commands is an open
issue: Naming the bots.

A CI service

There are various CI services that provide free support for open source
projects hosted on GitHub. After experimenting with a couple CI
services, the decision was made to go with Travis[50].

The current CI service for Python is Pypatcher[51]. A request can be
made in IRC to try a patch from bugs.python.org[52]. The results can be
viewed at https://ci.centos.org/job/cPython-build-patch/ .

Work for this is being tracked at
https://github.com/python/core-workflow/issues/1.

Test coverage report

Getting an up-to-date test coverage report for Python's standard library
would be extremely beneficial as generating such a report can take quite
a while to produce.

There are a couple pre-existing services that provide free test coverage
for open source projects. In the end, Codecov[53] was chosen as the best
option.

Work for this is being tracked at
https://github.com/python/core-workflow/issues/2.

Notifying issues of pull request comments

The current development process does not include notifying an issue on
bugs.python.org[54] when a review comment is left on Rietveld[55]. It
would be nice to fix this so that people can subscribe only to comments
at bugs.python.org and not GitHub and yet still know when something
occurs on GitHub in terms of review comments on relevant pull requests.
Current thinking is to post a comment to bugs.python.org to the relevant
issue when at least one review comment has been made over a certain
period of time (e.g., 15 or 30 minutes, although with GitHub now
supporting reviews the time aspect may be unnecessary). This keeps the
email volume down for those that receive both GitHub and bugs.python.org
email notifications while still making sure that those only following
bugs.python.org know when there might be a review comment to address.

Allow bugs.python.org to use GitHub as a login provider

As of right now, bugs.python.org[56] allows people to log in using
Google, Launchpad, or OpenID credentials. It would be good to expand
this to GitHub credentials.

Web hooks for re-generating web content

The content at https://docs.python.org/,
https://docs.python.org/devguide, and https://www.python.org/dev/peps/
are all derived from files kept in one of the repositories to be moved
as part of this migration. As such, it would be nice to set up
appropriate webhooks to trigger rebuilding the appropriate web content
when the files they are based on change instead of having to wait for,
e.g., a cronjob to trigger.

This can partially be solved if the documentation is a Sphinx project as
then the site can have an unofficial mirror on Read the Docs, e.g.
http://cpython-devguide.readthedocs.io/.

Work for this is being tracked at
https://github.com/python/core-workflow/issues/9.

Link web content back to files that it is generated from

It would be helpful for people who find issues with any of the
documentation that is generated from a file to have a link on each page
which points back to the file on GitHub that stores the content of the
page. That would allow for quick pull requests to fix simple things such
as spelling mistakes.

Work for this is being tracked at http://bugs.python.org/issue28929.

Splitting out parts of the documentation into their own repositories

While certain parts of the documentation at https://docs.python.org
change with the code, other parts are fairly static and are not tightly
bound to the CPython code itself. The following sections of the
documentation fit this category of slow-changing, loosely-coupled:

-   Tutorial
-   Python Setup and Usage
-   HOWTOs
-   Installing Python Modules
-   Distributing Python Modules
-   Extending and Embedding
-   FAQs

These parts of the documentation could be broken out into their own
repositories to simplify their maintenance and to expand who has commit
rights to them to ease in their maintenance.

It has also been suggested to split out the What's New documents. That
would require deciding whether a workflow could be developed where it
would be difficult to forget to update What's New (potentially through a
label added to PRs, like "What's New needed").

Backup of Git repositories

While not necessary, it would be good to have official backups of the
various Git repositories for disaster protection. It will be up to the
PSF infrastructure committee to decide if this is worthwhile or
unnecessary.

Identify potential new core developers

The Python development team has long-standing guidelines for selecting
new core developers. The key part of the guidelines is that a person
needs to have contributed multiple patches which have been accepted and
are high enough quality and size to demonstrate an understanding of
Python's development process. A bot could be written which tracks patch
acceptance rates and generates a report to help identify contributors
who warrant consideration for becoming core developers. This work
doesn't even necessarily require GitHub integration as long as the
committer field in all git commits is filled in properly.

Work is being tracked at
https://github.com/python/core-workflow/issues/10.

Status

Requirements for migrating the devinabox[57] repository:

-   Completed
    -   Adding GitHub username support to bugs.python.org (Maciej Szulik
        and Ezio Melotti)
    -   A bot to enforce CLA signing:
        https://github.com/python/the-knights-who-say-ni (Brett Cannon)
    -   Create a 'Python core' team
    -   Define commands to move a Mercurial repository to Git:
        https://github.com/orsenthil/cpython-hg-to-git (Senthil Kumaran)

Repositories whose build steps need updating:

-   Completed
    -   peps[58]
    -   devguide[59]

cpython repo[60]

Required:

-   Not started
    -   Update PEP 101 (commitment from Ned Deily to do this;
        non-blocker)
-   In progress
    -   Deprecate sys._mercurial (http://bugs.python.org/issue27593;
        review committal from Ned Deily; non-blocker)
    -   Update the linking service for mapping commit IDs to URLs (code
        ready, needs deployment once the hg repository is made
        read-only;
        https://gist.github.com/brettcannon/f8d97c92b0df264cd4db008ffd32daf9;
        post-migration)
-   Completed
    -   Notify the issue if a commit is made
        (http://psf.upfronthosting.co.za/roundup/meta/issue611)
    -   Track PR status in appropriate issue
        (http://psf.upfronthosting.co.za/roundup/meta/issue590)
    -   Update the devguide, including Document steps to commit a pull
        request (https://github.com/python/devguide/milestone/1)
    -   Update commit hash detection on b.p.o to support 10- and
        11-character hashes
        (http://psf.upfronthosting.co.za/roundup/meta/issue610)
    -   Linking a pull request to an issue
        (http://psf.upfronthosting.co.za/roundup/meta/issue589)
    -   Email python-checkins for each commit (PR or direct)
        (https://help.github.com/articles/managing-notifications-for-pushes-to-a-repository/)
    -   Message #python-dev for each commit (PR or direct)
        (https://github.com/python/cpython/settings/hooks/new?service=irc)
    -   Get docs built from git
        (https://github.com/python/docsbuild-scripts/blob/main/build_docs.py
        already updated; https://github.com/python/psf-salt/pull/91 to
        switch)
    -   Migrate buildbots to be triggered and pull from GitHub

Optional features:

-   Not started
    -   Check for whitespace abnormalities as part of CI
    -   Create https://git.python.org
    -   Backup of pull request data
    -   Handling Misc/ACKS
    -   Pull request commit queue
    -   Allow bugs.python.org to use GitHub as a login provider
    -   Web hooks for re-generating web content
    -   Splitting out parts of the documentation into their own
        repositories
    -   Backup of Git repositories
-   In progress
    -   Notifying issues of pull request comments
        (http://psf.upfronthosting.co.za/roundup/meta/issue592)
    -   Convert b.p.o patches to GitHub PRs
        (http://psf.upfronthosting.co.za/roundup/meta/issue600)
-   Completed
    -   A CI Service
    -   Test coverage report
    -   Link web content back to files that it is generated from
    -   Handling Misc/NEWS
    -   Bot to generate cherry-pick pull requests
    -   Write .github/CONTRIBUTING.md (to prevent PRs that are
        inappropriate from even showing up and pointing to the devguide)

Open Issues

For this PEP, open issues are ones where a decision needs to be made to
how to approach or solve a problem. Open issues do not entail
coordination issues such as who is going to write a certain bit of code.

The fate of hg.python.org

With the code repositories moving over to Git[61], there is no technical
need to keep hg.python.org[62] running. Having said that, some in the
community would like to have it stay functioning as a Mercurial[63]
mirror of the Git repositories. Others have said that they still want a
mirror, but one using Git.

As maintaining hg.python.org is not necessary, it will be up to the PSF
infrastructure committee to decide if they want to spend the time and
resources to keep it running. They may also choose whether they want to
host a Git mirror on PSF infrastructure.

Depending on the decision reached, other ancillary repositories will
either be forced to migration or they can choose to simply stay on
hg.python.org.

Git CLI commands for committing a pull request to cpython

Because Git[64] may be a new version control system for core developers,
the commands people are expected to run will need to be written down.
These commands also need to keep a linear history while giving proper
attribution to the pull request author.

Another set of commands will also be necessary for when working with a
patch file uploaded to bugs.python.org[65]. Here the linear history will
be kept implicitly, but it will need to make sure to keep/add
attribution.

Naming the bots

As naming things can lead to bikeshedding of epic proportions, Brett
Cannon will choose the final name of the various bots (the name of the
project for the bots themselves can be anything, this is purely for the
name used in giving commands to the bot or the account name). The names
must come from Monty Python, which is only fitting since Python is named
after the comedy troupe.

Rejected Ideas

Separate Python 2 and Python 3 repositories

It was discussed whether separate repositories for Python 2 and Python 3
were desired. The thinking was that this would shrink the overall
repository size which benefits people with slow Internet connections or
small bandwidth caps.

In the end it was decided that it was easier logistically to simply keep
all of CPython's history in a single repository.

Commit multi-release changes in bugfix branch first

As the current development process has changes committed in the oldest
branch first and then merged up to the default branch, the question came
up as to whether this workflow should be perpetuated. In the end it was
decided that committing in the newest branch and then cherry-picking
changes into older branches would work best as most people will
instinctively work off the newest branch and it is a more common
workflow when using Git[66].

Cherry-picking is also more bot-friendly for an in-browser workflow. In
the merge-up scenario, if you were to request a bot to do a merge and it
failed, then you would have to make sure to immediately solve the merge
conflicts if you still allowed the main commit, else you would need to
postpone the entire commit until all merges could be handled. With a
cherry-picking workflow, the main commit could proceed while postponing
the merge-failing cherry-picks. This allows for possibly distributing
the work of managing conflicting merges.

Lastly, cherry-picking should help avoid merge races. Currently, when
one is doing work that spans branches, it takes time to commit in the
older branch, possibly push to another clone representing the default
branch, merge the change, and then push upstream. Cherry-picking should
decouple this so that you don't have to rush your multi-branch changes
as the cherry-pick can be done separately.

Deriving Misc/NEWS from the commit logs

As part of the discussion surrounding Handling Misc/NEWS, the suggestion
has come up of deriving the file from the commit logs itself. In this
scenario, the first line of a commit message would be taken to represent
the news entry for the change. Some heuristic to tie in whether a change
warranted a news entry would be used, e.g., whether an issue number is
listed.

This idea has been rejected due to some core developers preferring to
write a news entry separate from the commit message. The argument is the
first line of a commit message compared to that of a news entry have
different requirements in terms of brevity, what should be said, etc.

Deriving Misc/NEWS from bugs.python.org

A rejected solution to the NEWS file problem was to specify the entry on
bugs.python.org[67]. This would mean an issue that is marked as
"resolved" could not be closed until a news entry is added in the "news"
field in the issue tracker. The benefit of tying the news entry to the
issue is it makes sure that all changes worthy of a news entry have an
accompanying issue. It also makes classifying a news entry automatic
thanks to the Component field of the issue. The Versions field of the
issue also ties the news entry to which Python releases were affected. A
script would be written to query bugs.python.org for relevant new
entries for a release and to produce the output needed to be checked
into the code repository. This approach is agnostic to whether a commit
was done by CLI or bot. A drawback is that there's a disconnect between
the actual commit that made the change and the news entry by having them
live in separate places (in this case, GitHub and bugs.python.org). This
would mean making a commit would then require remembering to go back to
bugs.python.org to add the news entry.

References

Copyright

This document has been placed in the public domain.

[1] Mercurial (https://www.mercurial-scm.org/)

[2] https://hg.python.org

[3] Git (https://git-scm.com/)

[4] GitHub (https://github.com)

[5] https://bugs.python.org

[6] https://hg.python.org

[7] Rietveld (https://github.com/rietveld-codereview/rietveld)

[8] https://bugs.python.org

[9] GitLab (https://about.gitlab.com/)

[10] core-workflow mailing list
(https://mail.python.org/mailman/listinfo/core-workflow)

[11] Guido van Rossum's keynote at PyCon US
(https://www.youtube.com/watch?v=G-uKNd5TSBw)

[12] Email to core-workflow outlining reasons why GitHub was selected
(https://mail.python.org/pipermail/core-workflow/2016-January/000345.html)

[13] Rietveld (https://github.com/rietveld-codereview/rietveld)

[14] Pythocat (https://octodex.github.com/pythocat)

[15] https://hg.python.org

[16] Mercurial repository for devinabox
(https://hg.python.org/devinabox/)

[17] Mercurial repository for the Unified Benchmark Suite
(https://hg.python.org/benchmarks/)

[18] Mercurial repository of the Python Enhancement Proposals
(https://hg.python.org/peps/)

[19] Mercurial repository for the Python Developer's Guide
(https://hg.python.org/devguide/)

[20] Mercurial repository for CPython (https://hg.python.org/cpython/)

[21] https://bugs.python.org

[22] Python organization on GitHub (https://github.com/python)

[23] GitHub repository permission levels
(https://help.github.com/enterprise/2.4/user/articles/repository-permission-levels-for-an-organization/)

[24] Git (https://git-scm.com/)

[25] Python Software Foundation Contributor Agreement
(https://www.python.org/psf/contrib/contrib-form/)

[26] https://bugs.python.org

[27] https://bugs.python.org

[28] https://bugs.python.org

[29] GitHub status checks
(https://developer.github.com/v3/repos/statuses/)

[30] Heroku (https://www.heroku.com/)

[31] The Knights Who Say Ni project
(https://github.com/python/the-knights-who-say-ni)

[32] Mercurial repository for the Python Developer's Guide
(https://hg.python.org/devguide/)

[33] Mercurial repository of the Python Enhancement Proposals
(https://hg.python.org/peps/)

[34] https://hg.python.org

[35] Mercurial repository for CPython (https://hg.python.org/cpython/)

[36] https://bugs.python.org

[37] https://bugs.python.org

[38] https://bugs.python.org

[39] Mercurial repository for CPython (https://hg.python.org/cpython/)

[40] Mercurial repository for CPython (https://hg.python.org/cpython/)

[41] https://bugs.python.org

[42] Wiki page for bugs.python.org feature development
(https://wiki.python.org/moin/TrackerDevelopmentPlanning)

[43] Misc/NEWS (https://hg.python.org/cpython/file/default/Misc/NEWS)

[44] Devguide instructions on how to merge across branches
(https://docs.python.org/devguide/committing.html#merging-between-different-branches-within-the-same-major-version)

[45] Misc/ACKS (https://hg.python.org/cpython/file/default/Misc/ACKS)

[46] https://hg.python.org

[47] Git (https://git-scm.com/)

[48] Homu (http://homu.io/)

[49] Zuul (http://docs.openstack.org/infra/zuul/)

[50] Travis (https://travis-ci.org/)

[51] Pypatcher (https://github.com/kushaldas/pypatcher)

[52] https://bugs.python.org

[53] Codecov (https://codecov.io/)

[54] https://bugs.python.org

[55] Rietveld (https://github.com/rietveld-codereview/rietveld)

[56] https://bugs.python.org

[57] Mercurial repository for devinabox
(https://hg.python.org/devinabox/)

[58] Mercurial repository of the Python Enhancement Proposals
(https://hg.python.org/peps/)

[59] Mercurial repository for the Python Developer's Guide
(https://hg.python.org/devguide/)

[60] Mercurial repository for CPython (https://hg.python.org/cpython/)

[61] Git (https://git-scm.com/)

[62] https://hg.python.org

[63] Mercurial (https://www.mercurial-scm.org/)

[64] Git (https://git-scm.com/)

[65] https://bugs.python.org

[66] Git (https://git-scm.com/)

[67] https://bugs.python.org


text:
PEP: 580 Title: The C call protocol Author: Jeroen Demeyer
<EMAIL_ADDRESS>BDFL-Delegate: Petr Viktorin Status: Rejected Type:
Standards Track Content-Type: text/x-rst Created: 14-Jun-2018
Python-Version: 3.8 Post-History: 20-Jun-2018, 22-Jun-2018, 16-Jul-2018

Rejection Notice

This PEP is rejected in favor of PEP 590, which proposes a simpler
public C API for callable objects.

Abstract

A new "C call" protocol is proposed. It is meant for classes
representing functions or methods which need to implement fast calling.
The goal is to generalize all existing optimizations for built-in
functions to arbitrary extension types.

In the reference implementation, this new protocol is used for the
existing classes builtin_function_or_method and method_descriptor.
However, in the future, more classes may implement it.

NOTE: This PEP deals only with the Python/C API, it does not affect the
Python language or standard library.

Motivation

The standard function/method classes builtin_function_or_method and
method_descriptor allow very efficiently calling C code. However, they
are not subclassable, making them unsuitable for many applications: for
example, they offer limited introspection support (signatures only using
__text_signature__, no arbitrary __qualname__, no inspect.getfile()).
It's also not possible to store additional data to implement something
like functools.partial or functools.lru_cache. So, there are many
reasons why users would want to implement custom function/method classes
(in a duck-typing sense) in C. Unfortunately, such custom classes are
necessarily slower than the standard CPython function classes: the
bytecode interpreter has various optimizations which are specific to
instances of builtin_function_or_method, method_descriptor, method and
function.

This PEP also allows to simplify existing code: checks for
builtin_function_or_method and method_descriptor could be replaced by
simply checking for and using the C call protocol. Future PEPs may
implement the C call protocol for more classes, enabling even further
simplifications.

We also design the C call protocol such that it can easily be extended
with new features in the future.

For more background and motivation, see PEP 579.

Overview

Currently, CPython has multiple optimizations for fast calling for a few
specific function classes. A good example is the implementation of the
opcode CALL_FUNCTION, which has the following structure (see the actual
code):

    if (PyCFunction_Check(func)) {
        return _PyCFunction_FastCallKeywords(func, stack, nargs, kwnames);
    }
    else if (Py_TYPE(func) == &PyMethodDescr_Type) {
        return _PyMethodDescr_FastCallKeywords(func, stack, nargs, kwnames);
    }
    else {
        if (PyMethod_Check(func) && PyMethod_GET_SELF(func) != NULL) {
            /* ... */
        }
        if (PyFunction_Check(func)) {
            return _PyFunction_FastCallKeywords(func, stack, nargs, kwnames);
        }
        else {
            return _PyObject_FastCallKeywords(func, stack, nargs, kwnames);
        }
    }

Calling instances of these special-cased classes using the tp_call slot
is slower than using the optimizations. The basic idea of this PEP is to
enable such optimizations for user C code, both as caller and as callee.

The existing class builtin_function_or_method and a few others use a
PyMethodDef structure for describing the underlying C function and its
signature. The first concrete change is that this is replaced by a new
structure PyCCallDef. This stores some of the same information as a
PyMethodDef, but with one important addition: the "parent" of the
function (the class or module where it is defined). Note that
PyMethodDef arrays are still used to construct functions/methods but no
longer for calling them.

Second, we want that every class can use such a PyCCallDef for
optimizing calls, so the PyTypeObject structure gains a tp_ccalloffset
field giving an offset to a PyCCallDef * in the object structure and a
flag Py_TPFLAGS_HAVE_CCALL indicating that tp_ccalloffset is valid.

Third, since we want to deal efficiently with unbound and bound methods
too (as opposed to only plain functions), we need to handle __self__ in
the protocol: after the PyCCallDef * in the object structure, there is a
PyObject *self field. These two fields together are referred to as a
PyCCallRoot structure.

The new protocol for efficiently calling objects using these new
structures is called the "C call protocol".

NOTE: In this PEP, the phrases "unbound method" and "bound method" refer
to generic behavior, not to specific classes. For example, an unbound
method gets turned into a bound method after applying __get__.

New data structures

The PyTypeObject structure gains a new field Py_ssize_t tp_ccalloffset
and a new flag Py_TPFLAGS_HAVE_CCALL. If this flag is set, then
tp_ccalloffset is assumed to be a valid offset inside the object
structure (similar to tp_dictoffset and tp_weaklistoffset). It must be a
strictly positive integer. At that offset, a PyCCallRoot structure
appears:

    typedef struct {
        const PyCCallDef *cr_ccall;
        PyObject         *cr_self;  /* __self__ argument for methods */
    } PyCCallRoot;

The PyCCallDef structure contains everything needed to describe how the
function can be called:

    typedef struct {
        uint32_t  cc_flags;
        PyCFunc   cc_func;    /* C function to call */
        PyObject *cc_parent;  /* class or module */
    } PyCCallDef;

The reason for putting __self__ outside of PyCCallDef is that PyCCallDef
is not meant to be changed after creating the function. A single
PyCCallDef can be shared by an unbound method and multiple bound
methods. This wouldn't work if we would put __self__ inside that
structure.

NOTE: unlike tp_dictoffset we do not allow negative numbers for
tp_ccalloffset to mean counting from the end. There does not seem to be
a use case for it and it would only complicate the implementation.

Parent

The cc_parent field (accessed for example by a __parent__ or
__objclass__ descriptor from Python code) can be any Python object, or
NULL. Custom classes are free to set cc_parent to whatever they want. It
is only used by the C call protocol if the CCALL_OBJCLASS flag is set.

For methods of extension types, cc_parent points to the class that
defines the method (which may be a superclass of type(self)). This is
currently non-trivial to retrieve from a method's code. In the future,
this can be used to access the module state via the defining class. See
the rationale of PEP 573 for details.

When the flag CCALL_OBJCLASS is set (as it will be for methods of
extension types), cc_parent is used for type checks like the following:

    >>> list.append({}, "x")
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: descriptor 'append' requires a 'list' object but received a 'dict'

For functions of modules, cc_parent is set to the module. Currently,
this is exactly the same as __self__. However, using __self__ for the
module is a quirk of the current implementation: in the future, we want
to allow functions which use __self__ in the normal way, for
implementing methods. Such functions can still use cc_parent instead to
refer to the module.

The parent would also typically be used to implement __qualname__. The
new C API function PyCCall_GenericGetQualname() does exactly that.

Using tp_print

We propose to replace the existing unused field tp_print by
tp_ccalloffset. Since Py_TPFLAGS_HAVE_CCALL would not be added to
Py_TPFLAGS_DEFAULT, this ensures full backwards compatibility for
existing extension modules setting tp_print. It also means that we can
require that tp_ccalloffset is a valid offset when Py_TPFLAGS_HAVE_CCALL
is specified: we do not need to check tp_ccalloffset != 0. In future
Python versions, we may decide that tp_print becomes tp_ccalloffset
unconditionally, drop the Py_TPFLAGS_HAVE_CCALL flag and instead check
for tp_ccalloffset != 0.

NOTE: the exact layout of PyTypeObject is not part of the
stable ABI <384>). Therefore, changing the tp_print field from a
printfunc (a function pointer) to a Py_ssize_t should not be a problem,
even if this changes the memory layout of the PyTypeObject structure.
Moreover, on all systems for which binaries are commonly built (Windows,
Linux, macOS), the size of printfunc and Py_ssize_t are the same, so the
issue of binary compatibility will not come up anyway.

The C call protocol

We say that a class implements the C call protocol if it has the
Py_TPFLAGS_HAVE_CCALL flag set (as explained above, it must then set
tp_ccalloffset > 0). Such a class must implement __call__ as described
in this section (in practice, this just means setting tp_call to
PyCCall_Call).

The cc_func field is a C function pointer, which plays the same role as
the existing ml_meth field of PyMethodDef. Its precise signature depends
on flags. The subset of flags influencing the signature of cc_func is
given by the bitmask CCALL_SIGNATURE. Below are the possible values for
cc_flags & CCALL_SIGNATURE together with the arguments that the C
function takes. The return value is always PyObject *. The following are
analogous to the existing PyMethodDef signature flags:

-   CCALL_VARARGS: cc_func(PyObject *self, PyObject *args)
-   CCALL_VARARGS | CCALL_KEYWORDS:
    cc_func(PyObject *self, PyObject *args, PyObject *kwds) (kwds is
    either NULL or a dict; this dict must not be modified by the callee)
-   CCALL_FASTCALL:
    cc_func(PyObject *self, PyObject *const *args, Py_ssize_t nargs)
-   CCALL_FASTCALL | CCALL_KEYWORDS:
    cc_func(PyObject *self, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames)
    (kwnames is either NULL or a non-empty tuple of keyword names)
-   CCALL_NOARGS: cc_func(PyObject *self, PyObject *unused) (second
    argument is always NULL)
-   CCALL_O: cc_func(PyObject *self, PyObject *arg)

The flag CCALL_DEFARG may be combined with any of these. If so, the C
function takes an additional argument as first argument before self,
namely a const pointer to the PyCCallDef structure used for this call.
For example, we have the following signature:

-   CCALL_DEFARG | CCALL_VARARGS:
    cc_func(const PyCCallDef *def, PyObject *self, PyObject *args)

One exception is CCALL_DEFARG | CCALL_NOARGS: the unused argument is
dropped, so the signature becomes

-   CCALL_DEFARG | CCALL_NOARGS:
    cc_func(const PyCCallDef *def, PyObject *self)

NOTE: unlike the existing METH_... flags, the CCALL_... constants do not
necessarily represent single bits. So checking
if (cc_flags & CCALL_VARARGS) is not a valid way for checking the
signature. There are also no guarantees of binary compatibility for
these flags between Python versions. This allows the implementation to
choose the most efficient numerical values of the flags. In the
reference implementation, the legal values for
cc_flags & CCALL_SIGNATURE form exactly the interval [0, …, 11]. This
means that the compiler can easily optimize a switch statement for those
cases using a computed goto.

Checking __objclass__

If the CCALL_OBJCLASS flag is set and if cr_self is NULL (this is the
case for unbound methods of extension types), then a type check is done:
the function must be called with at least one positional argument and
the first (typically called self) must be an instance of cc_parent
(which must be a class). If not, a TypeError is raised.

Self slicing

If cr_self is not NULL or if the flag CCALL_SELFARG is not set in
cc_flags, then the argument passed as self is simply cr_self.

If cr_self is NULL and the flag CCALL_SELFARG is set, then the first
positional argument is removed from args and instead passed as self
argument to the C function. Effectively, the first positional argument
is treated as __self__. If there are no positional arguments, TypeError
is raised.

This process is called "self slicing" and a function is said to have
self slicing if cr_self is NULL and CCALL_SELFARG is set.

Note that a CCALL_NOARGS function with self slicing effectively has one
argument, namely self. Analogously, a CCALL_O function with self slicing
has two arguments.

Descriptor behavior

Classes supporting the C call protocol must implement the descriptor
protocol in a specific way.

This is required for an efficient implementation of bound methods: if
other code can make assumptions on what __get__ does, it enables
optimizations which would not be possible otherwise. In particular, we
want to allow sharing the PyCCallDef structure between bound and unbound
methods. We also need a correct implementation of _PyObject_GetMethod
which is used by the LOAD_METHOD/CALL_METHOD optimization.

First of all, if func supports the C call protocol, then func.__set__
and func.__delete__ must not be implemented.

Second, func.__get__ must behave as follows:

-   If cr_self is not NULL, then __get__ must be a no-op in the sense
    that func.__get__(obj, cls)(*args, **kwds) behaves exactly the same
    as func(*args, **kwds). It is also allowed for __get__ to be not
    implemented at all.
-   If cr_self is NULL, then func.__get__(obj, cls)(*args, **kwds) (with
    obj not None) must be equivalent to func(obj, *args, **kwds). In
    particular, __get__ must be implemented in this case. This is
    unrelated to self slicing: obj may be passed as self argument to the
    C function or it may be the first positional argument.
-   If cr_self is NULL, then func.__get__(None, cls)(*args, **kwds) must
    be equivalent to func(*args, **kwds).

There are no restrictions on the object func.__get__(obj, cls). The
latter is not required to implement the C call protocol for example. We
only specify what func.__get__(obj, cls).__call__ does.

For classes that do not care about __self__ and __get__ at all, the
easiest solution is to assign cr_self = Py_None (or any other non-NULL
value).

The __name__ attribute

The C call protocol requires that the function has a __name__ attribute
which is of type str (not a subclass).

Furthermore, the object returned by __name__ must be stored somewhere;
it cannot be a temporary object. This is required because
PyEval_GetFuncName uses a borrowed reference to the __name__ attribute
(see also[1]).

Generic API functions

This section lists the new public API functions or macros dealing with
the C call protocol.

-   int PyCCall_Check(PyObject *op): return true if op implements the C
    call protocol.

All the functions and macros below apply to any instance supporting the
C call protocol. In other words, PyCCall_Check(func) must be true.

-   PyObject *PyCCall_Call(PyObject *func, PyObject *args, PyObject *kwds):
    call func with positional arguments args and keyword arguments kwds
    (kwds may be NULL). This function is meant to be put in the tp_call
    slot.
-   PyObject *PyCCall_FastCall(PyObject *func, PyObject *const *args, Py_ssize_t nargs, PyObject *kwds):
    call func with nargs positional arguments given by args[0], …,
    args[nargs-1]. The parameter kwds can be NULL (no keyword
    arguments), a dict with name:value items or a tuple with keyword
    names. In the latter case, the keyword values are stored in the args
    array, starting at args[nargs].

Macros to access the PyCCallRoot and PyCCallDef structures:

-   const PyCCallRoot *PyCCall_CCALLROOT(PyObject *func): pointer to the
    PyCCallRoot structure inside func.
-   const PyCCallDef *PyCCall_CCALLDEF(PyObject *func): shorthand for
    PyCCall_CCALLROOT(func)->cr_ccall.
-   uint32_t PyCCall_FLAGS(PyObject *func): shorthand for
    PyCCall_CCALLROOT(func)->cr_ccall->cc_flags.
-   PyObject *PyCCall_SELF(PyOject *func): shorthand for
    PyCCall_CCALLROOT(func)->cr_self.

Generic getters, meant to be put into the tp_getset array:

-   PyObject *PyCCall_GenericGetParent(PyObject *func, void *closure):
    return cc_parent. Raise AttributeError if cc_parent is NULL.
-   PyObject *PyCCall_GenericGetQualname(PyObject *func, void *closure):
    return a string suitable for using as __qualname__. This uses the
    __qualname__ of cc_parent if possible. It also uses the __name__
    attribute.

Profiling

The profiling events c_call, c_return and c_exception are only generated
when calling actual instances of builtin_function_or_method or
method_descriptor. This is done for simplicity and also for backwards
compatibility (such that the profile function does not receive objects
that it does not recognize). In a future PEP, we may extend C-level
profiling to arbitrary classes implementing the C call protocol.

Changes to built-in functions and methods

The reference implementation of this PEP changes the existing classes
builtin_function_or_method and method_descriptor to use the C call
protocol. In fact, those two classes are almost merged: the
implementation becomes very similar, but they remain separate classes
(mostly for backwards compatibility). The PyCCallDef structure is simply
stored as part of the object structure. Both classes use
PyCFunctionObject as object structure. This is the new layout for both
classes:

    typedef struct {
        PyObject_HEAD
        PyCCallDef  *m_ccall;
        PyObject    *m_self;         /* Passed as 'self' arg to the C function */
        PyCCallDef   _ccalldef;      /* Storage for m_ccall */
        PyObject    *m_name;         /* __name__; str object (not NULL) */
        PyObject    *m_module;       /* __module__; can be anything */
        const char  *m_doc;          /* __text_signature__ and __doc__ */
        PyObject    *m_weakreflist;  /* List of weak references */
    } PyCFunctionObject;

For functions of a module and for unbound methods of extension types,
m_ccall points to the _ccalldef field. For bound methods, m_ccall points
to the PyCCallDef of the unbound method.

NOTE: the new layout of method_descriptor changes it such that it no
longer starts with PyDescr_COMMON. This is purely an implementation
detail and it should cause few (if any) compatibility problems.

C API functions

The following function is added (also to the stable ABI <384>):

-   PyObject * PyCFunction_ClsNew(PyTypeObject *cls, PyMethodDef *ml, PyObject *self, PyObject *module, PyObject *parent):
    create a new object with object structure PyCFunctionObject and
    class cls. The entries of the PyMethodDef structure are used to
    construct the new object, but the pointer to the PyMethodDef
    structure is not stored. The flags for the C call protocol are
    automatically determined in terms of ml->ml_flags, self and parent.

The existing functions PyCFunction_New, PyCFunction_NewEx and
PyDescr_NewMethod are implemented in terms of PyCFunction_ClsNew.

The undocumented functions PyCFunction_GetFlags and
PyCFunction_GET_FLAGS are deprecated. They are still artificially
supported by storing the original METH_... flags in a bitfield inside
cc_flags. Despite the fact that PyCFunction_GetFlags is technically part
of the stable ABI <384>, it is highly unlikely to be used that way:
first of all, it is not even documented. Second, the flag METH_FASTCALL
is not part of the stable ABI but it is very common (because of Argument
Clinic). So, if one cannot support METH_FASTCALL, it is hard to imagine
a use case for PyCFunction_GetFlags. The fact that PyCFunction_GET_FLAGS
and PyCFunction_GetFlags are not used at all by CPython outside of
Objects/call.c further shows that these functions are not particularly
useful.

Inheritance

Extension types inherit the type flag Py_TPFLAGS_HAVE_CCALL and the
value tp_ccalloffset from the base class, provided that they implement
tp_call and tp_descr_get the same way as the base class. Heap types
never inherit the C call protocol because that would not be safe (heap
types can be changed dynamically).

Performance

This PEP should not impact the performance of existing code (in the
positive or negative sense). It is meant to allow efficient new code to
be written, not to make existing code faster.

Here are a few pointers to the python-dev mailing list where performance
improvements are discussed:

-   https://mail.python.org/pipermail/python-dev/2018-July/154571.html
-   https://mail.python.org/pipermail/python-dev/2018-July/154740.html
-   https://mail.python.org/pipermail/python-dev/2018-July/154775.html
-   https://mail.python.org/pipermail/python-dev/2019-April/156954.html

Stable ABI

The function PyCFunction_ClsNew is added to the stable ABI <384>.

None of the functions, structures or constants dealing with the C call
protocol are added to the stable ABI.

There are two reasons for this: first of all, the most useful feature of
the C call protocol is probably the METH_FASTCALL calling convention.
Given that this is not even part of the public API (see also PEP 579,
issue 6), it would be strange to add anything else from the C call
protocol to the stable ABI.

Second, we want the C call protocol to be extensible in the future. By
not adding anything to the stable ABI, we are free to do that without
restrictions.

Backwards compatibility

There is no difference at all for the Python interface, nor for the
documented C API (in the sense that all functions remain supported with
the same functionality).

The only potential breakage is with C code which accesses the internals
of PyCFunctionObject and PyMethodDescrObject. We expect very few
problems because of this.

Rationale

Why is this better than PEP 575?

One of the major complaints of PEP 575 was that is was coupling
functionality (the calling and introspection protocol) with the class
hierarchy: a class could only benefit from the new features if it was a
subclass of base_function. It may be difficult for existing classes to
do that because they may have other constraints on the layout of the C
object structure, coming from an existing base class or implementation
details. For example, functools.lru_cache cannot implement PEP 575
as-is.

It also complicated the implementation precisely because changes were
needed both in the implementation details and in the class hierarchy.

The current PEP does not have these problems.

Why store the function pointer in the instance?

The actual information needed for calling an object is stored in the
instance (in the PyCCallDef structure) instead of the class. This is
different from the tp_call slot or earlier attempts at implementing a
tp_fastcall slot[2].

The main use case is built-in functions and methods. For those, the C
function to be called does depend on the instance.

Note that the current protocol makes it easy to support the case where
the same C function is called for all instances: just use a single
static PyCCallDef structure for every instance.

Why CCALL_OBJCLASS?

The flag CCALL_OBJCLASS is meant to support various cases where the
class of a self argument must be checked, such as:

    >>> list.append({}, None)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: append() requires a 'list' object but received a 'dict'

    >>> list.__len__({})
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: descriptor '__len__' requires a 'list' object but received a 'dict'

    >>> float.__dict__["fromhex"](list, "0xff")
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: descriptor 'fromhex' for type 'float' doesn't apply to type 'list'

In the reference implementation, only the first of these uses the new
code. The other examples show that these kind of checks appear in
multiple places, so it makes sense to add generic support for them.

Why CCALL_SELFARG?

The flag CCALL_SELFARG and the concept of self slicing are needed to
support methods: the C function should not care whether it is called as
unbound method or as bound method. In both cases, there should be a self
argument and this is simply the first positional argument of an unbound
method call.

For example, list.append is a METH_O method. Both the calls
list.append([], 42) and [].append(42) should translate to the C call
list_append([], 42).

Thanks to the proposed C call protocol, we can support this in such a
way that both the unbound and the bound method share a PyCCallDef
structure (with the CCALL_SELFARG flag set).

So, CCALL_SELFARG has two advantages: there is no extra layer of
indirection for calling methods and constructing bound methods does not
require setting up a PyCCallDef structure.

Another minor advantage is that we could make the error messages for a
wrong call signature more uniform between Python methods and built-in
methods. In the following example, Python is undecided whether a method
takes 1 or 2 arguments:

    >>> class List(list):
    ...     def myappend(self, item):
    ...         self.append(item)
    >>> List().myappend(1, 2)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: myappend() takes 2 positional arguments but 3 were given
    >>> List().append(1, 2)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: append() takes exactly one argument (2 given)

It is currently impossible for PyCFunction_Call to know the actual
number of user-visible arguments since it cannot distinguish at runtime
between a function (without self argument) and a bound method (with self
argument). The CCALL_SELFARG flag makes this difference explicit.

Why CCALL_DEFARG?

The flag CCALL_DEFARG gives the callee access to the PyCCallDef *. There
are various use cases for this:

1.  The callee can use the cc_parent field, which is useful for PEP 573.
2.  Applications are free to extend the PyCCallDef structure with
    user-defined fields, which can then be accessed analogously.
3.  In the case where the PyCCallDef structure is part of the object
    structure (this is true for example for PyCFunctionObject), an
    appropriate offset can be subtracted from the PyCCallDef pointer to
    get a pointer to the callable object defining that PyCCallDef.

An earlier version of this PEP defined a flag CCALL_FUNCARG instead of
CCALL_DEFARG which would pass the callable object to the callee. This
had similar use cases, but there was some ambiguity for bound methods:
should the "callable object" be the bound method object or the original
function wrapped by the method? By passing the PyCCallDef * instead,
this ambiguity is gone since the bound method uses the PyCCallDef * from
the wrapped function.

Replacing tp_print

We repurpose tp_print as tp_ccalloffset because this makes it easier for
external projects to backport the C call protocol to earlier Python
versions. In particular, the Cython project has shown interest in doing
that (see
https://mail.python.org/pipermail/python-dev/2018-June/153927.html).

Alternative suggestions

PEP 576 is an alternative approach to solving the same problem as this
PEP. See
https://mail.python.org/pipermail/python-dev/2018-July/154238.html for
comments on the difference between PEP 576 and PEP 580.

Discussion

Links to threads on the python-dev mailing list where this PEP has been
discussed:

-   https://mail.python.org/pipermail/python-dev/2018-June/153938.html
-   https://mail.python.org/pipermail/python-dev/2018-June/153984.html
-   https://mail.python.org/pipermail/python-dev/2018-July/154238.html
-   https://mail.python.org/pipermail/python-dev/2018-July/154470.html
-   https://mail.python.org/pipermail/python-dev/2018-July/154571.html
-   https://mail.python.org/pipermail/python-dev/2018-September/155166.html
-   https://mail.python.org/pipermail/python-dev/2018-October/155403.html
-   https://mail.python.org/pipermail/python-dev/2019-March/156853.html
-   https://mail.python.org/pipermail/python-dev/2019-March/156879.html

Reference implementation

The reference implementation can be found at
https://github.com/jdemeyer/cpython/tree/pep580

For an example of using the C call protocol, the following branch
implements functools.lru_cache using PEP 580:
https://github.com/jdemeyer/cpython/tree/lru580

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] Bad C API, https://pythoncapi.readthedocs.io/bad_api.html#bad-c-api

[2] Add tp_fastcall to PyTypeObject: support FASTCALL calling convention
for all callable objects, https://bugs.python.org/issue29259

license: Public Domain


text:
PEP: 583 Title: A Concurrency Memory Model for Python Version:
$Revision: 56116 $ Last-Modified: $Date: 2007-06-28 12:53:41 -0700 (Thu,
28 Jun 2007) $ Author: Jeffrey Yasskin<EMAIL_ADDRESS>Status:
Withdrawn Type: Informational Content-Type: text/x-rst Created:
22-Mar-2008 Post-History:

Abstract

This PEP describes how Python programs may behave in the presence of
concurrent reads and writes to shared variables from multiple threads.
We use a happens before relation to define when variable accesses are
ordered or concurrent. Nearly all programs should simply use locks to
guard their shared variables, and this PEP highlights some of the
strange things that can happen when they don't, but programmers often
assume that it's ok to do "simple" things without locking, and it's
somewhat unpythonic to let the language surprise them. Unfortunately,
avoiding surprise often conflicts with making Python run quickly, so
this PEP tries to find a good tradeoff between the two.

Rationale

So far, we have 4 major Python implementations -- CPython, Jython,
IronPython, and PyPy -- as well as lots of minor ones. Some of these
already run on platforms that do aggressive optimizations. In general,
these optimizations are invisible within a single thread of execution,
but they can be visible to other threads executing concurrently. CPython
currently uses a GIL to ensure that other threads see the results they
expect, but this limits it to a single processor. Jython and IronPython
run on Java's or .NET's threading system respectively, which allows them
to take advantage of more cores but can also show surprising values to
other threads.

So that threaded Python programs continue to be portable between
implementations, implementers and library authors need to agree on some
ground rules.

A couple definitions

Variable

    A name that refers to an object. Variables are generally introduced
    by assigning to them, and may be destroyed by passing them to del.
    Variables are fundamentally mutable, while objects may not be. There
    are several varieties of variables: module variables (often called
    "globals" when accessed from within the module), class variables,
    instance variables (also known as fields), and local variables. All
    of these can be shared between threads (the local variables if
    they're saved into a closure). The object in which the variables are
    scoped notionally has a dict whose keys are the variables' names.

Object

    A collection of instance variables (a.k.a. fields) and methods. At
    least, that'll do for this PEP.

Program Order

    The order that actions (reads and writes) happen within a thread,
    which is very similar to the order they appear in the text.

Conflicting actions

    Two actions on the same variable, at least one of which is a write.

Data race

    A situation in which two conflicting actions happen at the same
    time. "The same time" is defined by the memory model.

Two simple memory models

Before talking about the details of data races and the surprising
behaviors they produce, I'll present two simple memory models. The first
is probably too strong for Python, and the second is probably too weak.

Sequential Consistency

In a sequentially-consistent concurrent execution, actions appear to
happen in a global total order with each read of a particular variable
seeing the value written by the last write that affected that variable.
The total order for actions must be consistent with the program order. A
program has a data race on a given input when one of its sequentially
consistent executions puts two conflicting actions next to each other.

This is the easiest memory model for humans to understand, although it
doesn't eliminate all confusion, since operations can be split in odd
places.

Happens-before consistency

The program contains a collection of synchronization actions, which in
Python currently include lock acquires and releases and thread starts
and joins. Synchronization actions happen in a global total order that
is consistent with the program order (they don't have to happen in a
total order, but it simplifies the description of the model). A lock
release synchronizes with all later acquires of the same lock.
Similarly, given t = threading.Thread(target=worker):

-   A call to t.start() synchronizes with the first statement in
    worker().
-   The return from worker() synchronizes with the return from t.join().
-   If the return from t.start() happens before (see below) a call to
    t.isAlive() that returns False, the return from worker()
    synchronizes with that call.

We call the source of the synchronizes-with edge a release operation on
the relevant variable, and we call the target an acquire operation.

The happens before order is the transitive closure of the program order
with the synchronizes-with edges. That is, action A happens before
action B if:

-   A falls before B in the program order (which means they run in the
    same thread)
-   A synchronizes with B
-   You can get to B by following happens-before edges from A.

An execution of a program is happens-before consistent if each read R
sees the value of a write W to the same variable such that:

-   R does not happen before W, and
-   There is no other write V that overwrote W before R got a chance to
    see it. (That is, it can't be the case that W happens before V
    happens before R.)

You have a data race if two conflicting actions aren't related by
happens-before.

An example

Let's use the rules from the happens-before model to prove that the
following program prints "[7]":

    class Queue:
        def __init__(self):
            self.l = []
            self.cond = threading.Condition()

        def get():
            with self.cond:
                while not self.l:
                    self.cond.wait()
                ret = self.l[0]
                self.l = self.l[1:]
                return ret

        def put(x):
            with self.cond:
                self.l.append(x)
                self.cond.notify()

    myqueue = Queue()

    def worker1():
        x = [7]
        myqueue.put(x)

    def worker2():
        y = myqueue.get()
        print y

    thread1 = threading.Thread(target=worker1)
    thread2 = threading.Thread(target=worker2)
    thread2.start()
    thread1.start()

1.  Because myqueue is initialized in the main thread before thread1 or
    thread2 is started, that initialization happens before worker1 and
    worker2 begin running, so there's no way for either to raise a
    NameError, and both myqueue.l and myqueue.cond are set to their
    final objects.
2.  The initialization of x in worker1 happens before it calls
    myqueue.put(), which happens before it calls myqueue.l.append(x),
    which happens before the call to myqueue.cond.release(), all because
    they run in the same thread.
3.  In worker2, myqueue.cond will be released and re-acquired until
    myqueue.l contains a value (x). The call to myqueue.cond.release()
    in worker1 happens before that last call to myqueue.cond.acquire()
    in worker2.
4.  That last call to myqueue.cond.acquire() happens before
    myqueue.get() reads myqueue.l, which happens before myqueue.get()
    returns, which happens before print y, again all because they run in
    the same thread.
5.  Because happens-before is transitive, the list initially stored in x
    in thread1 is initialized before it is printed in thread2.

Usually, we wouldn't need to look all the way into a thread-safe queue's
implementation in order to prove that uses were safe. Its interface
would specify that puts happen before gets, and we'd reason directly
from that.

Surprising behaviors with races

Lots of strange things can happen when code has data races. It's easy to
avoid all of these problems by just protecting shared variables with
locks. This is not a complete list of race hazards; it's just a
collection that seem relevant to Python.

In all of these examples, variables starting with r are local variables,
and other variables are shared between threads.

Zombie values

This example comes from the Java memory model:

  Initially p is q and p.x == 0.

    Thread 1    Thread 2
    ----------- ----------
    r1 = p      r6 = p
    r2 = r1.x   r6.x = 3
    r3 = q
    r4 = r3.x
    r5 = r1.x

  Can produce r2 == r5 == 0 but r4 == 3, proving that p.x went from 0 to
  3 and back to 0.

A good compiler would like to optimize out the redundant load of p.x in
initializing r5 by just re-using the value already loaded into r2. We
get the strange result if thread 1 sees memory in this order:

    Evaluation   Computes   Why
    ------------ ---------- ----------------------------------------------
    r1 = p
    r2 = r1.x    r2 == 0
    r3 = q       r3 is p
    p.x = 3                 Side-effect of thread 2
    r4 = r3.x    r4 == 3
    r5 = r2      r5 == 0    Optimized from r5 = r1.x because r2 == r1.x.

Inconsistent Orderings

From N2177: Sequential Consistency for Atomics, and also known as
Independent Read of Independent Write (IRIW).

  Initially, a == b == 0.

    Thread 1   Thread 2   Thread 3   Thread 4
    ---------- ---------- ---------- ----------
    r1 = a     r3 = b     a = 1      b = 1
    r2 = b     r4 = a

  We may get r1 == r3 == 1 and r2 == r4 == 0, proving both that a was
  written before b (thread 1's data), and that b was written before a
  (thread 2's data). See Special Relativity for a real-world example.

This can happen if thread 1 and thread 3 are running on processors that
are close to each other, but far away from the processors that threads 2
and 4 are running on and the writes are not being transmitted all the
way across the machine before becoming visible to nearby threads.

Neither acquire/release semantics nor explicit memory barriers can help
with this. Making the orders consistent without locking requires
detailed knowledge of the architecture's memory model, but Java requires
it for volatiles so we could use documentation aimed at its
implementers.

A happens-before race that's not a sequentially-consistent race

From the POPL paper about the Java memory model [#JMM-popl].

  Initially, x == y == 0.

  +-------------+-------------+
  | Thread 1    | Thread 2    |
  +=============+=============+
  | r1 = x      | r2 = y      |
  +-------------+-------------+
  | if r1 != 0: | if r2 != 0: |
  +-------------+-------------+
  |   y = 42    |   x = 42    |
  +-------------+-------------+

  Can r1 == r2 == 42???

In a sequentially-consistent execution, there's no way to get an
adjacent read and write to the same variable, so the program should be
considered correctly synchronized (albeit fragile), and should only
produce r1 == r2 == 0. However, the following execution is
happens-before consistent:

  +-------------+-------+--------+
  | Statement   | Value | Thread |
  +=============+=======+========+
  | r1 = x      | 42    | 1      |
  +-------------+-------+--------+
  | if r1 != 0: | true  | 1      |
  +-------------+-------+--------+
  |   y = 42    |       | 1      |
  +-------------+-------+--------+
  | r2 = y      | 42    | 2      |
  +-------------+-------+--------+
  | if r2 != 0: | true  | 2      |
  +-------------+-------+--------+
  |   x = 42    |       | 2      |
  +-------------+-------+--------+

WTF, you are asking yourself. Because there were no inter-thread
happens-before edges in the original program, the read of x in thread 1
can see any of the writes from thread 2, even if they only happened
because the read saw them. There are data races in the happens-before
model.

We don't want to allow this, so the happens-before model isn't enough
for Python. One rule we could add to happens-before that would prevent
this execution is:

  If there are no data races in any sequentially-consistent execution of
  a program, the program should have sequentially consistent semantics.

Java gets this rule as a theorem, but Python may not want all of the
machinery you need to prove it.

Self-justifying values

Also from the POPL paper about the Java memory model [#JMM-popl].

  Initially, x == y == 0.

    Thread 1   Thread 2
    ---------- ----------
    r1 = x     r2 = y
    y = r1     x = r2

  Can x == y == 42???

In a sequentially consistent execution, no. In a happens-before
consistent execution, yes: The read of x in thread 1 is allowed to see
the value written in thread 2 because there are no happens-before
relations between the threads. This could happen if the compiler or
processor transforms the code into:

  +--------------+----------+
  | Thread 1     | Thread 2 |
  +==============+==========+
  | y = 42       | r2 = y   |
  +--------------+----------+
  | r1 = x       | x = r2   |
  +--------------+----------+
  | if r1 != 42: |          |
  +--------------+----------+
  |   y = r1     |          |
  +--------------+----------+

It can produce a security hole if the speculated value is a secret
object, or points to the memory that an object used to occupy. Java
cares a lot about such security holes, but Python may not.

Uninitialized values (direct)

From several classic double-checked locking examples.

  Initially, d == None.

    Thread 1            Thread 2
    ------------------- ------------
    while not d: pass   d = [3, 4]
    assert d[1] == 4

  This could raise an IndexError, fail the assertion, or, without some
  care in the implementation, cause a crash or other undefined behavior.

Thread 2 may actually be implemented as:

    r1 = list()
    r1.append(3)
    r1.append(4)
    d = r1

Because the assignment to d and the item assignments are independent,
the compiler and processor may optimize that to:

    r1 = list()
    d = r1
    r1.append(3)
    r1.append(4)

Which is obviously incorrect and explains the IndexError. If we then
look deeper into the implementation of r1.append(3), we may find that it
and d[1] cannot run concurrently without causing their own race
conditions. In CPython (without the GIL), those race conditions would
produce undefined behavior.

There's also a subtle issue on the reading side that can cause the value
of d[1] to be out of date. Somewhere in the implementation of list, it
stores its contents as an array in memory. This array may happen to be
in thread 1's cache. If thread 1's processor reloads d from main memory
without reloading the memory that ought to contain the values 3 and 4,
it could see stale values instead. As far as I know, this can only
actually happen on Alphas and maybe Itaniums, and we probably have to
prevent it anyway to avoid crashes.

Uninitialized values (flag)

From several more double-checked locking examples.

  Initially, d == dict() and initialized == False.

    Thread 1                      Thread 2
    ----------------------------- --------------------
    while not initialized: pass   d['a'] = 3
    r1 = d['a']                   initialized = True
    r2 = r1 == 3
    assert r2

  This could raise a KeyError, fail the assertion, or, without some care
  in the implementation, cause a crash or other undefined behavior.

Because d and initialized are independent (except in the programmer's
mind), the compiler and processor can rearrange these almost
arbitrarily, except that thread 1's assertion has to stay after the
loop.

Inconsistent guarantees from relying on data dependencies

This is a problem with Java final variables and the proposed
data-dependency ordering in C++0x.

  First execute:

      g = []
      def Init():
          g.extend([1,2,3])
          return [1,2,3]
      h = None

  Then in two threads:

    Thread 1              Thread 2
    --------------------- -------------
    while not h: pass     r1 = Init()
    assert h == [1,2,3]   freeze(r1)
    assert h == g         h = r1

  If h has semantics similar to a Java final variable (except for being
  write-once), then even though the first assertion is guaranteed to
  succeed, the second could fail.

Data-dependent guarantees like those final provides only work if the
access is through the final variable. It's not even safe to access the
same object through a different route. Unfortunately, because of how
processors work, final's guarantees are only cheap when they're weak.

The rules for Python

The first rule is that Python interpreters can't crash due to race
conditions in user code. For CPython, this means that race conditions
can't make it down into C. For Jython, it means that
NullPointerExceptions can't escape the interpreter.

Presumably we also want a model at least as strong as happens-before
consistency because it lets us write a simple description of how
concurrent queues and thread launching and joining work.

Other rules are more debatable, so I'll present each one with pros and
cons.

Data-race-free programs are sequentially consistent

We'd like programmers to be able to reason about their programs as if
they were sequentially consistent. Since it's hard to tell whether
you've written a happens-before race, we only want to require
programmers to prevent sequential races. The Java model does this
through a complicated definition of causality, but if we don't want to
include that, we can just assert this property directly.

No security holes from out-of-thin-air reads

If the program produces a self-justifying value, it could expose access
to an object that the user would rather the program not see. Again,
Java's model handles this with the causality definition. We might be
able to prevent these security problems by banning speculative writes to
shared variables, but I don't have a proof of that, and Python may not
need those security guarantees anyway.

Restrict reorderings instead of defining happens-before

The .NET [#CLR-msdn] and x86 [#x86-model] memory models are based on
defining which reorderings compilers may allow. I think that it's easier
to program to a happens-before model than to reason about all of the
possible reorderings of a program, and it's easier to insert enough
happens-before edges to make a program correct, than to insert enough
memory fences to do the same thing. So, although we could layer some
reordering restrictions on top of the happens-before base, I don't think
Python's memory model should be entirely reordering restrictions.

Atomic, unordered assignments

Assignments of primitive types are already atomic. If you assign
3<<72 + 5 to a variable, no thread can see only part of the value.
Jeremy Manson suggested that we extend this to all objects. This allows
compilers to reorder operations to optimize them, without allowing some
of the more confusing uninitialized values. The basic idea here is that
when you assign a shared variable, readers can't see any changes made to
the new value before the assignment, or to the old value after the
assignment. So, if we have a program like:

  Initially, (d.a, d.b) == (1, 2), and (e.c, e.d) == (3, 4). We also
  have class Obj(object): pass.

    Thread 1     Thread 2
    ------------ ---------------------
    r1 = Obj()   r3 = d
    r1.a = 3     r4, r5 = r3.a, r3.b
    r1.b = 4     r6 = e
    d = r1       r7, r8 = r6.c, r6.d
    r2 = Obj()
    r2.c = 6
    r2.d = 7
    e = r2

  (r4, r5) can be (1, 2) or (3, 4) but nothing else, and (r7, r8) can be
  either (3, 4) or (6, 7) but nothing else. Unlike if writes were
  releases and reads were acquires, it's legal for thread 2 to see
  (e.c, e.d) == (6, 7) and (d.a, d.b) == (1, 2) (out of order).

This allows the compiler a lot of flexibility to optimize without
allowing users to see some strange values. However, because it relies on
data dependencies, it introduces some surprises of its own. For example,
the compiler could freely optimize the above example to:

    Thread 1     Thread 2
    ------------ ---------------------
    r1 = Obj()   r3 = d
    r2 = Obj()   r6 = e
    r1.a = 3     r4, r7 = r3.a, r6.c
    r2.c = 6     r5, r8 = r3.b, r6.d
    r2.d = 7
    e = r2
    r1.b = 4
    d = r1

As long as it didn't let the initialization of e move above any of the
initializations of members of r2, and similarly for d and r1.

This also helps to ground happens-before consistency. To see the
problem, imagine that the user unsafely publishes a reference to an
object as soon as she gets it. The model needs to constrain what values
can be read through that reference. Java says that every field is
initialized to 0 before anyone sees the object for the first time, but
Python would have trouble defining "every field". If instead we say that
assignments to shared variables have to see a value at least as up to
date as when the assignment happened, then we don't run into any trouble
with early publication.

Two tiers of guarantees

Most other languages with any guarantees for unlocked variables
distinguish between ordinary variables and volatile/atomic variables.
They provide many more guarantees for the volatile ones. Python can't
easily do this because we don't declare variables. This may or may not
matter, since python locks aren't significantly more expensive than
ordinary python code. If we want to get those tiers back, we could:

1.  Introduce a set of atomic types similar to Java's[1] or C++'s[2].
    Unfortunately, we couldn't assign to them with =.
2.  Without requiring variable declarations, we could also specify that
    all of the fields on a given object are atomic.
3.  Extend the __slots__ mechanism[3] with a parallel __volatiles__
    list, and maybe a __finals__ list.

Sequential Consistency

We could just adopt sequential consistency for Python. This avoids all
of the hazards mentioned above, but it prohibits lots of optimizations
too. As far as I know, this is the current model of CPython, but if
CPython learned to optimize out some variable reads, it would lose this
property.

If we adopt this, Jython's dict implementation may no longer be able to
use ConcurrentHashMap because that only promises to create appropriate
happens-before edges, not to be sequentially consistent (although maybe
the fact that Java volatiles are totally ordered carries over). Both
Jython and IronPython would probably need to use AtomicReferenceArray or
the equivalent for any __slots__ arrays.

Adapt the x86 model

The x86 model is:

1.  Loads are not reordered with other loads.
2.  Stores are not reordered with other stores.
3.  Stores are not reordered with older loads.
4.  Loads may be reordered with older stores to different locations but
    not with older stores to the same location.
5.  In a multiprocessor system, memory ordering obeys causality (memory
    ordering respects transitive visibility).
6.  In a multiprocessor system, stores to the same location have a total
    order.
7.  In a multiprocessor system, locked instructions have a total order.
8.  Loads and stores are not reordered with locked instructions.

In acquire/release terminology, this appears to say that every store is
a release and every load is an acquire. This is slightly weaker than
sequential consistency, in that it allows inconsistent orderings, but it
disallows zombie values and the compiler optimizations that produce
them. We would probably want to weaken the model somehow to explicitly
allow compilers to eliminate redundant variable reads. The x86 model may
also be expensive to implement on other platforms, although because x86
is so common, that may not matter much.

Upgrading or downgrading to an alternate model

We can adopt an initial memory model without totally restricting future
implementations. If we start with a weak model and want to get stronger
later, we would only have to change the implementations, not programs.
Individual implementations could also guarantee a stronger memory model
than the language demands, although that could hurt interoperability. On
the other hand, if we start with a strong model and want to weaken it
later, we can add a from __future__ import weak_memory statement to
declare that some modules are safe.

Implementation Details

The required model is weaker than any particular implementation. This
section tries to document the actual guarantees each implementation
provides, and should be updated as the implementations change.

CPython

Uses the GIL to guarantee that other threads don't see funny
reorderings, and does few enough optimizations that I believe it's
actually sequentially consistent at the bytecode level. Threads can
switch between any two bytecodes (instead of only between statements),
so two threads that concurrently execute:

    i = i + 1

with i initially 0 could easily end up with i==1 instead of the expected
i==2. If they execute:

    i += 1

instead, CPython 2.6 will always give the right answer, but it's easy to
imagine another implementation in which this statement won't be atomic.

PyPy

Also uses a GIL, but probably does enough optimization to violate
sequential consistency. I know very little about this implementation.

Jython

Provides true concurrency under the Java memory model and stores all
object fields (except for those in __slots__?) in a ConcurrentHashMap,
which provides fairly strong ordering guarantees. Local variables in a
function may have fewer guarantees, which would become visible if they
were captured into a closure that was then passed to another thread.

IronPython

Provides true concurrency under the CLR memory model, which probably
protects it from uninitialized values. IronPython uses a locked map to
store object fields, providing at least as many guarantees as Jython.

References

Acknowledgements

Thanks to Jeremy Manson and Alex Martelli for detailed discussions on
what this PEP should look like.

Copyright

This document has been placed in the public domain.

[1] Package java.util.concurrent.atomic
(http://java.sun.com/javase/6/docs/api/java/util/concurrent/atomic/package-summary.html)

[2] C++ Atomic Types and Operations, Hans Boehm and Lawrence Crowl
(http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2427.html)

[3] __slots__ (http://docs.python.org/ref/slots.html)