text:
PEP: 481 Title: Migrate CPython to Git, Github, and Phabricator Author:
Donald Stufft<EMAIL_ADDRESS>Status: Withdrawn Type: Process
Content-Type: text/x-rst Created: 29-Nov-2014 Post-History: 29-Nov-2014

Abstract

Note

This PEP has been withdrawn, if you're looking for the PEP documenting
the move to Github, please refer to PEP 512.

This PEP proposes migrating the repository hosting of CPython and the
supporting repositories to Git and Github. It also proposes adding
Phabricator as an alternative to Github Pull Requests to handle
reviewing changes. This particular PEP is offered as an alternative to
PEP 474 and PEP 462 which aims to achieve the same overall benefits but
restricts itself to tools that support Mercurial and are completely Open
Source.

Rationale

CPython is an open source project which relies on a number of volunteers
donating their time. As an open source project it relies on attracting
new volunteers as well as retaining existing ones in order to continue
to have a healthy amount of manpower available. In addition to
increasing the amount of manpower that is available to the project, it
also needs to allow for effective use of what manpower is available.

The current toolchain of the CPython project is a custom and unique
combination of tools which mandates a workflow that is similar to one
found in a lot of older projects, but which is becoming less and less
popular as time goes on.

The one-off nature of the CPython toolchain and workflow means that any
new contributor is going to need spend time learning the tools and
workflow before they can start contributing to CPython. Once a new
contributor goes through the process of learning the CPython workflow
they also are unlikely to be able to take that knowledge and apply it to
future projects they wish to contribute to. This acts as a barrier to
contribution which will scare off potential new contributors.

In addition the tooling that CPython uses is under-maintained,
antiquated, and it lacks important features that enable committers to
more effectively use their time when reviewing and approving changes.
The fact that it is under-maintained means that bugs are likely to last
for longer, if they ever get fixed, as well as it's more likely to go
down for extended periods of time. The fact that it is antiquated means
that it doesn't effectively harness the capabilities of the modern web
platform. Finally the fact that it lacks several important features such
as a lack of pre-testing commits and the lack of an automatic merge tool
means that committers have to do needless busy work to commit even the
simplest of changes.

Version Control System

The first decision that needs to be made is the VCS of the primary
server side repository. Currently the CPython repository, as well as a
number of supporting repositories, uses Mercurial. When evaluating the
VCS we must consider the capabilities of the VCS itself as well as the
network effect and mindshare of the community around that VCS.

There are really only two real options for this, Mercurial and Git.
Between the two of them the technical capabilities are largely
equivalent. For this reason this PEP will largely ignore the technical
arguments about the VCS system and will instead focus on the social
aspects.

It is not possible to get exact numbers for the number of projects or
people which are using a particular VCS, however we can infer this by
looking at several sources of information for what VCS projects are
using.

The Open Hub (previously Ohloh) statistics[1] show that 37% of the
repositories indexed by The Open Hub are using Git (second only to SVN
which has 48%) while Mercurial has just 2% (beating only bazaar which
has 1%). This has Git being just over 18 times as popular as Mercurial
on The Open Hub.

Another source of information on the popular of the difference VCSs is
PyPI itself. This source is more targeted at the Python community itself
since it represents projects developed for Python. Unfortunately PyPI
does not have a standard location for representing this information, so
this requires manual processing. If we limit our search to the top 100
projects on PyPI (ordered by download counts) we can see that 62% of
them use Git while 22% of them use Mercurial while 13% use something
else. This has Git being just under 3 times as popular as Mercurial for
the top 100 projects on PyPI.

Obviously from these numbers Git is by far the more popular DVCS for
open source projects and choosing the more popular VCS has a number of
positive benefits.

For new contributors it increases the likelihood that they will have
already learned the basics of Git as part of working with another
project or if they are just now learning Git, that they'll be able to
take that knowledge and apply it to other projects. Additionally a
larger community means more people writing how to guides, answering
questions, and writing articles about Git which makes it easier for a
new user to find answers and information about the tool they are trying
to learn.

Another benefit is that by nature of having a larger community, there
will be more tooling written around it. This increases options for
everything from GUI clients, helper scripts, repository hosting, etc.

Repository Hosting

This PEP proposes allowing GitHub Pull Requests to be submitted, however
GitHub does not have a way to submit Pull Requests against a repository
that is not hosted on GitHub. This PEP also proposes that in addition to
GitHub Pull Requests Phabricator's Differential app can also be used to
submit proposed changes and Phabricator does allow submitting changes
against a repository that is not hosted on Phabricator.

For this reason this PEP proposes using GitHub as the canonical location
of the repository with a read-only mirror located in Phabricator. If at
some point in the future GitHub is no longer desired, then repository
hosting can easily be moved to solely in Phabricator and the ability to
accept GitHub Pull Requests dropped.

In addition to hosting the repositories on Github, a read only copy of
all repositories will also be mirrored onto the PSF Infrastructure.

Code Review

Currently CPython uses a custom fork of Rietveld which has been modified
to not run on Google App Engine which is really only able to be
maintained currently by one person. In addition it is missing out on
features that are present in many modern code review tools.

This PEP proposes allowing both Github Pull Requests and Phabricator
changes to propose changes and review code. It suggests both so that
contributors can select which tool best enables them to submit changes,
and reviewers can focus on reviewing changes in the tooling they like
best.

GitHub Pull Requests

GitHub is a very popular code hosting site and is increasingly becoming
the primary place people look to contribute to a project. Enabling users
to contribute through GitHub is enabling contributors to contribute
using tooling that they are likely already familiar with and if they are
not they are likely to be able to apply to another project.

GitHub Pull Requests have a fairly major advantage over the older
"submit a patch to a bug tracker" model. It allows developers to work
completely within their VCS using standard VCS tooling so it does not
require creating a patch file and figuring out what the right location
is to upload it to. This lowers the barrier for sending a change to be
reviewed.

On the reviewing side, GitHub Pull Requests are far easier to review,
they have nice syntax highlighted diffs which can operate in either
unified or side by side views. They allow expanding the context on a
diff up to and including the entire file. Finally they allow commenting
inline and on the pull request as a whole and they present that in a
nice unified way which will also hide comments which no longer apply.
Github also provides a "rendered diff" view which enables easily viewing
a diff of rendered markup (such as rst) instead of needing to review the
diff of the raw markup.

The Pull Request work flow also makes it trivial to enable the ability
to pre-test a change before actually merging it. Any particular pull
request can have any number of different types of "commit statuses"
applied to it, marking the commit (and thus the pull request) as either
in a pending, successful, errored, or failure state. This makes it easy
to see inline if the pull request is passing all of the tests, if the
contributor has signed a CLA, etc.

Actually merging a Github Pull Request is quite simple, a core reviewer
simply needs to press the "Merge" button once the status of all the
checks on the Pull Request are green for successful.

GitHub also has a good workflow for submitting pull requests to a
project completely through their web interface. This would enable the
Python documentation to have "Edit on GitHub" buttons on every page and
people who discover things like typos, inaccuracies, or just want to
make improvements to the docs they are currently writing can simply hit
that button and get an in browser editor that will let them make changes
and submit a pull request all from the comfort of their browser.

Phabricator

In addition to GitHub Pull Requests this PEP also proposes setting up a
Phabricator instance and pointing it at the GitHub hosted repositories.
This will allow utilizing the Phabricator review applications of
Differential and Audit.

Differential functions similarly to GitHub pull requests except that
they require installing the arc command line tool to upload patches to
Phabricator.

Whether to enable Phabricator for any particular repository can be
chosen on a case-by-case basis, this PEP only proposes that it must be
enabled for the CPython repository, however for smaller repositories
such as the PEP repository it may not be worth the effort.

Criticism

X is not written in Python

One feature that the current tooling (Mercurial, Rietveld) has is that
the primary language for all of the pieces are written in Python. It is
this PEPs belief that we should focus on the best tools for the job and
not the best tools that happen to be written in Python. Volunteer time
is a precious resource to any open source project and we can best
respect and utilize that time by focusing on the benefits and downsides
of the tools themselves rather than what language their authors happened
to write them in.

One concern is the ability to modify tools to work for us, however one
of the Goals here is to not modify software to work for us and instead
adapt ourselves to a more standard workflow. This standardization pays
off in the ability to re-use tools out of the box freeing up developer
time to actually work on Python itself as well as enabling knowledge
sharing between projects.

However, if we do need to modify the tooling, Git itself is largely
written in C the same as CPython itself is. It can also have commands
written for it using any language, including Python. Phabricator is
written in PHP which is a fairly common language in the web world and
fairly easy to pick up. GitHub itself is largely written in Ruby but
given that it's not Open Source there is no ability to modify it so it's
implementation language is completely meaningless.

GitHub is not Free/Open Source

GitHub is a big part of this proposal and someone who tends more to
ideology rather than practicality may be opposed to this PEP on that
grounds alone. It is this PEPs belief that while using entirely
Free/Open Source software is an attractive idea and a noble goal, that
valuing the time of the contributors by giving them good tooling that is
well maintained and that they either already know or if they learn it
they can apply to other projects is a more important concern than
treating whether something is Free/Open Source is a hard requirement.

However, history has shown us that sometimes benevolent proprietary
companies can stop being benevolent. This is hedged against in a few
ways:

-   We are not utilizing the GitHub Issue Tracker, both because it is
    not powerful enough for CPython but also because for the primary
    CPython repository the ability to take our issues and put them
    somewhere else if we ever need to leave GitHub relies on GitHub
    continuing to allow API access.
-   We are utilizing the GitHub Pull Request workflow, however all of
    those changes live inside of Git. So a mirror of the GitHub
    repositories can easily contain all of those Pull Requests. We would
    potentially lose any comments if GitHub suddenly turned "evil", but
    the changes themselves would still exist.
-   We are utilizing the GitHub repository hosting feature, however
    since this is just git moving away from GitHub is as simple as
    pushing the repository to a different location. Data portability for
    the repository itself is extremely high.
-   We are also utilizing Phabricator to provide an alternative for
    people who do not wish to use GitHub. This also acts as a fallback
    option which will already be in place if we ever need to stop using
    GitHub.

Relying on GitHub comes with a number of benefits beyond just the
benefits of the platform itself. Since it is a commercially backed
venture it has a full-time staff responsible for maintaining its
services. This includes making sure they stay up, making sure they stay
patched for various security vulnerabilities, and further improving the
software and infrastructure as time goes on.

Mercurial is better than Git

Whether Mercurial or Git is better on a technical level is a highly
subjective opinion. This PEP does not state whether the mechanics of Git
or Mercurial is better and instead focuses on the network effect that is
available for either option. Since this PEP proposes switching to Git
this leaves the people who prefer Mercurial out, however those users can
easily continue to work with Mercurial by using the hg-git[2] extension
for Mercurial which will let it work with a repository which is Git on
the serverside.

CPython Workflow is too Complicated

One sentiment that came out of previous discussions was that the multi
branch model of CPython was too complicated for Github Pull Requests. It
is the belief of this PEP that statement is not accurate.

Currently any particular change requires manually creating a patch for
2.7 and 3.x which won't change at all in this regards.

If someone submits a fix for the current stable branch (currently 3.4)
the GitHub Pull Request workflow can be used to create, in the browser,
a Pull Request to merge the current stable branch into the master branch
(assuming there is no merge conflicts). If there is a merge conflict
that would need to be handled locally. This provides an improvement over
the current situation where the merge must always happen locally.

Finally if someone submits a fix for the current development branch
currently then this has to be manually applied to the stable branch if
it desired to include it there as well. This must also happen locally as
well in the new workflow, however for minor changes it could easily be
accomplished in the GitHub web editor.

Looking at this, I do not believe that any system can hide the
complexities involved in maintaining several long running branches. The
only thing that the tooling can do is make it as easy as possible to
submit changes.

Example: Scientific Python

One of the key ideas behind the move to both git and Github is that a
feature of a DVCS, the repository hosting, and the workflow used is the
social network and size of the community using said tools. We can see
this is true by looking at an example from a sub-community of the Python
community: The Scientific Python community. They have already migrated
most of the key pieces of the SciPy stack onto Github using the Pull
Request-based workflow. This process started with IPython, and as more
projects moved over it became a natural default for new projects in the
community.

They claim to have seen a great benefit from this move, in that it
enables casual contributors to easily move between different projects
within their sub-community without having to learn a special, bespoke
workflow and a different toolchain for each project. They've found that
when people can use their limited time on actually contributing instead
of learning the different tools and workflows, not only do they
contribute more to one project, but that they also expand out and
contribute to other projects. This move has also been attributed to the
increased tendency for members of that community to go so far as
publishing their research and educational materials on Github as well.

This example showcases the real power behind moving to a highly popular
toolchain and workflow, as each variance introduces yet another hurdle
for new and casual contributors to get past and it makes the time spent
learning that workflow less reusable with other projects.

References

Copyright

This document has been placed in the public domain.

[1] Open Hub Statistics

[2] Hg-Git mercurial plugin


text:
PEP: 701 Title: Syntactic formalization of f-strings Author: Pablo
Galindo<EMAIL_ADDRESS>Batuhan Taskaya<EMAIL_ADDRESS>Lysandros Nikolaou<EMAIL_ADDRESS>Marta Gómez Macías
<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pep-701-syntactic-formalization-of-f-strings/22046
Status: Accepted Type: Standards Track Content-Type: text/x-rst Created:
15-Nov-2022 Python-Version: 3.12 Post-History: 19-Dec-2022, Resolution:
14-Mar-2023

Abstract

This document proposes to lift some of the restrictions originally
formulated in PEP 498 and to provide a formalized grammar for f-strings
that can be integrated into the parser directly. The proposed syntactic
formalization of f-strings will have some small side-effects on how
f-strings are parsed and interpreted, allowing for a considerable number
of advantages for end users and library developers, while also
dramatically reducing the maintenance cost of the code dedicated to
parsing f-strings.

Motivation

When f-strings were originally introduced in PEP 498, the specification
was provided without providing a formal grammar for f-strings.
Additionally, the specification contains several restrictions that are
imposed so the parsing of f-strings could be implemented into CPython
without modifying the existing lexer. These limitations have been
recognized previously and previous attempts have been made to lift them
in PEP 536, but none of this work was ever implemented. Some of these
limitations (collected originally by PEP 536) are:

1.  It is impossible to use the quote character delimiting the f-string
    within the expression portion:

        >>> f'Magic wand: { bag['wand'] }'
                                     ^
        SyntaxError: invalid syntax

2.  A previously considered way around it would lead to escape sequences
    in executed code and is prohibited in f-strings:

        >>> f'Magic wand { bag[\'wand\'] } string'
        SyntaxError: f-string expression portion cannot include a backslash

3.  Comments are forbidden even in multi-line f-strings:

        >>> f'''A complex trick: {
        ... bag['bag']  # recursive bags!
        ... }'''
        SyntaxError: f-string expression part cannot include '#'

4.  Arbitrary nesting of expressions without expansion of escape
    sequences is available in many other languages that employ a string
    interpolation method that uses expressions instead of just variable
    names. Some examples:

        # Ruby
        "#{ "#{1+2}" }"

        # JavaScript
        `${`${1+2}`}`

        # Swift
        "\("\(1+2)")"

        # C#
        $"{$"{1+2}"}"

These limitations serve no purpose from a language user perspective and
can be lifted by giving f-string literals a regular grammar without
exceptions and implementing it using dedicated parse code.

The other issue that f-strings have is that the current implementation
in CPython relies on tokenising f-strings as STRING tokens and a post
processing of these tokens. This has the following problems:

1.  It adds a considerable maintenance cost to the CPython parser. This
    is because the parsing code needs to be written by hand, which has
    historically led to a considerable number of inconsistencies and
    bugs. Writing and maintaining parsing code by hand in C has always
    been considered error prone and dangerous as it needs to deal with a
    lot of manual memory management over the original lexer buffers.
2.  The f-string parsing code is not able to use the new improved error
    message mechanisms that the new PEG parser, originally introduced in
    PEP 617, has allowed. The improvements that these error messages
    brought has been greatly celebrated but unfortunately f-strings
    cannot benefit from them because they are parsed in a separate piece
    of the parsing machinery. This is especially unfortunate, since
    there are several syntactical features of f-strings that can be
    confusing due to the different implicit tokenization that happens
    inside the expression part (for instance f"{y:=3}" is not an
    assignment expression).
3.  Other Python implementations have no way to know if they have
    implemented f-strings correctly because contrary to other language
    features, they are not part of the
    official Python grammar <python:f-strings>. This is important
    because several prominent alternative implementations are using
    CPython's PEG parser, such as PyPy, and/or are basing their grammars
    on the official PEG grammar. The fact that f-strings use a separate
    parser prevents these alternative implementations from leveraging
    the official grammar and benefiting from improvements in error
    messages derived from the grammar.

A version of this proposal was originally discussed on Python-Dev and
presented at the Python Language Summit 2022 where it was
enthusiastically received.

Rationale

By building on top of the new Python PEG Parser (PEP 617), this PEP
proposes to redefine “f-strings”, especially emphasizing the clear
separation of the string component and the expression (or replacement,
{...}) component. PEP 498 summarizes the syntactical part of “f-strings”
as the following:

  In Python source code, an f-string is a literal string, prefixed with
  ‘f’, which contains expressions inside braces. The expressions are
  replaced with their values.

However, PEP 498 also contained a formal list of exclusions on what can
or cannot be contained inside the expression component (primarily due to
the limitations of the existing parser). By clearly establishing the
formal grammar, we now also have the ability to define the expression
component of an f-string as truly "any applicable Python expression" (in
that particular context) without being bound by the limitations imposed
by the details of our implementation.

The formalization effort and the premise above also has a significant
benefit for Python programmers due to its ability to simplify and
eliminate the obscure limitations. This reduces the mental burden and
the cognitive complexity of f-string literals (as well as the Python
language in general).

1.  The expression component can include any string literal that a
    normal Python expression can include. This opens up the possibility
    of nesting string literals (formatted or not) inside the expression
    component of an f-string with the same quote type (and length):

        >>> f"These are the things: {", ".join(things)}"

        >>> f"{source.removesuffix(".py")}.c: $(srcdir)/{source}"

        >>> f"{f"{f"infinite"}"}" + " " + f"{f"nesting!!!"}"

    This "feature" is not universally agreed to be desirable, and some
    users find this unreadable. For a discussion on the different views
    on this, see the considerations regarding quote reuse section.

2.  Another issue that has felt unintuitive to most is the lack of
    support for backslashes within the expression component of an
    f-string. One example that keeps coming up is including a newline
    character in the expression part for joining containers. For
    example:

        >>> a = ["hello", "world"]
        >>> f"{'\n'.join(a)}"
        File "<stdin>", line 1
            f"{'\n'.join(a)}"
                            ^
        SyntaxError: f-string expression part cannot include a backslash

    A common work-around for this was to either assign the newline to an
    intermediate variable or pre-create the whole string prior to
    creating the f-string:

        >>> a = ["hello", "world"]
        >>> joined = '\n'.join(a)
        >>> f"{joined}"
        'hello\nworld'

    It only feels natural to allow backslashes in the expression part
    now that the new PEG parser can easily support it.

      >>> a = ["hello", "world"] >>> f"{'n'.join(a)}" 'hellonworld'

3.  Before the changes proposed in this document, there was no explicit
    limit in how f-strings can be nested, but the fact that string
    quotes cannot be reused inside the expression component of f-strings
    made it impossible to nest f-strings arbitrarily. In fact, this is
    the most nested-fstring that can be written:

        >>> f"""{f'''{f'{f"{1+1}"}'}'''}"""
        '2'

    As this PEP allows placing any valid Python expression inside the
    expression component of the f-strings, it is now possible to reuse
    quotes and therefore is possible to nest f-strings arbitrarily:

        >>> f"{f"{f"{f"{f"{f"{1+1}"}"}"}"}"}"
        '2'

    Although this is just a consequence of allowing arbitrary
    expressions, the authors of this PEP do not believe that this is a
    fundamental benefit and we have decided that the language
    specification will not explicitly mandate that this nesting can be
    arbitrary. This is because allowing arbitrarily-deep nesting imposes
    a lot of extra complexity to the lexer implementation (particularly
    as lexer/parser pipelines need to allow "untokenizing" to support
    the 'f-string debugging expressions' and this is especially taxing
    when arbitrary nesting is allowed). Implementations are therefore
    free to impose a limit on the nesting depth if they need to. Note
    that this is not an uncommon situation, as the CPython
    implementation already imposes several limits all over the place,
    including a limit on the nesting depth of parentheses and brackets,
    a limit on the nesting of the blocks, a limit in the number of
    branches in if statements, a limit on the number of expressions in
    star-unpacking, etc.

Specification

The formal proposed PEG grammar specification for f-strings is (see PEP
617 for details on the syntax):

    fstring
        | FSTRING_START fstring_middle* FSTRING_END
    fstring_middle
        | fstring_replacement_field
        | FSTRING_MIDDLE
    fstring_replacement_field
        | '{' (yield_expr | star_expressions) "="? [ "!" NAME ] [ ':' fstring_format_spec* ] '}'
    fstring_format_spec:
        | FSTRING_MIDDLE
        | fstring_replacement_field

The new tokens (FSTRING_START, FSTRING_MIDDLE, FSTRING_END) are defined
later in this document.

This PEP leaves up to the implementation the level of f-string nesting
allowed (f-strings within the expression parts of other f-strings) but
specifies a lower bound of 5 levels of nesting. This is to ensure that
users can have a reasonable expectation of being able to nest f-strings
with "reasonable" depth. This PEP implies that limiting nesting is not
part of the language specification but also the language specification
doesn't mandate arbitrary nesting.

Similarly, this PEP leaves up to the implementation the level of
expression nesting in format specifiers but specifies a lower bound of 2
levels of nesting. This means that the following should always be valid:

    f"{'':*^{1:{1}}}"

but the following can be valid or not depending on the implementation:

    f"{'':*^{1:{1:{1}}}}"

The new grammar will preserve the Abstract Syntax Tree (AST) of the
current implementation. This means that no semantic changes will be
introduced by this PEP on existing code that uses f-strings.

Handling of f-string debug expressions

Since Python 3.8, f-strings can be used to debug expressions by using
the = operator. For example:

    >>> a = 1
    >>> f"{1+1=}"
    '1+1=2'

This semantics were not introduced formally in a PEP and they were
implemented in the current string parser as a special case in bpo-36817
and documented in the f-string lexical analysis section.

This feature is not affected by the changes proposed in this PEP but is
important to specify that the formal handling of this feature requires
the lexer to be able to "untokenize" the expression part of the
f-string. This is not a problem for the current string parser as it can
operate directly on the string token contents. However, incorporating
this feature into a given parser implementation requires the lexer to
keep track of the raw string contents of the expression part of the
f-string and make them available to the parser when the parse tree is
constructed for f-string nodes. A pure "untokenization" is not enough
because as specified currently, f-string debug expressions preserve
whitespace in the expression, including spaces after the { and the =
characters. This means that the raw string contents of the expression
part of the f-string must be kept intact and not just the associated
tokens.

How parser/lexer implementations deal with this problem is of course up
to the implementation.

New tokens

Three new tokens are introduced: FSTRING_START, FSTRING_MIDDLE and
FSTRING_END. Different lexers may have different implementations that
may be more efficient than the ones proposed here given the context of
the particular implementation. However, the following definitions will
be used as part of the public APIs of CPython (such as the tokenize
module) and are also provided as a reference so that the reader can have
a better understanding of the proposed grammar changes and how the
tokens are used:

-   FSTRING_START: This token includes the f-string prefix (f/F/fr) and
    the opening quote(s).
-   FSTRING_MIDDLE: This token includes a portion of text inside the
    string that's not part of the expression part and isn't an opening
    or closing brace. This can include the text between the opening
    quote and the first expression brace ({), the text between two
    expression braces (} and {) and the text between the last expression
    brace (}) and the closing quote.
-   FSTRING_END: This token includes the closing quote.

These tokens are always string parts and they are semantically
equivalent to the STRING token with the restrictions specified. These
tokens must be produced by the lexer when lexing f-strings. This means
that the tokenizer cannot produce a single token for f-strings anymore.
How the lexer emits this token is not specified as this will heavily
depend on every implementation (even the Python version of the lexer in
the standard library is implemented differently to the one used by the
PEG parser).

As an example:

    f'some words {a+b:.3f} more words {c+d=} final words'

will be tokenized as:

    FSTRING_START - "f'"
    FSTRING_MIDDLE - 'some words '
    LBRACE - '{'
    NAME - 'a'
    PLUS - '+'
    NAME - 'b'
    OP - ':'
    FSTRING_MIDDLE - '.3f'
    RBRACE - '}'
    FSTRING_MIDDLE - ' more words '
    LBRACE - '{'
    NAME - 'c'
    PLUS - '+'
    NAME - 'd'
    OP - '='
    RBRACE - '}'
    FSTRING_MIDDLE - ' final words'
    FSTRING_END - "'"

while f"""some words""" will be tokenized simply as:

    FSTRING_START - 'f"""'
    FSTRING_MIDDLE - 'some words'
    FSTRING_END - '"""'

Changes to the tokenize module

The tokenize module will be adapted to emit these tokens as described in
the previous section when parsing f-strings so tools can take advantage
of this new tokenization schema and avoid having to implement their own
f-string tokenizer and parser.

How to produce these new tokens

One way existing lexers can be adapted to emit these tokens is to
incorporate a stack of "lexer modes" or to use a stack of different
lexers. This is because the lexer needs to switch from "regular Python
lexing" to "f-string lexing" when it encounters an f-string start token
and as f-strings can be nested, the context needs to be preserved until
the f-string closes. Also, the "lexer mode" inside an f-string
expression part needs to behave as a "super-set" of the regular Python
lexer (as it needs to be able to switch back to f-string lexing when it
encounters the } terminator for the expression part as well as handling
f-string formatting and debug expressions). For reference, here is a
draft of the algorithm to modify a CPython-like tokenizer to emit these
new tokens:

1.  If the lexer detects that an f-string is starting (by detecting the
    letter 'f/F' and one of the possible quotes) keep advancing until a
    valid quote is detected (one of ", """, ' or ''') and emit a
    FSTRING_START token with the contents captured (the 'f/F' and the
    starting quote). Push a new tokenizer mode to the tokenizer mode
    stack for "F-string tokenization". Go to step 2.

2.  Keep consuming tokens until a one of the following is encountered:

    -   A closing quote equal to the opening quote.
    -   If in "format specifier mode" (see step 3), an opening brace
        ({), a closing brace (}), or a newline token (\n).
    -   If not in "format specifier mode" (see step 3), an opening brace
        ({) or a closing brace (}) that is not immediately followed by
        another opening/closing brace.

    In all cases, if the character buffer is not empty, emit a
    FSTRING_MIDDLE token with the contents captured so far but transform
    any double opening/closing braces into single opening/closing
    braces. Now, proceed as follows depending on the character
    encountered:

    -   If a closing quote matching the opening quite is encountered go
        to step 4.
    -   If an opening bracket (not immediately followed by another
        opening bracket) is encountered, go to step 3.
    -   If a closing bracket (not immediately followed by another
        closing bracket) is encountered, emit a token for the closing
        bracket and go to step 2.

3.  Push a new tokenizer mode to the tokenizer mode stack for "Regular
    Python tokenization within f-string" and proceed to tokenize with
    it. This mode tokenizes as the "Regular Python tokenization" until a
    : or a } character is encountered with the same level of nesting as
    the opening bracket token that was pushed when we enter the f-string
    part. Using this mode, emit tokens until one of the stop points are
    reached. When this happens, emit the corresponding token for the
    stopping character encountered and, pop the current tokenizer mode
    from the tokenizer mode stack and go to step 2. If the stopping
    point is a : character, enter step 2 in "format specifier" mode.

4.  Emit a FSTRING_END token with the contents captured and pop the
    current tokenizer mode (corresponding to "F-string tokenization")
    and go back to "Regular Python mode".

Of course, as mentioned before, it is not possible to provide a precise
specification of how this should be done for an arbitrary tokenizer as
it will depend on the specific implementation and nature of the lexer to
be changed.

Consequences of the new grammar

All restrictions mentioned in the PEP are lifted from f-string literals,
as explained below:

-   Expression portions may now contain strings delimited with the same
    kind of quote that is used to delimit the f-string literal.

-   Backslashes may now appear within expressions just like anywhere
    else in Python code. In case of strings nested within f-string
    literals, escape sequences are expanded when the innermost string is
    evaluated.

-   New lines are now allowed within expression brackets. This means
    that these are now allowed:

        >>> x = 1
        >>> f"___{
        ...     x
        ... }___"
        '___1___'

        >>> f"___{(
        ...     x
        ... )}___"
        '___1___'

-   Comments, using the # character, are allowed within the expression
    part of an f-string. Note that comments require that the closing
    bracket (}) of the expression part to be present in a different line
    as the one the comment is in or otherwise it will be ignored as part
    of the comment.

Considerations regarding quote reuse

One of the consequences of the grammar proposed here is that, as
mentioned above, f-string expressions can now contain strings delimited
with the same kind of quote that is used to delimit the external
f-string literal. For example:

  >>> f" something { my_dict["key"] } something else "

In the discussion thread for this PEP, several concerns have been raised
regarding this aspect and we want to collect them here, as these should
be taken into consideration when accepting or rejecting this PEP.

Some of these objections include:

-   Many people find quote reuse within the same string confusing and
    hard to read. This is because allowing quote reuse will violate a
    current property of Python as it stands today: the fact that strings
    are fully delimited by two consecutive pairs of the same kind of
    quote, which by itself is a very simple rule. One of the reasons
    quote reuse may be harder for humans to parse, leading to less
    readable code, is that the quote character is the same for both
    start and end (as opposed to other delimiters).
-   Some users have raised concerns that quote reuse may break some
    lexer and syntax highlighting tools that rely on simple mechanisms
    to detect strings and f-strings, such as regular expressions or
    simple delimiter matching tools. Introducing quote reuse in
    f-strings will either make it trickier to keep these tools working
    or will break the tools altogether (as, for instance, regular
    expressions cannot parse arbitrary nested structures with
    delimiters). The IDLE editor, included in the standard library, is
    an example of a tool which may need some work to correctly apply
    syntax highlighting to f-strings.

Here are some of the arguments in favour:

-   Many languages that allow similar syntactic constructs (normally
    called "string interpolation") allow quote reuse and arbitrary
    nesting. These languages include JavaScript, Ruby, C#, Bash, Swift
    and many others. The fact that many languages allow quote reuse can
    be a compelling argument in favour of allowing it in Python. This is
    because it will make the language more familiar to users coming from
    other languages.

-   As many other popular languages allow quote reuse in string
    interpolation constructs, this means that editors that support
    syntax highlighting for these languages will already have the
    necessary tools to support syntax highlighting for f-strings with
    quote reuse in Python. This means that although the files that
    handle syntax highlighting for Python will need to be updated to
    support this new feature, is not expected to be impossible or very
    hard to do.

-   One advantage of allowing quote reuse is that it composes cleanly
    with other syntax. Sometimes this is referred to as "referential
    transparency". An example of this is that if we have f(x+1),
    assuming a is a brand new variable, it should behave the same as
    a = x+1; f(a). And vice versa. So if we have:

        def py2c(source):
            prefix = source.removesuffix(".py")
            return f"{prefix}.c"

    It should be expected that if we replace the variable prefix with
    its definition, the answer should be the same:

        def py2c(source):
            return f"{source.removesuffix(".py")}.c"

-   Code generators (like ast.unparse from standard library) in their
    current form rely on complicated algorithms to ensure expressions
    within an f-string are properly suited for the context in which they
    are being used. These non-trivial algorithms come with challenges
    such as finding an unused quote type (by tracking the outer quotes),
    and generating string representations which would not include
    backslashes if possible. Allowing quote reuse and backslashes would
    simplify the code generators which deal with f-strings considerably,
    as the regular Python expression logic can be used inside and
    outside of f-strings without any special treatment.

-   Limiting quote reuse will considerably increase the complexity of
    the implementation of the proposed changes. This is because it will
    force the parser to have the context that is parsing an expression
    part of an f-string with a given quote in order to know if it needs
    to reject an expression that reuses the quote. Carrying this context
    around is not trivial in parsers that can backtrack arbitrarily
    (such as the PEG parser). The issue becomes even more complex if we
    consider that f-strings can be arbitrarily nested and therefore
    several quote types may need to be rejected.

    To gather feedback from the community, a poll has been initiated to
    get a sense of how the community feels about this aspect of the PEP.

Backwards Compatibility

This PEP does not introduce any backwards incompatible syntactic or
semantic changes to the Python language. However, the tokenize module (a
quasi-public part of the standard library) will need to be updated to
support the new f-string tokens (to allow tool authors to correctly
tokenize f-strings). See changes to the tokenize module for more details
regarding how the public API of tokenize will be affected.

How to Teach This

As the concept of f-strings is already ubiquitous in the Python
community, there is no fundamental need for users to learn anything new.
However, as the formalized grammar allows some new possibilities, it is
important that the formal grammar is added to the documentation and
explained in detail, explicitly mentioning what constructs are possible
since this PEP is aiming to avoid confusion.

It is also beneficial to provide users with a simple framework for
understanding what can be placed inside an f-string expression. In this
case the authors think that this work will make it even simpler to
explain this aspect of the language, since it can be summarized as:

  You can place any valid Python expression inside an f-string
  expression.

With the changes in this PEP, there is no need to clarify that string
quotes are limited to be different from the quotes of the enclosing
string, because this is now allowed: as an arbitrary Python string can
contain any possible choice of quotes, so can any f-string expression.
Additionally there is no need to clarify that certain things are not
allowed in the expression part because of implementation restrictions
such as comments, new line characters or backslashes.

The only "surprising" difference is that as f-strings allow specifying a
format, expressions that allow a : character at the top level still need
to be enclosed in parenthesis. This is not new to this work, but it is
important to emphasize that this restriction is still in place. This
allows for an easier modification of the summary:

  You can place any valid Python expression inside an f-string
  expression, and everything after a : character at the top level will
  be identified as a format specification.

Reference Implementation

A reference implementation can be found in the implementation fork.

Rejected Ideas

1.  Although we think the readability arguments that have been raised
    against allowing quote reuse in f-string expressions are valid and
    very important, we have decided to propose not rejecting quote reuse
    in f-strings at the parser level. The reason is that one of the
    cornerstones of this PEP is to reduce the complexity and maintenance
    of parsing f-strings in CPython and this will not only work against
    that goal, but it may even make the implementation even more complex
    than the current one. We believe that forbidding quote reuse should
    be done in linters and code style tools and not in the parser, the
    same way other confusing or hard-to-read constructs in the language
    are handled today.

2.  We have decided not to lift the restriction that some expression
    portions need to wrap ':' and '!' in parentheses at the top level,
    e.g.:

        >>> f'Useless use of lambdas: { lambda x: x*2 }'
        SyntaxError: unexpected EOF while parsing

    The reason is that this will introduce a considerable amount of
    complexity for no real benefit. This is due to the fact that the :
    character normally separates the f-string format specification. This
    format specification is currently tokenized as a string. As the
    tokenizer MUST tokenize what's on the right of the : as either a
    string or a stream of tokens, this won't allow the parser to
    differentiate between the different semantics as that would require
    the tokenizer to backtrack and produce a different set of tokens
    (this is, first try as a stream of tokens, and if it fails, try as a
    string for a format specifier).

    As there is no fundamental advantage in being able to allow lambdas
    and similar expressions at the top level, we have decided to keep
    the restriction that these must be parenthesized if needed:

        >>> f'Useless use of lambdas: { (lambda x: x*2) }'

3.  We have decided to disallow (for the time being) using escaped
    braces (\{ and \}) in addition to the {{ and }} syntax. Although the
    authors of the PEP believe that allowing escaped braces is a good
    idea, we have decided to not include it in this PEP, as it is not
    strictly necessary for the formalization of f-strings proposed here,
    and it can be added independently in a regular CPython issue.

Open Issues

None yet

Footnotes

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

license: Public Domain


text:
PEP: 8002 Title: Open Source Governance Survey Author: Barry Warsaw
<EMAIL_ADDRESS>Łukasz Langa<EMAIL_ADDRESS>Antoine Pitrou
<EMAIL_ADDRESS>Doug Hellmann<EMAIL_ADDRESS>Carol
Willing<EMAIL_ADDRESS>Status: Final Type: Informational Topic:
Governance Content-Type: text/x-rst Created: 24-Aug-2018

Abstract

This PEP surveys existing and similar open source and free software
projects for their governance models, providing summaries that will
serve as useful references for Python's own selection of a new
governance model in the wake of Guido's retirement.

Rather than an individual PEP for each of these community surveys, they
will all be collected here in this PEP.

Rationale

CPython is not the first open source project to undergo a governance
crisis. Other projects have experimented various governance options,
sometimes several times during their existence. There are useful lessons
to take away of their experience, which will help inform our own
decision.

Project choice

There are many open source projects out there, but it will be most
fruitful to survey those which are similar enough to CPython on a couple
key metrics:

1.  the number of contributors and their activity (there are scaling
    issues that don't make the governance models of very small projects
    very enlightening for our purposes) ;
2.  being mostly or partly community-driven (company-driven projects can
    afford different governance options, since the company hierarchy has
    power over the main participants) ;
3.  being faced with important design decisions that require a somewhat
    formal decision process.

Rust

The governance structure is documented in Rust RFC #1068.

The effective governance process grows organically over time without
being entirely codified as RFCs, especially in case of day-to-day
operation details. One example is the formation of Domain Working Groups
in February 2018.

Key people and their functions

In the Rust project there are teams responsible for certain areas. For
language features there is a "lang team", for tooling there's "dev
tools" and "Cargo", and so on. Contentious issues have facilitators to
drive discussion who often aren't the decision makers. Typically the
facilitators are authors of the proposed changes (see "Controversial
decision process" below). They ensure all key decision makers are
involved along with interested community members. They push towards an
agreeable outcome via iteration.

In practice this means decisions are rarely escalated to the core team.

The most common role of a contributor is team membership. Issue
triage/code review privileges without team membership is rare.
Contributors have full commit access, code ownership separation is based
on trust. Writing to the compiler repository is frowned upon, all
changes go through pull requests and get merged by an integration bot
after they were reviewed and approved.

New team members are added by nomination by an existing team member.

Regular decision process

Primary work happens via GitHub issues and pull requests. Approving a
pull request by any team member allows it to be merged without further
process. All merged pull requests end up in the next stable version of
Rust.

Notifying relevant people by mentions is important. Listening to the
firehose of e-mails for all GitHub activity is not popular.

There are planning and triage meetings open to the public happening on
IRC and Discord. They are not very popular because most of work happens
through GitHub. Discussions also happen on official Rust forums
(https://users.rust-lang.org/ and https://internals.rust-lang.org/).
There is a dedicated moderation team responsible for taking notes and
enforcing code of conduct.

Controversial decision process

Larger or controversial work goes through a RFC process. It allows
everyone to express their thoughts and iterates towards a resolution. At
some point when all blocking concerns of relevant team members are
addressed, they sign off on the RFC and it reaches a "final comment
period". That does not require consensus amongst all participants,
rather there should not be a strong consensus against the proposal.

After 10 days the RFC is merged unless any new blocking concerns are
raised by team members. A "merge" signifies that work towards
implementing the feature and integrating it can now happen without
interruption. An RFC doesn't have to have a reference implementation for
it to be accepted.

The other possible results of the "final comment period" are to:

-   postpone the RFC (similar to the Deferred status in PEPs),
-   get it back into discussion if blocking concerns can be addressed,
    or
-   close it if blocking concerns are not solvable. When an RFC is
    marked as closed, there is a 7-day grace period to debate whether it
    should be closed.

In practice registering concerns with an RFC happens very often
initially but rarely causes for the RFC to be entirely killed.

This process scales well for small-contention changes and/or smaller
changes. For the largest controversial changes the discussion gets
unwieldy. This is a topic currently (as of August 2018) on the minds of
the Rust team (see: "Listening and Trust, part 1", "Listening and Trust,
part 2", "Listening and Trust, part 3", "Proposal for a staged RFC
process").

Planning a new release

Every six weeks the Rust compiler is released with whatever it contained
at the time. There are no LTS channels or releases yet but this concept
is planned to make redistributors able to keep up with development
better.

Every few years a so-called "Edition" is released. Those are milestone
releases with full sets of updated documentation and tooling. They can
be backwards incompatible with previous editions. External packages opt
into breaking changes in their crate metadata. The Rust compiler
supports all editions that existed prior to its release. Linking between
crates of any supported edition is possible.

Changes in the process over time

The Rust programming language was started by Graydon Hoare who developed
it as a personal project for a few years. When Mozilla started
sponsoring the project, the team slowly grew with Graydon as a
BDFL-style figure. He left the project in 2013. Rust functions without a
BDFL since. The RFC process was put in place later. Initially some
design discussions happened during closed-door weekly video meetings
which was shut down in May 2015 (before the 1.0 release of Rust),
organically replaced with open discussion and direct influence of teams.

The number of teams is growing in time. The number of technical
decisions made by the core team is decreasing, instead those get
delegated to respective teams.

The concept of a "final comment period" was introduced to encourage more
public discussion and enable reacting to a change about to being made,
instead of having to revert a rushed decision that was already made.

OpenStack

The OpenStack Foundation Bylaws lay out the basic structure for project
governance, with Article IV delegating day-to-day management of the open
source project to the OpenStack Technical Committee (TC), and The TC
member policy defining broadly how the Technical Committee shall be
elected. The TC publishes a set of more detailed governance documents,
including the TC charter, which describes the team structure, precise
rules for establishing eligibility to run for office, and criteria for
establishing the various electorates.

Key people and their functions

The OpenStack community is made up of many distinct project teams,
responsible for producing different components of the software (block
storage management, compute management, etc.) or managing different
parts of the processes the community follows (such as tracking the
release schedule). Each team is led by a Project Team Lead (PTL),
elected by the Active Project Contributors for that project.

Active Project Contributors (APCs) are recent contributors to a given
project team. APC status formally requires two things: becoming an
individual member of the OpenStack Foundation (membership is free) and
having a change merged within the last year (two development cycles) in
a repository managed by a project team.

The elected PTL serves a term equal to one development cycle (roughly 6
months). There is no restriction on the number of consecutive terms a
person may serve as PTL, and it is common for someone to serve for
several terms in a row. It is also not unusual for a team to have only
one candidate volunteer to serve as PTL for a given cycle, in which case
there is no need for an election.

The PTL represents the team in all cases except where they have
explicitly delegated some responsibility. For example, many teams
designate a separate release liaison to manage the release process for a
development cycle. The PTL also serves as a final decision maker in
cases where consensus cannot be reached between the team members.

While the APCs all vote for the PTL of a team, in many other cases only
the core reviewer team will be consulted on policy decisions for the
team. Anyone may review any patch for any OpenStack project. After
someone demonstrates that they have a good grasp of the technical issues
of a project, that they provide useful feedback on reviews, and that
they understand the direction the project is going, they may be invited
to become a member of the core review team. Unlike in many other
communities, this status does not grant them the right to submit code
without having it reviewed. Rather, it asks them to commit to reviewing
code written by other contributors, and to participate in team
decision-making discussions. Asking someone to become a member of the
core review team is a strong indication of trust.

The Technical Committee (TC) is responsible for managing the development
of OpenStack as a whole. The 13 members of the Technical Committee are
directly elected by APCs from all project teams. Each member serves a
term of two development cycles (roughly 1 year), with the elections
split so that only about half of the members' terms expire at any time,
to ensure continuity. The TC establishes overall policies, such as the
criteria for adding new project teams, the deprecation policy for Python
2, testing requirements, etc.

Regular decision process

All elections for PTL or TC members use https://civs.cs.cornell.edu to
run a Condorcet election. This system was selected because it emphasizes
consensus candidates over strict popularity.

The OpenStack contributor community relies on 3 primary tools for
discussion: the openstack-dev mailing list, a gerrit code review
instance at https://review.openstack.org, and a set of
OpenStack-specific IRC channels on Freenode. There are a few teams whose
contributors are based primarily in China, and they have trouble
accessing IRC. Those teams tend to use alternative platforms such as
WeChat, instead.

The tool used for discussing any given decision will vary based on its
weight and impact. Everyone is encouraged to use either the mailing list
or gerrit to support asynchronous discussion across a wider range of
timezones and firewalls, especially for publicizing final decisions for
the rest of the community.

Policy decisions limited to a single team are usually made by the core
review team for a project, and the policies and decision processes may
vary between teams. Some groups write down their team policies in their
documentation repository, and use the code review tool (gerrit) to vote
on them. Some teams discuss policies on IRC, either ad hoc or during a
regularly scheduled meeting, and make decisions there. Some teams use
the mailing list for those discussions. The PTL for the team is
responsible for ensuring the discussion is managed and the outcome is
communicated (either by doing so directly or ensuring that the task is
delegated to someone else).

All team policy decisions need to be compatible with the overall
policies set by the Technical Committee. Because the TC tends to make
broader governance decisions that apply to the entire contributor
community, the process for discussing and voting on those decisions is
described more formally, including specifying the number of votes needed
to pass and the minimum length of time required for discussion. For
example, most motions require 1/3 of the members (5) to pass and must
stay open at least 3 days after receiving sufficient votes to pass,
ensuring that there is time for dissent to be registered. See the
Technical Committee Charter and house rules for more details.

Significant design decisions are usually discussed by reviewing a
specification document, somewhat similar to a PEP, that covers the
requirements, alternatives, and implementation details. Feedback is
solicited from all contributors, and then specifications are eventually
approved or rejected by members of the core review team for a project.
Some teams require only 2 reviewers to approve a design, while other
teams require a stronger indication of consensus before a design is
approved. Each team sets a deadline for approving specifications within
each development cycle, to encourage contributors to work out designs
for significant new features early and avoid risk from changes late in
the cycle.

Smaller technical decisions are typically made by reviewing the
patch(es) needed to implement the change. Anyone may review any patch
and provide technical feedback, but ultimately two core reviewers for a
team are needed to approve most changes (exceptions are often made for
trivial changes such as typos or for fixes that unblock the CI gating
system).

Controversial decision process

Controversial, or merely complicated, decisions frequently expand
outside of specification reviews to mailing list discussions. They often
also result in discussions at one of the regularly scheduled in-person
community gatherings. Because many members of the community cannot
attend these events, the discussions are summarized and final decisions
are made using on-line tools as much as possible.

The PTL is responsible for deciding when consensus has been reached for
decisions that affect a single team, and to make a final call in rare
cases where consensus has not been reached and a decision absolutely
needs to be made. The TC acts as a similar decision-making group of last
resort for cases where issues between teams cannot be resolved in
another way. Such escalation of decision-making ends up being rarely
necessary, because the contributors directly involved generally prefer
to come to a consensual agreement rather than escalate the decision to
others.

Planning a new release

OpenStack has a major release about every 6 months. These are
coordinated date-based releases, which include the work finished up to
that point in time in all of the member projects. Some project teams
release more often than every 6 months (this is especially true for
teams building libraries consumed by other teams). Those smaller
releases tend to be produced when there is content (new features or bug
fixes) to justify them.

The schedule for each development cycle, with deadlines and a final
release date, is proposed by the release management team, in
coordination with the Foundation staff (releases are generally aligned
with the calendar of in-person events), and then the community has an
opportunity to provide feedback before the final dates are set.

Decisions about priorities for each development cycle are made at the
team level and the TC level. Core review teams prioritize internal work,
such as fixing bugs and implementing new features. The TC selects
community goals, which usually require some amount of work from all
teams. Agreeing to these priorities at the start of each cycle helps the
teams coordinate their work, which is especially important because the
implementation will require reviews from multiple team members.

Changes in the process over time

Over the last 8 years the number of OpenStack project teams has grown
from 2 to 63. The makeup of the Technical Committee has changed to
accommodate that growth. Originally the TC was made up of PTLs, but as
the membership grew it became impractical for the group to function
effectively.

The community also used to be organized around "program areas" rather
than project teams. A program area covered a feature set, such as
gathering telemetry or managing block storage. This organization failed
when multiple teams of people wanted to work on the same feature set
using different solutions. Organizing teams around the code they deliver
allows different teams to have different interpretations of the same
requirements. For example, there are now several teams working on
different deployment tools.

Jupyter

The governance structure is documented in the Main Governance Document
within the Jupyter Governance repo.

The effective governance process grows organically over time as the
needs of the project evolve. Formal changes to the Governance Document
are submitted via Pull Request, with an open period for comments. After
the open period, a Steering Council may call for a vote to ratify the PR
changes. Acceptance requires a minimum of 80% of the Steering Council to
vote and at least 2/3 of the vote must be positive. The BDFL can act
alone to accept or reject changes or override the Steering Council
decision; though this would be an extremely rare event.

Key people and their functions

The key people in Jupyter's Governance are the BDFL, Fernando Perez, and
the Steering Council. Contributors can be given a special status of core
contributor. Some may also be Institutional Contributors, who are
individuals who contribute to the project as part of their official
duties at an Institutional Partner.

Fernando Perez, the project founder, is the current and first BDFL. The
BDFL may serve as long as desired. The BDFL succession plan is described
in the Main Governance Document. In summary, the BDFL may appoint the
next BDFL. As a courtesy, it is expected that the BDFL will consult with
the Steering Council. In the event that the BDFL can not appoint a
successor, the Steering Council will recommend one.

Core contributors are individuals who are given rights, such as commit
privileges, to act in the best interest of the project within their area
of expertise or subproject. An existing core contributor typically
recommends someone be given core contributor rights by gathering
consensus from project leads, who are experienced core contributors as
listed in the README of the project repo.

To be recommended and invited as a Steering Council member, an
individual must be a Project Contributor who has produced contributions
that are substantial in quality and quantity, and sustained over at
least one year. Potential Council Members are nominated by existing
Council members and voted upon by the existing Council after asking if
the potential Member is interested and willing to serve in that
capacity.

Regular decision process

Project Jupyter is made up of a number of GitHub organizations and
subprojects within those organizations. Primary work happens via GitHub
issues and pull requests. Approving a pull request by any team member
allows it to be merged without further process. All merged pull requests
end up in the next stable release of a subproject.

There is a weekly, public Project-wide meeting that is recorded and
posted on YouTube. Some larger GitHub organizations, which are
subprojects of Project Jupyter, e.g. JupyterLab and JupyterHub, may have
additional public team meetings on a weekly or monthly schedule.
Discussions occur on Gitter, the Jupyter mailing list, and most
frequently an open issue and/or pull request on GitHub.

Controversial decision process

The foundations of Project Jupyter's governance are:

-   Openness & Transparency
-   Active Contribution
-   Institutional Neutrality

During the everyday project activities, Steering Council members
participate in all discussions, code review and other project activities
as peers with all other Contributors and the Community. In these
everyday activities, Council Members do not have any special power or
privilege through their membership on the Council. However, it is
expected that because of the quality and quantity of their contributions
and their expert knowledge of the Project Software and Services that
Council Members will provide useful guidance, both technical and in
terms of project direction, to potentially less experienced
contributors.

For controversial issues, the contributor community works together to
refine potential solutions, iterate as necessary, and build consensus by
sharing information and views constructively and openly. The Steering
Council may make decisions when regular community discussion doesn't
produce consensus on an issue in a reasonable time frame.

Voting

Rarely, if ever, is voting done for technical decisions.

For other Project issues, the Steering Council may call for a vote for a
decision via a Governance PR or email proposal. Acceptance requires a
minimum of 80% of the Steering Council to vote and at least 2/3 of the
vote must be positive.

The BDFL can act alone to accept or reject changes or override the
Steering Council decision; though this would be an extremely rare event.
As Benevolent, the BDFL, in practice chooses to defer that authority to
the consensus of the community discussion channels and the Steering
Council.

Planning releases

Since Project Jupyter has a number of projects, not just a single
project, the release planning is largely driven by the core contributors
of a project.

Changes in the process over time

The process has remained consistent over time, and the approach has
served us well. Moving forward The Project leadership will consist of a
BDFL and Steering Council. This governance model was a formalization of
what the Project was doing (prior to 2015 when the Main Governance
Document was adopted by the Steering Council), rather than a change in
direction.

Django

The governance structure is documented in Organization of the Django
Project.

Key people and their functions

The project recognizes three kinds of contributors. Members of the core
team, the Technical Board, and Fellows. Regular core committers no
longer exercise their "commit bit", instead they rely on pull requests
being reviewed and accepted. The Technical Board steers technical
choices. Fellows are hired contractors who triage new tickets, review
and merge patches from the committers and community, including
non-trivial ones.

Core team members are added by nomination and vote within the core team,
with technical board veto (so far not exercised). Technical board is
elected by and from the core team membership every 18 months (every
major Django release). Sub-teams within the core team are self-selected
by interest.

Regular decision process

Most day-to-day decisions are made by Fellows and sometimes other active
core team members.

The core team votes on new members which requires a 4/5 majority of
votes cast, no quorum requirement. The Technical Board has veto power.
This power was never exercised

Controversial decision process

The Technical Board occasionally approves Django Enhancement Proposals
(DEPs) but those are rare. The DEP process is roughly modeled after PEPs
and documented in DEP 1. DEPs are mostly used to design major new
features, but also for information on general guidelines and process.

An idea for a DEP should be first publicly vetted on the
django-developers mailing list. After it was roughly validated, the
author forms a team with three roles:

-   authors who write the DEP and steers the discussion;
-   implementers who prepare the implementation of the DEP;
-   a shepherd who is a core developer and will be the primary reviewer
    of the DEP.

The DEP's draft is submitted, assigned a number, and discussed. Authors
collect feedback and steer discussion as they see fit. Suggested venues
to avoid endless open-ended discussions are: separate mailing lists,
Wiki pages, working off of pull requests on the DEP.

Once the feedback round is over, the shepherd asks the Technical Board
for review and pronouncement. The Board can rule on a DEP as a team or
designate one member to review and decide.

In any case where consensus can't be reached, the Technical Board has
final say. This was never exercised.

Differences between DEPs and PEPs

The main difference is that the entire workflow is based on pull
requests rather than e-mail. They are pronounced upon by the Technical
Board. They need to have the key roles identified before submission and
throughout the process. The shepherd role exists to guide a DEP to
completion without engaging the Technical Board.

Those changes to the process make it more distributed and workable in a
governance model without a BDFL.

Planning a new release

Releases are done on a fixed time-based schedule, with a major version
every 18 months. With paid Fellows to ensure the necessary work gets
down, on-time releases are routine.

Changes in the process over time

Django originally had two BDFLs: Jacob Kaplan-Moss and Adrian Holovaty.
They retired (Adrian's post, Jacob's post) 9 years into the project's
history. Following the stepping down, the DEP process was defined.

TypeScript

The governance structure is not externally documented besides the
CONTRIBUTING.md document in the main TypeScript repository.

Key people and their functions

There is a formal design team and a release management team working at
Microsoft. The main person behind the project is currently Anders
Hejlsberg as some of the original members of the team have left the
company.

Regular decision process

Microsoft, where the project is developed, has a strong planning culture
so development roadmaps are released long in advanced, notes from design
discussions held at Microsoft get published quickly and meetings are
sometimes broadcast using Skype.

External contributions are encouraged through pull requests on GitHub.
Suggestions for new use cases or features are given by issues on GitHub.
This serves like an ad-hoc PEP-like process. There is some discussion
over social media (Twitter) as well.

Controversial decision process

Hejlsberg is the central figure of the project in terms of language
design, synthesizing community needs into a cohesive whole. There is no
formal process to externally contribute to the design of the language.

The TypeScript team filters through and integrates community
suggestions. The main advantages of this setup are that there is strong
and consistent design with dependable scheduling and execution. While
there is transparency of intentions and plans, the disadvantage of this
model is that community involvement is limited to pull requests and
suggestions.

Planning a new release

Microsoft determines the release schedule, communicates dates and
features well in advance. Nightly builds are usually stable (with a
significant portion of users on this release form).

Versioned releases are done every 1 - 3 months, with a roadmap available
on GitHub.

Changes in the process over time

TypeScript is likely the first notable project by Microsoft developed
fully in the open (versus source-available).

Open-sourcing of TypeScript by Microsoft was a planned feature from the
inception of the project. Before the first open release was made, the
language was driven fully by needs identified by the original teams and
the early in-house users. The initial open-sourcing happened via the
now-defunct Microsoft CodePlex platform. It didn't have a well-defined
routine of accepting external contributions. Community engagement rose
significantly after the project got moved.

Astropy

Key people and their functions

The Astropy Project team's responsibilities are spread over many
different roles[1], though frequently a person will have several roles.

The main body overseeing the Astropy Project is the Astropy Coordination
Committee (CoCo) . Its key roles are dealing with any financial issues,
approving new packages wanting to join the Astropy Project, approving or
rejecting Astropy Proposals for Enhancement (APEs)[2], and generally
anything that's "leadership"-oriented or time-sensitive. As of this
writing, the committee has four members, and might grow or shrink as the
demands on the committee change.

Regular decision process

Code-level decisions

The Astropy Project includes the core Astropy package and other
affiliated packages. For the sake of simplicity, we will avoid
discussing affiliated packages, which can have their own rules.
Therefore, everything below will concern the core Astropy package.

The core Astropy package is organized as sub-packages. Each sub-package
has an official maintainer as well as one or more deputies, who are
responsible for ensuring code is reviewed and generally architecting the
subpackage. Code-level decisions are therefore made in GitHub issues or
pull requests (PRs), usually on the basis of consensus, moderated by the
maintainer and deputies of that sub-package.

When there is specific disagreement, majority vote of those who are
involved in the discussion (e.g. PR) determines the winner, with the
CoCo called on to break ties or mediate disagreements.

Non-code decisions

Non-code decisions (like sprint scheduling, bugfix release timing, etc)
are usually announced on the astropy-dev mailing list[3] with a
vote-by-message format, or a "if there are no objections"-style message
for highly uncontroversial items. In general, on astropy-dev the
expectation is a concrete proposal which other members are welcome to
comment or vote on.

Voting

Voting usually involves either using the +1/-1 format on GitHub or the
astropy-dev mailing list. There, any interested person can vote
regardless of their official role in the project, or lack thereof.
Furthermore, there is no veto mechanism for the CoCo to override
decisions of the majority.

Controversial decision process

Simpler controversial decisions are generally discussed on the
astropy-dev mailing list[4], and after a reasonable time either there is
a clear consensus/compromise (this happens most of the time), or the
CoCo makes a decision to avoid stalling.

More complicated decisions follow the APE process, which is modeled
after the PEP process. Here, the CoCo makes the final decision after a
discussion period, open to everyone, has passed. Generally the CoCo
would follow the consensus or majority will.

Ethical issues

The Project has an Ombudsperson who ensures there is an alternate
contact for sensitive issues, such as Code of Conduct violations,
independent from the Coordination Committee. In practice, the CoCo, the
Community engagement coordinators and the Ombudsperson would work
together privately to try and communicate with the violator to address
the situation.

Planning a new release

The major release timing is on a fixed schedule (every 6 months);
whatever is in at that time goes in.

Changes in the process over time

The CoCo and the "Open Development" ethos came from the inception of the
Project after a series of votes by interested Python-oriented
astronomers and allied software engineers. The core results of that
initial discussion were embodied in the Vision for Astropy document[5].

The existence of the formal roles and most of the rest of the above came
as evolutionary steps as the community grew larger, each following
either the APE process, or the regular process of a proposal being
brought up for discussion and vote in astropy-dev[6]. In general all
evolved as sort of ratification of already-existing practices, only
after they were first tested in the wild.

Self-appreciation

The fact that anyone who has the time can step in and suggest something
(usually via PR) or vote on their preference, leads to a sense that "we
are all in this together", leading to better-coordinated effort.

Additionally, the function of the CoCo as mostly a tie-breaking body
means that there's no sense of a dictator who enforces their will, while
still giving clear points of contact for external organizations that are
leery of fully-democratic governance models.

References

Bonus: Microsoft

Despite the selection process for "relevant projects" described above,
it is worthwhile considering how companies that are held financially
accountable for their decisions go about making them. This is not
intended as a readily-usable model for Python, but as additional insight
that may influence the final design or selection.

This section is not taken from any official documentation, but has been
abstracted by Steve Dower, a current Microsoft employee, to reflect the
processes that are most applicable to individual projects in the
engineering departments. Role titles are used (and defined) rather than
identifying specific individuals, and all names are examples and should
not be taken as a precise description of the company at any particular
time in history.

This is also highly simplified and idealised. There are plenty of
unhealthy teams that do not look like this description, and those
typically have high attrition (people leave the team more frequently
than other teams). Teams that retain their people are usually closer to
the model described here, but ultimately everything involving humans is
imperfect and Microsoft is no exception.

Key people and their functions

Microsoft has a hierarchy that ultimately reports to the CEO. Below the
CEO are a number of organisations, some of which are focused on
engineering projects (as opposed to sales, marketing or other
functions). These engineering organisations roughly break down into
significant product families - for example, there has been a "Windows
group", an "Xbox group", and a "server and tools group". These are
typically led by Executive Vice Presidents (EVPs), who report to the
CEO.

Below each EVP are many Corporate Vice Presidents (CVPs), each of which
is responsible for one or more products. This level is where the
hierarchy becomes relevant for the purposes of this PEP - the CEO and
EVPs are rarely involved in most decision processes, but set the
direction under which CVPs make decisions.

Each product under a CVP has a team consisting of Program Managers (PMs)
and Engineering Managers. Engineering Managers have teams of engineers
who are largely uninvolved in decision making, though may be used as
specialists in some cases. For the rest of this section, Engineering
refers to anyone from the engineering team who is contributing with a
technical-focus, and PM refers to anyone from the program management
team contributing with a customer-focus. After decisions are made,
Engineering does the implementation and testing work, and PM validates
with users that their problem has been solved.

(This is actually a huge simplification, to the point where some people
in these roles are offended by this characterisation. In reality, most
people in PM or Engineering do work that crosses the boundary between
the two roles, and so they should be treated as a term describing the
work that somebody is doing in the moment, rather than an identifier or
restriction for a person.)

Teams generally represent a feature, while the CVP represents a product.
For example, Visual Studio Code has a CVP who is ultimately responsible
for decisions about that product and its overall direction (in the
context set by their EVP). But many teams contribute features into
Visual Studio Code.

For complete clarity, the CEO, EVPs, and CVPs do not ever directly
modify source code. Their entire role is to provide direction for
whoever is immediately below them and to adjudicate on controversial
decisions.

Regular decision process

Changes to product code that are not visible to external users are made
solely by Engineering. Individual engineers will be assigned tasks by a
designated engineering manager, or may self-assign. Promotion to
increasingly senior positions generally reflects trust in the
individual's decision-making ability, and more senior engineers are
trusted to make decisions with less validation from the rest of the
team. Most bugs are covered by this process (that is, fixing a
user-visible problem without changing the intended experience is an
Engineering decision).

Decisions affecting users of a particular feature are made by the PM
team for that feature. They will use whatever data sources available to
identify an issue, experiment with alternatives, and ultimately prepare
a design document. Senior members from PM and Engineering will review
designs to clarify the details, and ultimately an artifact is created
that the feature team agrees on. Engineering will use this artifact to
implement the work, and PM will later use this artifact to validate that
the original issue has been resolved.

Senior members of Engineering and PM teams for a feature are expected to
make decisions in the spirit of the direction set by their CVP. Teams
have regular meetings with their CVP to discuss recent decisions and
ensure consistency. Decisions that are not obviously in line with CVP
expectations are escalated to the controversial process.

Controversial decision process

When decisions require cross-team coordination, or do not obviously
align with previous CVP guidance, teams will escalate decision making.
These often include decisions that involve changing direction,
attempting to reach a new or different group of users, deprecating and
removing significant features (or on a short timeframe), or changes that
require quick releases.

In general, CVPs are not intimately familiar with all aspects of the
feature team's work. As a result, the feature team must provide both a
recommendation and sufficient context for the decision that the CVP can
decide without additional knowledge. Most of the time, the first attempt
results in a series of questions from the CVP, which the team will
research, answer and attempt the decision again at a later date.

Common questions asked by CVPs are:

-   how many users are affected by this decision?
-   what is the plan for minimizing impact on current users?
-   how will the change be "sold"/described to potential users?

CVPs are expected to have a strong understanding of the entire field, so
that they can evaluate some questions for themselves, such as:

-   what similar decisions have been made by other projects within
    Microsoft?
-   what other projects have plans that may impact this decision?
-   what similar decisions have been made by projects outside Microsoft?
-   do users need it?
-   is it in line with the direction set by their EVP?

Decisions made by CVPs are generally arbitrary and final, though they
typically will provide their rationale.

Planning a new release

Releases involve coordinating a number of feature teams, and so rarely
attempt to include input from all teams. A schedule will be determined
based on broader ecosystem needs, such as planned events/conferences or
opportunities to take advantage of media attention.

Teams are informed of the release date, the theme of the release, and
make their own plans around it following the above decision making
process. Changing the release date is considered a controversial
decision.

Acknowledgements

Thank you to Alex Crichton from the Rust team for an extensive
explanation of how the core team governs the project.

Jeremy Stanley, Chris Dent, Julia Kreger, Sean McGinnis, Emmet Hikory,
and Thierry Carrez contributed to the OpenStack section.

The Project Jupyter Steering Council created the Main Governance
Document for Project Jupyter, and Carol Willing summarized the key
points of that document for the Jupyter section.

Thank you to Carl Meyer from the Django team for explanation how their
project's governance is set up.

The TypeScript and Swift sections were created after conversations with
Joe Pamer and Vlad Matveev. Thanks!

Answers about the Astropy project were kindly contributed, in
significant detail, by Erik Tollerud and reviewed by other members of
the project.

Annex 1: Template questions

The following set of questions was used as a template to guide
evaluation and interaction with the surveyed projects:

1.  Do you have any open documentation on how the governance model is
    set up?
2.  How does the process look like in practice?
    -   Who are the key people?
    -   What "special statuses" can contributors have?
    -   How are they elected/how are the statuses assigned?
    -   How are regular decisions made?
    -   How are controversial decisions made?
    -   Is there a voting mechanism? how does it work? how often do
        votes actually happen?
    -   Is there a veto mechanism? how often was it actually used?
3.  How do you like the process?
    -   Which parts work well?
    -   Which parts could work better?
    -   When it doesn't work well, how does it look like?
    -   What would you change if it were only up to you?
4.  Related project work:
    -   How do you decide when a release happens and what goes into it?
    -   How do you decide who gets commit access?
    -   Where do you hold discussions? (GitHub, mailing lists,
        face-to-face meetings, and so on)
    -   Do you have a RFC/PEP-like process?
    -   Who has access to those discussion channels?
    -   How is this access granted/revoked?
    -   Who moderates those discussions?
    -   Do you (and how) censure participants and how?
5.  Process evolution
    -   How did this process evolve historically?
    -   How can it be changed in the future?

Copyright

This document has been placed in the public domain.

[1] Astropy roles and responsibilities https://www.astropy.org/team.html

[2] Astropy Proposals for Enhancement
https://github.com/astropy/astropy-APEs

[3] Astropy-dev mailing list
https://groups.google.com/forum/#!forum/astropy-dev

[4] Astropy-dev mailing list
https://groups.google.com/forum/#!forum/astropy-dev

[5] Vision for a Common Astronomy Python Package
https://docs.astropy.org/en/stable/development/vision.html

[6] Astropy-dev mailing list
https://groups.google.com/forum/#!forum/astropy-dev

authors:
Barry Warsaw
Łukasz Langa