text:
PEP: 287 Title: reStructuredText Docstring Format Version: $Revision$
Last-Modified: $Date$ Author: David Goodger<EMAIL_ADDRESS>Discussions-To<EMAIL_ADDRESS>Status: Active Type: Informational
Content-Type: text/x-rst Created: 25-Mar-2002 Post-History: 02-Apr-2002
Replaces: 216

Abstract

When plaintext hasn't been expressive enough for inline documentation,
Python programmers have sought out a format for docstrings. This PEP
proposes that the reStructuredText markup be adopted as a standard
markup format for structured plaintext documentation in Python
docstrings, and for PEPs and ancillary documents as well.
reStructuredText is a rich and extensible yet easy-to-read,
what-you-see-is-what-you-get plaintext markup syntax.

Only the low-level syntax of docstrings is addressed here. This PEP is
not concerned with docstring semantics or processing at all (see PEP 256
for a "Road Map to the Docstring PEPs"). Nor is it an attempt to
deprecate pure plaintext docstrings, which are always going to be
legitimate. The reStructuredText markup is an alternative for those who
want more expressive docstrings.

Benefits

Programmers are by nature a lazy breed. We reuse code with functions,
classes, modules, and subsystems. Through its docstring syntax, Python
allows us to document our code from within. The "holy grail" of the
Python Documentation Special Interest Group (Doc-SIG) has been a markup
syntax and toolset to allow auto-documentation, where the docstrings of
Python systems can be extracted in context and processed into useful,
high-quality documentation for multiple purposes.

Document markup languages have three groups of customers: the authors
who write the documents, the software systems that process the data, and
the readers, who are the final consumers and the most important group.
Most markups are designed for the authors and software systems; readers
are only meant to see the processed form, either on paper or via browser
software. ReStructuredText is different: it is intended to be easily
readable in source form, without prior knowledge of the markup.
ReStructuredText is entirely readable in plaintext format, and many of
the markup forms match common usage (e.g., *emphasis*), so it reads
quite naturally. Yet it is rich enough to produce complex documents, and
extensible so that there are few limits. Of course, to write
reStructuredText documents some prior knowledge is required.

The markup offers functionality and expressivity, while maintaining easy
readability in the source text. The processed form (HTML etc.) makes it
all accessible to readers: inline live hyperlinks; live links to and
from footnotes; automatic tables of contents (with live links!); tables;
images for diagrams etc.; pleasant, readable styled text.

The reStructuredText parser is available now, part of the Docutils
project. Standalone reStructuredText documents and PEPs can be converted
to HTML; other output format writers are being worked on and will become
available over time. Work is progressing on a Python source "Reader"
which will implement auto-documentation from docstrings. Authors of
existing auto-documentation tools are encouraged to integrate the
reStructuredText parser into their projects, or better yet, to join
forces to produce a world-class toolset for the Python standard library.

Tools will become available in the near future, which will allow
programmers to generate HTML for online help, XML for multiple purposes,
and eventually PDF, DocBook, and LaTeX for printed documentation,
essentially "for free" from the existing docstrings. The adoption of a
standard will, at the very least, benefit docstring processing tools by
preventing further "reinventing the wheel".

Eventually PyDoc, the one existing standard auto-documentation tool,
could have reStructuredText support added. In the interim it will have
no problem with reStructuredText markup, since it treats all docstrings
as preformatted plaintext.

Goals

These are the generally accepted goals for a docstring format, as
discussed in the Doc-SIG:

1.  It must be readable in source form by the casual observer.
2.  It must be easy to type with any standard text editor.
3.  It must not need to contain information which can be deduced from
    parsing the module.
4.  It must contain sufficient information (structure) so it can be
    converted to any reasonable markup format.
5.  It must be possible to write a module's entire documentation in
    docstrings, without feeling hampered by the markup language.

reStructuredText meets and exceeds all of these goals, and sets its own
goals as well, even more stringent. See Docstring-Significant Features
below.

The goals of this PEP are as follows:

1.  To establish reStructuredText as a standard structured plaintext
    format for docstrings (inline documentation of Python modules and
    packages), PEPs, README-type files and other standalone documents.
    "Accepted" status will be sought through Python community consensus
    and eventual BDFL pronouncement.

    Please note that reStructuredText is being proposed as a standard,
    not the only standard. Its use will be entirely optional. Those who
    don't want to use it need not.

2.  To solicit and address any related concerns raised by the Python
    community.

3.  To encourage community support. As long as multiple competing
    markups are out there, the development community remains fractured.
    Once a standard exists, people will start to use it, and momentum
    will inevitably gather.

4.  To consolidate efforts from related auto-documentation projects. It
    is hoped that interested developers will join forces and work on a
    joint/merged/common implementation.

Once reStructuredText is a Python standard, effort can be focused on
tools instead of arguing for a standard. Python needs a standard set of
documentation tools.

With regard to PEPs, one or both of the following strategies may be
applied:

a)  Keep the existing PEP section structure constructs (one-line section
    headers, indented body text). Subsections can either be forbidden,
    or supported with reStructuredText-style underlined headers in the
    indented body text.
b)  Replace the PEP section structure constructs with the
    reStructuredText syntax. Section headers will require underlines,
    subsections will be supported out of the box, and body text need not
    be indented (except for block quotes).

Strategy (b) is recommended, and its implementation is complete.

Support for 2822 headers has been added to the reStructuredText parser
for PEPs (unambiguous given a specific context: the first contiguous
block of the document). It may be desired to concretely specify what
over/underline styles are allowed for PEP section headers, for
uniformity.

Rationale

The lack of a standard syntax for docstrings has hampered the
development of standard tools for extracting and converting docstrings
into documentation in standard formats (e.g., HTML, DocBook, TeX). There
have been a number of proposed markup formats and variations, and many
tools tied to these proposals, but without a standard docstring format
they have failed to gain a strong following and/or floundered
half-finished.

Throughout the existence of the Doc-SIG, consensus on a single standard
docstring format has never been reached. A lightweight, implicit markup
has been sought, for the following reasons (among others):

1.  Docstrings written within Python code are available from within the
    interactive interpreter, and can be "print"ed. Thus the use of
    plaintext for easy readability.
2.  Programmers want to add structure to their docstrings, without
    sacrificing raw docstring readability. Unadorned plaintext cannot be
    transformed ("up-translated") into useful structured formats.
3.  Explicit markup (like XML or TeX) is widely considered unreadable by
    the uninitiated.
4.  Implicit markup is aesthetically compatible with the clean and
    minimalist Python syntax.

Many alternative markups for docstrings have been proposed on the
Doc-SIG over the years; a representative sample is listed below. Each is
briefly analyzed in terms of the goals stated above. Please note that
this is not intended to be an exclusive list of all existing markup
systems; there are many other markups (Texinfo, Doxygen, TIM, YODL, AFT,
...) which are not mentioned.

-   XML, SGML, DocBook, HTML, XHTML

    XML and SGML are explicit, well-formed meta-languages suitable for
    all kinds of documentation. XML is a variant of SGML. They are best
    used behind the scenes, because to untrained eyes they are verbose,
    difficult to type, and too cluttered to read comfortably as source.
    DocBook, HTML, and XHTML are all applications of SGML and/or XML,
    and all share the same basic syntax and the same shortcomings.

-   TeX

    TeX is similar to XML/SGML in that it's explicit, but not very easy
    to write, and not easy for the uninitiated to read.

-   Perl POD

    Most Perl modules are documented in a format called POD (Plain Old
    Documentation). This is an easy-to-type, very low level format with
    strong integration with the Perl parser. Many tools exist to turn
    POD documentation into other formats: info, HTML and man pages,
    among others. However, the POD syntax takes after Perl itself in
    terms of readability.

-   JavaDoc

    Special comments before Java classes and functions serve to document
    the code. A program to extract these, and turn them into HTML
    documentation is called javadoc, and is part of the standard Java
    distribution. However, JavaDoc has a very intimate relationship with
    HTML, using HTML tags for most markup. Thus it shares the
    readability problems of HTML.

-   Setext, StructuredText

    Early on, variants of Setext (Structure Enhanced Text), including
    Zope Corp's StructuredText, were proposed for Python docstring
    formatting. Hereafter these variants will collectively be called
    "STexts". STexts have the advantage of being easy to read without
    special knowledge, and relatively easy to write.

    Although used by some (including in most existing Python
    auto-documentation tools), until now STexts have failed to become
    standard because:

    -   STexts have been incomplete. Lacking "essential" constructs that
        people want to use in their docstrings, STexts are rendered less
        than ideal. Note that these "essential" constructs are not
        universal; everyone has their own requirements.
    -   STexts have been sometimes surprising. Bits of text are
        unexpectedly interpreted as being marked up, leading to user
        frustration.
    -   SText implementations have been buggy.
    -   Most STexts have no formal specification except for the
        implementation itself. A buggy implementation meant a buggy
        spec, and vice-versa.
    -   There has been no mechanism to get around the SText markup rules
        when a markup character is used in a non-markup context. In
        other words, no way to escape markup.

Proponents of implicit STexts have vigorously opposed proposals for
explicit markup (XML, HTML, TeX, POD, etc.), and the debates have
continued off and on since 1996 or earlier.

reStructuredText is a complete revision and reinterpretation of the
SText idea, addressing all of the problems listed above.

Specification

The specification and user documentation for reStructuredText is quite
extensive. Rather than repeating or summarizing it all here, links to
the originals are provided.

Please first take a look at A ReStructuredText Primer, a short and
gentle introduction. The Quick reStructuredText user reference quickly
summarizes all of the markup constructs. For complete and extensive
details, please refer to the following documents:

-   An Introduction to reStructuredText
-   reStructuredText Markup Specification
-   reStructuredText Directives

In addition, Problems With StructuredText explains many markup decisions
made with regards to StructuredText, and A Record of reStructuredText
Syntax Alternatives records markup decisions made independently.

Docstring-Significant Features

-   A markup escaping mechanism.

    Backslashes (\) are used to escape markup characters when needed for
    non-markup purposes. However, the inline markup recognition rules
    have been constructed in order to minimize the need for
    backslash-escapes. For example, although asterisks are used for
    emphasis, in non-markup contexts such as "*" or "()" or "x y", the
    asterisks are not interpreted as markup and are left unchanged. For
    many non-markup uses of backslashes (e.g., describing regular
    expressions), inline literals or literal blocks are applicable; see
    the next item.

-   Markup to include Python source code and Python interactive
    sessions: inline literals, literal blocks, and doctest blocks.

    Inline literals use double-backquotes to indicate program I/O or
    code snippets. No markup interpretation (including backslash-escape
    [\] interpretation) is done within inline literals.

    Literal blocks (block-level literal text, such as code excerpts or
    ASCII graphics) are indented, and indicated with a double-colon
    ("::") at the end of the preceding paragraph (right here -->):

        if literal_block:
            text = 'is left as-is'
            spaces_and_linebreaks = 'are preserved'
            markup_processing = None

    Doctest blocks begin with ">>> " and end with a blank line. Neither
    indentation nor literal block double-colons are required. For
    example:

        Here's a doctest block:

        >>> print 'Python-specific usage examples; begun with ">>>"'
        Python-specific usage examples; begun with ">>>"
        >>> print '(cut and pasted from interactive sessions)'
        (cut and pasted from interactive sessions)

-   Markup that isolates a Python identifier: interpreted text.

    Text enclosed in single backquotes is recognized as "interpreted
    text", whose interpretation is application-dependent. In the context
    of a Python docstring, the default interpretation of interpreted
    text is as Python identifiers. The text will be marked up with a
    hyperlink connected to the documentation for the identifier given.
    Lookup rules are the same as in Python itself: LGB namespace lookups
    (local, global, builtin). The "role" of the interpreted text
    (identifying a class, module, function, etc.) is determined
    implicitly from the namespace lookup. For example:

        class Keeper(Storer):

            """
            Keep data fresher longer.

            Extend `Storer`.  Class attribute `instances` keeps track
            of the number of `Keeper` objects instantiated.
            """

            instances = 0
            """How many `Keeper` objects are there?"""

            def __init__(self):
                """
                Extend `Storer.__init__()` to keep track of
                instances.  Keep count in `self.instances` and data
                in `self.data`.
                """
                Storer.__init__(self)
                self.instances += 1

                self.data = []
                """Store data in a list, most recent last."""

            def storedata(self, data):
                """
                Extend `Storer.storedata()`; append new `data` to a
                list (in `self.data`).
                """
                self.data = data

    Each piece of interpreted text is looked up according to the local
    namespace of the block containing its docstring.

-   Markup that isolates a Python identifier and specifies its type:
    interpreted text with roles.

    Although the Python source context reader is designed not to require
    explicit roles, they may be used. To classify identifiers
    explicitly, the role is given along with the identifier in either
    prefix or suffix form:

        Use :method:`Keeper.storedata` to store the object's data in
        `Keeper.data`:instance_attribute:.

    The syntax chosen for roles is verbose, but necessarily so (if
    anyone has a better alternative, please post it to the Doc-SIG). The
    intention of the markup is that there should be little need to use
    explicit roles; their use is to be kept to an absolute minimum.

-   Markup for "tagged lists" or "label lists": field lists.

    Field lists represent a mapping from field name to field body. These
    are mostly used for extension syntax, such as "bibliographic field
    lists" (representing document metadata such as author, date, and
    version) and extension attributes for directives (see below). They
    may be used to implement methodologies (docstring semantics), such
    as identifying parameters, exceptions raised, etc.; such usage is
    beyond the scope of this PEP.

    A modified 2822 syntax is used, with a colon before as well as after
    the field name. Field bodies are more versatile as well; they may
    contain multiple field bodies (even nested field lists). For
    example:

        :Date: 2002-03-22
        :Version: 1
        :Authors:
            - Me
            - Myself
            - I

    Standard 2822 header syntax cannot be used for this construct
    because it is ambiguous. A word followed by a colon at the beginning
    of a line is common in written text.

-   Markup extensibility: directives and substitutions.

    Directives are used as an extension mechanism for reStructuredText,
    a way of adding support for new block-level constructs without
    adding new syntax. Directives for images, admonitions (note,
    caution, etc.), and tables of contents generation (among others)
    have been implemented. For example, here's how to place an image:

        .. image:: mylogo.png

    Substitution definitions allow the power and flexibility of
    block-level directives to be shared by inline text. For example:

        The |biohazard| symbol must be used on containers used to
        dispose of medical waste.

        .. |biohazard| image:: biohazard.png

-   Section structure markup.

    Section headers in reStructuredText use adornment via underlines
    (and possibly overlines) rather than indentation. For example:

        This is a Section Title
        =======================

        This is a Subsection Title
        --------------------------

        This paragraph is in the subsection.

        This is Another Section Title
        =============================

        This paragraph is in the second section.

Questions & Answers

1.  Is reStructuredText rich enough?

    Yes, it is for most people. If it lacks some construct that is
    required for a specific application, it can be added via the
    directive mechanism. If a useful and common construct has been
    overlooked and a suitably readable syntax can be found, it can be
    added to the specification and parser.

2.  Is reStructuredText too rich?

    For specific applications or individuals, perhaps. In general, no.

    Since the very beginning, whenever a docstring markup syntax has
    been proposed on the Doc-SIG, someone has complained about the lack
    of support for some construct or other. The reply was often
    something like, "These are docstrings we're talking about, and
    docstrings shouldn't have complex markup." The problem is that a
    construct that seems superfluous to one person may be absolutely
    essential to another.

    reStructuredText takes the opposite approach: it provides a rich set
    of implicit markup constructs (plus a generic extension mechanism
    for explicit markup), allowing for all kinds of documents. If the
    set of constructs is too rich for a particular application, the
    unused constructs can either be removed from the parser (via
    application-specific overrides) or simply omitted by convention.

3.  Why not use indentation for section structure, like StructuredText
    does? Isn't it more "Pythonic"?

    Guido van Rossum wrote the following in a 2001-06-13 Doc-SIG post:

      I still think that using indentation to indicate sectioning is
      wrong. If you look at how real books and other print publications
      are laid out, you'll notice that indentation is used frequently,
      but mostly at the intra-section level. Indentation can be used to
      offset lists, tables, quotations, examples, and the like. (The
      argument that docstrings are different because they are input for
      a text formatter is wrong: the whole point is that they are also
      readable without processing.)

      I reject the argument that using indentation is Pythonic: text is
      not code, and different traditions and conventions hold. People
      have been presenting text for readability for over 30 centuries.
      Let's not innovate needlessly.

    See Section Structure via Indentation__ in Problems With
    StructuredText for further elaboration.

    __ http://docutils.sourceforge.net/docs/dev/rst/problems.html

        #section-structure-via-indentation

4.  Why use reStructuredText for PEPs? What's wrong with the existing
    standard?

    The existing standard for PEPs is very limited in terms of general
    expressibility, and referencing is especially lacking for such a
    reference-rich document type. PEPs are currently converted into
    HTML, but the results (mostly monospaced text) are less than
    attractive, and most of the value-added potential of HTML
    (especially inline hyperlinks) is untapped.

    Making reStructuredText a standard markup for PEPs will enable much
    richer expression, including support for section structure, inline
    markup, graphics, and tables. In several PEPs there are ASCII
    graphics diagrams, which are all that plaintext documents can
    support. Since PEPs are made available in HTML form, the ability to
    include proper diagrams would be immediately useful.

    Current PEP practices allow for reference markers in the form "[1]"
    in the text, and the footnotes/references themselves are listed in a
    section toward the end of the document. There is currently no
    hyperlinking between the reference marker and the footnote/reference
    itself (it would be possible to add this to pep2html.py, but the
    "markup" as it stands is ambiguous and mistakes would be
    inevitable). A PEP with many references (such as this one ;-)
    requires a lot of flipping back and forth. When revising a PEP,
    often new references are added or unused references deleted. It is
    painful to renumber the references, since it has to be done in two
    places and can have a cascading effect (insert a single new
    reference 1, and every other reference has to be renumbered; always
    adding new references to the end is suboptimal). It is easy for
    references to go out of sync.

    PEPs use references for two purposes: simple URL references and
    footnotes. reStructuredText differentiates between the two. A PEP
    might contain references like this:

        Abstract

            This PEP proposes adding frungible doodads [1] to the core.
            It extends PEP 9876 [2] via the BCA [3] mechanism.

        ...

        References and Footnotes

            [1] http://www.example.org/

            [2] PEP 9876, Let's Hope We Never Get Here
                http://peps.python.org/pep-9876/

            [3] "Bogus Complexity Addition"

    Reference 1 is a simple URL reference. Reference 2 is a footnote
    containing text and a URL. Reference 3 is a footnote containing text
    only. Rewritten using reStructuredText, this PEP could look like
    this:

        Abstract
        ========

        This PEP proposes adding `frungible doodads`_ to the core.  It
        extends PEP 9876 [#pep9876]_ via the BCA [#]_ mechanism.

        ...

        References & Footnotes
        ======================

        .. _frungible doodads: http://www.example.org/

        .. [#pep9876] PEP 9876, Let's Hope We Never Get Here

        .. [#] "Bogus Complexity Addition"

    URLs and footnotes can be defined close to their references if
    desired, making them easier to read in the source text, and making
    the PEPs easier to revise. The "References and Footnotes" section
    can be auto-generated with a document tree transform. Footnotes from
    throughout the PEP would be gathered and displayed under a standard
    header. If URL references should likewise be written out explicitly
    (in citation form), another tree transform could be used.

    URL references can be named ("frungible doodads"), and can be
    referenced from multiple places in the document without additional
    definitions. When converted to HTML, references will be replaced
    with inline hyperlinks (HTML <a> tags). The two footnotes are
    automatically numbered, so they will always stay in sync. The first
    footnote also contains an internal reference name, "pep9876", so
    it's easier to see the connection between reference and footnote in
    the source text. Named footnotes can be referenced multiple times,
    maintaining consistent numbering.

    The "#pep9876" footnote could also be written in the form of a
    citation:

        It extends PEP 9876 [PEP9876]_ ...

        .. [PEP9876] PEP 9876, Let's Hope We Never Get Here

    Footnotes are numbered, whereas citations use text for their
    references.

5.  Wouldn't it be better to keep the docstring and PEP proposals
    separate?

    The PEP markup proposal may be removed if it is deemed that there is
    no need for PEP markup, or it could be made into a separate PEP. If
    accepted, PEP 1, PEP Purpose and Guidelines, and PEP 9, Sample PEP
    Template will be updated.

    It seems natural to adopt a single consistent markup standard for
    all uses of structured plaintext in Python, and to propose it all in
    one place.

6.  The existing pep2html.py script converts the existing PEP format to
    HTML. How will the new-format PEPs be converted to HTML?

    A new version of pep2html.py with integrated reStructuredText
    parsing has been completed. The Docutils project supports PEPs with
    a "PEP Reader" component, including all functionality currently in
    pep2html.py (auto-recognition of PEP & RFC references, email
    masking, etc.).

7.  Who's going to convert the existing PEPs to reStructuredText?

    PEP authors or volunteers may convert existing PEPs if they like,
    but there is no requirement to do so. The reStructuredText-based
    PEPs will coexist with the old PEP standard. The pep2html.py
    mentioned in answer 6 processes both old and new standards.

8.  Why use reStructuredText for README and other ancillary files?

    The reasoning given for PEPs in answer 4 above also applies to
    README and other ancillary files. By adopting a standard markup,
    these files can be converted to attractive cross-referenced HTML and
    put up on python.org. Developers of other projects can also take
    advantage of this facility for their own documentation.

9.  Won't the superficial similarity to existing markup conventions
    cause problems, and result in people writing invalid markup (and not
    noticing, because the plaintext looks natural)? How forgiving is
    reStructuredText of "not quite right" markup?

    There will be some mis-steps, as there would be when moving from one
    programming language to another. As with any language, proficiency
    grows with experience. Luckily, reStructuredText is a very little
    language indeed.

    As with any syntax, there is the possibility of syntax errors. It is
    expected that a user will run the processing system over their input
    and check the output for correctness.

    In a strict sense, the reStructuredText parser is very unforgiving
    (as it should be; "In the face of ambiguity, refuse the temptation
    to guess" <20> applies to parsing markup as well as computer
    languages). Here's design goal 3 from An Introduction to
    reStructuredText:

      Unambiguous. The rules for markup must not be open for
      interpretation. For any given input, there should be one and only
      one possible output (including error output).

    While unforgiving, at the same time the parser does try to be
    helpful by producing useful diagnostic output ("system messages").
    The parser reports problems, indicating their level of severity
    (from least to most: debug, info, warning, error, severe). The user
    or the client software can decide on reporting thresholds; they can
    ignore low-level problems or cause high-level problems to bring
    processing to an immediate halt. Problems are reported during the
    parse as well as included in the output, often with two-way links
    between the source of the problem and the system message explaining
    it.

10. Will the docstrings in the Python standard library modules be
    converted to reStructuredText?

    No. Python's library reference documentation is maintained
    separately from the source. Docstrings in the Python standard
    library should not try to duplicate the library reference
    documentation. The current policy for docstrings in the Python
    standard library is that they should be no more than concise hints,
    simple and markup-free (although many do contain ad-hoc implicit
    markup).

11. I want to write all my strings in Unicode. Will anything break?

    The parser fully supports Unicode. Docutils supports arbitrary input
    and output encodings.

12. Why does the community need a new structured text design?

    The existing structured text designs are deficient, for the reasons
    given in "Rationale" above. reStructuredText aims to be a complete
    markup syntax, within the limitations of the "readable plaintext"
    medium.

13. What is wrong with existing documentation methodologies?

    What existing methodologies? For Python docstrings, there is no
    official standard markup format, let alone a documentation
    methodology akin to JavaDoc. The question of methodology is at a
    much higher level than syntax (which this PEP addresses). It is
    potentially much more controversial and difficult to resolve, and is
    intentionally left out of this discussion.

References & Footnotes

Copyright

This document has been placed in the public domain.

Acknowledgements

Some text is borrowed from PEP 216, Docstring Format, by Moshe Zadka.

Special thanks to all members past & present of the Python Doc-SIG.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 End:


text:
PEP: 3003 Title: Python Language Moratorium Version: $Revision$
Last-Modified: $Date$ Author: Brett Cannon, Jesse Noller, Guido van
Rossum Status: Final Type: Process Content-Type: text/x-rst Created:
21-Oct-2009 Post-History: 03-Nov-2009

Abstract

This PEP proposes a temporary moratorium (suspension) of all changes to
the Python language syntax, semantics, and built-ins for a period of at
least two years from the release of Python 3.1. In particular, the
moratorium would include Python 3.2 (to be released 18-24 months after
3.1) but allow Python 3.3 (assuming it is not released prematurely) to
once again include language changes.

This suspension of features is designed to allow non-CPython
implementations to "catch up" to the core implementation of the
language, help ease adoption of Python 3.x, and provide a more stable
base for the community.

Rationale

This idea was proposed by Guido van Rossum on the python-ideas[1]
mailing list. The premise of his email was to slow the alteration of the
Python core syntax, builtins and semantics to allow non-CPython
implementations to catch up to the current state of Python, both 2.x and
3.x.

Python, as a language is more than the core implementation --CPython --
with a rich, mature and vibrant community of implementations, such as
Jython[2], IronPython[3] and PyPy[4] that are a benefit not only to the
community, but to the language itself.

Still others, such as Unladen Swallow[5] (a branch of CPython) seek not
to create an alternative implementation, but rather they seek to enhance
the performance and implementation of CPython itself.

Python 3.x was a large part of the last several years of Python's
development. Its release, as well as a bevy of changes to the language
introduced by it and the previous 2.6.x releases, puts alternative
implementations at a severe disadvantage in "keeping pace" with core
python development.

Additionally, many of the changes put into the recent releases of the
language as implemented by CPython have not yet seen widespread usage by
the general user population. For example, most users are limited to the
version of the interpreter (typically CPython) which comes pre-installed
with their operating system. Most OS vendors are just barely beginning
to ship Python 2.6 -- even fewer are shipping Python 3.x.

As it is expected that Python 2.7 be the effective "end of life" of the
Python 2.x code line, with Python 3.x being the future, it is in the
best interest of Python core development to temporarily suspend the
alteration of the language itself to allow all of these external
entities to catch up and to assist in the adoption of, and migration to,
Python 3.x

Finally, the moratorium is intended to free up cycles within core
development to focus on other issues, such as the CPython interpreter
and improvements therein, the standard library, etc.

This moratorium does not allow for exceptions -- once accepted, any
pending changes to the syntax or semantics of the language will be
postponed until the moratorium is lifted.

This moratorium does not attempt to apply to any other Python
implementation meaning that if desired other implementations may add
features which deviate from the standard implementation.

Details

Cannot Change

-   New built-ins

-

    Language syntax

        The grammar file essentially becomes immutable apart from
        ambiguity fixes.

-

    General language semantics

        The language operates as-is with only specific exemptions (see
        below).

-

    New __future__ imports

        These are explicitly forbidden, as they effectively change the
        language syntax and/or semantics (albeit using a compiler
        directive).

Case-by-Case Exemptions

-

    New methods on built-ins

        The case for adding a method to a built-in object can be made.

-

    Incorrect language semantics

        If the language semantics turn out to be ambiguous or improperly
        implemented based on the intention of the original design then
        the semantics may change.

-

    Language semantics that are difficult to implement

        Because other VMs have not begun implementing Python 3.x
        semantics there is a possibility that certain semantics are too
        difficult to replicate. In those cases they can be changed to
        ease adoption of Python 3.x by the other VMs.

Allowed to Change

-

    C API

        It is entirely acceptable to change the underlying C code of
        CPython as long as other restrictions of this moratorium are not
        broken. E.g. removing the GIL would be fine assuming certain
        operations that are currently atomic remain atomic.

-

    The standard library

        As the standard library is not directly tied to the language
        definition it is not covered by this moratorium.

-

    Backports of 3.x features to 2.x

        The moratorium only affects features that would be new in 3.x.

-

    Import semantics

        For example, PEP 382. After all, import semantics vary between
        Python implementations anyway.

Retroactive

It is important to note that the moratorium covers all changes since the
release of Python 3.1. This rule is intended to avoid features being
rushed or smuggled into the CPython source tree while the moratorium is
being discussed. A review of the NEWS file for the py3k development
branch showed no commits would need to be rolled back in order to meet
this goal.

Extensions

The time period of the moratorium can only be extended through a new
PEP.

Copyright

This document has been placed in the public domain.

References

[1] https://mail.python.org/pipermail/python-ideas/2009-October/006305.html

[2] http://www.jython.org/

[3] http://www.codeplex.com/IronPython

[4] http://codespeak.net/pypy/

[5] http://code.google.com/p/unladen-swallow/

license: Public Domain

authors:
Brett Cannon
Guido van Rossum
Jesse Noller


text:
PEP: 667 Title: Consistent views of namespaces Author: Mark Shannon
<EMAIL_ADDRESS>Tian Gao<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/46631 Status: Accepted Type: Standards
Track Created: 30-Jul-2021 Python-Version: 3.13 Post-History:
20-Aug-2021 Resolution:
https://discuss.python.org/t/pep-667-consistent-views-of-namespaces/46631/25

Abstract

In early versions of Python all namespaces, whether in functions,
classes or modules, were all implemented the same way: as a dictionary.

For performance reasons, the implementation of function namespaces was
changed. Unfortunately this meant that accessing these namespaces
through locals() and frame.f_locals ceased to be consistent and some odd
bugs crept in over the years as threads, generators and coroutines were
added.

This PEP proposes making these namespaces consistent once more.
Modifications to frame.f_locals will always be visible in the underlying
variables. Modifications to local variables will immediately be visible
in frame.f_locals, and they will be consistent regardless of threading
or coroutines.

The locals() function will act the same as it does now for class and
modules scopes. For function scopes it will return an instantaneous
snapshot of the underlying frame.f_locals.

Motivation

The current implementation of locals() and frame.f_locals is slow,
inconsistent and buggy. We want to make it faster, consistent, and most
importantly fix the bugs.

For example:

    class C:
        x = 1
        sys._getframe().f_locals['x'] = 2
        print(x)

prints 2

but:

    def f():
        x = 1
        sys._getframe().f_locals['x'] = 2
        print(x)
    f()

prints 1

This is inconsistent, and confusing. With this PEP both examples would
print 2.

Worse than that, the current behavior can result in strange bugs.

There are no compensating advantages for the current behavior; it is
unreliable and slow.

Rationale

The current implementation of frame.f_locals returns a dictionary that
is created on the fly from the array of local variables. This can result
in the array and dictionary getting out of sync with each other. Writes
to the f_locals may not show up as modifications to local variables.
Writes to local variables can get lost.

By making frame.f_locals return a view on the underlying frame, these
problems go away. frame.f_locals is always in sync with the frame
because it is a view of it, not a copy of it.

Specification

Python

frame.f_locals will return a view object on the frame that implements
the collections.abc.Mapping interface.

For module and class scopes frame.f_locals will be a dictionary, for
function scopes it will be a custom class.

locals() will be defined as:

    def locals():
        frame = sys._getframe(1)
        f_locals = frame.f_locals
        if frame.is_function():
            f_locals = dict(f_locals)
        return f_locals

All writes to the f_locals mapping will be immediately visible in the
underlying variables. All changes to the underlying variables will be
immediately visible in the mapping. The f_locals object will be a full
mapping, and can have arbitrary key-value pairs added to it.

For example:

    def l():
        "Get the locals of caller"
        return sys._getframe(1).f_locals

    def test():
        if 0: y = 1 # Make 'y' a local variable
        x = 1
        l()['x'] = 2
        l()['y'] = 4
        l()['z'] = 5
        y
        print(locals(), x)

test() will print {'x': 2, 'y': 4, 'z': 5} 2.

In Python 3.10, the above will fail with an UnboundLocalError, as the
definition of y by l()['y'] = 4 is lost.

If the second-to-last line were changed from y to z, this would be a
NameError, as it is today. Keys added to frame.f_locals that are not
lexically local variables remain visible in frame.f_locals, but do not
dynamically become local variables.

C-API

Extensions to the API

Three new C-API functions will be added:

    PyObject *PyEval_GetFrameLocals(void)
    PyObject *PyEval_GetFrameGlobals(void)
    PyObject *PyEval_GetFrameBuiltins(void)

PyEval_GetFrameLocals() is equivalent to: locals().
PyEval_GetFrameGlobals() is equivalent to: globals().

All these functions will return a new reference.

Changes to existing APIs

PyFrame_GetLocals(f) is equivalent to f.f_locals, and hence its return
value will change as described above for accessing f.f_locals.

The following C-API functions will be deprecated, as they return
borrowed references:

    PyEval_GetLocals()
    PyEval_GetGlobals()
    PyEval_GetBuiltins()

The following functions should be used instead:

    PyEval_GetFrameLocals()
    PyEval_GetFrameGlobals()
    PyEval_GetFrameBuiltins()

which return new references.

The semantics of PyEval_GetLocals() is changed as it now returns a proxy
for the frame locals in optimized frames, not a dictionary.

The following three functions will become no-ops, and will be
deprecated:

    PyFrame_FastToLocalsWithError()
    PyFrame_FastToLocals()
    PyFrame_LocalsToFast()

Behavior of f_locals for optimized functions

Although f.f_locals behaves as if it were the namespace of the function,
there will be some observable differences. For example,
f.f_locals is f.f_locals may be False.

However f.f_locals == f.f_locals will be True, and all changes to the
underlying variables, by any means, will always be visible.

Backwards Compatibility

Python

The current implementation has many corner cases and oddities. Code that
works around those may need to be changed. Code that uses locals() for
simple templating, or print debugging, will continue to work correctly.
Debuggers and other tools that use f_locals to modify local variables,
will now work correctly, even in the presence of threaded code,
coroutines and generators.

C-API

PyEval_GetLocals

Because PyEval_GetLocals() returns a borrowed reference, it requires the
proxy mapping to be cached on the frame, extending its lifetime and
creating a cycle. PyEval_GetFrameLocals() should be used instead.

This code:

    locals = PyEval_GetLocals();
    if (locals == NULL) {
        goto error_handler;
    }
    Py_INCREF(locals);

should be replaced with:

    locals = PyEval_GetFrameLocals();
    if (locals == NULL) {
        goto error_handler;
    }

Implementation

Each read of frame.f_locals will create a new proxy object that gives
the appearance of being the mapping of local (including cell and free)
variable names to the values of those local variables.

A possible implementation is sketched out below. All attributes that
start with an underscore are invisible and cannot be accessed directly.
They serve only to illustrate the proposed design.

    NULL: Object # NULL is a singleton representing the absence of a value.

    class CodeType:

        _name_to_offset_mapping_impl: dict | NULL
        _cells: frozenset # Set of indexes of cell and free variables
        ...

        def __init__(self, ...):
            self._name_to_offset_mapping_impl = NULL
            self._variable_names = deduplicate(
                self.co_varnames + self.co_cellvars + self.co_freevars
            )
            ...

        @property
        def _name_to_offset_mapping(self):
            "Mapping of names to offsets in local variable array."
            if self._name_to_offset_mapping_impl is NULL:
                self._name_to_offset_mapping_impl = {
                    name: index for (index, name) in enumerate(self._variable_names)
                }
            return self._name_to_offset_mapping_impl

    class FrameType:

        _locals : array[Object] # The values of the local variables, items may be NULL.
        _extra_locals: dict | NULL # Dictionary for storing extra locals not in _locals.
        _locals_cache: FrameLocalsProxy | NULL # required to support PyEval_GetLocals()

        def __init__(self, ...):
            self._extra_locals = NULL
            self._locals_cache = NULL
            ...

        @property
        def f_locals(self):
            return FrameLocalsProxy(self)

    class FrameLocalsProxy:
        "Implements collections.MutableMapping."

        __slots__ = ("_frame", )

        def __init__(self, frame:FrameType):
            self._frame = frame

        def __getitem__(self, name):
            f = self._frame
            co = f.f_code
            if name in co._name_to_offset_mapping:
                index = co._name_to_offset_mapping[name]
                val = f._locals[index]
                if val is NULL:
                    raise KeyError(name)
                if index in co._cells
                    val = val.cell_contents
                    if val is NULL:
                        raise KeyError(name)
                return val
            else:
                if f._extra_locals is NULL:
                    raise KeyError(name)
                return f._extra_locals[name]

        def __setitem__(self, name, value):
            f = self._frame
            co = f.f_code
            if name in co._name_to_offset_mapping:
                index = co._name_to_offset_mapping[name]
                kind = co._local_kinds[index]
                if index in co._cells
                    cell = f._locals[index]
                    cell.cell_contents = val
                else:
                    f._locals[index] = val
            else:
                if f._extra_locals is NULL:
                    f._extra_locals = {}
                f._extra_locals[name] = val

        def __iter__(self):
            f = self._frame
            co = f.f_code
            yield from iter(f._extra_locals)
            for index, name in enumerate(co._variable_names):
                val = f._locals[index]
                if val is NULL:
                    continue
                if index in co._cells:
                    val = val.cell_contents
                    if val is NULL:
                        continue
                yield name

        def __contains__(self, item):
            f = self._frame
            if item in f._extra_locals:
                return True
            return item in co._variable_names

        def __len__(self):
            f = self._frame
            co = f.f_code
            res = 0
            for index, _ in enumerate(co._variable_names):
                val = f._locals[index]
                if val is NULL:
                    continue
                if index in co._cells:
                    if val.cell_contents is NULL:
                        continue
                res += 1
            return len(self._extra_locals) + res

C API

PyEval_GetLocals() will be implemented roughly as follows:

    PyObject *PyEval_GetLocals(void) {
        PyFrameObject * = ...; // Get the current frame.
        if (frame->_locals_cache == NULL) {
            frame->_locals_cache = PyEval_GetFrameLocals();
        }
        return frame->_locals_cache;
    }

As with all functions that return a borrowed reference, care must be
taken to ensure that the reference is not used beyond the lifetime of
the object.

Impact on PEP 709 inlined comprehensions

For inlined comprehensions within a function, locals() currently behaves
the same inside or outside of the comprehension, and this will not
change. The behavior of locals() inside functions will generally change
as specified in the rest of this PEP.

For inlined comprehensions at module or class scope, currently calling
locals() within the inlined comprehension returns a new dictionary for
each call. This PEP will make locals() within a function also always
return a new dictionary for each call, improving consistency; class or
module scope inlined comprehensions will appear to behave as if the
inlined comprehension is still a distinct function.

Comparison with PEP 558

This PEP and PEP 558 share a common goal: to make the semantics of
locals() and frame.f_locals() intelligible, and their operation
reliable.

The key difference between this PEP and PEP 558 is that PEP 558 keeps an
internal copy of the local variables, whereas this PEP does not.

PEP 558 does not specify exactly when the internal copy is updated,
making the behavior of PEP 558 impossible to reason about.

Implementation

The implementation is in development as a draft pull request on GitHub.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 223 Title: Change the Meaning of \x Escapes Author: Tim Peters
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 20-Aug-2000 Python-Version: 2.0 Post-History:
23-Aug-2000

Abstract

Change \x escapes, in both 8-bit and Unicode strings, to consume exactly
the two hex digits following. The proposal views this as correcting an
original design flaw, leading to clearer expression in all flavors of
string, a cleaner Unicode story, better compatibility with Perl regular
expressions, and with minimal risk to existing code.

Syntax

The syntax of \x escapes, in all flavors of non-raw strings, becomes :

    \xhh

where h is a hex digit (0-9, a-f, A-F). The exact syntax in 1.5.2 is not
clearly specified in the Reference Manual; it says :

    \xhh...

implying "two or more" hex digits, but one-digit forms are also accepted
by the 1.5.2 compiler, and a plain \x is "expanded" to itself (i.e., a
backslash followed by the letter x). It's unclear whether the Reference
Manual intended either of the 1-digit or 0-digit behaviors.

Semantics

In an 8-bit non-raw string, :

    \xij

expands to the character :

    chr(int(ij, 16))

Note that this is the same as in 1.6 and before.

In a Unicode string, :

    \xij

acts the same as :

    \u00ij

i.e. it expands to the obvious Latin-1 character from the initial
segment of the Unicode space.

An \x not followed by at least two hex digits is a compile-time error,
specifically ValueError in 8-bit strings, and UnicodeError (a subclass
of ValueError) in Unicode strings. Note that if an \x is followed by
more than two hex digits, only the first two are "consumed". In 1.6 and
before all but the last two were silently ignored.

Example

In 1.5.2:

    >>> "\x123465"  # same as "\x65"
    'e'
    >>> "\x65"
    'e'
    >>> "\x1"
    '\001'
    >>> "\x\x"
    '\\x\\x'
    >>>

In 2.0:

    >>> "\x123465" # \x12 -> \022, "3456" left alone
    '\0223456'
    >>> "\x65"
    'e'
    >>> "\x1"
    [ValueError is raised]
    >>> "\x\x"
    [ValueError is raised]
    >>>

History and Rationale

\x escapes were introduced in C as a way to specify variable-width
character encodings. Exactly which encodings those were, and how many
hex digits they required, was left up to each implementation. The
language simply stated that \x "consumed" all hex digits following, and
left the meaning up to each implementation. So, in effect, \x in C is a
standard hook to supply platform-defined behavior.

Because Python explicitly aims at platform independence, the \x escape
in Python (up to and including 1.6) has been treated the same way across
all platforms: all except the last two hex digits were silently ignored.
So the only actual use for \x escapes in Python was to specify a single
byte using hex notation.

Larry Wall appears to have realized that this was the only real use for
\x escapes in a platform-independent language, as the proposed rule for
Python 2.0 is in fact what Perl has done from the start (although you
need to run in Perl -w mode to get warned about \x escapes with fewer
than 2 hex digits following -- it's clearly more Pythonic to insist on 2
all the time).

When Unicode strings were introduced to Python, \x was generalized so as
to ignore all but the last four hex digits in Unicode strings. This
caused a technical difficulty for the new regular expression engine: SRE
tries very hard to allow mixing 8-bit and Unicode patterns and strings
in intuitive ways, and it no longer had any way to guess what, for
example, r"\x123456" should mean as a pattern: is it asking to match the
8-bit character \x56 or the Unicode character \u3456?

There are hacky ways to guess, but it doesn't end there. The ISO C99
standard also introduces 8-digit \U12345678 escapes to cover the entire
ISO 10646 character space, and it's also desired that Python 2 support
that from the start. But then what are \x escapes supposed to mean? Do
they ignore all but the last eight hex digits then? And if less than 8
following in a Unicode string, all but the last 4? And if less than 4,
all but the last 2?

This was getting messier by the minute, and the proposal cuts the
Gordian knot by making \x simpler instead of more complicated. Note that
the 4-digit generalization to \xijkl in Unicode strings was also
redundant, because it meant exactly the same thing as \uijkl in Unicode
strings. It's more Pythonic to have just one obvious way to specify a
Unicode character via hex notation.

Development and Discussion

The proposal was worked out among Guido van Rossum, Fredrik Lundh and
Tim Peters in email. It was subsequently explained and discussed on
Python-Dev under subject "Go x yourself"[1], starting 2000-08-03.
Response was overwhelmingly positive; no objections were raised.

Backward Compatibility

Changing the meaning of \x escapes does carry risk of breaking existing
code, although no instances of incompatibility have yet been discovered.
The risk is believed to be minimal.

Tim Peters verified that, except for pieces of the standard test suite
deliberately provoking end cases, there are no instances of \xabcdef...
with fewer or more than 2 hex digits following, in either the Python CVS
development tree, or in assorted Python packages sitting on his machine.

It's unlikely there are any with fewer than 2, because the Reference
Manual implied they weren't legal (although this is debatable!). If
there are any with more than 2, Guido is ready to argue they were buggy
anyway <0.9 wink>.

Guido reported that the O'Reilly Python books already document that
Python works the proposed way, likely due to their Perl editing heritage
(as above, Perl worked (very close to) the proposed way from its start).

Finn Bock reported that what JPython does with \x escapes is
unpredictable today. This proposal gives a clear meaning that can be
consistently and easily implemented across all Python implementations.

Effects on Other Tools

Believed to be none. The candidates for breakage would mostly be parsing
tools, but the author knows of none that worry about the internal
structure of Python strings beyond the approximation "when there's a
backslash, swallow the next character". Tim Peters checked
python-mode.el, the std tokenize.py and pyclbr.py, and the IDLE syntax
coloring subsystem, and believes there's no need to change any of them.
Tools like tabnanny.py and checkappend.py inherit their immunity from
tokenize.py.

Reference Implementation

The code changes are so simple that a separate patch will not be
produced. Fredrik Lundh is writing the code, is an expert in the area,
and will simply check the changes in before 2.0b1 is released.

BDFL Pronouncements

Yes, ValueError, not SyntaxError. "Problems with literal interpretations
traditionally raise 'runtime' exceptions rather than syntax errors."

References

Copyright

This document has been placed in the public domain.

[1] Tim Peters, Go x yourself
https://mail.python.org/pipermail/python-dev/2000-August/007825.html


text:
PEP: 731 Title: C API Working Group Charter Author: Guido van Rossum
<EMAIL_ADDRESS>Petr Viktorin<EMAIL_ADDRESS>Victor Stinner
<EMAIL_ADDRESS>Steve Dower<EMAIL_ADDRESS>Irit
Katriel<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pep-731-c-api-working-group-charter/36117
Status: Active Type: Process Topic: Governance Created: 11-Oct-2023
Post-History: 13-Oct-2023, 23-May-2024, 19-Jun-2024 Resolution:
https://discuss.python.org/t/pep-731-c-api-working-group-charter/36117/12

Abstract

This PEP proposes to establish the C API Working Group: a small
committee of Python core developers responsible for overseeing and
coordinating the development and maintenance of the Python C API.

The working group will maintain documentation, test suites and tooling
related to Python's C API. As delegated by the Steering Council it is
the deciding body for changes to the C API, from the addition or removal
of individual API functions, types, etc., to the acceptance of new
designs of a more or less radical nature.

The working group's mandate is to represent the interests of all Python
users, but especially all maintainers of code that uses Python's C API,
whether in the context of CPython, using an alternate Python
implementation, or using a binding framework for other programming
languages (such as C++ and Rust).

The working group serves at the pleasure of the Python Steering Council.
This document serves as the working group's charter.

Epigraph

KEEPER
Stop! Who would cross the Bridge of Death must answer me these questions
three, ere the other side he see.

1.  What was Python named after?
2.  What was Python 2's EOL?
3.  What is the optimal strategy to evolve the CPython C API?

LANCELOT
Auuuuuuuugh!

Motivation

Despite many discussions and in-person meetings at core developer
sprints and Language Summits, and a thorough inventory of the problems
and stakeholders of the C API, no consensus has been reached about many
contentious issues, including, but not limited to:

-   Conventions for designing new API functions;
-   How to deal with compatibility;
-   What's the best strategy for handling errors;
-   The future of the Stable ABI and the Limited API;
-   Whether to switch to a handle-based API convention (and how).

The general feeling is that there are too many stakeholders, proposals,
requirements, constraints, and conventions, to make progress without
having a small trusted group of deciders.

At the 2023 Language Summit in Salt Lake City it was decided to start
work on an inventory of problems. At the 2023 core developer sprint in
Brno this work is more or less finished and after a discussion it
appeared that the next step is to establish a working group to ensure
that we're not stymied forever.

The Steering Council has indicated its desire to delegate decisions
about the C API to such a working group, anticipating its formal
establishment.

Specification

We propose the creation of a new group, the C API Working Group. This
group will be responsible for overseeing and coordinating the
development and maintenance of the Python C API. It will do this by
establishing the principles underpinning the work and publishing
guidelines that the core developers can refer to.

The "operations and process" section below describes how the working
group operates and how it is governed.

Members

The members of the working group are:

-   Erlend Aasland
-   Michael Droettboom
-   Petr Viktorin
-   Serhiy Storchaka
-   Steve Dower
-   Victor Stinner

Mandate

The working group's mandate is to ensure that the Python C API is
suitable for all users of and contributors to the API, without unduly
preferencing one group over another. The working group will identify
exemplar stakeholders, their needs and preferences, and will determine a
plan for meeting these needs equitably and sustainably. It will oversee
execution of the plan.

Operations and process

The working group has at least three members, comprised of prominent
Python core developers. The members should consider the needs of the
various stakeholders carefully.

The Steering Council appoints the initial working group. There is no
term limit for working group members. Working group members may resign
their position at any time, for any reason. There is an expectation that
the membership will change over time.

To determine replacements, nominations will be collected from the core
developer community. Self-nominations are allowed. The existing working
group will then decide the replacement member(s) from the nominees. The
expectation is that this will be done by fiat, but the working group can
choose a replacement by any means they see fit, including a vote.

The working group remains accountable to the Steering Council. At any
point, for any reason, the Steering Council could (publicly or
privately) make a specific change or request a non-specific change to
the composition of the working group.

We acknowledge that this is not a particularly democratic structure and
puts a lot of faith in the working group. However, the Python community
has a long history of success with structures that are not fully
democratic! We believe that self-governance, cycling of membership, and
accountability to the Steering Council will be sufficient to ensure that
the C API workgroup is meeting the needs of the community.

The working group may operate primarily through reviews of GitHub issues
and PRs. Regular meetings are likely not necessary, but the working
group may set up video calls, a private chat, or whatever other
mechanism they decide upon internally.

The working group should aim for transparency, posting all decisions
publicly on discuss.python.org, with a rationale if possible. Before
making a decision, the working group should give all interested
community members (as examples of different categories of stakeholders)
a chance to weigh in. There should be at least a week between the start
of a discussion and the working group's decision.

Relationship with the Steering Council

Just like today, the Python Steering Council remains responsible for the
overall direction of the Python C API and continues to decide on PEPs
related to the C API, using the standard PEP review process (community
discussion, etc.). The C API working group provides written opinions and
recommendations to the Steering Council on PEPs related to the C API.

However, the working group can make smaller C API changes directly. The
Steering Council may also choose to delegate decisions on some PEPs to
the working group (exactly as any other PEP delegation).

Amendments

This PEP serves as a charter for the working group. Changes to its
operation can be made either through a new PEP or through a change to
this PEP. In either case, the change will be decided upon by the
Steering Council after discussion in the community.

Contact

To ask the C API Working Group for a decision, community members may
open an issue in the capi-workgroup/decisions repository.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.


text:
PEP: 355 Title: Path - Object oriented filesystem paths Version:
$Revision$ Last-Modified: $Date$ Author: Björn Lindqvist
<EMAIL_ADDRESS>Status: Rejected Type: Standards Track Content-Type:
text/x-rst Created: 24-Jan-2006 Python-Version: 2.5 Post-History:

Rejection Notice

This PEP has been rejected (in this form). The proposed path class is
the ultimate kitchen sink; but the notion that it's better to implement
all functionality that uses a path as a method on a single class is an
anti-pattern. (E.g. why not open()? Or execfile()?) Subclassing from str
is a particularly bad idea; many string operations make no sense when
applied to a path. This PEP has lingered, and while the discussion
flares up from time to time, it's time to put this PEP out of its
misery. A less far-fetched proposal might be more palatable.

Abstract

This PEP describes a new class, Path, to be added to the os module, for
handling paths in an object oriented fashion. The "weak" deprecation of
various related functions is also discussed and recommended.

Background

The ideas expressed in this PEP are not recent, but have been debated in
the Python community for many years. Many have felt that the API for
manipulating file paths as offered in the os.path module is inadequate.
The first proposal for a Path object was raised by Just van Rossum on
python-dev in 2001[1]. In 2003, Jason Orendorff released version 1.0 of
the "path module" which was the first public implementation that used
objects to represent paths[2].

The path module quickly became very popular and numerous attempts were
made to get the path module included in the Python standard
library;[3],[4],[5],[6].

This PEP summarizes the ideas and suggestions people have expressed
about the path module and proposes that a modified version should be
included in the standard library.

Motivation

Dealing with filesystem paths is a common task in any programming
language, and very common in a high-level language like Python. Good
support for this task is needed, because:

-   Almost every program uses paths to access files. It makes sense that
    a task, that is so often performed, should be as intuitive and as
    easy to perform as possible.
-   It makes Python an even better replacement language for
    over-complicated shell scripts.

Currently, Python has a large number of different functions scattered
over half a dozen modules for handling paths. This makes it hard for
newbies and experienced developers to choose the right method.

The Path class provides the following enhancements over the current
common practice:

-   One "unified" object provides all functionality from previous
    functions.

-   Subclassability - the Path object can be extended to support paths
    other than filesystem paths. The programmer does not need to learn a
    new API, but can reuse their knowledge of Path to deal with the
    extended class.

-   With all related functionality in one place, the right approach is
    easier to learn as one does not have to hunt through many different
    modules for the right functions.

-   Python is an object oriented language. Just like files, datetimes
    and sockets are objects so are paths, they are not merely strings to
    be passed to functions. Path objects is inherently a pythonic idea.

-   Path takes advantage of properties. Properties make for more
    readable code:

        if imgpath.ext == 'jpg':
            jpegdecode(imgpath)

    Is better than:

        if os.path.splitexit(imgpath)[1] == 'jpg':
            jpegdecode(imgpath)

Rationale

The following points summarize the design:

-   Path extends from string, therefore all code which expects string
    pathnames need not be modified and no existing code will break.
-   A Path object can be created either by using the classmethod
    Path.cwd, by instantiating the class with a string representing a
    path or by using the default constructor which is equivalent to
    Path(".").
-   Path provides common pathname manipulation, pattern expansion,
    pattern matching and other high-level file operations including
    copying. Basically Path provides everything path-related except the
    manipulation of file contents, for which file objects are better
    suited.
-   Platform incompatibilities are dealt with by not instantiating
    system specific methods.

Specification

This class defines the following public interface (docstrings have been
extracted from the reference implementation, and shortened for brevity;
see the reference implementation for more detail):

    class Path(str):

        # Special Python methods:
        def __new__(cls, *args) => Path
            """
            Creates a new path object concatenating the *args.  *args
            may only contain Path objects or strings.  If *args is
            empty, Path(os.curdir) is created.
            """
        def __repr__(self): ...
        def __add__(self, more): ...
        def __radd__(self, other): ...

        # Alternative constructor.
        def cwd(cls): ...

        # Operations on path strings:
        def abspath(self) => Path
            """Returns the absolute path of self as a new Path object."""
        def normcase(self): ...
        def normpath(self): ...
        def realpath(self): ...
        def expanduser(self): ...
        def expandvars(self): ...
        def basename(self): ...
        def expand(self): ...
        def splitpath(self) => (Path, str)
            """p.splitpath() -> Return (p.parent, p.name)."""
        def stripext(self) => Path
            """p.stripext() -> Remove one file extension from the path."""
        def splitunc(self): ...  # See footnote [1]
        def splitall(self): ...
        def relpath(self): ...
        def relpathto(self, dest): ...

        # Properties about the path:
        parent => Path
            """This Path's parent directory as a new path object."""
        name => str
            """The name of this file or directory without the full path."""
        ext => str
            """
            The file extension or an empty string if Path refers to a
            file without an extension or a directory.
            """
        drive => str
            """
            The drive specifier.  Always empty on systems that don't
            use drive specifiers.
            """
        namebase => str
            """
            The same as path.name, but with one file extension
            stripped off.
            """
        uncshare[1]

        # Operations that return lists of paths:
        def listdir(self, pattern = None): ...
        def dirs(self, pattern = None): ...
        def files(self, pattern = None): ...
        def walk(self, pattern = None): ...
        def walkdirs(self, pattern = None): ...
        def walkfiles(self, pattern = None): ...
        def match(self, pattern) => bool
            """Returns True if self.name matches the given pattern."""

        def matchcase(self, pattern) => bool
            """
            Like match() but is guaranteed to be case sensitive even
            on platforms with case insensitive filesystems.
            """
        def glob(self, pattern):

        # Methods for retrieving information about the filesystem
        # path:
        def exists(self): ...
        def isabs(self): ...
        def isdir(self): ...
        def isfile(self): ...
        def islink(self): ...
        def ismount(self): ...
        def samefile(self, other): ...  # See footnote [1]
        def atime(self): ...
            """Last access time of the file."""
        def mtime(self): ...
            """Last-modified time of the file."""
        def ctime(self): ...
            """
            Return the system's ctime which, on some systems (like
            Unix) is the time of the last change, and, on others (like
            Windows), is the creation time for path.
            """
        def size(self): ...
        def access(self, mode): ...  # See footnote [1]
        def stat(self): ...
        def lstat(self): ...
        def statvfs(self): ...  # See footnote [1]
        def pathconf(self, name): ...  # See footnote [1]

        # Methods for manipulating information about the filesystem
        # path.
        def utime(self, times) => None
        def chmod(self, mode) => None
        def chown(self, uid, gid) => None # See footnote [1]
        def rename(self, new) => None
        def renames(self, new) => None

        # Create/delete operations on directories
        def mkdir(self, mode = 0777): ...
        def makedirs(self, mode = 0777): ...
        def rmdir(self): ...
        def removedirs(self): ...

        # Modifying operations on files
        def touch(self): ...
        def remove(self): ...
        def unlink(self): ...

        # Modifying operations on links
        def link(self, newpath): ...
        def symlink(self, newlink): ...
        def readlink(self): ...
        def readlinkabs(self): ...

        # High-level functions from shutil
        def copyfile(self, dst): ...
        def copymode(self, dst): ...
        def copystat(self, dst): ...
        def copy(self, dst): ...
        def copy2(self, dst): ...
        def copytree(self, dst, symlinks = True): ...
        def move(self, dst): ...
        def rmtree(self, ignore_errors = False, onerror = None): ...

        # Special stuff from os
        def chroot(self): ...  # See footnote [1]
        def startfile(self): ...  # See footnote [1]

Replacing older functions with the Path class

In this section, "a ==> b" means that b can be used as a replacement for
a.

In the following examples, we assume that the Path class is imported
with from path import Path.

-   Replacing os.path.join:

        os.path.join(os.getcwd(), "foobar")
        ==>
        Path(Path.cwd(), "foobar")

        os.path.join("foo", "bar", "baz")
        ==>
        Path("foo", "bar", "baz")

-   Replacing os.path.splitext:

        fname = "Python2.4.tar.gz"
        os.path.splitext(fname)[1]
        ==>
        fname = Path("Python2.4.tar.gz")
        fname.ext

    Or if you want both parts:

        fname = "Python2.4.tar.gz"
        base, ext = os.path.splitext(fname)
        ==>
        fname = Path("Python2.4.tar.gz")
        base, ext = fname.namebase, fname.extx

-   Replacing glob.glob:

        lib_dir = "/lib"
        libs = glob.glob(os.path.join(lib_dir, "*s.o"))
        ==>
        lib_dir = Path("/lib")
        libs = lib_dir.files("*.so")

Deprecations

Introducing this module to the standard library introduces a need for
the "weak" deprecation of a number of existing modules and functions.
These modules and functions are so widely used that they cannot be truly
deprecated, as in generating DeprecationWarning. Here "weak deprecation"
means notes in the documentation only.

The table below lists the existing functionality that should be
deprecated.

  Path method/property   Deprecates function
  ---------------------- ---------------------------------
  normcase()             os.path.normcase()
  normpath()             os.path.normpath()
  realpath()             os.path.realpath()
  expanduser()           os.path.expanduser()
  expandvars()           os.path.expandvars()
  parent                 os.path.dirname()
  name                   os.path.basename()
  splitpath()            os.path.split()
  drive                  os.path.splitdrive()
  ext                    os.path.splitext()
  splitunc()             os.path.splitunc()
  __new__()              os.path.join(), os.curdir
  listdir()              os.listdir() [fnmatch.filter()]
  match()                fnmatch.fnmatch()
  matchcase()            fnmatch.fnmatchcase()
  glob()                 glob.glob()
  exists()               os.path.exists()
  isabs()                os.path.isabs()
  isdir()                os.path.isdir()
  isfile()               os.path.isfile()
  islink()               os.path.islink()
  ismount()              os.path.ismount()
  samefile()             os.path.samefile()
  atime()                os.path.getatime()
  ctime()                os.path.getctime()
  mtime()                os.path.getmtime()
  size()                 os.path.getsize()
  cwd()                  os.getcwd()
  access()               os.access()
  stat()                 os.stat()
  lstat()                os.lstat()
  statvfs()              os.statvfs()
  pathconf()             os.pathconf()
  utime()                os.utime()
  chmod()                os.chmod()
  chown()                os.chown()
  rename()               os.rename()
  renames()              os.renames()
  mkdir()                os.mkdir()
  makedirs()             os.makedirs()
  rmdir()                os.rmdir()
  removedirs()           os.removedirs()
  remove()               os.remove()
  unlink()               os.unlink()
  link()                 os.link()
  symlink()              os.symlink()
  readlink()             os.readlink()
  chroot()               os.chroot()
  startfile()            os.startfile()
  copyfile()             shutil.copyfile()
  copymode()             shutil.copymode()
  copystat()             shutil.copystat()
  copy()                 shutil.copy()
  copy2()                shutil.copy2()
  copytree()             shutil.copytree()
  move()                 shutil.move()
  rmtree()               shutil.rmtree()

The Path class deprecates the whole of os.path, shutil, fnmatch and
glob. A big chunk of os is also deprecated.

Closed Issues

A number contentious issues have been resolved since this PEP first
appeared on python-dev:

-   The __div__() method was removed. Overloading the / (division)
    operator may be "too much magic" and make path concatenation appear
    to be division. The method can always be re-added later if the BDFL
    so desires. In its place, __new__() got an *args argument that
    accepts both Path and string objects. The *args are concatenated
    with os.path.join() which is used to construct the Path object.
    These changes obsoleted the problematic joinpath() method which was
    removed.

-   The methods and the properties getatime()/atime, getctime()/ctime,
    getmtime()/mtime and getsize()/size duplicated each other. These
    methods and properties have been merged to atime(), ctime(), mtime()
    and size(). The reason they are not properties instead, is because
    there is a possibility that they may change unexpectedly. The
    following example is not guaranteed to always pass the assertion:

        p = Path("foobar")
        s = p.size()
        assert p.size() == s

Open Issues

Some functionality of Jason Orendorff's path module have been omitted:

-   Function for opening a path - better handled by the builtin open().

-   Functions for reading and writing whole files - better handled by
    file objects' own read() and write() methods.

-   A chdir() function may be a worthy inclusion.

-   A deprecation schedule needs to be set up. How much functionality
    should Path implement? How much of existing functionality should it
    deprecate and when?

-   The name obviously has to be either "path" or "Path," but where
    should it live? In its own module or in os?

-   Due to Path subclassing either str or unicode, the following
    non-magic, public methods are available on Path objects:

        capitalize(), center(), count(), decode(), encode(),
        endswith(), expandtabs(), find(), index(), isalnum(),
        isalpha(), isdigit(), islower(), isspace(), istitle(),
        isupper(), join(), ljust(), lower(), lstrip(), replace(),
        rfind(), rindex(), rjust(), rsplit(), rstrip(), split(),
        splitlines(), startswith(), strip(), swapcase(), title(),
        translate(), upper(), zfill()

    On python-dev it has been argued whether this inheritance is sane or
    not. Most persons debating said that most string methods doesn't
    make sense in the context of filesystem paths --they are just dead
    weight. The other position, also argued on python-dev, is that
    inheriting from string is very convenient because it allows code to
    "just work" with Path objects without having to be adapted for them.

    One of the problems is that at the Python level, there is no way to
    make an object "string-like enough," so that it can be passed to the
    builtin function open() (and other builtins expecting a string or
    buffer), unless the object inherits from either str or unicode.
    Therefore, to not inherit from string requires changes in CPython's
    core.

The functions and modules that this new module is trying to replace
(os.path, shutil, fnmatch, glob and parts of os) are expected to be
available in future Python versions for a long time, to preserve
backwards compatibility.

Reference Implementation

Currently, the Path class is implemented as a thin wrapper around the
standard library modules fnmatch, glob, os, os.path and shutil. The
intention of this PEP is to move functionality from the aforementioned
modules to Path while they are being deprecated.

For more detail and an implementation see:

  http://wiki.python.org/moin/PathModule

Examples

In this section, "a ==> b" means that b can be used as a replacement for
a.

-   Make all python files in the a directory executable:

        DIR = '/usr/home/guido/bin'
        for f in os.listdir(DIR):
            if f.endswith('.py'):
                path = os.path.join(DIR, f)
                os.chmod(path, 0755)
        ==>
        for f in Path('/usr/home/guido/bin').files("*.py"):
            f.chmod(0755)

-   Delete emacs backup files:

        def delete_backups(arg, dirname, names):
            for name in names:
                if name.endswith('~'):
                    os.remove(os.path.join(dirname, name))
        os.path.walk(os.environ['HOME'], delete_backups, None)
        ==>
        d = Path(os.environ['HOME'])
        for f in d.walkfiles('*~'):
            f.remove()

-   Finding the relative path to a file:

        b = Path('/users/peter/')
        a = Path('/users/peter/synergy/tiki.txt')
        a.relpathto(b)

-   Splitting a path into directory and filename:

        os.path.split("/path/to/foo/bar.txt")
        ==>
        Path("/path/to/foo/bar.txt").splitpath()

-   List all Python scripts in the current directory tree:

        list(Path().walkfiles("*.py"))

References and Footnotes

[1] Method is not guaranteed to be available on all platforms.

Copyright

This document has been placed in the public domain.

[1] "(idea) subclassable string: path object?", van Rossum, 2001
https://mail.python.org/pipermail/python-dev/2001-August/016663.html

[2] "path module v1.0 released", Orendorff, 2003
https://mail.python.org/pipermail/python-announce-list/2003-January/001984.html

[3] "Some RFE for review", Birkenfeld, 2005
https://mail.python.org/pipermail/python-dev/2005-June/054438.html

[4] "path module", Orendorff, 2003
https://mail.python.org/pipermail/python-list/2003-July/174289.html

[5] "PRE-PEP: new Path class", Roth, 2004
https://mail.python.org/pipermail/python-list/2004-January/201672.html

[6] http://wiki.python.org/moin/PathClass