text:
PEP: 258 Title: Docutils Design Specification Version: $Revision$
Last-Modified: $Date$ Author: David Goodger<EMAIL_ADDRESS>Discussions-To<EMAIL_ADDRESS>Status: Rejected Type: Standards
Track Content-Type: text/x-rst Requires: 256, 257 Created: 31-May-2001
Post-History: 13-Jun-2001

Rejection Notice

While this may serve as an interesting design document for the
now-independent docutils, it is no longer slated for inclusion in the
standard library.

Abstract

This PEP documents design issues and implementation details for
Docutils, a Python Docstring Processing System (DPS). The rationale and
high-level concepts of a DPS are documented in PEP 256, "Docstring
Processing System Framework". Also see PEP 256 for a "Road Map to the
Docstring PEPs".

Docutils is being designed modularly so that any of its components can
be replaced easily. In addition, Docutils is not limited to the
processing of Python docstrings; it processes standalone documents as
well, in several contexts.

No changes to the core Python language are required by this PEP. Its
deliverables consist of a package for the standard library and its
documentation.

Specification

Docutils Project Model

Project components and data flow:

    +---------------------------+
    |        Docutils:          |
    | docutils.core.Publisher,  |
    | docutils.core.publish_*() |
    +---------------------------+
     /            |            \
    /             |             \
    1,3,5   /        6     |              \ 7
    +--------+       +-------------+       +--------+
    | READER | ----> | TRANSFORMER | ====> | WRITER |
    +--------+       +-------------+       +--------+
    /     \\                                  |
    /       \\                                 |
    2    /      4  \\                             8  |
    +-------+   +--------+                        +--------+
    | INPUT |   | PARSER |                        | OUTPUT |
    +-------+   +--------+                        +--------+

The numbers above each component indicate the path a document's data
takes. Double-width lines between Reader & Parser and between
Transformer & Writer indicate that data sent along these paths should be
standard (pure & unextended) Docutils doc trees. Single-width lines
signify that internal tree extensions or completely unrelated
representations are possible, but they must be supported at both ends.

Publisher

The docutils.core module contains a "Publisher" facade class and several
convenience functions: "publish_cmdline()" (for command-line front
ends), "publish_file()" (for programmatic use with file-like I/O), and
"publish_string()" (for programmatic use with string I/O). The Publisher
class encapsulates the high-level logic of a Docutils system. The
Publisher class has overall responsibility for processing, controlled by
the Publisher.publish() method:

1.  Set up internal settings (may include config files & command-line
    options) and I/O objects.
2.  Call the Reader object to read data from the source Input object and
    parse the data with the Parser object. A document object is
    returned.
3.  Set up and apply transforms via the Transformer object attached to
    the document.
4.  Call the Writer object which translates the document to the final
    output format and writes the formatted data to the destination
    Output object. Depending on the Output object, the output may be
    returned from the Writer, and then from the publish() method.

Calling the "publish" function (or instantiating a "Publisher" object)
with component names will result in default behavior. For custom
behavior (customizing component settings), create custom component
objects first, and pass them to the Publisher or publish_* convenience
functions.

Readers

Readers understand the input context (where the data is coming from),
send the whole input or discrete "chunks" to the parser, and provide the
context to bind the chunks together back into a cohesive whole.

Each reader is a module or package exporting a "Reader" class with a
"read" method. The base "Reader" class can be found in the
docutils/readers/__init__.py module.

Most Readers will have to be told what parser to use. So far (see the
list of examples below), only the Python Source Reader ("PySource";
still incomplete) will be able to determine the parser on its own.

Responsibilities:

-   Get input text from the source I/O.
-   Pass the input text to the parser, along with a fresh document tree
    root.

Examples:

-   Standalone (Raw/Plain): Just read a text file and process it. The
    reader needs to be told which parser to use.

    The "Standalone Reader" has been implemented in module
    docutils.readers.standalone.

-   Python Source: See Python Source Reader below. This Reader is
    currently in development in the Docutils sandbox.

-   Email: 822 headers, quoted excerpts, signatures, MIME parts.

-   PEP: 822 headers, "PEP xxxx" and "RFC xxxx" conversion to URIs. The
    "PEP Reader" has been implemented in module docutils.readers.pep;
    see PEP 287 and PEP 12.

-   Wiki: Global reference lookups of "wiki links" incorporated into
    transforms. (CamelCase only or unrestricted?) Lazy indentation?

-   Web Page: As standalone, but recognize meta fields as meta tags.
    Support for templates of some sort? (After <body>, before </body>?)

-   FAQ: Structured "question & answer(s)" constructs.

-   Compound document: Merge chapters into a book. Master manifest file?

Parsers

Parsers analyze their input and produce a Docutils document tree. They
don't know or care anything about the source or destination of the data.

Each input parser is a module or package exporting a "Parser" class with
a "parse" method. The base "Parser" class can be found in the
docutils/parsers/__init__.py module.

Responsibilities: Given raw input text and a doctree root node, populate
the doctree by parsing the input text.

Example: The only parser implemented so far is for the reStructuredText
markup. It is implemented in the docutils/parsers/rst/ package.

The development and integration of other parsers is possible and
encouraged.

Transformer

The Transformer class, in docutils/transforms/__init__.py, stores
transforms and applies them to documents. A transformer object is
attached to every new document tree. The Publisher calls
Transformer.apply_transforms() to apply all stored transforms to the
document tree. Transforms change the document tree from one form to
another, add to the tree, or prune it. Transforms resolve references and
footnote numbers, process interpreted text, and do other
context-sensitive processing.

Some transforms are specific to components (Readers, Parser, Writers,
Input, Output). Standard component-specific transforms are specified in
the default_transforms attribute of component classes. After the Reader
has finished processing, the Publisher calls
Transformer.populate_from_components() with a list of components and all
default transforms are stored.

Each transform is a class in a module in the docutils/transforms/
package, a subclass of docutils.transforms.Transform. Transform classes
each have a default_priority attribute which is used by the Transformer
to apply transforms in order (low to high). The default priority can be
overridden when adding transforms to the Transformer object.

Transformer responsibilities:

-   Apply transforms to the document tree, in priority order.
-   Store a mapping of component type name ('reader', 'writer', etc.) to
    component objects. These are used by certain transforms (such as
    "components.Filter") to determine suitability.

Transform responsibilities:

-   Modify a doctree in-place, either purely transforming one structure
    into another, or adding new structures based on the doctree and/or
    external data.

Examples of transforms (in the docutils/transforms/ package):

-   frontmatter.DocInfo: Conversion of document metadata (bibliographic
    information).
-   references.AnonymousHyperlinks: Resolution of anonymous references
    to corresponding targets.
-   parts.Contents: Generates a table of contents for a document.
-   document.Merger: Combining multiple populated doctrees into one.
    (Not yet implemented or fully understood.)
-   document.Splitter: Splits a document into a tree-structure of
    subdocuments, perhaps by section. It will have to transform
    references appropriately. (Neither implemented not remotely
    understood.)
-   components.Filter: Includes or excludes elements which depend on a
    specific Docutils component.

Writers

Writers produce the final output (HTML, XML, TeX, etc.). Writers
translate the internal document tree structure into the final data
format, possibly running Writer-specific transforms first.

By the time the document gets to the Writer, it should be in final form.
The Writer's job is simply (and only) to translate from the Docutils
doctree structure to the target format. Some small transforms may be
required, but they should be local and format-specific.

Each writer is a module or package exporting a "Writer" class with a
"write" method. The base "Writer" class can be found in the
docutils/writers/__init__.py module.

Responsibilities:

-   Translate doctree(s) into specific output formats.
    -   Transform references into format-native forms.
-   Write the translated output to the destination I/O.

Examples:

-   XML: Various forms, such as:
    -   Docutils XML (an expression of the internal document tree,
        implemented as docutils.writers.docutils_xml).
    -   DocBook (being implemented in the Docutils sandbox).
-   HTML (XHTML implemented as docutils.writers.html4css1).
-   PDF (a ReportLabs interface is being developed in the Docutils
    sandbox).
-   TeX (a LaTeX Writer is being implemented in the sandbox).
-   Docutils-native pseudo-XML (implemented as
    docutils.writers.pseudoxml, used for testing).
-   Plain text
-   reStructuredText?

Input/Output

I/O classes provide a uniform API for low-level input and output.
Subclasses will exist for a variety of input/output mechanisms. However,
they can be considered an implementation detail. Most applications
should be satisfied using one of the convenience functions associated
with the Publisher.

I/O classes are currently in the preliminary stages; there's a lot of
work yet to be done. Issues:

-   How to represent multi-file input (files & directories) in the API?
-   How to represent multi-file output? Perhaps "Writer" variants, one
    for each output distribution type? Or Output objects with associated
    transforms?

Responsibilities:

-   Read data from the input source (Input objects) or write data to the
    output destination (Output objects).

Examples of input sources:

-   A single file on disk or a stream (implemented as
    docutils.io.FileInput).
-   Multiple files on disk (MultiFileInput?).
-   Python source files: modules and packages.
-   Python strings, as received from a client application (implemented
    as docutils.io.StringInput).

Examples of output destinations:

-   A single file on disk or a stream (implemented as
    docutils.io.FileOutput).
-   A tree of directories and files on disk.
-   A Python string, returned to a client application (implemented as
    docutils.io.StringOutput).
-   No output; useful for programmatic applications where only a portion
    of the normal output is to be used (implemented as
    docutils.io.NullOutput).
-   A single tree-shaped data structure in memory.
-   Some other set of data structures in memory.

Docutils Package Structure

-   Package "docutils".
    -   Module "__init__.py" contains: class "Component", a base class
        for Docutils components; class "SettingsSpec", a base class for
        specifying runtime settings (used by docutils.frontend); and
        class "TransformSpec", a base class for specifying transforms.

    -   Module "docutils.core" contains facade class "Publisher" and
        convenience functions. See Publisher above.

    -   Module "docutils.frontend" provides runtime settings support,
        for programmatic use and front-end tools (including
        configuration file support, and command-line argument and option
        processing).

    -   Module "docutils.io" provides a uniform API for low-level input
        and output. See Input/Output above.

    -   Module "docutils.nodes" contains the Docutils document tree
        element class library plus tree-traversal Visitor pattern base
        classes. See Document Tree below.

    -   Module "docutils.statemachine" contains a finite state machine
        specialized for regular-expression-based text filters and
        parsers. The reStructuredText parser implementation is based on
        this module.

    -   Module "docutils.urischemes" contains a mapping of known URI
        schemes ("http", "ftp", "mail", etc.).

    -   Module "docutils.utils" contains utility functions and classes,
        including a logger class ("Reporter"; see Error Handling below).

    -   Package "docutils.parsers": markup parsers.

        -   Function "get_parser_class(parser_name)" returns a parser
            module by name. Class "Parser" is the base class of specific
            parsers. (docutils/parsers/__init__.py)
        -   Package "docutils.parsers.rst": the reStructuredText parser.
        -   Alternate markup parsers may be added.

        See Parsers above.

    -   Package "docutils.readers": context-aware input readers.

        -   Function "get_reader_class(reader_name)" returns a reader
            module by name or alias. Class "Reader" is the base class of
            specific readers. (docutils/readers/__init__.py)
        -   Module "docutils.readers.standalone" reads independent
            document files.
        -   Module "docutils.readers.pep" reads PEPs (Python Enhancement
            Proposals).
        -   Readers to be added for: Python source code (structure &
            docstrings), email, FAQ, and perhaps Wiki and others.

        See Readers above.

    -   Package "docutils.writers": output format writers.

        -   Function "get_writer_class(writer_name)" returns a writer
            module by name. Class "Writer" is the base class of specific
            writers. (docutils/writers/__init__.py)
        -   Module "docutils.writers.html4css1" is a simple HyperText
            Markup Language document tree writer for HTML 4.01 and CSS1.
        -   Module "docutils.writers.docutils_xml" writes the internal
            document tree in XML form.
        -   Module "docutils.writers.pseudoxml" is a simple internal
            document tree writer; it writes indented pseudo-XML.
        -   Writers to be added: HTML 3.2 or 4.01-loose, XML (various
            forms, such as DocBook), PDF, TeX, plaintext,
            reStructuredText, and perhaps others.

        See Writers above.

    -   Package "docutils.transforms": tree transform classes.

        -   Class "Transformer" stores transforms and applies them to
            document trees. (docutils/transforms/__init__.py)
        -   Class "Transform" is the base class of specific transforms.
            (docutils/transforms/__init__.py)
        -   Each module contains related transform classes.

        See Transforms above.

    -   Package "docutils.languages": Language modules contain
        language-dependent strings and mappings. They are named for
        their language identifier (as defined in Choice of Docstring
        Format below), converting dashes to underscores.

        -   Function "get_language(language_code)", returns matching
            language module. (docutils/languages/__init__.py)
        -   Modules: en.py (English), de.py (German), fr.py (French),
            it.py (Italian), sk.py (Slovak), sv.py (Swedish).
        -   Other languages to be added.
-   Third-party modules: "extras" directory. These modules are installed
    only if they're not already present in the Python installation.
    -   extras/optparse.py and extras/textwrap.py provide option parsing
        and command-line help; from Greg Ward's http://optik.sf.net/
        project, included for convenience.
    -   extras/roman.py contains Roman numeral conversion routines.

Front-End Tools

The tools/ directory contains several front ends for common Docutils
processing. See Docutils Front-End Tools for details.

Document Tree

A single intermediate data structure is used internally by Docutils, in
the interfaces between components; it is defined in the docutils.nodes
module. It is not required that this data structure be used internally
by any of the components, just between components as outlined in the
diagram in the Docutils Project Model above.

Custom node types are allowed, provided that either (a) a transform
converts them to standard Docutils nodes before they reach the Writer
proper, or (b) the custom node is explicitly supported by certain
Writers, and is wrapped in a filtered "pending" node. An example of
condition (a) is the Python Source Reader (see below), where a "stylist"
transform converts custom nodes. The HTML <meta> tag is an example of
condition (b); it is supported by the HTML Writer but not by others. The
reStructuredText "meta" directive creates a "pending" node, which
contains knowledge that the embedded "meta" node can only be handled by
HTML-compatible writers. The "pending" node is resolved by the
docutils.transforms.components.Filter transform, which checks that the
calling writer supports HTML; if it doesn't, the "pending" node (and
enclosed "meta" node) is removed from the document.

The document tree data structure is similar to a DOM tree, but with
specific node names (classes) instead of DOM's generic nodes. The schema
is documented in an XML DTD (eXtensible Markup Language Document Type
Definition), which comes in two parts:

-   the Docutils Generic DTD, docutils.dtd, and
-   the OASIS Exchange Table Model, soextbl.dtd.

The DTD defines a rich set of elements, suitable for many input and
output formats. The DTD retains all information necessary to reconstruct
the original input text, or a reasonable facsimile thereof.

See The Docutils Document Tree for details (incomplete).

Error Handling

When the parser encounters an error in markup, it inserts a system
message (DTD element "system_message"). There are five levels of system
messages:

-   Level-0, "DEBUG": an internal reporting issue. There is no effect on
    the processing. Level-0 system messages are handled separately from
    the others.
-   Level-1, "INFO": a minor issue that can be ignored. There is little
    or no effect on the processing. Typically level-1 system messages
    are not reported.
-   Level-2, "WARNING": an issue that should be addressed. If ignored,
    there may be minor problems with the output. Typically level-2
    system messages are reported but do not halt processing
-   Level-3, "ERROR": a major issue that should be addressed. If
    ignored, the output will contain unpredictable errors. Typically
    level-3 system messages are reported but do not halt processing
-   Level-4, "SEVERE": a critical error that must be addressed.
    Typically level-4 system messages are turned into exceptions which
    halt processing. If ignored, the output will contain severe errors.

Although the initial message levels were devised independently, they
have a strong correspondence to VMS error condition severity levels; the
names in quotes for levels 1 through 4 were borrowed from VMS. Error
handling has since been influenced by the log4j project.

Python Source Reader

The Python Source Reader ("PySource") is the Docutils component that
reads Python source files, extracts docstrings in context, then parses,
links, and assembles the docstrings into a cohesive whole. It is a major
and non-trivial component, currently under experimental development in
the Docutils sandbox. High-level design issues are presented here.

Processing Model

This model will evolve over time, incorporating experience and
discoveries.

1.  The PySource Reader uses an Input class to read in Python packages
    and modules, into a tree of strings.
2.  The Python modules are parsed, converting the tree of strings into a
    tree of abstract syntax trees with docstring nodes.
3.  The abstract syntax trees are converted into an internal
    representation of the packages/modules. Docstrings are extracted, as
    well as code structure details. See AST Mining below. Namespaces are
    constructed for lookup in step 6.
4.  One at a time, the docstrings are parsed, producing standard
    Docutils doctrees.
5.  PySource assembles all the individual docstrings' doctrees into a
    Python-specific custom Docutils tree paralleling the
    package/module/class structure; this is a custom Reader-specific
    internal representation (see the Docutils Python Source DTD).
    Namespaces must be merged: Python identifiers, hyperlink targets.
6.  Cross-references from docstrings (interpreted text) to Python
    identifiers are resolved according to the Python namespace lookup
    rules. See Identifier Cross-References below.
7.  A "Stylist" transform is applied to the custom doctree (by the
    Transformer), custom nodes are rendered using standard nodes as
    primitives, and a standard document tree is emitted. See Stylist
    Transforms below.
8.  Other transforms are applied to the standard doctree by the
    Transformer.
9.  The standard doctree is sent to a Writer, which translates the
    document into a concrete format (HTML, PDF, etc.).
10. The Writer uses an Output class to write the resulting data to its
    destination (disk file, directories and files, etc.).

AST Mining

Abstract Syntax Tree mining code will be written (or adapted) that scans
a parsed Python module, and returns an ordered tree containing the
names, docstrings (including attribute and additional docstrings; see
below), and additional info (in parentheses below) of all of the
following objects:

-   packages
-   modules
-   module attributes (+ initial values)
-   classes (+ inheritance)
-   class attributes (+ initial values)
-   instance attributes (+ initial values)
-   methods (+ parameters & defaults)
-   functions (+ parameters & defaults)

(Extract comments too? For example, comments at the start of a module
would be a good place for bibliographic field lists.)

In order to evaluate interpreted text cross-references, namespaces for
each of the above will also be required.

See the python-dev/docstring-develop thread "AST mining", started on
2001-08-14.

Docstring Extraction Rules

1.  What to examine:

    a)  If the "__all__" variable is present in the module being
        documented, only identifiers listed in "__all__" are examined
        for docstrings.
    b)  In the absence of "__all__", all identifiers are examined,
        except those whose names are private (names begin with "_" but
        don't begin and end with "__").
    c)  1a and 1b can be overridden by runtime settings.

2.  Where:

    Docstrings are string literal expressions, and are recognized in the
    following places within Python modules:

    a)  At the beginning of a module, function definition, class
        definition, or method definition, after any comments. This is
        the standard for Python __doc__ attributes.
    b)  Immediately following a simple assignment at the top level of a
        module, class definition, or __init__ method definition, after
        any comments. See Attribute Docstrings below.
    c)  Additional string literals found immediately after the
        docstrings in (a) and (b) will be recognized, extracted, and
        concatenated. See Additional Docstrings below.
    d)  @@@ 2.2-style "properties" with attribute docstrings? Wait for
        syntax?

3.  How:

    Whenever possible, Python modules should be parsed by Docutils, not
    imported. There are several reasons:

    -   Importing untrusted code is inherently insecure.
    -   Information from the source is lost when using introspection to
        examine an imported module, such as comments and the order of
        definitions.
    -   Docstrings are to be recognized in places where the byte-code
        compiler ignores string literal expressions (2b and 2c above),
        meaning importing the module will lose these docstrings.

    Of course, standard Python parsing tools such as the "parser"
    library module should be used.

    When the Python source code for a module is not available (i.e. only
    the .pyc file exists) or for C extension modules, to access
    docstrings the module can only be imported, and any limitations must
    be lived with.

Since attribute docstrings and additional docstrings are ignored by the
Python byte-code compiler, no namespace pollution or runtime bloat will
result from their use. They are not assigned to __doc__ or to any other
attribute. The initial parsing of a module may take a slight performance
hit.

Attribute Docstrings

(This is a simplified version of PEP 224.)

A string literal immediately following an assignment statement is
interpreted by the docstring extraction machinery as the docstring of
the target of the assignment statement, under the following conditions:

1.  The assignment must be in one of the following contexts:

    a)  At the top level of a module (i.e., not nested inside a compound
        statement such as a loop or conditional): a module attribute.
    b)  At the top level of a class definition: a class attribute.
    c)  At the top level of the "__init__" method definition of a class:
        an instance attribute. Instance attributes assigned in other
        methods are assumed to be implementation details. (@@@ __new__
        methods?)
    d)  A function attribute assignment at the top level of a module or
        class definition.

    Since each of the above contexts are at the top level (i.e., in the
    outermost suite of a definition), it may be necessary to place dummy
    assignments for attributes assigned conditionally or in a loop.

2.  The assignment must be to a single target, not to a list or a tuple
    of targets.

3.  The form of the target:

    a)  For contexts 1a and 1b above, the target must be a simple
        identifier (not a dotted identifier, a subscripted expression,
        or a sliced expression).
    b)  For context 1c above, the target must be of the form
        "self.attrib", where "self" matches the "__init__" method's
        first parameter (the instance parameter) and "attrib" is a
        simple identifier as in 3a.
    c)  For context 1d above, the target must be of the form
        "name.attrib", where "name" matches an already-defined function
        or method name and "attrib" is a simple identifier as in 3a.

Blank lines may be used after attribute docstrings to emphasize the
connection between the assignment and the docstring.

Examples:

    g = 'module attribute (module-global variable)'
    """This is g's docstring."""

    class AClass:

        c = 'class attribute'
        """This is AClass.c's docstring."""

        def __init__(self):
            """Method __init__'s docstring."""

            self.i = 'instance attribute'
            """This is self.i's docstring."""

    def f(x):
        """Function f's docstring."""
        return x**2

    f.a = 1
    """Function attribute f.a's docstring."""

Additional Docstrings

(This idea was adapted from PEP 216.)

Many programmers would like to make extensive use of docstrings for API
documentation. However, docstrings do take up space in the running
program, so some programmers are reluctant to "bloat up" their code.
Also, not all API documentation is applicable to interactive
environments, where __doc__ would be displayed.

Docutils' docstring extraction tools will concatenate all string literal
expressions which appear at the beginning of a definition or after a
simple assignment. Only the first strings in definitions will be
available as __doc__, and can be used for brief usage text suitable for
interactive sessions; subsequent string literals and all attribute
docstrings are ignored by the Python byte-code compiler and may contain
more extensive API information.

Example:

    def function(arg):
        """This is __doc__, function's docstring."""
        """
        This is an additional docstring, ignored by the byte-code
        compiler, but extracted by Docutils.
        """
        pass

Issue: from __future__ import

This would break "from __future__ import" statements introduced in
Python 2.1 for multiple module docstrings (main docstring plus
additional docstring(s)). The Python Reference Manual specifies:

  A future statement must appear near the top of the module. The only
  lines that can appear before a future statement are:

  -   the module docstring (if any),
  -   comments,
  -   blank lines, and
  -   other future statements.

Resolution?

1.  Should we search for docstrings after a __future__ statement? Very
    ugly.
2.  Redefine __future__ statements to allow multiple preceding string
    literals?
3.  Or should we not even worry about this? There probably shouldn't be
    __future__ statements in production code, after all. Perhaps modules
    with __future__ statements will simply have to put up with the
    single-docstring limitation.

Choice of Docstring Format

Rather than force everyone to use a single docstring format, multiple
input formats are allowed by the processing system. A special variable,
__docformat__, may appear at the top level of a module before any
function or class definitions. Over time or through decree, a standard
format or set of formats should emerge.

A module's __docformat__ variable only applies to the objects defined in
the module's file. In particular, the __docformat__ variable in a
package's __init__.py file does not apply to objects defined in
subpackages and submodules.

The __docformat__ variable is a string containing the name of the format
being used, a case-insensitive string matching the input parser's module
or package name (i.e., the same name as required to "import" the module
or package), or a registered alias. If no __docformat__ is specified,
the default format is "plaintext" for now; this may be changed to the
standard format if one is ever established.

The __docformat__ string may contain an optional second field, separated
from the format name (first field) by a single space: a case-insensitive
language identifier as defined in 1766. A typical language identifier
consists of a 2-letter language code from ISO 639 (3-letter codes used
only if no 2-letter code exists; 1766 is currently being revised to
allow 3-letter codes). If no language identifier is specified, the
default is "en" for English. The language identifier is passed to the
parser and can be used for language-dependent markup features.

Identifier Cross-References

In Python docstrings, interpreted text is used to classify and mark up
program identifiers, such as the names of variables, functions, classes,
and modules. If the identifier alone is given, its role is inferred
implicitly according to the Python namespace lookup rules. For functions
and methods (even when dynamically assigned), parentheses ('()') may be
included:

    This function uses `another()` to do its work.

For class, instance and module attributes, dotted identifiers are used
when necessary. For example (using reStructuredText markup):

    class Keeper(Storer):

        """
        Extend `Storer`.  Class attribute `instances` keeps track
        of the number of `Keeper` objects instantiated.
        """

        instances = 0
        """How many `Keeper` objects are there?"""

        def __init__(self):
            """
            Extend `Storer.__init__()` to keep track of instances.

            Keep count in `Keeper.instances`, data in `self.data`.
            """
            Storer.__init__(self)
            Keeper.instances += 1

            self.data = []
            """Store data in a list, most recent last."""

        def store_data(self, data):
            """
            Extend `Storer.store_data()`; append new `data` to a
            list (in `self.data`).
            """
            self.data = data

Each of the identifiers quoted with backquotes ("`") will become
references to the definitions of the identifiers themselves.

Stylist Transforms

Stylist transforms are specialized transforms specific to the PySource
Reader. The PySource Reader doesn't have to make any decisions as to
style; it just produces a logically constructed document tree, parsed
and linked, including custom node types. Stylist transforms understand
the custom nodes created by the Reader and convert them into standard
Docutils nodes.

Multiple Stylist transforms may be implemented and one can be chosen at
runtime (through a "--style" or "--stylist" command-line option). Each
Stylist transform implements a different layout or style; thus the name.
They decouple the context-understanding part of the Reader from the
layout-generating part of processing, resulting in a more flexible and
robust system. This also serves to "separate style from content", the
SGML/XML ideal.

By keeping the piece of code that does the styling small and modular, it
becomes much easier for people to roll their own styles. The "barrier to
entry" is too high with existing tools; extracting the stylist code will
lower the barrier considerably.

References and Footnotes

Project Web Site

A SourceForge project has been set up for this work at
http://docutils.sourceforge.net/.

Copyright

This document has been placed in the public domain.

Acknowledgements

This document borrows ideas from the archives of the Python Doc-SIG.
Thanks to all members past & present.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 End:


text:
PEP: 2 Title: Procedure for Adding New Modules Version: $Revision$
Last-Modified: $Date$ Author: Brett Cannon<EMAIL_ADDRESS>Martijn
Faassen<EMAIL_ADDRESS>Status: Active Type: Process Content-Type:
text/x-rst Created: 07-Jul-2001 Post-History: 07-Jul-2001, 09-Mar-2002

Introduction

The Python Standard Library contributes significantly to Python's
success. The language comes with "batteries included", so it is easy for
people to become productive with just the standard library alone. It is
therefore important that the usefulness of the standard library be
maintained.

Due to the visibility and importance of the standard library, it must be
maintained thoughtfully. As such, any code within it must be maintained
by Python's development team which leads to a perpetual cost to each
addition made. There is also added cognitive load for users in
familiarizing themselves with what is in the standard library to be
considered.

New functionality is commonly added to the library in the form of new
modules. This PEP will describe the procedure for the addition of new
modules. PEP 4 deals with procedures for deprecation of modules; the
removal of old and unused modules from the standard library.

Acceptance Procedure

For top-level modules/packages, a PEP is required. The procedure for
writing a PEP is covered in PEP 1.

For submodules of a preexisting package in the standard library,
additions are at the discretion of the general Python development team
and its members.

General guidance on what modules typically are accepted into the
standard library, the overall process, etc. are covered in the
developer's guide.

Maintenance Procedure

Anything accepted into the standard library is expected to be primarily
maintained there, within Python's development infrastructure. While some
members of the development team may choose to maintain a backport of a
module outside of the standard library, it is up to them to keep their
external code in sync as appropriate.

Copyright

This document has been placed in the public domain.

license: Public Domain


text:
PEP: 222 Title: Web Library Enhancements Version: $Revision$
Last-Modified: $Date$ Author: A.M. Kuchling<EMAIL_ADDRESS>Status:
Deferred Type: Standards Track Content-Type: text/x-rst Created:
18-Aug-2000 Python-Version: 2.1 Post-History: 22-Dec-2000

Abstract

This PEP proposes a set of enhancements to the CGI development
facilities in the Python standard library. Enhancements might be new
features, new modules for tasks such as cookie support, or removal of
obsolete code.

The original intent was to make improvements to Python 2.1. However,
there seemed little interest from the Python community, and time was
lacking, so this PEP has been deferred to some future Python release.

Open Issues

This section lists changes that have been suggested, but about which no
firm decision has yet been made. In the final version of this PEP, this
section should be empty, as all the changes should be classified as
accepted or rejected.

cgi.py: We should not be told to create our own subclass just so we can
handle file uploads. As a practical matter, I have yet to find the time
to do this right, so I end up reading cgi.py's temp file into, at best,
another file. Some of our legacy code actually reads it into a second
temp file, then into a final destination! And even if we did, that would
mean creating yet another object with its __init__ call and associated
overhead.

cgi.py: Currently, query data with no = are ignored. Even if
keep_blank_values is set, queries like ...?value=&... are returned with
blank values but queries like ...?value&... are completely lost. It
would be great if such data were made available through the FieldStorage
interface, either as entries with None as values, or in a separate list.

Utility function: build a query string from a list of 2-tuples

Dictionary-related utility classes: NoKeyErrors (returns an empty
string, never a KeyError), PartialStringSubstitution (returns the
original key string, never a KeyError)

New Modules

This section lists details about entire new packages or modules that
should be added to the Python standard library.

-   fcgi.py : A new module adding support for the FastCGI protocol.
    Robin Dunn's code needs to be ported to Windows, though.

Major Changes to Existing Modules

This section lists details of major changes to existing modules, whether
in implementation or in interface. The changes in this section therefore
carry greater degrees of risk, either in introducing bugs or a backward
incompatibility.

The cgi.py module would be deprecated. (XXX A new module or package name
hasn't been chosen yet: 'web'? 'cgilib'?)

Minor Changes to Existing Modules

This section lists details of minor changes to existing modules. These
changes should have relatively small implementations, and have little
risk of introducing incompatibilities with previous versions.

Rejected Changes

The changes listed in this section were proposed for Python 2.1, but
were rejected as unsuitable. For each rejected change, a rationale is
given describing why the change was deemed inappropriate.

-   An HTML generation module is not part of this PEP. Several such
    modules exist, ranging from HTMLgen's purely programming interface
    to ASP-inspired simple templating to DTML's complex templating.
    There's no indication of which templating module to enshrine in the
    standard library, and that probably means that no module should be
    so chosen.
-   cgi.py: Allowing a combination of query data and POST data. This
    doesn't seem to be standard at all, and therefore is dubious
    practice.

Proposed Interface

XXX open issues: naming convention (studlycaps or underline-separated?);
need to look at the cgi.parse*() functions and see if they can be
simplified, too.

Parsing functions: carry over most of the parse* functions from cgi.py

    # The Response class borrows most of its methods from Zope's
    # HTTPResponse class.

    class Response:
        """
        Attributes:
        status: HTTP status code to return
        headers: dictionary of response headers
        body: string containing the body of the HTTP response
        """

        def __init__(self, status=200, headers={}, body=""):
            pass

        def setStatus(self, status, reason=None):
            "Set the numeric HTTP response code"
            pass

        def setHeader(self, name, value):
            "Set an HTTP header"
            pass

        def setBody(self, body):
            "Set the body of the response"
            pass

        def setCookie(self, name, value,
                      path = '/',
                      comment = None,
                      domain = None,
                      max-age = None,
                      expires = None,
                      secure = 0
                      ):
            "Set a cookie"
            pass

        def expireCookie(self, name):
            "Remove a cookie from the user"
            pass

        def redirect(self, url):
            "Redirect the browser to another URL"
            pass

        def __str__(self):
            "Convert entire response to a string"
            pass

        def dump(self):
            "Return a string representation useful for debugging"
            pass

        # XXX methods for specific classes of error:serverError,
        # badRequest, etc.?

    class Request:

        """
        Attributes:

        XXX should these be dictionaries, or dictionary-like objects?
        .headers : dictionary containing HTTP headers
        .cookies : dictionary of cookies
        .fields  : data from the form
        .env     : environment dictionary
        """

        def __init__(self, environ=os.environ, stdin=sys.stdin,
                     keep_blank_values=1, strict_parsing=0):
            """Initialize the request object, using the provided environment
            and standard input."""
            pass

        # Should people just use the dictionaries directly?
        def getHeader(self, name, default=None):
            pass

        def getCookie(self, name, default=None):
            pass

        def getField(self, name, default=None):
            "Return field's value as a string (even if it's an uploaded file)"
            pass

        def getUploadedFile(self, name):
            """Returns a file object that can be read to obtain the contents
            of an uploaded file.  XXX should this report an error if the
            field isn't actually an uploaded file?  Or should it wrap
            a StringIO around simple fields for consistency?
            """

        def getURL(self, n=0, query_string=0):
            """Return the URL of the current request, chopping off 'n' path
            components from the right.  Eg. if the URL is
            "http://foo.com/bar/baz/quux", n=2 would return
            "http://foo.com/bar".  Does not include the query string (if
            any)
            """

        def getBaseURL(self, n=0):
            """Return the base URL of the current request, adding 'n' path
            components to the end to recreate more of the whole URL.

            Eg. if the request URL is
            "http://foo.com/q/bar/baz/qux", n=0 would return
            "http://foo.com/", and n=2 "http://foo.com/q/bar".

            Returned URL does not include the query string, if any.
            """

        def dump(self):
            "String representation suitable for debugging output"
            pass

        # Possibilities?  I don't know if these are worth doing in the
        # basic objects.
        def getBrowser(self):
            "Returns Mozilla/IE/Lynx/Opera/whatever"

        def isSecure(self):
            "Return true if this is an SSLified request"

    # Module-level function
    def wrapper(func, logfile=sys.stderr):
        """
        Calls the function 'func', passing it the arguments
        (request, response, logfile).  Exceptions are trapped and
        sent to the file 'logfile'.
        """
        # This wrapper will detect if it's being called from the command-line,
        # and if so, it will run in a debugging mode; name=value pairs
        # can be entered on standard input to set field values.
        # (XXX how to do file uploads in this syntax?)

Copyright

This document has been placed in the public domain.


text:
PEP: 502 Title: String Interpolation - Extended Discussion Author: Mike
G. Miller Status: Rejected Type: Informational Content-Type: text/x-rst
Created: 10-Aug-2015 Python-Version: 3.6

Abstract

PEP 498: Literal String Interpolation, which proposed "formatted
strings" was accepted September 9th, 2015. Additional background and
rationale given during its design phase is detailed below.

To recap that PEP, a string prefix was introduced that marks the string
as a template to be rendered. These formatted strings may contain one or
more expressions built on the existing syntax of str.format().[1][2] The
formatted string expands at compile-time into a conventional string
format operation, with the given expressions from its text extracted and
passed instead as positional arguments.

At runtime, the resulting expressions are evaluated to render a string
to given specifications:

    >>> location = 'World'
    >>> f'Hello, {location} !'      # new prefix: f''
    'Hello, World !'                # interpolated result

Format-strings may be thought of as merely syntactic sugar to simplify
traditional calls to str.format().

PEP Status

This PEP was rejected based on its using an opinion-based tone rather
than a factual one. This PEP was also deemed not critical as PEP 498 was
already written and should be the place to house design decision
details.

Motivation

Though string formatting and manipulation features are plentiful in
Python, one area where it falls short is the lack of a convenient string
interpolation syntax. In comparison to other dynamic scripting languages
with similar use cases, the amount of code necessary to build similar
strings is substantially higher, while at times offering lower
readability due to verbosity, dense syntax, or identifier duplication.

These difficulties are described at moderate length in the original post
to python-ideas that started the snowball (that became PEP 498)
rolling.[3]

Furthermore, replacement of the print statement with the more consistent
print function of Python 3 (PEP 3105) has added one additional minor
burden, an additional set of parentheses to type and read. Combined with
the verbosity of current string formatting solutions, this puts an
otherwise simple language at an unfortunate disadvantage to its peers:

    echo "Hello, user: $user, id: $id, on host: $hostname"              # bash
    say  "Hello, user: $user, id: $id, on host: $hostname";             # perl
    puts "Hello, user: #{user}, id: #{id}, on host: #{hostname}\n"      # ruby
                                                                        # 80 ch -->|
    # Python 3, str.format with named parameters
    print('Hello, user: {user}, id: {id}, on host: {hostname}'.format(**locals()))

    # Python 3, worst case
    print('Hello, user: {user}, id: {id}, on host: {hostname}'.format(user=user,
                                                                      id=id,
                                                                      hostname=
                                                                        hostname))

In Python, the formatting and printing of a string with multiple
variables in a single line of code of standard width is noticeably
harder and more verbose, with indentation exacerbating the issue.

For use cases such as smaller projects, systems programming, shell
script replacements, and even one-liners, where message formatting
complexity has yet to be encapsulated, this verbosity has likely lead a
significant number of developers and administrators to choose other
languages over the years.

Rationale

Goals

The design goals of format strings are as follows:

1.  Eliminate need to pass variables manually.
2.  Eliminate repetition of identifiers and redundant parentheses.
3.  Reduce awkward syntax, punctuation characters, and visual noise.
4.  Improve readability and eliminate mismatch errors, by preferring
    named parameters to positional arguments.
5.  Avoid need for locals() and globals() usage, instead parsing the
    given string for named parameters, then passing them
    automatically.[4][5]

Limitations

In contrast to other languages that take design cues from Unix and its
shells, and in common with Javascript, Python specified both single (')
and double (") ASCII quote characters to enclose strings. It is not
reasonable to choose one of them now to enable interpolation, while
leaving the other for uninterpolated strings. Other characters, such as
the "Backtick" (or grave accent ) are also `constrained by history_ as a
shortcut for repr().

This leaves a few remaining options for the design of such a feature:

-   An operator, as in printf-style string formatting via %.
-   A class, such as string.Template().
-   A method or function, such as str.format().
-   New syntax, or
-   A new string prefix marker, such as the well-known r'' or u''.

The first three options above are mature. Each has specific use cases
and drawbacks, yet also suffer from the verbosity and visual noise
mentioned previously. All options are discussed in the next sections.

Background

Formatted strings build on several existing techniques and proposals and
what we've collectively learned from them. In keeping with the design
goals of readability and error-prevention, the following examples
therefore use named, not positional arguments.

Let's assume we have the following dictionary, and would like to print
out its items as an informative string for end users:

    >>> params = {'user': 'nobody', 'id': 9, 'hostname': 'darkstar'}

Printf-style formatting, via operator

This venerable technique continues to have its uses, such as with
byte-based protocols, simplicity in simple cases, and familiarity to
many programmers:

    >>> 'Hello, user: %(user)s, id: %(id)s, on host: %(hostname)s' % params
    'Hello, user: nobody, id: 9, on host: darkstar'

In this form, considering the prerequisite dictionary creation, the
technique is verbose, a tad noisy, yet relatively readable. Additional
issues are that an operator can only take one argument besides the
original string, meaning multiple parameters must be passed in a tuple
or dictionary. Also, it is relatively easy to make an error in the
number of arguments passed, the expected type, have a missing key, or
forget the trailing type, e.g. (s or d).

string.Template Class

The string.Template class from PEP 292 (Simpler String Substitutions) is
a purposely simplified design, using familiar shell interpolation
syntax, with safe-substitution feature, that finds its main use cases in
shell and internationalization tools:

    Template('Hello, user: $user, id: ${id}, on host: $hostname').substitute(params)

While also verbose, the string itself is readable. Though functionality
is limited, it meets its requirements well. It isn't powerful enough for
many cases, and that helps keep inexperienced users out of trouble, as
well as avoiding issues with moderately-trusted input (i18n) from
third-parties. It unfortunately takes enough code to discourage its use
for ad-hoc string interpolation, unless encapsulated in a convenience
library such as flufl.i18n.

PEP 215 - String Interpolation

PEP 215 was a former proposal of which this one shares a lot in common.
Apparently, the world was not ready for it at the time, but considering
recent support in a number of other languages, its day may have come.

The large number of dollar sign ($) characters it included may have led
it to resemble Python's arch-nemesis Perl, and likely contributed to the
PEP's lack of acceptance. It was superseded by the following proposal.

str.format() Method

The str.format() syntax of PEP 3101 is the most recent and modern of the
existing options. It is also more powerful and usually easier to read
than the others. It avoids many of the drawbacks and limits of the
previous techniques.

However, due to its necessary function call and parameter passing, it
runs from verbose to very verbose in various situations with string
literals:

    >>> 'Hello, user: {user}, id: {id}, on host: {hostname}'.format(**params)
    'Hello, user: nobody, id: 9, on host: darkstar'

    # when using keyword args, var name shortening sometimes needed to fit :/
    >>> 'Hello, user: {user}, id: {id}, on host: {host}'.format(user=user,
                                                                id=id,
                                                                host=hostname)
    'Hello, user: nobody, id: 9, on host: darkstar'

The verbosity of the method-based approach is illustrated here.

PEP 498 -- Literal String Formatting

PEP 498 defines and discusses format strings, as also described in the
Abstract above.

It also, somewhat controversially to those first exposed, introduces the
idea that format-strings shall be augmented with support for arbitrary
expressions. This is discussed further in the Restricting Syntax section
under Rejected Ideas.

PEP 501 -- Translation ready string interpolation

The complimentary PEP 501 brings internationalization into the
discussion as a first-class concern, with its proposal of the i-prefix,
string.Template syntax integration compatible with ES6 (Javascript),
deferred rendering, and an object return value.

Implementations in Other Languages

String interpolation is now well supported by various programming
languages used in multiple industries, and is converging into a standard
of sorts. It is centered around str.format() style syntax in minor
variations, with the addition of arbitrary expressions to expand
utility.

In the Motivation section it was shown how convenient interpolation
syntax existed in Bash, Perl, and Ruby. Let's take a look at their
expression support.

Bash

Bash supports a number of arbitrary, even recursive constructs inside
strings:

    > echo "user: $USER, id: $((id + 6)) on host: $(echo is $(hostname))"
    user: nobody, id: 15 on host: is darkstar

-   Explicit interpolation within double quotes.
-   Direct environment variable access supported.
-   Arbitrary expressions are supported.[6]
-   External process execution and output capture supported.[7]
-   Recursive expressions are supported.

Perl

Perl also has arbitrary expression constructs, perhaps not as well
known:

    say "I have @{[$id + 6]} guanacos.";                # lists
    say "I have ${\($id + 6)} guanacos.";               # scalars
    say "Hello { @names.join(', ') } how are you?";     # Perl 6 version

-   Explicit interpolation within double quotes.
-   Arbitrary expressions are supported.[8][9]

Ruby

Ruby allows arbitrary expressions in its interpolated strings:

    puts "One plus one is two: #{1 + 1}\n"

-   Explicit interpolation within double quotes.
-   Arbitrary expressions are supported.[10][11]
-   Possible to change delimiter chars with %.
-   See the Reference Implementation(s) section for an implementation in
    Python.

Others

Let's look at some less-similar modern languages recently implementing
string interpolation.

Scala

Scala interpolation is directed through string prefixes. Each prefix has
a different result:

    s"Hello, $name ${1 + 1}"                    # arbitrary
    f"$name%s is $height%2.2f meters tall"      # printf-style
    raw"a\nb"                                   # raw, like r''

These prefixes may also be implemented by the user, by extending Scala's
StringContext class.

-   Explicit interpolation within double quotes with literal prefix.
-   User implemented prefixes supported.
-   Arbitrary expressions are supported.

ES6 (Javascript)

Designers of Template strings faced the same issue as Python where
single and double quotes were taken. Unlike Python however, "backticks"
were not. Despite their issues, they were chosen as part of the
ECMAScript 2015 (ES6) standard:

    console.log(`Fifteen is ${a + b} and\nnot ${2 * a + b}.`);

Custom prefixes are also supported by implementing a function the same
name as the tag:

    function tag(strings, ...values) {
        console.log(strings.raw[0]);    // raw string is also available
        return "Bazinga!";
    }
    tag`Hello ${ a + b } world ${ a * b}`;

-   Explicit interpolation within backticks.
-   User implemented prefixes supported.
-   Arbitrary expressions are supported.

C#, Version 6

C# has a useful new interpolation feature as well, with some ability to
customize interpolation via the IFormattable interface:

    $"{person.Name, 20} is {person.Age:D3} year{(p.Age == 1 ? "" : "s")} old.";

-   Explicit interpolation with double quotes and $ prefix.
-   Custom interpolations are available.
-   Arbitrary expressions are supported.

Apple's Swift

Arbitrary interpolation under Swift is available on all strings:

    let multiplier = 3
    let message = "\(multiplier) times 2.5 is \(Double(multiplier) * 2.5)"
    // message is "3 times 2.5 is 7.5"

-   Implicit interpolation with double quotes.
-   Arbitrary expressions are supported.
-   Cannot contain CR/LF.

Additional examples

A number of additional examples of string interpolation may be found at
Wikipedia.

Now that background and history have been covered, let's continue on for
a solution.

New Syntax

This should be an option of last resort, as every new syntax feature has
a cost in terms of real-estate in a brain it inhabits. There is however
one alternative left on our list of possibilities, which follows.

New String Prefix

Given the history of string formatting in Python and
backwards-compatibility, implementations in other languages, avoidance
of new syntax unless necessary, an acceptable design is reached through
elimination rather than unique insight. Therefore, marking interpolated
string literals with a string prefix is chosen.

We also choose an expression syntax that reuses and builds on the
strongest of the existing choices, str.format() to avoid further
duplication of functionality:

    >>> location = 'World'
    >>> f'Hello, {location} !'      # new prefix: f''
    'Hello, World !'                # interpolated result

PEP 498 -- Literal String Formatting, delves into the mechanics and
implementation of this design.

Additional Topics

Safety

In this section we will describe the safety situation and precautions
taken in support of format-strings.

1.  Only string literals have been considered for format-strings, not
    variables to be taken as input or passed around, making external
    attacks difficult to accomplish.

    str.format() and alternatives already handle this use-case.

2.  Neither locals() nor globals() are necessary nor used during the
    transformation, avoiding leakage of information.

3.  To eliminate complexity as well as RuntimeError (s) due to recursion
    depth, recursive interpolation is not supported.

However, mistakes or malicious code could be missed inside string
literals. Though that can be said of code in general, that these
expressions are inside strings means they are a bit more likely to be
obscured.

Mitigation via Tools

The idea is that tools or linters such as pyflakes, pylint, or Pycharm,
may check inside strings with expressions and mark them up
appropriately. As this is a common task with programming languages
today, multi-language tools won't have to implement this feature solely
for Python, significantly shortening time to implementation.

Farther in the future, strings might also be checked for constructs that
exceed the safety policy of a project.

Style Guide/Precautions

As arbitrary expressions may accomplish anything a Python expression is
able to, it is highly recommended to avoid constructs inside
format-strings that could cause side effects.

Further guidelines may be written once usage patterns and true problems
are known.

Reference Implementation(s)

The say module on PyPI implements string interpolation as described here
with the small burden of a callable interface:

    ＞ pip install say

    from say import say
    nums = list(range(4))
    say("Nums has {len(nums)} items: {nums}")

A Python implementation of Ruby interpolation is also available. It uses
the codecs module to do its work:

    ＞ pip install interpy

    # coding: interpy
    location = 'World'
    print("Hello #{location}.")

Backwards Compatibility

By using existing syntax and avoiding current or historical features,
format strings were designed so as to not interfere with existing code
and are not expected to cause any issues.

Postponed Ideas

Internationalization

Though it was highly desired to integrate internationalization support,
(see PEP 501), the finer details diverge at almost every point, making a
common solution unlikely:[12]

-   Use-cases differ
-   Compile vs. run-time tasks
-   Interpolation syntax needs
-   Intended audience
-   Security policy

Rejected Ideas

Restricting Syntax to str.format() Only

The common arguments against support of arbitrary expressions were:

1.  YAGNI, "You aren't gonna need it."
2.  The feature is not congruent with historical Python conservatism.
3.  Postpone - can implement in a future version if need is
    demonstrated.

Support of only str.format() syntax however, was deemed not enough of a
solution to the problem. Often a simple length or increment of an
object, for example, is desired before printing.

It can be seen in the Implementations in Other Languages section that
the developer community at large tends to agree. String interpolation
with arbitrary expressions is becoming an industry standard in modern
languages due to its utility.

Additional/Custom String-Prefixes

As seen in the Implementations in Other Languages section, many modern
languages have extensible string prefixes with a common interface. This
could be a way to generalize and reduce lines of code in common
situations. Examples are found in ES6 (Javascript), Scala, Nim, and C#
(to a lesser extent). This was rejected by the BDFL.[13]

Automated Escaping of Input Variables

While helpful in some cases, this was thought to create too much
uncertainty of when and where string expressions could be used safely or
not. The concept was also difficult to describe to others.[14]

Always consider format string variables to be unescaped, unless the
developer has explicitly escaped them.

Environment Access and Command Substitution

For systems programming and shell-script replacements, it would be
useful to handle environment variables and capture output of commands
directly in an expression string. This was rejected as not important
enough, and looking too much like bash/perl, which could encourage bad
habits.[15]

Acknowledgements

-   Eric V. Smith for the authoring and implementation of PEP 498.
-   Everyone on the python-ideas mailing list for rejecting the various
    crazy ideas that came up, helping to keep the final design in focus.

References

Copyright

This document has been placed in the public domain.

[1] Python Str.Format Syntax
(https://docs.python.org/3.6/library/string.html#format-string-syntax)

[2] Python Format-Spec Mini Language
(https://docs.python.org/3.6/library/string.html#format-specification-mini-language)

[3] Briefer String Format
(https://mail.python.org/pipermail/python-ideas/2015-July/034659.html)

[4] Briefer String Format
(https://mail.python.org/pipermail/python-ideas/2015-July/034669.html)

[5] Briefer String Format
(https://mail.python.org/pipermail/python-ideas/2015-July/034701.html)

[6] Bash Docs (https://tldp.org/LDP/abs/html/arithexp.html)

[7] Bash Docs (https://tldp.org/LDP/abs/html/commandsub.html)

[8] Perl Cookbook
(https://docstore.mik.ua/orelly/perl/cookbook/ch01_11.htm)

[9] Perl Docs
(https://web.archive.org/web/20121025185907/https://perl6maven.com/perl6-scalar-array-and-hash-interpolation)

[10] Ruby Docs
(http://ruby-doc.org/core-2.1.1/doc/syntax/literals_rdoc.html#label-Strings)

[11] Ruby Docs
(https://en.wikibooks.org/wiki/Ruby_Programming/Syntax/Literals#Interpolation)

[12] Literal String Formatting
(https://mail.python.org/pipermail/python-dev/2015-August/141289.html)

[13] Extensible String Prefixes
(https://mail.python.org/pipermail/python-ideas/2015-August/035336.html)

[14] Escaping of Input Variables
(https://mail.python.org/pipermail/python-ideas/2015-August/035532.html)

[15] Environment Access and Command Substitution
(https://mail.python.org/pipermail/python-ideas/2015-August/035554.html)


text:
PEP: 208 Title: Reworking the Coercion Model Author: Neil Schemenauer
<EMAIL_ADDRESS>Marc-André Lemburg<EMAIL_ADDRESS>Status: Final
Type: Standards Track Content-Type: text/x-rst Created: 04-Dec-2000
Python-Version: 2.1 Post-History:

Abstract

Many Python types implement numeric operations. When the arguments of a
numeric operation are of different types, the interpreter tries to
coerce the arguments into a common type. The numeric operation is then
performed using this common type. This PEP proposes a new type flag to
indicate that arguments to a type's numeric operations should not be
coerced. Operations that do not support the supplied types indicate it
by returning a new singleton object. Types which do not set the type
flag are handled in a backwards compatible manner. Allowing operations
handle different types is often simpler, more flexible, and faster than
having the interpreter do coercion.

Rationale

When implementing numeric or other related operations, it is often
desirable to provide not only operations between operands of one type
only, e.g. integer + integer, but to generalize the idea behind the
operation to other type combinations as well, e.g. integer + float.

A common approach to this mixed type situation is to provide a method of
"lifting" the operands to a common type (coercion) and then use that
type's operand method as execution mechanism. Yet, this strategy has a
few drawbacks:

-   the "lifting" process creates at least one new (temporary) operand
    object,
-   since the coercion method is not being told about the operation that
    is to follow, it is not possible to implement operation specific
    coercion of types,
-   there is no elegant way to solve situations were a common type is
    not at hand, and
-   the coercion method will always have to be called prior to the
    operation's method itself.

A fix for this situation is obviously needed, since these drawbacks make
implementations of types needing these features very cumbersome, if not
impossible. As an example, have a look at the DateTime and
DateTimeDelta[1] types, the first being absolute, the second relative.
You can always add a relative value to an absolute one, giving a new
absolute value. Yet, there is no common type which the existing coercion
mechanism could use to implement that operation.

Currently, PyInstance types are treated specially by the interpreter in
that their numeric methods are passed arguments of different types.
Removing this special case simplifies the interpreter and allows other
types to implement numeric methods that behave like instance types. This
is especially useful for extension types like ExtensionClass.

Specification

Instead of using a central coercion method, the process of handling
different operand types is simply left to the operation. If the
operation finds that it cannot handle the given operand type
combination, it may return a special singleton as indicator.

Note that "numbers" (anything that implements the number protocol, or
part of it) written in Python already use the first part of this
strategy - it is the C level API that we focus on here.

To maintain nearly 100% backward compatibility we have to be very
careful to make numbers that don't know anything about the new strategy
(old style numbers) work just as well as those that expect the new
scheme (new style numbers). Furthermore, binary compatibility is a must,
meaning that the interpreter may only access and use new style
operations if the number indicates the availability of these.

A new style number is considered by the interpreter as such if and only
if it sets the type flag Py_TPFLAGS_CHECKTYPES. The main difference
between an old style number and a new style one is that the numeric slot
functions can no longer assume to be passed arguments of identical type.
New style slots must check all arguments for proper type and implement
the necessary conversions themselves. This may seem to cause more work
on the behalf of the type implementor, but is in fact no more difficult
than writing the same kind of routines for an old style coercion slot.

If a new style slot finds that it cannot handle the passed argument type
combination, it may return a new reference of the special singleton
Py_NotImplemented to the caller. This will cause the caller to try the
other operands operation slots until it finds a slot that does implement
the operation for the specific type combination. If none of the possible
slots succeed, it raises a TypeError.

To make the implementation easy to understand (the whole topic is
esoteric enough), a new layer in the handling of numeric operations is
introduced. This layer takes care of all the different cases that need
to be taken into account when dealing with all the possible combinations
of old and new style numbers. It is implemented by the two static
functions binary_op() and ternary_op(), which are both internal
functions that only the functions in Objects/abstract.c have access to.
The numeric API (PyNumber_*) is easy to adapt to this new layer.

As a side-effect all numeric slots can be NULL-checked (this has to be
done anyway, so the added feature comes at no extra cost).

The scheme used by the layer to execute a binary operation is as
follows:

  v     w     Action taken
  ----- ----- -----------------------------------
  new   new   v.op(v,w), w.op(v,w)
  new   old   v.op(v,w), coerce(v,w), v.op(v,w)
  old   new   w.op(v,w), coerce(v,w), v.op(v,w)
  old   old   coerce(v,w), v.op(v,w)

The indicated action sequence is executed from left to right until
either the operation succeeds and a valid result (!= Py_NotImplemented)
is returned or an exception is raised. Exceptions are returned to the
calling function as-is. If a slot returns Py_NotImplemented, the next
item in the sequence is executed.

Note that coerce(v,w) will use the old style nb_coerce slot methods via
a call to PyNumber_Coerce().

Ternary operations have a few more cases to handle:

  v     w     z     Action taken
  ----- ----- ----- ------------------------------------------------------
  new   new   new   v.op(v,w,z), w.op(v,w,z), z.op(v,w,z)
  new   old   new   v.op(v,w,z), z.op(v,w,z), coerce(v,w,z), v.op(v,w,z)
  old   new   new   w.op(v,w,z), z.op(v,w,z), coerce(v,w,z), v.op(v,w,z)
  old   old   new   z.op(v,w,z), coerce(v,w,z), v.op(v,w,z)
  new   new   old   v.op(v,w,z), w.op(v,w,z), coerce(v,w,z), v.op(v,w,z)
  new   old   old   v.op(v,w,z), coerce(v,w,z), v.op(v,w,z)
  old   new   old   w.op(v,w,z), coerce(v,w,z), v.op(v,w,z)
  old   old   old   coerce(v,w,z), v.op(v,w,z)

The same notes as above, except that coerce(v,w,z) actually does:

    if z != Py_None:
        coerce(v,w), coerce(v,z), coerce(w,z)
    else:
        # treat z as absent variable
        coerce(v,w)

The current implementation uses this scheme already (there's only one
ternary slot: nb_pow(a,b,c)).

Note that the numeric protocol is also used for some other related
tasks, e.g. sequence concatenation. These can also benefit from the new
mechanism by implementing right-hand operations for type combinations
that would otherwise fail to work. As an example, take string
concatenation: currently you can only do string + string. With the new
mechanism, a new string-like type could implement new_type + string and
string + new_type, even though strings don't know anything about
new_type.

Since comparisons also rely on coercion (every time you compare an
integer to a float, the integer is first converted to float and then
compared...), a new slot to handle numeric comparisons is needed:

    PyObject *nb_cmp(PyObject *v, PyObject *w)

This slot should compare the two objects and return an integer object
stating the result. Currently, this result integer may only be -1, 0, 1.
If the slot cannot handle the type combination, it may return a
reference to Py_NotImplemented. [XXX Note that this slot is still in
flux since it should take into account rich comparisons (i.e. PEP 207).]

Numeric comparisons are handled by a new numeric protocol API:

    PyObject *PyNumber_Compare(PyObject *v, PyObject *w)

This function compare the two objects as "numbers" and return an integer
object stating the result. Currently, this result integer may only be
-1, 0, 1. In case the operation cannot be handled by the given objects,
a TypeError is raised.

The PyObject_Compare() API needs to adjusted accordingly to make use of
this new API.

Other changes include adapting some of the built-in functions (e.g.
cmp()) to use this API as well. Also, PyNumber_CoerceEx() will need to
check for new style numbers before calling the nb_coerce slot. New style
numbers don't provide a coercion slot and thus cannot be explicitly
coerced.

Reference Implementation

A preliminary patch for the CVS version of Python is available through
the Source Forge patch manager[2].

Credits

This PEP and the patch are heavily based on work done by Marc-André
Lemburg[3].

Copyright

This document has been placed in the public domain.

References

[1] http://www.lemburg.com/files/python/mxDateTime.html

[2] http://sourceforge.net/patch/?func=detailpatch&patch_id=102652&group_id=5470

[3] http://www.lemburg.com/files/python/CoercionProposal.html

authors:
Marc-André Lemburg
Neil Schemenauer