text:
PEP: 501 Title: General purpose template literal strings Author: Alyssa
Coghlan<EMAIL_ADDRESS>Nick Humrich<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pep-501-reopen-general-purpose-string-template-literals/24625
Status: Draft Type: Standards Track Content-Type: text/x-rst Requires:
701 Created: 08-Aug-2015 Python-Version: 3.12 Post-History: 08-Aug-2015,
05-Sep-2015, 09-Mar-2023,

Abstract

Though easy and elegant to use, Python f-strings can be vulnerable to
injection attacks when used to construct shell commands, SQL queries,
HTML snippets and similar (for example,
os.system(f"echo {message_from_user}")). This PEP introduces template
literal strings (or "t-strings"), which have syntax and semantics that
are similar to f-strings, but with rendering deferred until format or
another template rendering function is called on them. This will allow
standard library calls, helper functions and third party tools to safety
and intelligently perform appropriate escaping and other string
processing on inputs while retaining the usability and convenience of
f-strings.

Relationship with other PEPs

This PEP is inpired by and builds on top of the f-string syntax first
implemented in PEP 498 and formalised in PEP 701.

This PEP complements the literal string typing support added to Python's
formal type system in PEP 675 by introducing a safe way to do dynamic
interpolation of runtime values into security sensitive strings.

This PEP competes with some aspects of the tagged string proposal in PEP
750 (most notably in whether template rendering is expressed as
render(t"template literal") or as render"template literal"), but also
shares many common features (after PEP 750 was published, this PEP was
updated with several new changes inspired by the tagged strings
proposal).

This PEP does NOT propose an alternative to PEP 292 for user interface
internationalization use cases (but does note the potential for future
syntactic enhancements aimed at that use case that would benefit from
the compiler-supported value interpolation machinery that this PEP and
PEP 750 introduce).

Motivation

PEP 498 added new syntactic support for string interpolation that is
transparent to the compiler, allowing name references from the
interpolation operation full access to containing namespaces (as with
any other expression), rather than being limited to explicit name
references. These are referred to in the PEP (and elsewhere) as
"f-strings" (a mnemonic for "formatted strings").

Since acceptance of PEP 498, f-strings have become well-established and
very popular. f-strings became even more useful and flexible with the
formalised grammar in PEP 701. While f-strings are great, eager
rendering has its limitations. For example, the eagerness of f-strings
has made code like the following unfortunately plausible:

    os.system(f"echo {message_from_user}")

This kind of code is superficially elegant, but poses a significant
problem if the interpolated value message_from_user is in fact provided
by an untrusted user: it's an opening for a form of code injection
attack, where the supplied user data has not been properly escaped
before being passed to the os.system call.

While the LiteralString type annotation introduced in PEP 675 means that
typecheckers are able to report a type error for this kind of unsafe
function usage, those errors don't help make it easier to write code
that uses safer alternatives (such as subprocess.run).

To address that problem (and a number of other concerns), this PEP
proposes the complementary introduction of "t-strings" (a mnemonic for
"template literal strings"), where format(t"Message with {data}") would
produce the same result as f"Message with {data}", but the template
literal instance can instead be passed to other template rendering
functions which process the contents of the template differently.

Proposal

Dedicated template literal syntax

This PEP proposes a new string prefix that declares the string to be a
template literal rather than an ordinary string:

    template = t"Substitute {names:>{field_width}} and {expressions()!r} at runtime"

This would be effectively interpreted as:

    template = TemplateLiteral(
        r"Substitute {names:>{field_width}} and {expressions()} at runtime",
        TemplateLiteralText(r"Substitute "),
        TemplateLiteralField("names", names, f">{field_width}", ""),
        TemplateLiteralText(r" and "),
        TemplateLiteralField("expressions()", expressions(), f"", "r"),
    )

(Note: this is an illustrative example implementation. The exact compile
time construction syntax of types.TemplateLiteral is considered an
implementation detail not specified by the PEP. In particular, the
compiler may bypass the default constructor's runtime logic that detects
consecutive text segments and merges them into a single text segment, as
well as checking the runtime types of all supplied arguments).

The __format__ method on types.TemplateLiteral would then implement the
following str.format inspired semantics:

    >>> import datetime
    >>> name = 'Jane'
    >>> age = 50
    >>> anniversary = datetime.date(1991, 10, 12)
    >>> format(t'My name is {name}, my age next year is {age+1}, my anniversary is {anniversary:%A, %B %d, %Y}.')
    'My name is Jane, my age next year is 51, my anniversary is Saturday, October 12, 1991.'
    >>> format(t'She said her name is {name!r}.')
    "She said her name is 'Jane'."

The syntax of template literals would be based on PEP 701, and largely
use the same syntax for the string portion of the template. Aside from
using a different prefix, the one other syntactic change is in the
definition and handling of conversion specifiers, both to allow !() as a
standard conversion specifier to request evaluation of a field at
rendering time, and to allow custom renderers to also define custom
conversion specifiers.

This PEP does not propose to remove or deprecate any of the existing
string formatting mechanisms, as those will remain valuable when
formatting strings that are not present directly in the source code of
the application.

Lazy field evaluation conversion specifier

In addition to the existing support for the a, r, and s conversion
specifiers, str.format, str.format_map, and string.Formatter will be
updated to accept () as a conversion specifier that means "call the
interpolated value".

To support application of the standard conversion specifiers in custom
template rendering functions, a new !operator.convert_field function
will be added.

The signature and behaviour of the format builtin will also be updated
to accept a conversion specifier as a third optional parameter. If a
non-empty conversion specifier is given, the value will be converted
with !operator.convert_field before looking up the __format__ method.

Custom conversion specifiers

To allow additional field-specific directives to be passed to custom
rendering functions in a way that still allows formatting of the
template with the default renderer, the conversion specifier field will
be allowed to contain a second ! character.

!operator.convert_field and format (and hence the default
TemplateLiteral.render template rendering method), will ignore that
character and any subsequent text in the conversion specifier field.

str.format, str.format_map, and string.Formatter will also be updated to
accept (and ignore) custom conversion specifiers.

Template renderer for POSIX shell commands

As both a practical demonstration of the benefits of delayed rendering
support, and as a valuable feature in its own right, a new sh template
renderer will be added to the shlex module. This renderer will produce
strings where all interpolated fields are escaped with shlex.quote.

The subprocess.Popen API (and higher level APIs that depend on it, such
as subprocess.run) will be updated to accept interpolation templates and
handle them in accordance with the new shlex.sh renderer.

Background

This PEP was initially proposed as a competitor to PEP 498. After it
became clear that the eager rendering proposal had sustantially more
immediate support, it then spent several years in a deferred state,
pending further experience with PEP 498's simpler approach of only
supporting eager rendering without the additional complexity of also
supporting deferred rendering.

Since then, f-strings have become very popular and PEP 701 was
introduced to tidy up some rough edges and limitations in their syntax
and semantics. The template literal proposal was updated in 2023 to
reflect current knowledge of f-strings, and improvements from PEP 701.

In 2024, PEP 750 was published, proposing a general purpose mechanism
for custom tagged string prefixes, rather than the narrower template
literal proposal in this PEP. This PEP was again updated, both to
incorporate new ideas inspired by the tagged strings proposal, and to
describe the perceived benefits of the narrower template literal syntax
proposal in this PEP over the more general tagged string proposal.

Summary of differences from f-strings

The key differences between f-strings and t-strings are:

-   the t (template literal) prefix indicates delayed rendering, but
    otherwise largely uses the same syntax and semantics as formatted
    strings
-   template literals are available at runtime as a new kind of object
    (types.TemplateLiteral)
-   the default rendering used by formatted strings is invoked on a
    template literal object by calling format(template) rather than
    being done implicitly in the compiled code
-   unlike f-strings (where conversion specifiers are handled directly
    in the compiler), t-string conversion specifiers are handled at
    rendering time by the rendering function
-   the new !() conversion specifier indicates that the field expression
    is a callable that should be called when using the default format
    rendering function. This specifier is specifically not being added
    to f-strings (since it is pointless there).
-   a second ! is allowed in t-string conversion specifiers (with any
    subsequent text being ignored) as a way to allow custom template
    rendering functions to accept custom conversion specifiers without
    breaking the default !TemplateLiteral.render rendering method. This
    feature is specifically not being added to f-strings (since it is
    pointless there).
-   while f-string f"Message {here}" would be semantically equivalent to
    format(t"Message {here}"), f-strings will continue to be supported
    directly in the compiler and hence avoid the runtime overhead of
    actually using the delayed rendering machinery that is needed for
    t-strings

Summary of differences from tagged strings

When tagged strings were first proposed, there were several notable
differences from the proposal in PEP 501 beyond the surface syntax
difference between whether rendering function invocations are written as
render(t"template literal") or as render"template literal".

Over the course of the initial PEP 750 discussion, many of those
differences were eliminated, either by PEP 501 adopting that aspect of
PEP 750's proposal (such as lazily applying conversion specifiers), or
by PEP 750 changing to retain some aspect of PEP 501's proposal (such as
defining a dedicated type to hold template segments rather than
representing them as simple sequences).

The main remaining significant difference is that this PEP argues that
adding only the t-string prefix is a sufficient enhancement to give all
the desired benefits described in PEP 750. The expansion to a
generalised "tagged string" syntax isn't necessary, and causes
additional problems that can be avoided.

The two PEPs also differ in their proposed approaches to handling lazy
evaluation of template fields.

While there are other differences between the two proposals, those
differences are more cosmetic than substantive. In particular:

-   this PEP proposes different names for the structural typing
    protocols
-   this PEP proposes specific names for the concrete implementation
    types
-   this PEP proposes exact details for the proposed APIs of the
    concrete implementation types (including concatenation and
    repetition support, which are not part of the structural typing
    protocols)
-   this PEP proposes changes to the existing format builtin to make it
    usable directly as a template field renderer

The two PEPs also differ in how they make their case for delayed
rendering support. This PEP focuses more on the concrete implementation
concept of using template literals to allow the "interpolation" and
"rendering" steps in f-string processing to be separated in time, and
then taking advantage of that to reduce the potential code injection
risks associated with misuse of f-strings. PEP 750 focuses more on the
way that native templating support allows behaviours that are difficult
or impossible to achieve via existing string based templating methods.
As with the cosmetic differences noted above, this is more a difference
in style than a difference in substance.

Rationale

f-strings (PEP 498) made interpolating values into strings with full
access to Python's lexical namespace semantics simpler, but it does so
at the cost of creating a situation where interpolating values into
sensitive targets like SQL queries, shell commands and HTML templates
will enjoy a much cleaner syntax when handled without regard for code
injection attacks than when they are handled correctly.

This PEP proposes to provide the option of delaying the actual rendering
of a template literal to a formatted string to its __format__ method,
allowing the use of other template renderers by passing the template
around as a first class object.

While very different in the technical details, the types.TemplateLiteral
interface proposed in this PEP is conceptually quite similar to the
FormattableString type underlying the native interpolation support
introduced in C# 6.0, as well as the JavaScript template literals
introduced in ES6.

While not the original motivation for developing the proposal, many of
the benefits for defining domain specific languages described in PEP 750
also apply to this PEP (including the potential for per-DSL semantic
highlighting in code editors based on the type specifications of
declared template variables and rendering function parameters).

Specification

This PEP proposes a new t string prefix that results in the creation of
an instance of a new type, types.TemplateLiteral.

Template literals are Unicode strings (bytes literals are not
permitted), and string literal concatenation operates as normal, with
the entire combined literal forming the template literal.

The template string is parsed into literals, expressions, format
specifiers, and conversion specifiers as described for f-strings in PEP
498 and PEP 701. The syntax for conversion specifiers is relaxed such
that arbitrary strings are accepted (excluding those containing {, } or
:) rather than being restricted to valid Python identifiers.

However, rather than being rendered directly into a formatted string,
these components are instead organised into instances of new types with
the following behaviour:

    class TemplateLiteralText(str):
        # This is a renamed and extended version of the DecodedConcrete type in PEP 750
        # Real type would be implemented in C, this is an API compatible Python equivalent
        _raw: str

        def __new__(cls, raw: str):
            decoded = raw.encode("utf-8").decode("unicode-escape")
            if decoded == raw:
                decoded = raw
            text = super().__new__(cls, decoded)
            text._raw = raw
            return text

        @staticmethod
        def merge(text_segments:Sequence[TemplateLiteralText]) -> TemplateLiteralText:
            if len(text_segments) == 1:
                return text_segments[0]
            return TemplateLiteralText("".join(t._raw for t in text_segments))

        @property
        def raw(self) -> str:
            return self._raw

        def __repr__(self) -> str:
            return f"{type(self).__name__}(r{self._raw!r})"

        def __add__(self, other:Any) -> TemplateLiteralText|NotImplemented:
            if isinstance(other, TemplateLiteralText):
                return TemplateLiteralText(self._raw + other._raw)
            return NotImplemented

        def __mul__(self, other:Any) -> TemplateLiteralText|NotImplemented:
            try:
                factor = operator.index(other)
            except TypeError:
                return NotImplemented
            return TemplateLiteralText(self._raw * factor)
        __rmul__ = __mul__

    class TemplateLiteralField(NamedTuple):
        # This is mostly a renamed version of the InterpolationConcrete type in PEP 750
        # However:
        #    - value is eagerly evaluated (values were all originally lazy in PEP 750)
        #    - conversion specifiers are allowed to be arbitrary strings
        #    - order of fields is adjusted so the text form is the first field and the
        #      remaining parameters match the updated signature of the `*format` builtin
        # Real type would be implemented in C, this is an API compatible Python equivalent

        expr: str
        value: Any
        format_spec: str | None = None
        conversion_spec: str | None = None

        def __repr__(self) -> str:
            return (f"{type(self).__name__}({self.expr}, {self.value!r}, "
                    f"{self.format_spec!r}, {self.conversion_spec!r})")

        def __str__(self) -> str:
            return format(self.value, self.format_spec, self.conversion_spec)

        def __format__(self, format_override) -> str:
            if format_override:
                format_spec = format_override
            else:
                format_spec = self.format_spec
            return format(self.value, format_spec, self.conversion_spec)

    class TemplateLiteral:
        # This type corresponds to the TemplateConcrete type in PEP 750
        # Real type would be implemented in C, this is an API compatible Python equivalent
        _raw_template: str
        _segments = tuple[TemplateLiteralText|TemplateLiteralField]

        def __new__(cls, raw_template:str, *segments:TemplateLiteralText|TemplateLiteralField):
            self = super().__new__(cls)
            self._raw_template = raw_template
            # Check if there are any adjacent text segments that need merging
            # or any empty text segments that need discarding
            type_err = "Template literal segments must be template literal text or field instances"
            text_expected = True
            needs_merge = False
            for segment in segments:
                match segment:
                    case TemplateLiteralText():
                        if not text_expected or not segment:
                            needs_merge = True
                            break
                        text_expected = False
                    case TemplateLiteralField():
                        text_expected = True
                    case _:
                        raise TypeError(type_err)
            if not needs_merge:
                # Match loop above will have checked all segments
                self._segments = segments
                return self
            # Merge consecutive runs of text fields and drop any empty text fields
            merged_segments:list[TemplateLiteralText|TemplateLiteralField] = []
            pending_merge:list[TemplateLiteralText] = []
            for segment in segments:
                match segment:
                    case TemplateLiteralText() as text_segment:
                        if text_segment:
                            pending_merge.append(text_segment)
                    case TemplateLiteralField():
                        if pending_merge:
                            merged_segments.append(TemplateLiteralText.merge(pending_merge))
                            pending_merge.clear()
                        merged_segments.append(segment)
                    case _:
                        # First loop above may not check all segments when a merge is needed
                        raise TypeError(type_err)
            if pending_merge:
                merged_segments.append(TemplateLiteralText.merge(pending_merge))
                pending_merge.clear()
            self._segments = tuple(merged_segments)
            return self

        @property
        def raw_template(self) -> str:
            return self._raw_template

        @property
        def segments(self) -> tuple[TemplateLiteralText|TemplateLiteralField]:
            return self._segments

        def __len__(self) -> int:
            return len(self._segments)

        def __iter__(self) -> Iterable[TemplateLiteralText|TemplateLiteralField]:
            return iter(self._segments)

        # Note: template literals do NOT define any relative ordering
        def __eq__(self, other):
            if not isinstance(other, TemplateLiteral):
                return NotImplemented
            return (
                self._raw_template == other._raw_template
                and self._segments == other._segments
                and self.field_values == other.field_values
                and self.format_specifiers == other.format_specifiers
            )

        def __repr__(self) -> str:
            return (f"{type(self).__name__}(r{self._raw!r}, "
                    f"{', '.join(map(repr, self._segments))})")

        def __format__(self, format_specifier) -> str:
            # When formatted, render to a string, and then use string formatting
            return format(self.render(), format_specifier)

        def render(self, *, render_template=''.join, render_text=str, render_field=format):
            ...  # See definition of the template rendering semantics below

        def __add__(self, other) -> TemplateLiteral|NotImplemented:
            if isinstance(other, TemplateLiteral):
                combined_raw_text = self._raw + other._raw
                combined_segments = self._segments + other._segments
                return TemplateLiteral(combined_raw_text, *combined_segments)
            if isinstance(other, str):
                # Treat the given string as a new raw text segment
                combined_raw_text = self._raw + other
                combined_segments = self._segments + (TemplateLiteralText(other),)
                return TemplateLiteral(combined_raw_text, *combined_segments)
            return NotImplemented

        def __radd__(self, other) -> TemplateLiteral|NotImplemented:
            if isinstance(other, str):
                # Treat the given string as a new raw text segment. This effectively
                # has precedence over string concatenation in CPython due to
                # https://github.com/python/cpython/issues/55686
                combined_raw_text = other + self._raw
                combined_segments = (TemplateLiteralText(other),) + self._segments
                return TemplateLiteral(combined_raw_text, *combined_segments)
            return NotImplemented

        def __mul__(self, other) -> TemplateLiteral|NotImplemented:
            try:
                factor = operator.index(other)
            except TypeError:
                return NotImplemented
            if not self or factor == 1:
                return self
            if factor < 1:
                return TemplateLiteral("")
            repeated_text = self._raw_template * factor
            repeated_segments = self._segments * factor
            return TemplateLiteral(repeated_text, *repeated_segments)
        __rmul__ = __mul__

(Note: this is an illustrative example implementation, the exact compile
time construction method and internal data management details of
types.TemplateLiteral are considered an implementation detail not
specified by the PEP. However, the expected post-construction behaviour
of the public APIs on types.TemplateLiteral instances is specified by
the above code, as is the constructor signature for building template
instances at runtime)

The result of a template literal expression is an instance of this type,
rather than an already rendered string. Rendering only takes place when
the instance's render method is called (either directly, or indirectly
via __format__).

The compiler will pass the following details to the template literal for
later use:

-   a string containing the raw template as written in the source code
-   a sequence of template segments, with each segment being either:
    -   a literal text segment (a regular Python string that also
        provides access to its raw form)
    -   a parsed template interpolation field, specifying the text of
        the interpolated expression (as a regular string), its evaluated
        result, the format specifier text (with any substitution fields
        eagerly evaluated as an f-string), and the conversion specifier
        text (as a regular string)

The raw template is just the template literal as a string. By default,
it is used to provide a human-readable representation for the template
literal, but template renderers may also use it for other purposes (e.g.
as a cache lookup key).

The parsed template structure is taken from PEP 750 and consists of a
sequence of template segments corresponding to the text segments and
interpolation fields in the template string.

This approach is designed to allow compilers to fully process each
segment of the template in order, before finally emitting code to pass
all of the template segments to the template literal constructor.

For example, assuming the following runtime values:

    names = ["Alice", "Bob", "Carol", "Eve"]
    field_width = 10
    def expressions():
        return 42

The template from the proposal section would be represented at runtime
as:

    TemplateLiteral(
        r"Substitute {names:>{field_width}} and {expressions()!r} at runtime",
        TemplateLiteralText(r"Substitute "),
        TemplateLiteralField("names", ["Alice", "Bob", "Carol", "Eve"], ">10", ""),
        TemplateLiteralText(r" and "),
        TemplateLiteralField("expressions()", 42, "", "r"),
    )

Rendering templates

The TemplateLiteral.render implementation defines the rendering process
in terms of the following renderers:

-   an overall render_template operation that defines how the sequence
    of rendered text and field segments are composed into a fully
    rendered result. The default template renderer is string
    concatenation using ''.join.
-   a per text segment render_text operation that receives the
    individual literal text segments within the template. The default
    text renderer is the builtin str constructor.
-   a per field segment render_field operation that receives the field
    value, format specifier, and conversion specifier for substitution
    fields within the template. The default field renderer is the format
    builtin.

Given the parsed template representation above, the semantics of
template rendering would then be equivalent to the following:

    def render(self, *, render_template=''.join, render_text=str, render_field=format):
        rendered_segments = []
        for segment in self._segments:
            match segment:
                case TemplateLiteralText() as text_segment:
                    rendered_segments.append(render_text(text_segment))
                case TemplateLiteralField() as field_segment:
                    rendered_segments.append(render_field(*field_segment[1:]))
        return render_template(rendered_segments)

Format specifiers

The syntax and processing of field specifiers in t-strings is defined to
be the same as it is for f-strings.

This includes allowing field specifiers to themselves contain f-string
substitution fields. The raw text of the field specifiers (without
processing any substitution fields) is retained as part of the full raw
template string.

The parsed field specifiers receive the field specifier string with
those substitutions already resolved. The : prefix is also omitted.

Aside from separating them out from the substitution expression during
parsing, format specifiers are otherwise treated as opaque strings by
the interpolation template parser - assigning semantics to those (or,
alternatively, prohibiting their use) is handled at rendering time by
the field renderer.

Conversion specifiers

In addition to the existing support for a, r, and s conversion
specifiers, str.format and str.format_map will be updated to accept ()
as a conversion specifier that means "call the interpolated value".

Where PEP 701 restricts conversion specifiers to NAME tokens, this PEP
will instead allow FSTRING_MIDDLE tokens (such that only {, } and : are
disallowed). This change is made primarily to support lazy field
rendering with the !() conversion specifier, but also allows custom
rendering functions more flexibility when defining their own conversion
specifiers in preference to those defined for the default format field
renderer.

Conversion specifiers are still handled as plain strings, and do NOT
support the use of substitution fields.

The parsed conversion specifiers receive the conversion specifier string
with the ! prefix omitted.

To allow custom template renderers to define their own custom conversion
specifiers without causing the default renderer to fail, conversion
specifiers will be permitted to contain a custom suffix prefixed with a
second ! character. That is, !!<custom>, !a!<custom>, !r!<custom>,
!s!<custom>, and !()!<custom> would all be valid conversion specifiers
in a template literal.

As described above, the default rendering supports the original !a, !r
and !s conversion specifiers defined in PEP 3101, together with the new
!() lazy field evaluation conversion specifier defined in this PEP. The
default rendering ignores any custom conversion specifier suffixes.

The full mapping between the standard conversion specifiers and the
special methods called on the interpolated value when the field is
rendered:

-   No conversion (empty string): __format__ (with format specifier as
    parameter)
-   a: __repr__ (as per the ascii builtin)
-   r: __repr__ (as per the repr builtin)
-   s: __str__ (as per the str builtin)
-   (): __call__ (with no parameters)

When a conversion occurs, __format__ (with the format specifier) is
called on the result of the conversion rather than being called on the
original object.

The changes to format and the addition of !operator.convert_field make
it straightforward for custom renderers to also support the standard
conversion specifiers.

f-strings themselves will NOT support the new !() conversion specifier
(as it is redundant when value interpolation and value rendering always
occur at the same time). They also will NOT support the use of custom
conversion specifiers (since the rendering function is known at compile
time and doesn't make use of the custom specifiers).

New field conversion API in the operator module

To support application of the standard conversion specifiers in custom
template rendering functions, a new !operator.convert_field function
will be added:

    def convert_field(value, conversion_spec=''):
        """Apply the given string formatting conversion specifier to the given value"""
        std_spec, sep, custom_spec = conversion_spec.partition("!")
        match std_spec:
            case '':
                return value
            case 'a':
                return ascii(value)
            case 'r':
                return repr(value)
            case 's':
                return str(value)
            case '()':
                return value()
        if not sep:
            err = f"Invalid conversion specifier {std_spec!r}"
        else:
            err = f"Invalid conversion specifier {std_spec!r} in {conversion_spec!r}"
        raise ValueError(f"{err}: expected '', 'a', 'r', 's' or '()')

Conversion specifier parameter added to format

The signature and behaviour of the format builtin will be updated:

    def format(value, format_spec='', conversion_spec=''):
        if conversion_spec:
            value_to_format = operator.convert_field(value)
        else:
            value_to_format = value
        return type(value_to_format).__format__(value, format_spec)

If a non-empty conversion specifier is given, the value will be
converted with !operator.convert_field before looking up the __format__
method.

The signature of the __format__ special method does NOT change (only
format specifiers are handled by the object being formatted).

Structural typing and duck typing

To allow custom renderers to accept alternative interpolation template
implementations (rather than being tightly coupled to the native
template literal types), the following structural protocols will be
added to the typing module:

    @runtime_checkable
    class TemplateText(Protocol):
        # Renamed version of PEP 750's Decoded protocol
        def __str__(self) -> str:
            ...

        raw: str

    @runtime_checkable
    class TemplateField(Protocol):
        # Renamed and modified version of PEP 750's Interpolation protocol
        def __len__(self):
            ...

        def __getitem__(self, index: int):
            ...

        def __str__(self) -> str:
            ...

        expr: str
        value: Any
        format_spec: str | None = None
        conversion_spec: str | None = None

    @runtime_checkable
    class InterpolationTemplate(Protocol):
        # Corresponds to PEP 750's Template protocol
        def __iter__(self) -> Iterable[TemplateText|TemplateField]:
            ...

        raw_template: str

Note that the structural protocol APIs are substantially narrower than
the full implementation APIs defined for TemplateLiteralText,
TemplateLiteralField, and TemplateLiteral.

Code that wants to accept interpolation templates and define specific
handling for them without introducing a dependency on the typing module,
or restricting the code to handling the concrete template literal types,
should instead perform an attribute existence check on raw_template.

Writing custom renderers

Writing a custom renderer doesn't require any special syntax. Instead,
custom renderers are ordinary callables that process an interpolation
template directly either by calling the render() method with alternate
render_template, render_text, and/or render_field implementations, or by
accessing the template's data attributes directly.

For example, the following function would render a template using
objects' repr implementations rather than their native formatting
support:

    def repr_format(template):
        def render_field(value, format_spec, conversion_spec):
            converted_value = operator.convert_field(value, conversion_spec)
            return format(repr(converted_value), format_spec)
        return template.render(render_field=render_field)

The customer renderer shown respects the conversion specifiers in the
original template, but it is also possible to ignore them and render the
interpolated values directly:

    def input_repr_format(template):
        def render_field(value, format_spec, __):
            return format(repr(value), format_spec)
        return template.render(render_field=render_field)

When writing custom renderers, note that the return type of the overall
rendering operation is determined by the return type of the passed in
render_template callable. While this will still be a string for
formatting related use cases, producing non-string objects is permitted.
For example, a custom SQL template renderer could involve an
sqlalchemy.sql.text call that produces an SQL Alchemy query object. A
subprocess invocation related template renderer could produce a string
sequence suitable for passing to subprocess.run, or it could even call
subprocess.run directly, and return the result.

Non-strings may also be returned from render_text and render_field, as
long as they are paired with a render_template implementation that
expects that behaviour.

Custom renderers using the pattern matching style described in PEP 750
are also supported:

    # Use the structural typing protocols rather than the concrete implementation types
    from typing import InterpolationTemplate, TemplateText, TemplateField

    def greet(template: InterpolationTemplate) -> str:
        """Render an interpolation template using structural pattern matching."""
        result = []
        for segment in template:
            match segment:
                match segment:
                    case TemplateText() as text_segment:
                        result.append(text_segment)
                    case TemplateField() as field_segment:
                        result.append(str(field_segment).upper())
        return f"{''.join(result)}!"

Expression evaluation

As with f-strings, the subexpressions that are extracted from the
interpolation template are evaluated in the context where the template
literal appears. This means the expression has full access to local,
nonlocal and global variables. Any valid Python expression can be used
inside {}, including function and method calls.

Because the substitution expressions are evaluated where the string
appears in the source code, there are no additional security concerns
related to the contents of the expression itself, as you could have also
just written the same expression and used runtime field parsing:

    >>> bar=10
    >>> def foo(data):
    ...   return data + 20
    ...
    >>> str(t'input={bar}, output={foo(bar)}')
    'input=10, output=30'

Is essentially equivalent to:

    >>> 'input={}, output={}'.format(bar, foo(bar))
    'input=10, output=30'

Handling code injection attacks

The PEP 498 formatted string syntax makes it potentially attractive to
write code like the following:

    runquery(f"SELECT {column} FROM {table};")
    runcommand(f"cat {filename}")
    return_response(f"<html><body>{response.body}</body></html>")

These all represent potential vectors for code injection attacks, if any
of the variables being interpolated happen to come from an untrusted
source. The specific proposal in this PEP is designed to make it
straightforward to write use case specific renderers that take care of
quoting interpolated values appropriately for the relevant security
context:

    runquery(sql(t"SELECT {column} FROM {table} WHERE column={value};"))
    runcommand(sh(t"cat {filename}"))
    return_response(html(t"<html><body>{response.body}</body></html>"))

This PEP does not cover adding all such renderers to the standard
library immediately (though one for shell escaping is proposed), but
rather proposes to ensure that they can be readily provided by third
party libraries, and potentially incorporated into the standard library
at a later date.

Over time, it is expected that APIs processing potentially dangerous
string inputs may be updated to accept interpolation templates natively,
allowing problematic code examples to be fixed simply by replacing the f
string prefix with a t:

    runquery(t"SELECT {column} FROM {table};")
    runcommand(t"cat {filename}")
    return_response(t"<html><body>{response.body}</body></html>")

It is proposed that a renderer is included in the shlex module, aiming
to offer a more POSIX shell style experience for accessing external
programs, without the significant risks posed by running os.system or
enabling the system shell when using the subprocess module APIs. This
renderer will provide an interface for running external programs
inspired by that offered by the Julia programming language, only with
the backtick based \`cat $filename\ syntax replaced by t"cat {filename}"
style template literals. See more in the :ref:`pep-501-shlex-module
section.

Error handling

Either compile time or run time errors can occur when processing
interpolation expressions. Compile time errors are limited to those
errors that can be detected when parsing a template string into its
component tuples. These errors all raise SyntaxError.

Unmatched braces:

    >>> t'x={x'
      File "<stdin>", line 1
          t'x={x'
             ^
    SyntaxError: missing '}' in template literal expression

Invalid expressions:

    >>> t'x={!x}'
      File "<fstring>", line 1
        !x
        ^
    SyntaxError: invalid syntax

Run time errors occur when evaluating the expressions inside a template
string before creating the template literal object. See PEP 498 for some
examples.

Different renderers may also impose additional runtime constraints on
acceptable interpolated expressions and other formatting details, which
will be reported as runtime exceptions.

Renderer for shell escaping added to shlex

As a reference implementation, a renderer for safe POSIX shell escaping
can be added to the shlex module. This renderer would be called sh and
would be equivalent to calling shlex.quote on each field value in the
template literal.

Thus:

    os.system(shlex.sh(t'cat {myfile}'))

would have the same behavior as:

    os.system('cat ' + shlex.quote(myfile)))

The implementation would be:

    def sh(template: TemplateLiteral):
        def render_field(value, format_spec, conversion_spec)
            field_text = format(value, format_spec, conversion_spec)
            return quote(field_text)
        return template.render(render_field=render_field)

The addition of shlex.sh will NOT change the existing admonishments in
the subprocess documentation that passing shell=True is best avoided,
nor the reference from the os.system documentation the higher level
subprocess APIs.

Changes to subprocess module

With the additional renderer in the shlex module, and the addition of
template literals, the subprocess module can be changed to handle
accepting template literals as an additional input type to Popen, as it
already accepts a sequence, or a string, with different behavior for
each.

With the addition of template literals, subprocess.Popen (and in return,
all its higher level functions such as subprocess.run) could accept
strings in a safe way (at least on
POSIX systems <pep-501-defer-non-posix-shells>).

For example:

    subprocess.run(t'cat {myfile}', shell=True)

would automatically use the shlex.sh renderer provided in this PEP.
Therefore, using shlex inside a subprocess.run call like so:

    subprocess.run(shlex.sh(t'cat {myfile}'), shell=True)

would be redundant, as run would automatically render any template
literals through shlex.sh

Alternatively, when subprocess.Popen is run without shell=True, it could
still provide subprocess with a more ergonomic syntax. For example:

    subprocess.run(t'cat {myfile} --flag {value}')

would be equivalent to:

    subprocess.run(['cat', myfile, '--flag', value])

or, more accurately:

    subprocess.run(shlex.split(f'cat {shlex.quote(myfile)} --flag {shlex.quote(value)}'))

It would do this by first using the shlex.sh renderer, as above, then
using shlex.split on the result.

The implementation inside subprocess.Popen._execute_child would look
like:

    if hasattr(args, "raw_template"):
        import shlex
        if shell:
            args = [shlex.sh(args)]
        else:
            args = shlex.split(shlex.sh(args))

How to Teach This

This PEP intentionally includes two standard renderers that will always
be available in teaching environments: the format builtin and the new
shlex.sh POSIX shell renderer.

Together, these two renderers can be used to build an initial
understanding of delayed rendering on top of a student's initial
introduction to string formatting with f-strings. This initial
understanding would have the goal of allowing students to use template
literals effectively, in combination with pre-existing template
rendering functions.

For example, f"{'some text'}", f"{value}", f"{value!r}", ,
f"{callable()}" could all be introduced.

Those same operations could then be rewritten as
format(t"{'some text'}"), format(t"{value}"), format(t"{value!r}"), ,
format(t"{callable()}") to illustrate the relationship between the eager
rendering form and the delayed rendering form.

The difference between "template definition time" (or "interpolation
time" ) and "template rendering time" can then be investigated further
by storing the template literals as local variables and looking at their
representations separately from the results of the format calls. At this
point, the t"{callable!()}" syntax can be introduced to distinguish
between field expressions that are called at template definition time
and those that are called at template rendering time.

Finally, the differences between the results of f"{'some text'}",
format(t"{'some text'}"), and shlex.sh(t"{'some text'}") could be
explored to illustrate the potential for differences between the default
rendering function and custom rendering functions.

Actually defining your own custom template rendering functions would
then be a separate more advanced topic (similar to the way students are
routinely taught to use decorators and context managers well before they
learn how to write their own custom ones).

PEP 750 includes further ideas for teaching aspects of the delayed
rendering topic.

Discussion

Refer to PEP 498 for previous discussion, as several of the points there
also apply to this PEP. PEP 750's design discussions are also highly
relevant, as that PEP inspired several aspects of the current design.

Support for binary interpolation

As f-strings don't handle byte strings, neither will t-strings.

Interoperability with str-only interfaces

For interoperability with interfaces that only accept strings,
interpolation templates can still be prerendered with format, rather
than delegating the rendering to the called function.

This reflects the key difference from PEP 498, which always eagerly
applies the default rendering, without any way to delegate the choice of
renderer to another section of the code.

Preserving the raw template string

Earlier versions of this PEP failed to make the raw template string
available on the template literal. Retaining it makes it possible to
provide a more attractive template representation, as well as providing
the ability to precisely reconstruct the original string, including both
the expression text and the details of any eagerly rendered substitution
fields in format specifiers.

Creating a rich object rather than a global name lookup

Earlier versions of this PEP used an __interpolate__ builtin, rather
than creating a new kind of object for later consumption by
interpolation functions. Creating a rich descriptive object with a
useful default renderer made it much easier to support customisation of
the semantics of interpolation.

Building atop f-strings rather than replacing them

Earlier versions of this PEP attempted to serve as a complete substitute
for PEP 498 (f-strings) . With the acceptance of that PEP and the more
recent PEP 701, this PEP can instead build a more flexible delayed
rendering capability on top of the existing f-string eager rendering.

Assuming the presence of f-strings as a supporting capability simplified
a number of aspects of the proposal in this PEP (such as how to handle
substitution fields in format specifiers).

Defining repetition and concatenation semantics

This PEP explicitly defines repetition and concatenation semantics for
TemplateLiteral and TemplateLiteralText. While not strictly necessary,
defining these is expected to make the types easier to work with in code
that historically only supported regular strings.

New conversion specifier for lazy field evaluation

The initially published version of PEP 750 defaulted to lazy evaluation
for all interpolation fields. While it was subsequently updated to
default to eager evaluation (as happens for f-strings and this PEP), the
discussions around the topic prompted the idea of providing a way to
indicate to rendering functions that the interpolated field value should
be called at rendering time rather than being used without modification.

Since PEP 750 also deferred the processing of conversion specifiers
until evaluation time, the suggestion was put forward that invoking
__call__ without arguments could be seen as similar to the existing
conversion specifiers that invoke __repr__ (!a, !r) or __str__ (!s).

Accordingly, this PEP was updated to also make conversion specifier
processing the responsibility of rendering functions, and to introduce
!() as a new conversion specifier for lazy evaluation.

Adding !operator.convert_field and updating the format builtin was than
a matter of providing appropriate support to rendering function
implementations that wanted to accept the default conversion specifiers.

Allowing arbitrary conversion specifiers in custom renderers

Accepting !() as a new conversion specifier necessarily requires
updating the syntax that the parser accepts for conversion specifiers
(they are currently restricted to identifiers). This then raised the
question of whether t-string compilation should enforce the additional
restriction that f-string compilation imposes: that the conversion
specifier be exactly one of !a, !r, or !s.

With t-strings already being updated to allow !() when compiled, it made
sense to treat conversion specifiers as relating to rendering function
similar to the way that format specifiers related to the formatting of
individual objects: aside from some characters that are excluded for
parsing reasons, they are otherwise free text fields with the meaning
decided by the consuming function or object. This reduces the temptation
to introduce renderer specific metaformatting into the template's format
specifiers (since any renderer specific information can be placed in the
conversion specifier instead).

Only reserving a single new string prefix

The primary difference between this PEP and PEP 750 is that the latter
aims to enable the use of arbitrary string prefixes, rather than
requiring the creation of template literal instances that are then
passed to other APIs. For example, PEP 750 would allow the sh render
described in this PEP to be used as sh"cat {somefile}" rather than
requiring the template literal to be created explicitly and then passed
to a regular function call (as in sh(t"cat {somefile}")).

The main reason the PEP authors prefer the second spelling is because it
makes it clearer to a reader what is going on: a template literal
instance is being created, and then passed to a callable that knows how
to do something useful with interpolation template instances.

A draft proposal from one of the PEP 750 authors also suggests that
static typecheckers will be able to infer the use of particular domain
specific languages just as readily from the form that uses an explicit
function call as they would be able to infer it from a directly tagged
string.

With the tagged string syntax at least arguably reducing clarity for
human readers without increasing the overall expressiveness of the
construct, it seems reasonable to start with the smallest viable
proposal (a single new string prefix), and then revisit the potential
value of generalising to arbitrary prefixes in the future.

As a lesser, but still genuine, consideration, only using a single new
string prefix for this use case leaves open the possibility of defining
alternate prefixes in the future that still produce TemplateLiteral
objects, but use a different syntax within the string to define the
interpolation fields (see the i18n discussion <pep-501-defer-i18n>
below).

Deferring consideration of more concise delayed evaluation syntax

During the discussions of delayed evaluation, {-> expr} was suggested as
potential syntactic sugar for the already supported lambda based syntax:
{(lambda: expr)} (the parentheses are required in the existing syntax to
avoid misinterpretation of the : character as indicating the start of
the format specifier).

While adding such a spelling would complement the rendering time
function call syntax proposed in this PEP (that is, writing {-> expr!()}
to evaluate arbitrary expressions at rendering time), it is a topic that
the PEP authors consider to be better left to a future PEP if this PEP
or PEP 750 is accepted.

Deferring consideration of possible logging integration

One of the challenges with the logging module has been that we have
previously been unable to devise a reasonable migration strategy away
from the use of printf-style formatting. While the logging module does
allow formatters to specify the use of str.format or string.Template
style substitution, it can be awkward to ensure that messages written
that way are only ever processed by log record formatters that are
expecting that syntax.

The runtime parsing and interpolation overhead for logging messages also
poses a problem for extensive logging of runtime events for monitoring
purposes.

While beyond the scope of this initial PEP, template literal support
could potentially be added to the logging module's event reporting APIs,
permitting relevant details to be captured using forms like:

    logging.debug(t"Event: {event}; Details: {data}")
    logging.critical(t"Error: {error}; Details: {data}")

Rather than the historical mod-formatting style:

    logging.debug("Event: %s; Details: %s", event, data)
    logging.critical("Error: %s; Details: %s", event, data)

As the template literal is passed in as an ordinary argument, other
keyword arguments would also remain available:

    logging.critical(t"Error: {error}; Details: {data}", exc_info=True)

The approach to standardising lazy field evaluation described in this
PEP is primarily based on the anticipated needs of this hypothetical
integration into the logging module:

    logging.debug(t"Eager evaluation of {expensive_call()}")
    logging.debug(t"Lazy evaluation of {expensive_call!()}")

    logging.debug(t"Eager evaluation of {expensive_call_with_args(x, y, z)}")
    logging.debug(t"Lazy evaluation of {(lambda: expensive_call_with_args(x, y, z))!()}")

It's an open question whether the definition of logging formatters would
be updated to support template strings, but if they were, the most
likely way of defining fields which should be
looked up on the log record <logrecord-attributes> instead of being
interpreted eagerly is simply to escape them so they're available as
part of the literal text:

    proc_id = get_process_id()
    formatter = logging.Formatter(t"{{asctime}}:{proc_id}:{{name}}:{{levelname}}{{message}}")

Deferring consideration of possible use in i18n use cases

The initial motivating use case for this PEP was providing a cleaner
syntax for i18n (internationalization) translation, as that requires
access to the original unmodified template. As such, it focused on
compatibility with the substitution syntax used in Python's
string.Template formatting and Mozilla's l20n project.

However, subsequent discussion revealed there are significant additional
considerations to be taken into account in the i18n use case, which
don't impact the simpler cases of handling interpolation into security
sensitive contexts (like HTML, system shells, and database queries), or
producing application debugging messages in the preferred language of
the development team (rather than the native language of end users).

Due to that realisation, the PEP was switched to use the str.format
substitution syntax originally defined in PEP 3101 and subsequently used
as the basis for PEP 498.

While it would theoretically be possible to update string.Template to
support the creation of instances from native template literals, and to
implement the structural typing.Template protocol, the PEP authors have
not identified any practical benefit in doing so.

However, one significant benefit of the "only one string prefix"
approach used in this PEP is that while it generalises the existing
f-string interpolation syntax to support delayed rendering through
t-strings, it doesn't imply that that should be the only compiler
supported interpolation syntax that Python should ever offer.

Most notably, it leaves the door open to an alternate "t$-string" syntax
that would allow TemplateLiteral instances to be created using a PEP 292
based interpolation syntax rather than a PEP 3101 based syntax:

  template = t$"Substitute $words and ${other_values} at runtime"

The only runtime distinction between templates created that way and
templates created from regular t-strings would be in the contents of
their raw_template attributes.

Deferring escaped rendering support for non-POSIX shells

shlex.quote works by classifying the regex character set [\w@%+=:,./-]
to be safe, deeming all other characters to be unsafe, and hence
requiring quoting of the string containing them. The quoting mechanism
used is then specific to the way that string quoting works in POSIX
shells, so it cannot be trusted when running a shell that doesn't follow
POSIX shell string quoting rules.

For example, running
subprocess.run(f'echo {shlex.quote(sys.argv[1])}', shell=True) is safe
when using a shell that follows POSIX quoting rules:

    $ cat > run_quoted.py
    import sys, shlex, subprocess
    subprocess.run(f"echo {shlex.quote(sys.argv[1])}", shell=True)
    $ python3 run_quoted.py pwd
    pwd
    $ python3 run_quoted.py '; pwd'
    ; pwd
    $ python3 run_quoted.py "'pwd'"
    'pwd'

but remains unsafe when running a shell from Python invokes cmd.exe (or
Powershell):

    S:\> echo import sys, shlex, subprocess > run_quoted.py
    S:\> echo subprocess.run(f"echo {shlex.quote(sys.argv[1])}", shell=True) >> run_quoted.py
    S:\> type run_quoted.py
    import sys, shlex, subprocess
    subprocess.run(f"echo {shlex.quote(sys.argv[1])}", shell=True)
    S:\> python3 run_quoted.py "echo OK"
    'echo OK'
    S:\> python3 run_quoted.py "'& echo Oh no!"
    ''"'"'
    Oh no!'

Resolving this standard library limitation is beyond the scope of this
PEP.

Acknowledgements

-   Eric V. Smith for creating PEP 498 and demonstrating the feasibility
    of arbitrary expression substitution in string interpolation
-   The authors of PEP 750 for the substantial design improvements that
    tagged strings inspired for this PEP, their general advocacy for the
    value of language level delayed template rendering support, and
    their efforts to ensure that any native interpolation template
    support lays a strong foundation for future efforts in providing
    robust syntax highlighting and static type checking support for
    domain specific languages
-   Barry Warsaw, Armin Ronacher, and Mike Miller for their
    contributions to exploring the feasibility of using this model of
    delayed rendering in i18n use cases (even though the ultimate
    conclusion was that it was a poor fit, at least for current
    approaches to i18n in Python)

References

-   %-formatting
-   str.format
-   string.Template documentation
-   PEP 215: String Interpolation
-   PEP 292: Simpler String Substitutions
-   PEP 3101: Advanced String Formatting
-   PEP 498: Literal string formatting
-   PEP 675: Arbitrary Literal String Type
-   PEP 701: Syntactic formalization of f-strings
-   FormattableString and C# native string interpolation
-   IFormattable interface in C# (see remarks for globalization notes)
-   TemplateLiterals in Javascript
-   Running external commands in Julia

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

authors:
Alyssa Coghlan
Nick Humrich


text:
PEP: 382 Title: Namespace Packages Author: Martin von Löwis
<EMAIL_ADDRESS>Status: Rejected Type: Standards Track
Content-Type: text/x-rst Created: 02-Apr-2009 Python-Version: 3.2
Post-History:

Rejection Notice

On the first day of sprints at US PyCon 2012 we had a long and fruitful
discussion about PEP 382 and PEP 402. We ended up rejecting both but a
new PEP will be written to carry on in the spirit of PEP 402. Martin von
Löwis wrote up a summary:[1].

Abstract

Namespace packages are a mechanism for splitting a single Python package
across multiple directories on disk. In current Python versions, an
algorithm to compute the packages __path__ must be formulated. With the
enhancement proposed here, the import machinery itself will construct
the list of directories that make up the package. An implementation of
this PEP is available at[2].

Terminology

Within this PEP, the term package refers to Python packages as defined
by Python's import statement. The term distribution refers to separately
installable sets of Python modules as stored in the Python package
index, and installed by distutils or setuptools. The term vendor package
refers to groups of files installed by an operating system's packaging
mechanism (e.g. Debian or Redhat packages install on Linux systems).

The term portion refers to a set of files in a single directory
(possibly stored in a zip file) that contribute to a namespace package.

Namespace packages today

Python currently provides the pkgutil.extend_path to denote a package as
a namespace package. The recommended way of using it is to put:

    from pkgutil import extend_path
    __path__ = extend_path(__path__, __name__)

in the package's __init__.py. Every distribution needs to provide the
same contents in its __init__.py, so that extend_path is invoked
independent of which portion of the package gets imported first. As a
consequence, the package's __init__.py cannot practically define any
names as it depends on the order of the package fragments on sys.path
which portion is imported first. As a special feature, extend_path reads
files named <packagename>.pkg which allow to declare additional
portions.

setuptools provides a similar function pkg_resources.declare_namespace
that is used in the form:

    import pkg_resources
    pkg_resources.declare_namespace(__name__)

In the portion's __init__.py, no assignment to __path__ is necessary, as
declare_namespace modifies the package __path__ through sys.modules. As
a special feature, declare_namespace also supports zip files, and
registers the package name internally so that future additions to
sys.path by setuptools can properly add additional portions to each
package.

setuptools allows declaring namespace packages in a distribution's
setup.py, so that distribution developers don't need to put the magic
__path__ modification into __init__.py themselves.

Rationale

The current imperative approach to namespace packages has lead to
multiple slightly-incompatible mechanisms for providing namespace
packages. For example, pkgutil supports *.pkg files; setuptools doesn't.
Likewise, setuptools supports inspecting zip files, and supports adding
portions to its _namespace_packages variable, whereas pkgutil doesn't.

In addition, the current approach causes problems for system vendors.
Vendor packages typically must not provide overlapping files, and an
attempt to install a vendor package that has a file already on disk will
fail or cause unpredictable behavior. As vendors might chose to package
distributions such that they will end up all in a single directory for
the namespace package, all portions would contribute conflicting
__init__.py files.

Specification

Rather than using an imperative mechanism for importing packages, a
declarative approach is proposed here: A directory whose name ends with
.pyp (for Python package) contains a portion of a package.

The import statement is extended so that computes the package's __path__
attribute for a package named P as consisting of optionally a single
directory name P containing a file __init__.py, plus all directories
named P.pyp, in the order in which they are found in the parent's
package __path__ (or sys.path). If either of these are found, search for
additional portions of the package continues.

A directory may contain both a package in the P/__init__.py and the
P.pyp form.

No other change to the importing mechanism is made; searching modules
(including __init__.py) will continue to stop at the first module
encountered. In summary, the process import a package foo works like
this:

1.  sys.path is searched for directories foo or foo.pyp, or a file
    foo.<ext>. If a file is found and no directory, it is treated as a
    module, and imported.
2.  If a directory foo is found, a check is made whether it contains
    __init__.py. If so, the location of the __init__.py is remembered.
    Otherwise, the directory is skipped. Once an __init__.py is found,
    further directories called foo are skipped.
3.  For both directories foo and foo.pyp, the directories are added to
    the package's __path__.
4.  If an __init__ module was found, it is imported, with __path__ being
    initialized to the path computed all .pyp directories.

Impact on Import Hooks

Both loaders and finders as defined in PEP 302 will need to be changed
to support namespace packages. Failure to conform to the protocol below
might cause a package not being recognized as a namespace package;
loaders and finders not supporting this protocol must raise
AttributeError when the functions below get accessed.

Finders need to support looking for *.pth files in step 1 of above
algorithm. To do so, a finder used as a path hook must support a method:

  finder.find_package_portion(fullname)

This method will be called in the same manner as find_module, and it
must return a string to be added to the package's __path__. If the
finder doesn't find a portion of the package, it shall return None.
Raising AttributeError from above call will be treated as
non-conformance with this PEP, and the exception will be ignored. All
other exceptions are reported.

A finder may report both success from find_module and from
find_package_portion, allowing for both a package containing an
__init__.py and a portion of the same package.

All strings returned from find_package_portion, along with all path
names of .pyp directories are added to the new package's __path__.

Discussion

Original versions of this specification proposed the addition of *.pth
files, similar to the way those files are used on sys.path. With a
wildcard marker (*), a package could indicate that the entire path is
derived by looking at the parent path, searching for properly-named
subdirectories.

People then observed that the support for the full .pth syntax is
inappropriate, and the .pth files were changed to be mere marker files,
indicating that a directories is a package. Peter Tröger suggested that
.pth is an unsuitable file extension, as all file extensions related to
Python should start with .py. Therefore, the marker file was renamed to
be .pyp.

Dinu Gherman then observed that using a marker file is not necessary,
and that a directory extension could well serve as a such as a marker.
This is what this PEP currently proposes.

Phillip Eby designed PEP 402 as an alternative approach to this PEP,
after comparing Python's package syntax with that found in other
languages. PEP 402 proposes not to use a marker file at all. At the
discussion at PyCon DE 2011, people remarked that having an explicit
declaration of a directory as contributing to a package is a desirable
property, rather than an obstacle. In particular, Jython developers
noticed that Jython could easily mistake a directory that is a Java
package as being a Python package, if there is no need to declare Python
packages.

Packages can stop filling out the namespace package's __init__.py. As a
consequence, extend_path and declare_namespace become obsolete.

Namespace packages can start providing non-trivial __init__.py
implementations; to do so, it is recommended that a single distribution
provides a portion with just the namespace package's __init__.py (and
potentially other modules that belong to the namespace package proper).

The mechanism is mostly compatible with the existing namespace
mechanisms. extend_path will be adjusted to this specification; any
other mechanism might cause portions to get added twice to __path__.

References

Copyright

This document has been placed in the public domain.

[1] Namespace Packages resolution
(https://mail.python.org/pipermail/import-sig/2012-March/000421.html)

[2] PEP 382 branch (http://hg.python.org/features/pep-382-2#pep-382)

license: Public Domain


text:
PEP: 230 Title: Warning Framework Author: Guido van Rossum
<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 28-Nov-2000 Python-Version: 2.1 Post-History:
05-Nov-2000

Abstract

This PEP proposes a C and Python level API, as well as command line
flags, to issue warning messages and control what happens to them. This
is mostly based on GvR's proposal posted to python-dev on 05-Nov-2000,
with some ideas (such as using classes to categorize warnings) merged in
from Paul Prescod's counter-proposal posted on the same date. Also, an
attempt to implement the proposal caused several small tweaks.

Motivation

With Python 3000 looming, it is necessary to start issuing warnings
about the use of obsolete or deprecated features, in addition to errors.
There are also lots of other reasons to be able to issue warnings, both
from C and from Python code, both at compile time and at run time.

Warnings aren't fatal, and thus it's possible that a program triggers
the same warning many times during a single execution. It would be
annoying if a program emitted an endless stream of identical warnings.
Therefore, a mechanism is needed that suppresses multiple identical
warnings.

It is also desirable to have user control over which warnings are
printed. While in general it is useful to see all warnings all the time,
there may be times where it is impractical to fix the code right away in
a production program. In this case, there should be a way to suppress
warnings.

It is also useful to be able to suppress specific warnings during
program development, e.g. when a warning is generated by a piece of 3rd
party code that cannot be fixed right away, or when there is no way to
fix the code (possibly a warning message is generated for a perfectly
fine piece of code). It would be unwise to offer to suppress all
warnings in such cases: the developer would miss warnings about the rest
of the code.

On the other hand, there are also situations conceivable where some or
all warnings are better treated as errors. For example, it may be a
local coding standard that a particular deprecated feature should not be
used. In order to enforce this, it is useful to be able to turn the
warning about this particular feature into an error, raising an
exception (without necessarily turning all warnings into errors).

Therefore, I propose to introduce a flexible "warning filter" which can
filter out warnings or change them into exceptions, based on:

-   Where in the code they are generated (per package, module, or
    function)
-   The warning category (warning categories are discussed below)
-   A specific warning message

The warning filter must be controllable both from the command line and
from Python code.

APIs For Issuing Warnings

-   To issue a warning from Python:

        import warnings
        warnings.warn(message[, category[, stacklevel]])

    The category argument, if given, must be a warning category class
    (see below); it defaults to warnings.UserWarning. This may raise an
    exception if the particular warning issued is changed into an error
    by the warnings filter. The stacklevel can be used by wrapper
    functions written in Python, like this:

        def deprecation(message):
            warn(message, DeprecationWarning, level=2)

    This makes the warning refer to the deprecation()'s caller, rather
    than to the source of deprecation() itself (since the latter would
    defeat the purpose of the warning message).

-   To issue a warning from C:

        int PyErr_Warn(PyObject *category, char *message);

    Return 0 normally, 1 if an exception is raised (either because the
    warning was transformed into an exception, or because of a
    malfunction in the implementation, such as running out of memory).
    The category argument must be a warning category class (see below)
    or NULL, in which case it defaults to PyExc_RuntimeWarning. When
    PyErr_Warn() function returns 1, the caller should do normal
    exception handling.

    The current C implementation of PyErr_Warn() imports the warnings
    module (implemented in Python) and calls its warn() function. This
    minimizes the amount of C code that needs to be added to implement
    the warning feature.

    [XXX Open Issue: what about issuing warnings during lexing or
    parsing, which don't have the exception machinery available?]

Warnings Categories

There are a number of built-in exceptions that represent warning
categories. This categorization is useful to be able to filter out
groups of warnings. The following warnings category classes are
currently defined:

-   Warning -- this is the base class of all warning category classes
    and it itself a subclass of Exception
-   UserWarning -- the default category for warnings.warn()
-   DeprecationWarning -- base category for warnings about deprecated
    features
-   SyntaxWarning -- base category for warnings about dubious syntactic
    features
-   RuntimeWarning -- base category for warnings about dubious runtime
    features

[XXX: Other warning categories may be proposed during the review period
for this PEP.]

These standard warning categories are available from C as PyExc_Warning,
PyExc_UserWarning, etc. From Python, they are available in the
__builtin__ module, so no import is necessary.

User code can define additional warning categories by subclassing one of
the standard warning categories. A warning category must always be a
subclass of the Warning class.

The Warnings Filter

The warnings filter control whether warnings are ignored, displayed, or
turned into errors (raising an exception).

There are three sides to the warnings filter:

-   The data structures used to efficiently determine the disposition of
    a particular warnings.warn() or PyErr_Warn() call.
-   The API to control the filter from Python source code.
-   The command line switches to control the filter.

The warnings filter works in several stages. It is optimized for the
(expected to be common) case where the same warning is issued from the
same place in the code over and over.

First, the warning filter collects the module and line number where the
warning is issued; this information is readily available through
sys._getframe().

Conceptually, the warnings filter maintains an ordered list of filter
specifications; any specific warning is matched against each filter
specification in the list in turn until a match is found; the match
determines the disposition of the match. Each entry is a tuple as
follows:

    (category, message, module, lineno, action)

-   category is a class (a subclass of warnings.Warning) of which the
    warning category must be a subclass in order to match
-   message is a compiled regular expression that the warning message
    must match (the match is case-insensitive)
-   module is a compiled regular expression that the module name must
    match
-   lineno is an integer that the line number where the warning occurred
    must match, or 0 to match all line numbers
-   action is one of the following strings:
    -   "error" -- turn matching warnings into exceptions
    -   "ignore" -- never print matching warnings
    -   "always" -- always print matching warnings
    -   "default" -- print the first occurrence of matching warnings for
        each location where the warning is issued
    -   "module" -- print the first occurrence of matching warnings for
        each module where the warning is issued
    -   "once" -- print only the first occurrence of matching warnings

Since the Warning class is derived from the built-in Exception class, to
turn a warning into an error we simply raise category(message).

Warnings Output And Formatting Hooks

When the warnings filter decides to issue a warning (but not when it
decides to raise an exception), it passes the information about the
function warnings.showwarning(message, category, filename, lineno). The
default implementation of this function writes the warning text to
sys.stderr, and shows the source line of the filename. It has an
optional 5th argument which can be used to specify a different file than
sys.stderr.

The formatting of warnings is done by a separate function,
warnings.formatwarning(message, category, filename, lineno). This
returns a string (that may contain newlines and ends in a newline) that
can be printed to get the identical effect of the showwarning()
function.

API For Manipulating Warning Filters

    warnings.filterwarnings(message, category, module, lineno, action)

This checks the types of the arguments, compiles the message and module
regular expressions, and inserts them as a tuple in front of the
warnings filter.

    warnings.resetwarnings()

Reset the warnings filter to empty.

Command Line Syntax

There should be command line options to specify the most common
filtering actions, which I expect to include at least:

-   suppress all warnings
-   suppress a particular warning message everywhere
-   suppress all warnings in a particular module
-   turn all warnings into exceptions

I propose the following command line option syntax:

    -Waction[:message[:category[:module[:lineno]]]]

Where:

-   'action' is an abbreviation of one of the allowed actions ("error",
    "default", "ignore", "always", "once", or "module")
-   'message' is a message string; matches warnings whose message text
    is an initial substring of 'message' (matching is case-insensitive)
-   'category' is an abbreviation of a standard warning category class
    name or a fully-qualified name for a user-defined warning category
    class of the form [package.]module.classname
-   'module' is a module name (possibly package.module)
-   'lineno' is an integral line number

All parts except 'action' may be omitted, where an empty value after
stripping whitespace is the same as an omitted value.

The C code that parses the Python command line saves the body of all -W
options in a list of strings, which is made available to the warnings
module as sys.warnoptions. The warnings module parses these when it is
first imported. Errors detected during the parsing of sys.warnoptions
are not fatal; a message is written to sys.stderr and processing
continues with the option.

Examples:

-Werror

    Turn all warnings into errors

-Wall

    Show all warnings

-Wignore

    Ignore all warnings

-Wi:hello

    Ignore warnings whose message text starts with "hello"

-We::Deprecation

    Turn deprecation warnings into errors

-Wi:::spam:10

    Ignore all warnings on line 10 of module spam

-Wi:::spam -Wd:::spam:10

    Ignore all warnings in module spam except on line 10

-We::Deprecation -Wd::Deprecation:spam

    Turn deprecation warnings into errors except in module spam

Open Issues

Some open issues off the top of my head:

-   What about issuing warnings during lexing or parsing, which don't
    have the exception machinery available?
-   The proposed command line syntax is a bit ugly (although the simple
    cases aren't so bad: -Werror, -Wignore, etc.). Anybody got a better
    idea?
-   I'm a bit worried that the filter specifications are too complex.
    Perhaps filtering only on category and module (not on message text
    and line number) would be enough?
-   There's a bit of confusion between module names and file names. The
    reporting uses file names, but the filter specification uses module
    names. Maybe it should allow filenames as well?
-   I'm not at all convinced that packages are handled right.
-   Do we need more standard warning categories? Fewer?
-   In order to minimize the start-up overhead, the warnings module is
    imported by the first call to PyErr_Warn(). It does the command line
    parsing for -W options upon import. Therefore, it is possible that
    warning-free programs will not complain about invalid -W options.

Rejected Concerns

Paul Prescod, Barry Warsaw and Fred Drake have brought up several
additional concerns that I feel aren't critical. I address them here
(the concerns are paraphrased, not exactly their words):

-   Paul: warn() should be a built-in or a statement to make it easily
    available.

    Response: "from warnings import warn" is easy enough.

-   Paul: What if I have a speed-critical module that triggers warnings
    in an inner loop. It should be possible to disable the overhead for
    detecting the warning (not just suppress the warning).

    Response: rewrite the inner loop to avoid triggering the warning.

-   Paul: What if I want to see the full context of a warning?

    Response: use -Werror to turn it into an exception.

-   Paul: I prefer ":*:*:" to ":::" for leaving parts of the warning
    spec out.

    Response: I don't.

-   Barry: It would be nice if lineno can be a range specification.

    Response: Too much complexity already.

-   Barry: I'd like to add my own warning action. Maybe if 'action'
    could be a callable as well as a string. Then in my IDE, I could set
    that to "mygui.popupWarningsDialog".

    Response: For that purpose you would override
    warnings.showwarning().

-   Fred: why do the Warning category classes have to be in __builtin__?

    Response: that's the simplest implementation, given that the warning
    categories must be available in C before the first PyErr_Warn()
    call, which imports the warnings module. I see no problem with
    making them available as built-ins.

Implementation

Here's a prototype implementation:
http://sourceforge.net/patch/?func=detailpatch&patch_id=102715&group_id=5470


text:
PEP: 3099 Title: Things that will Not Change in Python 3000 Version:
$Revision$ Last-Modified: $Date$ Author: Georg Brandl<EMAIL_ADDRESS>Status: Final Type: Process Content-Type: text/x-rst Created:
04-Apr-2006 Post-History:

Abstract

Some ideas are just bad. While some thoughts on Python evolution are
constructive, some go against the basic tenets of Python so egregiously
that it would be like asking someone to run in a circle: it gets you
nowhere, even for Python 3000, where extraordinary proposals are
allowed. This PEP tries to list all BDFL pronouncements on Python 3000
that refer to changes that will not happen and new features that will
not be introduced, sorted by topics, along with a short explanation or a
reference to the relevant thread on the python-3000 mailing list.

If you think you should suggest any of the listed ideas it would be
better to just step away from the computer, go outside, and enjoy
yourself. Being active outdoors by napping in a nice patch of grass is
more productive than bringing up a beating-a-dead-horse idea and having
people tell you how dead the idea is. Consider yourself warned.

Core language

-   Python 3000 will not be case-insensitive.

-   Python 3000 will not be a rewrite from scratch.

      It will also not use C++ or another language different from C as
      implementation language. Rather, there will be a gradual
      transmogrification of the codebase. There's an excellent essay by
      Joel Spolsky explaining why:
      http://www.joelonsoftware.com/articles/fog0000000069.html

-   self will not become implicit.

      Having self be explicit is a good thing. It makes the code clear
      by removing ambiguity about how a variable resolves. It also makes
      the difference between functions and methods small.

      Thread: "Draft proposal: Implicit self in Python 3.0"
      https://mail.python.org/pipermail/python-dev/2006-January/059468.html

-   lambda will not be renamed.

      At one point lambda was slated for removal in Python 3000.
      Unfortunately no one was able to come up with a better way of
      providing anonymous functions. And so lambda is here to stay.

      But it is here to stay as-is. Adding support for statements is a
      non-starter. It would require allowing multi-line lambda
      expressions which would mean a multi-line expression could
      suddenly exist. That would allow for multi-line arguments to
      function calls, for instance. That is just plain ugly.

      Thread: "genexp syntax / lambda",
      https://mail.python.org/pipermail/python-3000/2006-April/001042.html

-   Python will not have programmable syntax.

      Thread: "It's a statement! It's a function! It's BOTH!",
      https://mail.python.org/pipermail/python-3000/2006-April/000286.html

-   There won't be a syntax for zip()-style parallel iteration.

      Thread: "Parallel iteration syntax",
      https://mail.python.org/pipermail/python-3000/2006-March/000210.html

-   Strings will stay iterable.

      Thread: "Making strings non-iterable",
      https://mail.python.org/pipermail/python-3000/2006-April/000759.html

-   There will be no syntax to sort the result of a generator expression
    or list comprehension. sorted() covers all use cases.

      Thread: "Adding sorting to generator comprehension",
      https://mail.python.org/pipermail/python-3000/2006-April/001295.html

-   Slices and extended slices won't go away (even if the __getslice__
    and __setslice__ APIs may be replaced) nor will they return views
    for the standard object types.

      Thread: Future of slices
      https://mail.python.org/pipermail/python-3000/2006-May/001563.html

-   It will not be forbidden to reuse a loop variable inside the loop's
    suite.

      Thread: elimination of scope bleeding of iteration variables
      https://mail.python.org/pipermail/python-dev/2006-May/064761.html

-   The parser won't be more complex than LL(1).

      Simple is better than complex. This idea extends to the parser.
      Restricting Python's grammar to an LL(1) parser is a blessing, not
      a curse. It puts us in handcuffs that prevent us from going
      overboard and ending up with funky grammar rules like some other
      dynamic languages that will go unnamed, such as Perl.

-   No braces.

      This is so obvious that it doesn't need a reference to a mailing
      list. Do from __future__ import braces to get a definitive answer
      on this subject.

-   No more backticks.

      Backticks (`) will no longer be used as shorthand for repr --but
      that doesn't mean they are available for other uses. Even ignoring
      the backwards compatibility confusion, the character itself causes
      too many problems (in some fonts, on some keyboards, when
      typesetting a book, etc).

      Thread: "new operators via backquoting",
      https://mail.python.org/pipermail/python-ideas/2007-January/000054.html

-   Referencing the global name foo will not be spelled globals.foo. The
    global statement will stay.

      Threads: "replace globals() and global statement with global
      builtin object",
      https://mail.python.org/pipermail/python-3000/2006-July/002485.html,
      "Explicit Lexical Scoping (pre-PEP?)",
      https://mail.python.org/pipermail/python-dev/2006-July/067111.html

-   There will be no alternative binding operators such as :=.

      Thread: "Explicit Lexical Scoping (pre-PEP?)",
      https://mail.python.org/pipermail/python-dev/2006-July/066995.html

-   We won't be removing container literals. That is, {expr: expr, ...},
    [expr, ...] and (expr, ...) will stay.

      Thread: "No Container Literals",
      https://mail.python.org/pipermail/python-3000/2006-July/002550.html

-   The else clause in while and for loops will not change semantics, or
    be removed.

      Thread: "for/except/else syntax"
      https://mail.python.org/pipermail/python-ideas/2009-October/006083.html

Builtins

-   zip() won't grow keyword arguments or other mechanisms to prevent it
    from stopping at the end of the shortest sequence.

      Thread: "have zip() raise exception for sequences of different
      lengths",
      https://mail.python.org/pipermail/python-3000/2006-August/003338.html

-   hash() won't become an attribute since attributes should be cheap to
    compute, which isn't necessarily the case for a hash.

      Thread: "hash as attribute/property",
      https://mail.python.org/pipermail/python-3000/2006-April/000362.html

Standard types

-   Iterating over a dictionary will continue to yield the keys.

      Thread: "Iterating over a dict",
      https://mail.python.org/pipermail/python-3000/2006-April/000283.html

      Thread: have iter(mapping) generate (key, value) pairs
      https://mail.python.org/pipermail/python-3000/2006-June/002368.html

-   There will be no frozenlist type.

      Thread: "Immutable lists",
      https://mail.python.org/pipermail/python-3000/2006-May/002219.html

-   int will not support subscripts yielding a range.

      Thread: "xrange vs. int.__getslice__",
      https://mail.python.org/pipermail/python-3000/2006-June/002450.html

Coding style

-   The (recommended) maximum line width will remain 80 characters, for
    both C and Python code.

      Thread: "C style guide",
      https://mail.python.org/pipermail/python-3000/2006-March/000131.html

Interactive Interpreter

-   The interpreter prompt (>>>) will not change. It gives Guido warm
    fuzzy feelings.

      Thread: "Low-hanging fruit: change interpreter prompt?",
      https://mail.python.org/pipermail/python-3000/2006-November/004891.html

Copyright

This document has been placed in the public domain.