text:
PEP: 659 Title: Specializing Adaptive Interpreter Author: Mark Shannon
<EMAIL_ADDRESS>Status: Draft Type: Informational Content-Type:
text/x-rst Created: 13-Apr-2021 Post-History: 11-May-2021

Abstract

In order to perform well, virtual machines for dynamic languages must
specialize the code that they execute to the types and values in the
program being run. This specialization is often associated with "JIT"
compilers, but is beneficial even without machine code generation.

A specializing, adaptive interpreter is one that speculatively
specializes on the types or values it is currently operating on, and
adapts to changes in those types and values.

Specialization gives us improved performance, and adaptation allows the
interpreter to rapidly change when the pattern of usage in a program
alters, limiting the amount of additional work caused by
mis-specialization.

This PEP proposes using a specializing, adaptive interpreter that
specializes code aggressively, but over a very small region, and is able
to adjust to mis-specialization rapidly and at low cost.

Adding a specializing, adaptive interpreter to CPython will bring
significant performance improvements. It is hard to come up with
meaningful numbers, as it depends very much on the benchmarks and on
work that has not yet happened. Extensive experimentation suggests
speedups of up to 50%. Even if the speedup were only 25%, this would
still be a worthwhile enhancement.

Motivation

Python is widely acknowledged as slow. Whilst Python will never attain
the performance of low-level languages like C, Fortran, or even Java, we
would like it to be competitive with fast implementations of scripting
languages, like V8 for Javascript or luajit for lua. Specifically, we
want to achieve these performance goals with CPython to benefit all
users of Python including those unable to use PyPy or other alternative
virtual machines.

Achieving these performance goals is a long way off, and will require a
lot of engineering effort, but we can make a significant step towards
those goals by speeding up the interpreter. Both academic research and
practical implementations have shown that a fast interpreter is a key
part of a fast virtual machine.

Typical optimizations for virtual machines are expensive, so a long
"warm up" time is required to gain confidence that the cost of
optimization is justified. In order to get speed-ups rapidly, without
noticeable warmup times, the VM should speculate that specialization is
justified even after a few executions of a function. To do that
effectively, the interpreter must be able to optimize and de-optimize
continually and very cheaply.

By using adaptive and speculative specialization at the granularity of
individual virtual machine instructions, we get a faster interpreter
that also generates profiling information for more sophisticated
optimizations in the future.

Rationale

There are many practical ways to speed-up a virtual machine for a
dynamic language. However, specialization is the most important, both in
itself and as an enabler of other optimizations. Therefore it makes
sense to focus our efforts on specialization first, if we want to
improve the performance of CPython.

Specialization is typically done in the context of a JIT compiler, but
research shows specialization in an interpreter can boost performance
significantly, even outperforming a naive compiler[1].

There have been several ways of doing this proposed in the academic
literature, but most attempt to optimize regions larger than a single
bytecode[2][3]. Using larger regions than a single instruction requires
code to handle de-optimization in the middle of a region. Specialization
at the level of individual bytecodes makes de-optimization trivial, as
it cannot occur in the middle of a region.

By speculatively specializing individual bytecodes, we can gain
significant performance improvements without anything but the most
local, and trivial to implement, de-optimizations.

The closest approach to this PEP in the literature is "Inline Caching
meets Quickening"[4]. This PEP has the advantages of inline caching, but
adds the ability to quickly de-optimize making the performance more
robust in cases where specialization fails or is not stable.

Performance

The speedup from specialization is hard to determine, as many
specializations depend on other optimizations. Speedups seem to be in
the range 10% - 60%.

-   Most of the speedup comes directly from specialization. The largest
    contributors are speedups to attribute lookup, global variables, and
    calls.
-   A small, but useful, fraction is from improved dispatch such as
    super-instructions and other optimizations enabled by quickening.

Implementation

Overview

Any instruction that would benefit from specialization will be replaced
by an "adaptive" form of that instruction. When executed, the adaptive
instructions will specialize themselves in response to the types and
values that they see. This process is known as "quickening".

Once an instruction in a code object has executed enough times, that
instruction will be "specialized" by replacing it with a new instruction
that is expected to execute faster for that operation.

Quickening

Quickening is the process of replacing slow instructions with faster
variants.

Quickened code has a number of advantages over immutable bytecode:

-   It can be changed at runtime.
-   It can use super-instructions that span lines and take multiple
    operands.
-   It does not need to handle tracing as it can fallback to the
    original bytecode for that.

In order that tracing can be supported, the quickened instruction format
should match the immutable, user visible, bytecode format: 16-bit
instructions of 8-bit opcode followed by 8-bit operand.

Adaptive instructions

Each instruction that would benefit from specialization is replaced by
an adaptive version during quickening. For example, the LOAD_ATTR
instruction would be replaced with LOAD_ATTR_ADAPTIVE.

Each adaptive instruction periodically attempts to specialize itself.

Specialization

CPython bytecode contains many instructions that represent high-level
operations, and would benefit from specialization. Examples include
CALL, LOAD_ATTR, LOAD_GLOBAL and BINARY_ADD.

By introducing a "family" of specialized instructions for each of these
instructions allows effective specialization, since each new instruction
is specialized to a single task. Each family will include an "adaptive"
instruction, that maintains a counter and attempts to specialize itself
when that counter reaches zero.

Each family will also include one or more specialized instructions that
perform the equivalent of the generic operation much faster provided
their inputs are as expected. Each specialized instruction will maintain
a saturating counter which will be incremented whenever the inputs are
as expected. Should the inputs not be as expected, the counter will be
decremented and the generic operation will be performed. If the counter
reaches the minimum value, the instruction is de-optimized by simply
replacing its opcode with the adaptive version.

Ancillary data

Most families of specialized instructions will require more information
than can fit in an 8-bit operand. To do this, a number of 16 bit entries
immediately following the instruction are used to store this data. This
is a form of inline cache, an "inline data cache". Unspecialized, or
adaptive, instructions will use the first entry of this cache as a
counter, and simply skip over the others.

Example families of instructions

LOAD_ATTR

The LOAD_ATTR instruction loads the named attribute of the object on top
of the stack, then replaces the object on top of the stack with the
attribute.

This is an obvious candidate for specialization. Attributes might belong
to a normal instance, a class, a module, or one of many other special
cases.

LOAD_ATTR would initially be quickened to LOAD_ATTR_ADAPTIVE which would
track how often it is executed, and call the _Py_Specialize_LoadAttr
internal function when executed enough times, or jump to the original
LOAD_ATTR instruction to perform the load. When optimizing, the kind of
the attribute would be examined, and if a suitable specialized
instruction was found, it would replace LOAD_ATTR_ADAPTIVE in place.

Specialization for LOAD_ATTR might include:

-   LOAD_ATTR_INSTANCE_VALUE A common case where the attribute is stored
    in the object's value array, and not shadowed by an overriding
    descriptor.
-   LOAD_ATTR_MODULE Load an attribute from a module.
-   LOAD_ATTR_SLOT Load an attribute from an object whose class defines
    __slots__.

Note how this allows optimizations that complement other optimizations.
The LOAD_ATTR_INSTANCE_VALUE works well with the "lazy dictionary" used
for many objects.

LOAD_GLOBAL

The LOAD_GLOBAL instruction looks up a name in the global namespace and
then, if not present in the global namespace, looks it up in the
builtins namespace. In 3.9 the C code for the LOAD_GLOBAL includes code
to check to see whether the whole code object should be modified to add
a cache, whether either the global or builtins namespace, code to lookup
the value in a cache, and fallback code. This makes it complicated and
bulky. It also performs many redundant operations even when supposedly
optimized.

Using a family of instructions makes the code more maintainable and
faster, as each instruction only needs to handle one concern.

Specializations would include:

-   LOAD_GLOBAL_ADAPTIVE would operate like LOAD_ATTR_ADAPTIVE above.
-   LOAD_GLOBAL_MODULE can be specialized for the case where the value
    is in the globals namespace. After checking that the keys of the
    namespace have not changed, it can load the value from the stored
    index.
-   LOAD_GLOBAL_BUILTIN can be specialized for the case where the value
    is in the builtins namespace. It needs to check that the keys of the
    global namespace have not been added to, and that the builtins
    namespace has not changed. Note that we don't care if the values of
    the global namespace have changed, just the keys.

See[5] for a full implementation.

Note

This PEP outlines the mechanisms for managing specialization, and does
not specify the particular optimizations to be applied. It is likely
that details, or even the entire implementation, may change as the code
is further developed.

Compatibility

There will be no change to the language, library or API.

The only way that users will be able to detect the presence of the new
interpreter is through timing execution, the use of debugging tools, or
measuring memory use.

Costs

Memory use

An obvious concern with any scheme that performs any sort of caching is
"how much more memory does it use?". The short answer is "not that
much".

Comparing memory use to 3.10

CPython 3.10 used 2 bytes per instruction, until the execution count
reached ~2000 when it allocates another byte per instruction and 32
bytes per instruction with a cache (LOAD_GLOBAL and LOAD_ATTR).

The following table shows the additional bytes per instruction to
support the 3.10 opcache or the proposed adaptive interpreter, on a 64
bit machine.

+----------------+----------------+----------------+----------------+
|   Version      | 3.10 cold      |   3.10 hot     |   3.11         |
+----------------+----------------+----------------+----------------+
|   Specialised  |   0%           |   ~15%         |   ~25%         |
+----------------+----------------+----------------+----------------+
| ------------   | ------------   | ------------   | ------------   |
+----------------+----------------+----------------+----------------+
|   code         |   2            |   2            |   2            |
+----------------+----------------+----------------+----------------+
|   opcache_map  |   0            |   1            |   0            |
+----------------+----------------+----------------+----------------+
|   opcache/data |   0            |   4.8          |   4            |
+----------------+----------------+----------------+----------------+
| ------------   | ------------   | ------------   | ------------   |
+----------------+----------------+----------------+----------------+
|   Total        |   2            |   7.8          |   6            |
+----------------+----------------+----------------+----------------+

3.10 cold is before the code has reached the ~2000 limit. 3.10 hot shows
the cache use once the threshold is reached.

The relative memory use depends on how much code is "hot" enough to
trigger creation of the cache in 3.10. The break even point, where the
memory used by 3.10 is the same as for 3.11 is ~70%.

It is also worth noting that the actual bytecode is only part of a code
object. Code objects also include names, constants and quite a lot of
debugging information.

In summary, for most applications where many of the functions are
relatively unused, 3.11 will consume more memory than 3.10, but not by
much.

Security Implications

None

Rejected Ideas

By implementing a specializing adaptive interpreter with inline data
caches, we are implicitly rejecting many alternative ways to optimize
CPython. However, it is worth emphasizing that some ideas, such as
just-in-time compilation, have not been rejected, merely deferred.

Storing data caches before the bytecode.

An earlier implementation of this PEP for 3.11 alpha used a different
caching scheme as described below:

  Quickened instructions will be stored in an array (it is neither
  necessary not desirable to store them in a Python object) with the
  same format as the original bytecode. Ancillary data will be stored in
  a separate array.

  Each instruction will use 0 or more data entries. Each instruction
  within a family must have the same amount of data allocated, although
  some instructions may not use all of it. Instructions that cannot be
  specialized, e.g. POP_TOP, do not need any entries. Experiments show
  that 25% to 30% of instructions can be usefully specialized. Different
  families will need different amounts of data, but most need 2 entries
  (16 bytes on a 64 bit machine).

  In order to support larger functions than 256 instructions, we compute
  the offset of the first data entry for instructions as
  (instruction offset)//2 + (quickened operand).

  Compared to the opcache in Python 3.10, this design:

  -   is faster; it requires no memory reads to compute the offset. 3.10
      requires two reads, which are dependent.
  -   uses much less memory, as the data can be different sizes for
      different instruction families, and doesn't need an additional
      array of offsets. can support much larger functions, up to about
      5000 instructions per function. 3.10 can support about 1000.

We rejected this scheme as the inline cache approach is both faster and
simpler.

References

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] The construction of high-performance virtual machines for dynamic
languages, Mark Shannon 2011.
https://theses.gla.ac.uk/2975/1/2011shannonphd.pdf

[2] The construction of high-performance virtual machines for dynamic
languages, Mark Shannon 2011.
https://theses.gla.ac.uk/2975/1/2011shannonphd.pdf

[3] Dynamic Interpretation for Dynamic Scripting Languages
https://www.scss.tcd.ie/publications/tech-reports/reports.09/TCD-CS-2009-37.pdf

[4] Inline Caching meets Quickening
https://www.unibw.de/ucsrl/pubs/ecoop10.pdf/view

[5] The adaptive and specialized instructions are implemented in
https://github.com/python/cpython/blob/main/Python/ceval.c

The optimizations are implemented in:
https://github.com/python/cpython/blob/main/Python/specialize.c


text:
PEP: 5 Title: Guidelines for Language Evolution Author: Paul Prescod
<EMAIL_ADDRESS>Status: Superseded Type: Process Content-Type:
text/x-rst Created: 26-Oct-2000 Post-History: Superseded-By: 387

Abstract

In the natural evolution of programming languages it is sometimes
necessary to make changes that modify the behavior of older programs.
This PEP proposes a policy for implementing these changes in a manner
respectful of the installed base of Python users.

Implementation Details

Implementation of this PEP requires the addition of a formal warning and
deprecation facility that will be described in another proposal.

Scope

These guidelines apply to future versions of Python that introduce
backward-incompatible behavior. Backward incompatible behavior is a
major deviation in Python interpretation from an earlier behavior
described in the standard Python documentation. Removal of a feature
also constitutes a change of behavior.

This PEP does not replace or preclude other compatibility strategies
such as dynamic loading of backwards-compatible parsers. On the other
hand, if execution of "old code" requires a special switch or pragma
then that is indeed a change of behavior from the point of view of the
user and that change should be implemented according to these
guidelines.

In general, common sense must prevail in the implementation of these
guidelines. For instance changing "sys.copyright" does not constitute a
backwards-incompatible change of behavior!

Steps For Introducing Backwards-Incompatible Features

1.  Propose backwards-incompatible behavior in a PEP. The PEP must
    include a section on backwards compatibility that describes in
    detail a plan to complete the remainder of these steps.
2.  Once the PEP is accepted as a productive direction, implement an
    alternate way to accomplish the task previously provided by the
    feature that is being removed or changed. For instance if the
    addition operator were scheduled for removal, a new version of
    Python could implement an "add()" built-in function.
3.  Formally deprecate the obsolete construct in the Python
    documentation.
4.  Add an optional warning mode to the parser that will inform users
    when the deprecated construct is used. In other words, all programs
    that will behave differently in the future must trigger warnings in
    this mode. Compile-time warnings are preferable to runtime warnings.
    The warning messages should steer people from the deprecated
    construct to the alternative construct.
5.  There must be at least a one-year transition period between the
    release of the transitional version of Python and the release of the
    backwards incompatible version. Users will have at least a year to
    test their programs and migrate them from use of the deprecated
    construct to the alternative one.

license: Public Domain


text:
PEP: 3142 Title: Add a "while" clause to generator expressions Version:
$Revision$ Last-Modified: $Date$ Author: Gerald Britton
<EMAIL_ADDRESS>Status: Rejected Type: Standards Track
Content-Type: text/x-rst Created: 12-Jan-2009 Python-Version: 3.0
Post-History: Resolution:
https://mail.python.org/pipermail/python-dev/2013-May/126136.html

Abstract

This PEP proposes an enhancement to generator expressions, adding a
"while" clause to complement the existing "if" clause.

Rationale

A generator expression (PEP 289) is a concise method to serve
dynamically-generated objects to list comprehensions (PEP 202). Current
generator expressions allow for an "if" clause to filter the objects
that are returned to those meeting some set of criteria. However, since
the "if" clause is evaluated for every object that may be returned, in
some cases it is possible that all objects would be rejected after a
certain point. For example:

    g = (n for n in range(100) if n*n < 50)

which is equivalent to the using a generator function (PEP 255):

    def __gen(exp):
        for n in exp:
            if n*n < 50:
                yield n
    g = __gen(iter(range(10)))

would yield 0, 1, 2, 3, 4, 5, 6 and 7, but would also consider the
numbers from 8 to 99 and reject them all since n*n >= 50 for numbers in
that range. Allowing for a "while" clause would allow the redundant
tests to be short-circuited:

    g = (n for n in range(100) while n*n < 50)

would also yield 0, 1, 2, 3, 4, 5, 6 and 7, but would stop at 8 since
the condition (n*n < 50) is no longer true. This would be equivalent to
the generator function:

    def __gen(exp):
        for n in exp:
            if n*n < 50:
                yield n
            else:
                break
    g = __gen(iter(range(100)))

Currently, in order to achieve the same result, one would need to either
write a generator function such as the one above or use the takewhile
function from itertools:

    from itertools import takewhile
    g = takewhile(lambda n: n*n < 50, range(100))

The takewhile code achieves the same result as the proposed syntax,
albeit in a longer (some would say "less-elegant") fashion. Also, the
takewhile version requires an extra function call (the lambda in the
example above) with the associated performance penalty. A simple test
shows that:

    for n in (n for n in range(100) if 1): pass

performs about 10% better than:

    for n in takewhile(lambda n: 1, range(100)): pass

though they achieve similar results. (The first example uses a
generator; takewhile is an iterator). If similarly implemented, a
"while" clause should perform about the same as the "if" clause does
today.

The reader may ask if the "if" and "while" clauses should be mutually
exclusive. There are good examples that show that there are times when
both may be used to good advantage. For example:

    p = (p for p in primes() if p > 100 while p < 1000)

should return prime numbers found between 100 and 1000, assuming I have
a primes() generator that yields prime numbers.

Adding a "while" clause to generator expressions maintains the compact
form while adding a useful facility for short-circuiting the expression.

Acknowledgements

Raymond Hettinger first proposed the concept of generator expressions in
January 2002.

Copyright

This document has been placed in the public domain.


text:
PEP: 666 Title: Reject Foolish Indentation Author: Laura Creighton
<EMAIL_ADDRESS>Status: Rejected Type: Standards Track Content-Type:
text/x-rst Created: 03-Dec-2001 Python-Version: 2.2 Post-History:
05-Dec-2001

Abstract

Everybody agrees that mixing tabs and spaces is a bad idea. Some people
want more than this. I propose that we let people define whatever Python
behaviour they want, so it will only run the way they like it, and will
not run the way they don't like it. We will do this with a command line
switch. Programs that aren't formatted the way the programmer wants
things will raise IndentationError.

-   python -TNone will refuse to run when there are any tabs.
-   python -Tn will refuse to run when tabs are not exactly n spaces
-   python -TOnly will refuse to run when blocks are indented by
    anything other than tabs

People who mix tabs and spaces, naturally, will find that their programs
do not run. Alas, we haven't found a way to give them an electric shock
as from a cattle prod remotely. (Though if somebody finds out a way to
do this, I will be pleased to add this option to the PEP.)

Rationale

<EMAIL_ADDRESS>(a.k.a. comp.lang.python) is periodically awash
with discussions about tabs and spaces. This is inevitable, given that
indentation is syntactically significant in Python. This has never
solved anything, and just makes various people frustrated and angry.
Eventually they start saying rude things to each other which is sad for
all of us. And it is also sad that they are wasting their valuable time
which they could spend creating something with Python. Moreover, for the
Python community as a whole, from a public relations point of view, this
is quite unfortunate. The people who aren't posting about tabs and
spaces, are, (unsurprisingly) invisible, while the people who are
posting make the rest of us look somewhat foolish.

The problem is that there is no polite way to say 'Stop wasting your
valuable time and mine.' People who are already in the middle of a flame
war are not well disposed to believe that you are acting out of
compassion for them, and quite rightly insist that their own time is
their own to do with as they please. They are stuck like flies in
treacle in this wretched argument, and it is self-evident that they
cannot disengage or they would have already done so.

But today I had to spend time cleaning my keyboard because the 'n' key
is sticking. So, in addition to feeling compassion for these people, I
am pretty annoyed. I figure if I make this PEP, we can then ask Guido to
quickly reject it, and then when this argument next starts up again, we
can say 'Guido isn't changing things to suit the tab-haters or the
only-tabbers, so this conversation is a waste of time.' Then everybody
can quietly believe that a) they are correct and b) other people are
fools and c) they are undeniably fortunate to not have to share a lab
with idiots, (which is something the arguers could do _now, but
apparently have forgotten).

And python-list can go back to worrying if it is too smug, rather than
whether it is too hostile for newcomers. Possibly somebody could get
around to explaining to me what is the difference between __getattr__
and __getattribute__ in non-Classic classes in 2.2, a question I have
foolishly posted in the middle of the current tab thread. I would like
to know the answer to that question[1].

This proposal, if accepted, will probably mean a heck of a lot of work
for somebody. But since I don't want it accepted, I don't care.

References

Copyright

This document has been placed in the public domain.

[1] Tim Peters already has (private correspondence). My early 2.2 didn't
have a __getattribute__, and __getattr__ was implemented like
__getattribute__ now is. This has been fixed. The important conclusion
is that my Decorator Pattern is safe and all is right with the world.


text:
PEP: 461 Title: Adding % formatting to bytes and bytearray Author: Ethan
Furman<EMAIL_ADDRESS>Status: Final Type: Standards Track
Content-Type: text/x-rst Created: 13-Jan-2014 Python-Version: 3.5
Post-History: 14-Jan-2014, 15-Jan-2014, 17-Jan-2014, 22-Feb-2014,
25-Mar-2014, 27-Mar-2014 Resolution:
https://mail.python.org/pipermail/python-dev/2014-March/133621.html

Abstract

This PEP proposes adding % formatting operations similar to Python 2's
str type to bytes and bytearray[1][2].

Rationale

While interpolation is usually thought of as a string operation, there
are cases where interpolation on bytes or bytearrays make sense, and the
work needed to make up for this missing functionality detracts from the
overall readability of the code.

Motivation

With Python 3 and the split between str and bytes, one small but
important area of programming became slightly more difficult, and much
more painful -- wire format protocols[3].

This area of programming is characterized by a mixture of binary data
and ASCII compatible segments of text (aka ASCII-encoded text). Bringing
back a restricted %-interpolation for bytes and bytearray will aid both
in writing new wire format code, and in porting Python 2 wire format
code.

Common use-cases include dbf and pdf file formats, email formats, and
FTP and HTTP communications, among many others.

Proposed semantics for bytes and bytearray formatting

%-interpolation

All the numeric formatting codes (d, i, o, u, x, X, e, E, f, F, g, G,
and any that are subsequently added to Python 3) will be supported, and
will work as they do for str, including the padding, justification and
other related modifiers (currently #, 0, -, space, and + (plus any added
to Python 3)). The only non-numeric codes allowed are c, b, a, and s
(which is a synonym for b).

For the numeric codes, the only difference between str and bytes (or
bytearray) interpolation is that the results from these codes will be
ASCII-encoded text, not unicode. In other words, for any numeric
formatting code %x:

    b"%x" % val

is equivalent to:

    ("%x" % val).encode("ascii")

Examples:

    >>> b'%4x' % 10
    b'   a'

    >>> b'%#4x' % 10
    ' 0xa'

    >>> b'%04X' % 10
    '000A'

%c will insert a single byte, either from an int in range(256), or from
a bytes argument of length 1, not from a str.

Examples:

    >>> b'%c' % 48
    b'0'

    >>> b'%c' % b'a'
    b'a'

%b will insert a series of bytes. These bytes are collected in one of
two ways:

-   input type supports Py_buffer[4]? use it to collect the necessary
    bytes
-   input type is something else? use its __bytes__ method[5] ; if there
    isn't one, raise a TypeError

In particular, %b will not accept numbers nor str. str is rejected as
the string to bytes conversion requires an encoding, and we are refusing
to guess; numbers are rejected because:

-   what makes a number is fuzzy (float? Decimal? Fraction? some user
    type?)
-   allowing numbers would lead to ambiguity between numbers and textual
    representations of numbers (3.14 vs '3.14')
-   given the nature of wire formats, explicit is definitely better than
    implicit

%s is included as a synonym for %b for the sole purpose of making 2/3
code bases easier to maintain. Python 3 only code should use %b.

Examples:

    >>> b'%b' % b'abc'
    b'abc'

    >>> b'%b' % 'some string'.encode('utf8')
    b'some string'

    >>> b'%b' % 3.14
    Traceback (most recent call last):
    ...
    TypeError: b'%b' does not accept 'float'

    >>> b'%b' % 'hello world!'
    Traceback (most recent call last):
    ...
    TypeError: b'%b' does not accept 'str'

%a will give the equivalent of
repr(some_obj).encode('ascii', 'backslashreplace') on the interpolated
value. Use cases include developing a new protocol and writing landmarks
into the stream; debugging data going into an existing protocol to see
if the problem is the protocol itself or bad data; a fall-back for a
serialization format; or any situation where defining __bytes__ would
not be appropriate but a readable/informative representation is
needed[6].

%r is included as a synonym for %a for the sole purpose of making 2/3
code bases easier to maintain. Python 3 only code use %a[7].

Examples:

    >>> b'%a' % 3.14
    b'3.14'

    >>> b'%a' % b'abc'
    b"b'abc'"

    >>> b'%a' % 'def'
    b"'def'"

Compatibility with Python 2

As noted above, %s and %r are being included solely to help ease
migration from, and/or have a single code base with, Python 2. This is
important as there are modules both in the wild and behind closed doors
that currently use the Python 2 str type as a bytes container, and hence
are using %s as a bytes interpolator.

However, %b and %a should be used in new, Python 3 only code, so %s and
%r will immediately be deprecated, but not removed from the 3.x series
[8].

Proposed variations

It has been proposed to automatically use .encode('ascii','strict') for
str arguments to %b.

-   Rejected as this would lead to intermittent failures. Better to have
    the operation always fail so the trouble-spot can be correctly
    fixed.

It has been proposed to have %b return the ascii-encoded repr when the
value is a str (b'%b' % 'abc' --> b"'abc'").

-   Rejected as this would lead to hard to debug failures far from the
    problem site. Better to have the operation always fail so the
    trouble-spot can be easily fixed.

Originally this PEP also proposed adding format-style formatting, but it
was decided that format and its related machinery were all strictly text
(aka str) based, and it was dropped.

Various new special methods were proposed, such as __ascii__,
__format_bytes__, etc.; such methods are not needed at this time, but
can be visited again later if real-world use shows deficiencies with
this solution.

A competing PEP, PEP 460 Add binary interpolation and formatting <460>,
also exists.

Objections

The objections raised against this PEP were mainly variations on two
themes:

-   the bytes and bytearray types are for pure binary data, with no
    assumptions about encodings
-   offering %-interpolation that assumes an ASCII encoding will be an
    attractive nuisance and lead us back to the problems of the Python 2
    str/unicode text model

As was seen during the discussion, bytes and bytearray are also used for
mixed binary data and ASCII-compatible segments: file formats such as
dbf and pdf, network protocols such as ftp and email, etc.

bytes and bytearray already have several methods which assume an ASCII
compatible encoding. upper(), isalpha(), and expandtabs() to name just a
few. %-interpolation, with its very restricted mini-language, will not
be any more of a nuisance than the already existing methods.

Some have objected to allowing the full range of numeric formatting
codes with the claim that decimal alone would be sufficient. However, at
least two formats (dbf and pdf) make use of non-decimal numbers.

Footnotes

Copyright

This document has been placed in the public domain.

[1] http://docs.python.org/2/library/stdtypes.html#string-formatting

[2] neither string.Template, format, nor str.format are under
consideration

[3] https://mail.python.org/pipermail/python-dev/2014-January/131518.html

[4] http://docs.python.org/3/c-api/buffer.html examples: memoryview,
array.array, bytearray, bytes

[5] http://docs.python.org/3/reference/datamodel.html#object.__bytes__

[6] https://mail.python.org/pipermail/python-dev/2014-February/132750.html

[7] http://bugs.python.org/issue23467 -- originally %r was not allowed,
but was added for consistency during the 3.5 alpha stage.

[8] http://bugs.python.org/issue23467 -- originally %r was not allowed,
but was added for consistency during the 3.5 alpha stage.


text:
PEP: 224 Title: Attribute Docstrings Author: Marc-André Lemburg
<EMAIL_ADDRESS>Status: Rejected Type: Standards Track Created:
23-Aug-2000 Python-Version: 2.1 Post-History:

See 224-rejection for more information.

Introduction

This PEP describes the "attribute docstring" proposal for Python 2.0.
This PEP tracks the status and ownership of this feature. It contains a
description of the feature and outlines changes necessary to support the
feature. The CVS revision history of this file contains the definitive
historical record.

Rationale

This PEP proposes a small addition to the way Python currently handles
docstrings embedded in Python code.

Python currently only handles the case of docstrings which appear
directly after a class definition, a function definition or as first
string literal in a module. The string literals are added to the objects
in question under the __doc__ attribute and are from then on available
for introspection tools which can extract the contained information for
help, debugging and documentation purposes.

Docstrings appearing in locations other than the ones mentioned are
simply ignored and don't result in any code generation.

Here is an example:

    class C:
        "class C doc-string"

        a = 1
        "attribute C.a doc-string (1)"

        b = 2
        "attribute C.b doc-string (2)"

The docstrings (1) and (2) are currently being ignored by the Python
byte code compiler, but could obviously be put to good use for
documenting the named assignments that precede them.

This PEP proposes to also make use of these cases by proposing semantics
for adding their content to the objects in which they appear under new
generated attribute names.

The original idea behind this approach which also inspired the above
example was to enable inline documentation of class attributes, which
can currently only be documented in the class's docstring or using
comments which are not available for introspection.

Implementation

Docstrings are handled by the byte code compiler as expressions. The
current implementation special cases the few locations mentioned above
to make use of these expressions, but otherwise ignores the strings
completely.

To enable use of these docstrings for documenting named assignments
(which is the natural way of defining e.g. class attributes), the
compiler will have to keep track of the last assigned name and then use
this name to assign the content of the docstring to an attribute of the
containing object by means of storing it in as a constant which is then
added to the object's namespace during object construction time.

In order to preserve features like inheritance and hiding of Python's
special attributes (ones with leading and trailing double underscores),
a special name mangling has to be applied which uniquely identifies the
docstring as belonging to the name assignment and allows finding the
docstring later on by inspecting the namespace.

The following name mangling scheme achieves all of the above:

    __doc_<attributename>__

To keep track of the last assigned name, the byte code compiler stores
this name in a variable of the compiling structure. This variable
defaults to NULL. When it sees a docstring, it then checks the variable
and uses the name as basis for the above name mangling to produce an
implicit assignment of the docstring to the mangled name. It then resets
the variable to NULL to avoid duplicate assignments.

If the variable does not point to a name (i.e. is NULL), no assignments
are made. These will continue to be ignored like before. All classical
docstrings fall under this case, so no duplicate assignments are done.

In the above example this would result in the following new class
attributes to be created:

    C.__doc_a__ == "attribute C.a doc-string (1)"
    C.__doc_b__ == "attribute C.b doc-string (2)"

A patch to the current CVS version of Python 2.0 which implements the
above is available on SourceForge at[1].

Caveats of the Implementation

Since the implementation does not reset the compiling structure variable
when processing a non-expression, e.g. a function definition, the last
assigned name remains active until either the next assignment or the
next occurrence of a docstring.

This can lead to cases where the docstring and assignment may be
separated by other expressions:

    class C:
        "C doc string"

        b = 2

        def x(self):
            "C.x doc string"
             y = 3
             return 1

        "b's doc string"

Since the definition of method "x" currently does not reset the used
assignment name variable, it is still valid when the compiler reaches
the docstring "b's doc string" and thus assigns the string to __doc_b__.

A possible solution to this problem would be resetting the name variable
for all non-expression nodes in the compiler.

Possible Problems

Even though highly unlikely, attribute docstrings could get accidentally
concatenated to the attribute's value:

    class C:
        x = "text" \
            "x's docstring"

The trailing slash would cause the Python compiler to concatenate the
attribute value and the docstring.

A modern syntax highlighting editor would easily make this accident
visible, though, and by simply inserting empty lines between the
attribute definition and the docstring you can avoid the possible
concatenation completely, so the problem is negligible.

Another possible problem is that of using triple quoted strings as a way
to uncomment parts of your code.

If there happens to be an assignment just before the start of the
comment string, then the compiler will treat the comment as docstring
attribute and apply the above logic to it.

Besides generating a docstring for an otherwise undocumented attribute
there is no breakage.

Comments from our BDFL

Early comments on the PEP from Guido:

  I "kinda" like the idea of having attribute docstrings (meaning it's
  not of great importance to me) but there are two things I don't like
  in your current proposal:

  1.  The syntax you propose is too ambiguous: as you say, stand-alone
      string literal are used for other purposes and could suddenly
      become attribute docstrings.
  2.  I don't like the access method either (__doc_<attrname>__).

The author's reply:

  > 1. The syntax you propose is too ambiguous: as you say, stand-alone
      >    string literal are used for other purposes and could suddenly
      >    become attribute docstrings.

  This can be fixed by introducing some extra checks in the compiler to
  reset the "doc attribute" flag in the compiler struct.

      > 2. I don't like the access method either (``__doc_<attrname>__``).

  Any other name will do. It will only have to match these criteria:

  -   must start with two underscores (to match __doc__)
  -   must be extractable using some form of inspection (e.g. by using a
      naming convention which includes some fixed name part)
  -   must be compatible with class inheritance (i.e. should be stored
      as attribute)

Later on in March, Guido pronounced on this PEP in March 2001 (on
python-dev). Here are his reasons for rejection mentioned in private
mail to the author of this PEP:

  ...

  It might be useful, but I really hate the proposed syntax.

      a = 1
      "foo bar"
      b = 1

  I really have no way to know whether "foo bar" is a docstring for a or
  for b.

  ...

  You can use this convention:

      a = 1
      __doc_a__ = "doc string for a"

  This makes it available at runtime.

      > Are you completely opposed to adding attribute documentation
      > to Python or is it just the way the implementation works ? I
      > find the syntax proposed in the PEP very intuitive and many
      > other users on c.l.p and in private emails have supported it
      > at the time I wrote the PEP.

  It's not the implementation, it's the syntax. It doesn't convey a
  clear enough coupling between the variable and the doc string.

Copyright

This document has been placed in the Public Domain.

References

[1] http://sourceforge.net/patch/?func=detailpatch&patch_id=101264&group_id=5470


text:
PEP: 639 Title: Improving License Clarity with Better Package Metadata
Author: Philippe Ombredanne<EMAIL_ADDRESS>C.A.M. Gerlach
<EMAIL_ADDRESS>Karolina Surma<EMAIL_ADDRESS>PEP-Delegate: Brett Cannon<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/53020 Status: Provisional Type: Standards
Track Topic: Packaging Created: 15-Aug-2019 Post-History: 15-Aug-2019,
17-Dec-2021, 10-May-2024, Resolution:
https://discuss.python.org/t/53020/106

Provisional Acceptance

This PEP has been provisionally accepted, with the following required
conditions before the PEP is made Final:

1.  An implementation of the PEP in two build back-ends.
2.  An implementation of the PEP in PyPI.

Abstract

This PEP defines a specification how licenses are documented in the
Python projects.

To achieve that, it:

-   Adopts the SPDX license expression syntax as a means of expressing
    the license for a Python project.
-   Defines how to include license files within the projects, source and
    built distributions.
-   Specifies the necessary changes to Core Metadata and the
    corresponding Pyproject Metadata keys
-   Describes the necessary changes to the source distribution (sdist),
    built distribution (wheel) and installed project standards.

This will make license declaration simpler and less ambiguous for
package authors to create, end users to understand, and tools to
programmatically process.

The changes will update the Core Metadata specification to version 2.4.

Goals

This PEP's scope is limited to covering new mechanisms for documenting
the license of a distribution package, specifically defining:

-   A means of specifying a SPDX license expression.
-   A method of including license texts in distribution packages and
    installed Projects.

The changes that this PEP requires have been designed to minimize impact
and maximize backward compatibility.

Non-Goals

This PEP doesn't recommend any particular license to be chosen by any
particular package author.

If projects decide not to use the new fields, no additional restrictions
are imposed by this PEP when uploading to PyPI.

This PEP also is not about license documentation for individual files,
though this is a surveyed topic <639-license-doc-source-files> in an
appendix, nor does it intend to cover cases where the
source distribution <Source Distribution (or "sdist")> and
binary distribution packages don't have
the same licenses <639-rejected-ideas-difference-license-source-binary>.

Motivation

Software must be licensed in order for anyone other than its creator to
download, use, share and modify it. Today, there are multiple fields
where licenses are documented in Core Metadata, and there are
limitations to what can be expressed in each of them. This often leads
to confusion both for package authors and end users, including
distribution re-packagers.

This has triggered a number of license-related discussions and issues,
including on outdated and ambiguous PyPI classifiers, license
interoperability with other ecosystems, too many confusing license
metadata options, limited support for license files in the Wheel
project, and the lack of precise license metadata.

As a result, on average, Python packages tend to have more ambiguous and
missing license information than other common ecosystems. This is
supported by the statistics page of the ClearlyDefined project, an Open
Source Initiative effort to help improve licensing clarity of other FOSS
projects, covering all packages from PyPI, Maven, npm and Rubygems.

The current license classifiers could be extended to include the full
range of the SPDX identifiers while deprecating the ambiguous
classifiers (such as License :: OSI Approved :: BSD License).

However, there are multiple arguments against such an approach:

-   It requires a great effort to duplicate the SPDX license list and
    keep it in sync.
-   It is a hard break in backward compatibility, forcing package
    authors to update to new classifiers immediately when PyPI
    deprecates the old ones.
-   It only covers packages under a single license; it doesn't address
    projects that vendor dependencies (e.g. Setuptools), offer a choice
    of licenses (e.g. Packaging) or were relicensed, adapt code from
    other projects or contain fonts, images, examples, binaries or other
    assets under other licenses.
-   It requires both authors and tools understand and implement the
    PyPI-specific classifier system.
-   It does not provide as clear an indicator that a package has adopted
    the new system, and should be treated accordingly.

Rationale

A survey was conducted to map the existing license metadata definitions
in the Python ecosystem <639-license-doc-python> and a
variety of other packaging systems, Linux distributions,
language ecosystems and applications <639-license-doc-other-projects>.

The takeaways from the survey have guided the recommendations of this
PEP:

-   SPDX and SPDX-like syntaxes are the most popular license expressions
    in many modern package systems.
-   Most Free and Open Source Software licenses require package authors
    to include their full text in a Distribution Package.

Therefore, this PEP introduces two new Core Metadata fields:

-   License-Expression <639-spec-field-license-expression> that provides
    an unambiguous way to express the license of a package using SPDX
    license expressions.
-   License-File <639-spec-field-license-file> that offers a
    standardized way to include the full text of the license(s) with the
    package when distributed, and allows other tools consuming the
    Core Metadata to locate a distribution archive's license files.

Furthermore, this specification builds upon existing practice in the
Setuptools and Wheel projects. An up-to-date version of the current
draft of this PEP is implemented in the Hatch packaging tool, and an
earlier draft of the license files portion <639-spec-field-license-file>
is implemented in Setuptools.

Terminology

The keywords "MUST", "MUST NOT", "REQUIRED", "SHOULD", "SHOULD NOT",
"RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be
interpreted as described in 2119.

License terms

The license-related terminology draws heavily from the SPDX Project,
particularly license identifier and license expression.

license classifier

    A PyPI Trove classifier (as
    described <packaging:core-metadata-classifier> in the Core Metadata
    specification) which begins with License ::.

license expression SPDX expression A string with valid SPDX license
expression syntax including one or more SPDX license identifier(s),
which describes a Project's license(s) and how they inter-relate.
Examples: GPL-3.0-or-later, MIT AND (Apache-2.0 OR BSD-2-clause)

license identifier SPDX identifier A valid SPDX short-form license
identifier, as described in the 639-spec-field-license-expression
section of this PEP. This includes all valid SPDX identifiers and the
custom LicenseRef-[idstring] strings conforming to the SPDX
specification, clause 10.1. Examples: MIT, GPL-3.0-only,
LicenseRef-My-Custom-License

root license directory license directory The directory under which
license files are stored in a project source tree, distribution archive
or installed project. Also, the root directory that their paths recorded
in the License-File <639-spec-field-license-file> Core Metadata field
are relative to. Defined to be the project root directory for a
project source tree or
source distribution <Source Distribution (or "sdist")>; and a
subdirectory named licenses of the directory containing the
built metadata— i.e., the .dist-info/licenses directory— for a
Built Distribution or installed project.

Specification

The changes necessary to implement this PEP include:

-   additions to Core Metadata <639-spec-core-metadata>, as defined in
    the specification.
-   additions to the author-provided
    project source metadata <639-spec-source-metadata>, as defined in
    the specification.
-   additions <639-spec-project-formats> to the source distribution
    (sdist), built distribution (wheel) and installed project
    specifications.
-   guide for tools <639-spec-converting-metadata> handling and
    converting legacy license metadata to license expressions, to ensure
    the results are consistent and correct.

Note that the guidance on errors and warnings is for tools' default
behavior; they MAY operate more strictly if users explicitly configure
them to do so, such as by a CLI flag or a configuration option.

SPDX license expression syntax

This PEP adopts the SPDX license expression syntax as documented in the
SPDX specification, either Version 2.2 or a later compatible version.

A license expression can use the following license identifiers:

-   Any SPDX-listed license short-form identifiers that are published in
    the SPDX License List, version 3.17 or any later compatible version.
    Note that the SPDX working group never removes any license
    identifiers; instead, they may choose to mark an identifier as
    "deprecated".
-   The custom LicenseRef-[idstring] string(s), where [idstring] is a
    unique string containing letters, numbers, . and/or -, to identify
    licenses that are not included in the SPDX license list. The custom
    identifiers must follow the SPDX specification, clause 10.1 of the
    given specification version.

Examples of valid SPDX expressions:

    MIT
    BSD-3-Clause
    MIT AND (Apache-2.0 OR BSD-2-Clause)
    MIT OR GPL-2.0-or-later OR (FSFUL AND BSD-2-Clause)
    GPL-3.0-only WITH Classpath-Exception-2.0 OR BSD-3-Clause
    LicenseRef-Special-License OR CC0-1.0 OR Unlicense
    LicenseRef-Proprietary

Examples of invalid SPDX expressions:

    Use-it-after-midnight
    Apache-2.0 OR 2-BSD-Clause
    LicenseRef-License with spaces
    LicenseRef-License_with_underscores

Core Metadata

The error and warning guidance in this section applies to build and
publishing tools; end-user-facing install tools MAY be less strict than
mentioned here when encountering malformed metadata that does not
conform to this specification.

As it adds new fields, this PEP updates the Core Metadata version to
2.4.

Add License-Expression field

The License-Expression optional Core Metadata field is specified to
contain a text string that is a valid SPDX license expression, as
defined above <639-spdx>.

Build and publishing tools SHOULD check that the License-Expression
field contains a valid SPDX expression, including the validity of the
particular license identifiers (as defined above <639-spdx>). Tools MAY
halt execution and raise an error when an invalid expression is found.
If tools choose to validate the SPDX expression, they also SHOULD store
a case-normalized version of the License-Expression field using the
reference case for each SPDX license identifier and uppercase for the
AND, OR and WITH keywords. Tools SHOULD report a warning and publishing
tools MAY raise an error if one or more license identifiers have been
marked as deprecated in the SPDX License List.

For all newly-uploaded distribution archives that include a
License-Expression field, the Python Package Index (PyPI) MUST validate
that they contain a valid, case-normalized license expression with valid
identifiers (as defined above <639-spdx>) and MUST reject uploads that
do not. Custom license identifiers which conform to the SPDX
specification are considered valid. PyPI MAY reject an upload for using
a deprecated license identifier, so long as it was deprecated as of the
above-mentioned SPDX License List version.

Add License-File field

License-File is an optional Core Metadata field. Each instance contains
the string representation of the path of a license-related file. The
path is located within the project source tree, relative to the
project root directory. It is a multi-use field that may appear zero or
more times and each instance lists the path to one such file. Files
specified under this field could include license text,
author/attribution information, or other legal notices that need to be
distributed with the package.

As specified by this PEP <639-spec-project-formats>, its value is also
that file's path relative to the root license directory in both
installed projects and the standardized Distribution Package types.

If a License-File is listed in a
Source Distribution <Source Distribution (or "sdist")> or
Built Distribution's Core Metadata:

-   That file MUST be included in the distribution archive at the
    specified path relative to the root license directory.
-   That file MUST be installed with the project at that same relative
    path.
-   The specified relative path MUST be consistent between project
    source trees, source distributions (sdists), built distributions
    (Wheels) and installed projects.
-   Inside the root license directory, packaging tools MUST reproduce
    the directory structure under which the source license files are
    located relative to the project root.
-   Path delimiters MUST be the forward slash character (/), and parent
    directory indicators (..) MUST NOT be used.
-   License file content MUST be UTF-8 encoded text.

Build tools MAY and publishing tools SHOULD produce an informative
warning if a built distribution's metadata contains no License-File
entries, and publishing tools MAY but build tools MUST NOT raise an
error.

For all newly-uploaded distribution archives that include one or more
License-File fields in their Core Metadata and declare a
Metadata-Version of 2.4 or higher, PyPI SHOULD validate that all
specified files are present in that distribution archives, and MUST
reject uploads that do not validate.

Deprecate License field

The legacy unstructured-text License Core Metadata field is deprecated
and replaced by the new License-Expression field. The fields are
mutually exclusive. Tools which generate Core Metadata MUST NOT create
both these fields. Tools which read Core Metadata, when dealing with
both these fields present at the same time, MUST read the value of
License-Expression and MUST disregard the value of the License field.

If only the License field is present, tools MAY issue a warning
informing users it is deprecated and recommending License-Expression
instead.

For all newly-uploaded distribution archives that include a
License-Expression field, the Python Package Index (PyPI) MUST reject
any that specify both License and License-Expression fields.

The License field may be removed from a new version of the specification
in a future PEP.

Deprecate license classifiers

Using license classifiers in the Classifier Core Metadata field
(described in the Core Metadata specification) is deprecated and
replaced by the more precise License-Expression field.

If the License-Expression field is present, build tools MAY raise an
error if one or more license classifiers is included in a Classifier
field, and MUST NOT add such classifiers themselves.

Otherwise, if this field contains a license classifier, tools MAY issue
a warning informing users such classifiers are deprecated, and
recommending License-Expression instead. For compatibility with existing
publishing and installation processes, the presence of license
classifiers SHOULD NOT raise an error unless License-Expression is also
provided.

New license classifiers MUST NOT be added to PyPI; users needing them
SHOULD use the License-Expression field instead. License classifiers may
be removed from a new version of the specification in a future PEP.

Project source metadata

This PEP specifies changes to the project's source metadata under a
[project] table in the pyproject.toml file.

Add string value to license key

license key in the [project] table is defined to contain a top-level
string value. It is a valid SPDX license expression as
defined in this PEP <639-spdx>. Its value maps to the License-Expression
field in the core metadata.

Build tools SHOULD validate and perform case normalization of the
expression as described in the 639-spec-field-license-expression
section, outputting an error or warning as specified.

Examples:

    [project]
    license = "MIT"

    [project]
    license = "MIT AND (Apache-2.0 OR BSD-2-clause)"

    [project]
    license = "MIT OR GPL-2.0-or-later OR (FSFUL AND BSD-2-Clause)"

    [project]
    license = "LicenseRef-Proprietary"

Add license-files key

A new license-files key is added to the [project] table for specifying
paths in the project source tree relative to pyproject.toml to file(s)
containing licenses and other legal notices to be distributed with the
package. It corresponds to the License-File fields in the Core Metadata.

Its value is an array of strings which MUST contain valid glob patterns,
as specified below:

-   Alphanumeric characters, underscores (_), hyphens (-) and dots (.)
    MUST be matched verbatim.
-   Special glob characters: *, ?, ** and character ranges: []
    containing only the verbatim matched characters MUST be supported.
    Within [...], the hyphen indicates a locale-agnostic range (e.g.
    a-z, order based on Unicode code points). Hyphens at the start or
    end are matched literally.
-   Path delimiters MUST be the forward slash character (/). Patterns
    are relative to the directory containing pyproject.toml, therefore
    the leading slash character MUST NOT be used.
-   Parent directory indicators (..) MUST NOT be used.

Any characters or character sequences not covered by this specification
are invalid. Projects MUST NOT use such values. Tools consuming this
field SHOULD reject invalid values with an error.

Tools MUST assume that license file content is valid UTF-8 encoded text,
and SHOULD validate this and raise an error if it is not.

Literal paths (e.g. LICENSE) are treated as valid globs which means they
can also be defined.

Build tools:

-   MUST treat each value as a glob pattern, and MUST raise an error if
    the pattern contains invalid glob syntax.
-   MUST include all files matched by a listed pattern in all
    distribution archives.
-   MUST list each matched file path under a License-File field in the
    Core Metadata.
-   MUST raise an error if any individual user-specified pattern does
    not match at least one file.

If the license-files key is present and is set to a value of an empty
array, then tools MUST NOT include any license files and MUST NOT raise
an error.

Examples of valid license files declaration:

    [project]
    license-files = ["LICEN[CS]E*", "AUTHORS*"]

    [project]
    license-files = ["licenses/LICENSE.MIT", "licenses/LICENSE.CC0"]

    [project]
    license-files = ["LICENSE.txt", "licenses/*"]

    [project]
    license-files = []

Examples of invalid license files declaration:

    [project]
    license-files = ["..\LICENSE.MIT"]

Reason: .. must not be used. \ is an invalid path delimiter, / must be
used.

    [project]
    license-files = ["LICEN{CSE*"]

Reason: "LICEN{CSE*" is not a valid glob.

Deprecate license key table subkeys

Table values for the license key in the [project] table, including the
text and file table subkeys, are now deprecated. If the new
license-files key is present, build tools MUST raise an error if the
license key is defined and has a value other than a single top-level
string.

If the new license-files key is not present and the text subkey is
present in a license table, tools SHOULD issue a warning informing users
it is deprecated and recommending a license expression as a top-level
string key instead.

Likewise, if the new license-files key is not present and the file
subkey is present in the license table, tools SHOULD issue a warning
informing users it is deprecated and recommending the license-files key
instead.

If the specified license file is present in the source tree, build tools
SHOULD use it to fill the License-File field in the core metadata, and
MUST include the specified file as if it were specified in a
license-file field. If the file does not exist at the specified path,
tools MUST raise an informative error as previously specified.

Table values for the license key MAY be removed from a new version of
the specification in a future PEP.

License files in project formats

A few additions will be made to the existing specifications.

Project source trees

    Per 639-spec-source-metadata section, the Declaring Project Metadata
    specification will be updated to reflect that license file paths
    MUST be relative to the project root directory; i.e. the directory
    containing the pyproject.toml (or equivalently, other legacy project
    configuration, e.g. setup.py, setup.cfg, etc).

Source distributions (sdists) <Source Distribution (or "sdist")>

    The sdist specification will be updated to reflect that if the
    Metadata-Version is 2.4 or greater, the sdist MUST contain any
    license files specified by the
    License-File field <639-spec-field-license-file> in the PKG-INFO at
    their respective paths relative to the of the sdist (containing the
    pyproject.toml and the PKG-INFO Core Metadata).

Built distributions (wheels)

    The Wheel specification will be updated to reflect that if the
    Metadata-Version is 2.4 or greater and one or more License-File
    fields is specified, the .dist-info directory MUST contain a
    licenses subdirectory, which MUST contain the files listed in the
    License-File fields in the METADATA file at their respective paths
    relative to the licenses directory.

Installed projects

    The Recording Installed Projects specification will be updated to
    reflect that if the Metadata-Version is 2.4 or greater and one or
    more License-File fields is specified, the .dist-info directory MUST
    contain a licenses subdirectory which MUST contain the files listed
    in the License-File fields in the METADATA file at their respective
    paths relative to the licenses directory, and that any files in this
    directory MUST be copied from wheels by install tools.

Converting legacy metadata

Tools MUST NOT use the contents of the license.text [project] key (or
equivalent tool-specific format), license classifiers or the value of
the Core Metadata License field to fill the top-level string value of
the license key or the Core Metadata License-Expression field without
informing the user and requiring unambiguous, affirmative user action to
select and confirm the desired license expression value before
proceeding.

Tool authors, who need to automatically convert license classifiers to
SPDX identifiers, can use the
recommendation <639-spec-mapping-classifiers-identifiers> prepared by
the PEP authors.

Backwards Compatibility

Adding a new License-Expression Core Metadata field and a top-level
string value for the license key in the pyproject.toml [project] table
unambiguously means support for the specification in this PEP. This
avoids the risk of new tooling misinterpreting a license expression as a
free-form license description or vice versa.

The legacy deprecated Core Metadata License field, license key table
subkeys (text and file) in the pyproject.toml [project] table and
license classifiers retain backwards compatibility. A removal is left to
a future PEP and a new version of the Core Metadata specification.

Specification of the new License-File Core Metadata field and adding the
files in the distribution is designed to be largely backwards-compatible
with the existing use of that field in many packaging tools. The new
license-files key in the [project] table of pyproject.toml will only
have an effect once users and tools adopt it.

This PEP specifies that license files should be placed in a dedicated
licenses subdir of .dist-info directory. This is new and ensures that
wheels following this PEP will have differently-located licenses
relative to those produced via the previous installer-specific behavior.
This is further supported by a new metadata version.

This also resolves current issues where license files are accidentally
replaced if they have the same names in different places, making wheels
undistributable without noticing. It also prevents conflicts with other
metadata files in the same directory.

The additions will be made to the source distribution (sdist), built
distribution (wheel) and installed project specifications. They document
behaviors allowed under their current specifications, and gate them
behind the new metadata version.

This PEP proposes PyPI implement validation of the new
License-Expression and License-File fields, which has no effect on new
and existing packages uploaded unless they explicitly opt in to using
these new fields and fail to follow the specification correctly.
Therefore, this does not have a backward compatibility impact, and
guarantees forward compatibility by ensuring all distributions uploaded
to PyPI with the new fields conform to the specification.

Security Implications

This PEP has no foreseen security implications: the License-Expression
field is a plain string and the License-File fields are file paths.
Neither introduces any known new security concerns.

How to Teach This

A majority of packages use a single license which makes the case simple:
a single license identifier is a valid license expression.

Users of packaging tools will learn the valid license expression of
their package through the messages issued by the tools when they detect
invalid ones, or when the deprecated License field or license
classifiers are used.

If an invalid License-Expression is used, the users will not be able to
publish their package to PyPI and an error message will help them
understand they need to use SPDX identifiers. It will be possible to
generate a distribution with incorrect license metadata, but not to
publish one on PyPI or any other index server that enforces
License-Expression validity. For authors using the now-deprecated
License field or license classifiers, packaging tools may warn them and
inform them of the replacement, License-Expression.

Tools may also help with the conversion and suggest a license expression
in many common cases:

-   The appendix 639-spec-mapping-classifiers-identifiers provides tool
    authors with recommendation on how to suggest a license expression
    produced from legacy classifiers.
-   Tools may be able to suggest how to update an existing License value
    in project source metadata and convert that to a license expression,
    as also specified in this PEP <639-spec-converting-metadata>.

Reference Implementation

Tools will need to support parsing and validating license expressions in
the License-Expression field if they decide to implement this part of
the specification. It's up to the tools whether they prefer to implement
the validation on their side (e.g. like hatch) or use one of the
available Python libraries (e.g. license-expression). This PEP does not
mandate using any specific library and leaves it to the tools authors to
choose the best implementation for their projects.

Rejected Ideas

Many alternative ideas were proposed and after a careful consideration,
rejected. The exhaustive list including the rationale for rejecting can
be found in a separate page <639-rejected-ideas-details>.

Appendices

A list of auxiliary documents is provided:

-   Detailed Licensing Examples <639-examples>,
-   User Scenarios <639-user-scenarios>,
-   License Documentation in Python and Other Projects <639-license-doc-python>,
-   Mapping License Classifiers to SPDX Identifiers <639-spec-mapping-classifiers-identifiers>,
-   Rejected Ideas <639-rejected-ideas-details> in detail.

References

Acknowledgments

-   Alyssa Coghlan
-   Kevin P. Fleming
-   Pradyun Gedam
-   Oleg Grenrus
-   Dustin Ingram
-   Chris Jerdonek
-   Cyril Roelandt
-   Luis Villa
-   Seth M. Larson
-   Ofek Lev

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.