text:
PEP: 213 Title: Attribute Access Handlers Author: Paul Prescod
<EMAIL_ADDRESS>Status: Deferred Type: Standards Track Content-Type:
text/x-rst Created: 21-Jul-2000 Python-Version: 2.1 Post-History:

Introduction

It is possible (and even relatively common) in Python code and in
extension modules to "trap" when an instance's client code attempts to
set an attribute and execute code instead. In other words, it is
possible to allow users to use attribute assignment/ retrieval/deletion
syntax even though the underlying implementation is doing some
computation rather than directly modifying a binding.

This PEP describes a feature that makes it easier, more efficient and
safer to implement these handlers for Python instances.

Justification

Scenario 1

You have a deployed class that works on an attribute named "stdout".
After a while, you think it would be better to check that stdout is
really an object with a "write" method at the moment of assignment.
Rather than change to a setstdout method (which would be incompatible
with deployed code) you would rather trap the assignment and check the
object's type.

Scenario 2

You want to be as compatible as possible with an object model that has a
concept of attribute assignment. It could be the W3C Document Object
Model or a particular COM interface (e.g. the PowerPoint interface). In
that case you may well want attributes in the model to show up as
attributes in the Python interface, even though the underlying
implementation may not use attributes at all.

Scenario 3

A user wants to make an attribute read-only.

In short, this feature allows programmers to separate the interface of
their module from the underlying implementation for whatever purpose.
Again, this is not a new feature but merely a new syntax for an existing
convention.

Current Solution

To make some attributes read-only:

    class foo:
        def __setattr__( self, name, val ):
            if name=="readonlyattr":
                raise TypeError
            elif name=="readonlyattr2":
                raise TypeError
        ...
        else:
            self.__dict__["name"]=val

This has the following problems:

1.  The creator of the method must be intimately aware of whether
    somewhere else in the class hierarchy __setattr__ has also been
    trapped for any particular purpose. If so, she must specifically
    call that method rather than assigning to the dictionary. There are
    many different reasons to overload __setattr__ so there is a decent
    potential for clashes. For instance object database implementations
    often overload setattr for an entirely unrelated purpose.
2.  The string-based switch statement forces all attribute handlers to
    be specified in one place in the code. They may then dispatch to
    task-specific methods (for modularity) but this could cause
    performance problems.
3.  Logic for the setting, getting and deleting must live in
    __getattr__, __setattr__ and __delattr__. Once again, this can be
    mitigated through an extra level of method call but this is
    inefficient.

Proposed Syntax

Special methods should declare themselves with declarations of the
following form:

    class x:
        def __attr_XXX__(self, op, val ):
            if op=="get":
                return someComputedValue(self.internal)
            elif op=="set":
                self.internal=someComputedValue(val)
            elif op=="del":
                del self.internal

Client code looks like this:

    fooval=x.foo
    x.foo=fooval+5
    del x.foo

Semantics

Attribute references of all three kinds should call the method. The op
parameter can be "get"/"set"/"del". Of course this string will be
interned so the actual checks for the string will be very fast.

It is disallowed to actually have an attribute named XXX in the same
instance as a method named __attr_XXX__.

An implementation of __attr_XXX__ takes precedence over an
implementation of __getattr__ based on the principle that __getattr__ is
supposed to be invoked only after finding an appropriate attribute has
failed.

An implementation of __attr_XXX__ takes precedence over an
implementation of __setattr__ in order to be consistent. The opposite
choice seems fairly feasible also, however. The same goes for __del_y__.

Proposed Implementation

There is a new object type called an attribute access handler. Objects
of this type have the following attributes:

    name (e.g. XXX, not __attr__XXX__)
    method (pointer to a method object)

In PyClass_New, methods of the appropriate form will be detected and
converted into objects (just like unbound method objects). These are
stored in the class __dict__ under the name XXX. The original method is
stored as an unbound method under its original name.

If there are any attribute access handlers in an instance at all, a flag
is set. Let's call it "I_have_computed_attributes" for now. Derived
classes inherit the flag from base classes. Instances inherit the flag
from classes.

A get proceeds as usual until just before the object is returned. In
addition to the current check whether the returned object is a method it
would also check whether a returned object is an access handler. If so,
it would invoke the getter method and return the value. To remove an
attribute access handler you could directly fiddle with the dictionary.

A set proceeds by checking the "I_have_computed_attributes" flag. If it
is not set, everything proceeds as it does today. If it is set then we
must do a dictionary get on the requested object name. If it returns an
attribute access handler then we call the setter function with the
value. If it returns any other object then we discard the result and
continue as we do today. Note that having an attribute access handler
will mildly affect attribute "setting" performance for all sets on a
particular instance, but no more so than today, using __setattr__. Gets
are more efficient than they are today with __getattr__.

The I_have_computed_attributes flag is intended to eliminate the
performance degradation of an extra "get" per "set" for objects not
using this feature. Checking this flag should have minuscule performance
implications for all objects.

The implementation of delete is analogous to the implementation of set.

Caveats

1.  You might note that I have not proposed any logic to keep the
    I_have_computed_attributes flag up to date as attributes are added
    and removed from the instance's dictionary. This is consistent with
    current Python. If you add a __setattr__ method to an object after
    it is in use, that method will not behave as it would if it were
    available at "compile" time. The dynamism is arguably not worth the
    extra implementation effort. This snippet demonstrates the current
    behavior:

        >>> def prn(*args):print args
        >>> class a:

        ...    __setattr__=prn
        >>> a().foo=5
        (<__main__.a instance at 882890>, 'foo', 5)

        >>> class b: pass
        >>> bi=b()
        >>> bi.__setattr__=prn
        >>> b.foo=5

2.  Assignment to __dict__["XXX"] can overwrite the attribute access
    handler for __attr_XXX__. Typically the access handlers will store
    information away in private __XXX variables

3.  An attribute access handler that attempts to call setattr or getattr
    on the object itself can cause an infinite loop (as with
    __getattr__) Once again, the solution is to use a special (typically
    private) variable such as __XXX.

Note

The descriptor mechanism described in PEP 252 is powerful enough to
support this more directly. A 'getset' constructor may be added to the
language making this possible:

    class C:
        def get_x(self):
            return self.__x
        def set_x(self, v):
            self.__x = v
        x = getset(get_x, set_x)

Additional syntactic sugar might be added, or a naming convention could
be recognized.


text:
PEP: 570 Title: Python Positional-Only Parameters Version: $Revision$
Last-Modified: $Date$ Author: Larry Hastings<EMAIL_ADDRESS>Pablo
Galindo<EMAIL_ADDRESS>Mario Corchero<EMAIL_ADDRESS>Eric N. Vander Weele<EMAIL_ADDRESS>BDFL-Delegate: Guido van Rossum
<EMAIL_ADDRESS>Discussions-To:
https://discuss.python.org/t/pep-570-python-positional-only-parameters/1078
Status: Final Type: Standards Track Content-Type: text/x-rst Created:
20-Jan-2018 Python-Version: 3.8

Abstract

This PEP proposes to introduce a new syntax, /, for specifying
positional-only parameters in Python function definitions.

Positional-only parameters have no externally-usable name. When a
function accepting positional-only parameters is called, positional
arguments are mapped to these parameters based solely on their order.

When designing APIs (application programming interfaces), library
authors try to ensure correct and intended usage of an API. Without the
ability to specify which parameters are positional-only, library authors
must be careful when choosing appropriate parameter names. This care
must be taken even for required parameters or when the parameters have
no external semantic meaning for callers of the API.

In this PEP, we discuss:

-   Python's history and current semantics for positional-only
    parameters
-   the problems encountered by not having them
-   how these problems are handled without language-intrinsic support
    for positional-only parameters
-   the benefits of having positional-only parameters

Within context of the motivation, we then:

-   discuss why positional-only parameters should be a feature intrinsic
    to the language
-   propose the syntax for marking positional-only parameters
-   present how to teach this new feature
-   note rejected ideas in further detail

Motivation

History of Positional-Only Parameter Semantics in Python

Python originally supported positional-only parameters. Early versions
of the language lacked the ability to call functions with arguments
bound to parameters by name. Around Python 1.0, parameter semantics
changed to be positional-or-keyword. Since then, users have been able to
provide arguments to a function either positionally or by the keyword
name specified in the function's definition.

In current versions of Python, many CPython "builtin" and standard
library functions only accept positional-only parameters. The resulting
semantics can be easily observed by calling one of these functions using
keyword arguments:

    >>> help(pow)
    ...
    pow(x, y, z=None, /)
    ...

    >>> pow(x=5, y=3)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: pow() takes no keyword arguments

pow() expresses that its parameters are positional-only via the /
marker. However, this is only a documentation convention; Python
developers cannot use this syntax in code.

There are functions with other interesting semantics:

-   range(), an overloaded function, accepts an optional parameter to
    the left of its required parameter.[1]
-   dict(), whose mapping/iterator parameter is optional and
    semantically must be positional-only. Any externally visible name
    for this parameter would occlude that name going into the **kwarg
    keyword variadic parameter dict.[2]

One can emulate these semantics in Python code by accepting
(*args, **kwargs) and parsing the arguments manually. However, this
results in a disconnect between the function definition and what the
function contractually accepts. The function definition does not match
the logic of the argument handling.

Additionally, the / syntax is used beyond CPython for specifying similar
semantics (i.e.,[3][4]); thus, indicating that these scenarios are not
exclusive to CPython and the standard library.

Problems Without Positional-Only Parameters

Without positional-only parameters, there are challenges for library
authors and users of APIs. The following subsections outline the
problems encountered by each entity.

Challenges for Library Authors

With positional-or-keyword parameters, the mix of calling conventions is
not always desirable. Authors may want to restrict usage of an API by
disallowing calling the API with keyword arguments, which exposes the
name of the parameter when part of the public API. This approach is
especially useful for required function parameters that already have
semantic meaning (e.g, namedtuple(typenames, field_names, …) or when the
parameter name has no true external meaning (e.g., arg1, arg2, …, etc
for min()). If a caller of an API starts using a keyword argument, the
library author cannot rename the parameter because it would be a
breaking change.

Positional-only parameters can be emulated by extracting arguments from
*args one by one. However, this approach is error-prone and is not
synonymous with the function definition, as previously mentioned. The
usage of the function is ambiguous and forces users to look at help(),
the associated auto-generated documentation, or source code to
understand what parameters the function contractually accepts.

Challenges for Users of an API

Users may be surprised when first encountering positional-only notation.
This is expected given that it has only recently been documented [5] and
it is not possible to use in Python code. For these reasons, this
notation is currently an outlier that appears only in CPython APIs
developed in C. Documenting the notation and making it possible to use
it in Python code would eliminate this disconnect.

Furthermore, the current documentation for positional-only parameters is
inconsistent:

-   Some functions denote optional groups of positional-only parameters
    by enclosing them in nested square brackets.[6]
-   Some functions denote optional groups of positional-only parameters
    by presenting multiple prototypes with varying numbers of
    parameters. [7]
-   Some functions use both of the above approaches.[8][9]

Another point the current documentation does not distinguish is whether
a function takes positional-only parameters. open() accepts keyword
arguments; however, ord() does not — there is no way of telling just by
reading the existing documentation.

Benefits of Positional-Only Parameters

Positional-only parameters give more control to library authors to
better express the intended usage of an API and allows the API to evolve
in a safe, backward-compatible way. Additionally, it makes the Python
language more consistent with existing documentation and the behavior of
various "builtin" and standard library functions.

Empowering Library Authors

Library authors would have the flexibility to change the name of
positional-only parameters without breaking callers. This flexibility
reduces the cognitive burden for choosing an appropriate public-facing
name for required parameters or parameters that have no true external
semantic meaning.

Positional-only parameters are useful in several situations such as:

-   when a function accepts any keyword argument but also can accept a
    positional one
-   when a parameter has no external semantic meaning
-   when an API's parameters are required and unambiguous

A key scenario is when a function accepts any keyword argument but can
also accepts a positional one. Prominent examples are Formatter.format
and dict.update. For instance, dict.update accepts a dictionary
(positionally), an iterable of key/value pairs (positionally), or
multiple keyword arguments. In this scenario, if the dictionary
parameter were not positional-only, the user could not use the name that
the function definition uses for the parameter or, conversely, the
function could not distinguish easily if the argument received is the
dictionary/iterable or a keyword argument for updating the key/value
pair.

Another scenario where positional-only parameters are useful is when the
parameter name has no true external semantic meaning. For example, let's
say we want to create a function that converts from one type to another:

    def as_my_type(x):
        ...

The name of the parameter provides no intrinsic value and forces the API
author to maintain its name forever since callers might pass x as a
keyword argument.

Additionally, positional-only parameters are useful when an API's
parameters are required and is unambiguous with respect to function. For
example:

    def add_to_queue(item: QueueItem):
        ...

The name of the function makes clear the argument expected. A keyword
argument provides minimal benefit and also limits the future evolution
of the API. Say at a later time we want this function to be able to take
multiple items, while preserving backwards compatibility:

    def add_to_queue(items: Union[QueueItem, List[QueueItem]]):
        ...

or to take them by using argument lists:

    def add_to_queue(*items: QueueItem):
        ...

the author would be forced to always keep the original parameter name to
avoid potentially breaking callers.

By being able to specify positional-only parameters, an author can
change the name of the parameters freely or even change them to *args,
as seen in the previous example. There are multiple function definitions
in the standard library which fall into this category. For example, the
required parameter to collections.defaultdict (called default_factory in
its documentation) can only be passed positionally. One special case of
this situation is the self parameter for class methods: it is
undesirable that a caller can bind by keyword to the name self when
calling the method from the class:

    io.FileIO.write(self=f, b=b"data")

Indeed, function definitions from the standard library implemented in C
usually take self as a positional-only parameter:

    >>> help(io.FileIO.write)
    Help on method_descriptor:

    write(self, b, /)
        Write buffer b to file, return number of bytes written.

Improving Language Consistency

The Python language would be more consistent with positional-only
parameters. If the concept is a normal feature of Python rather than a
feature exclusive to extension modules, it would reduce confusion for
users encountering functions with positional-only parameters. Some major
third-party packages are already using the / notation in their function
definitions[10][11].

Bridging the gap found between "builtin" functions which specify
positional-only parameters and pure Python implementations that lack the
positional syntax would improve consistency. The / syntax is already
exposed in the existing documentation such as when builtins and
interfaces are generated by the argument clinic.

Another essential aspect to consider is PEP 399, which mandates that
pure Python versions of modules in the standard library must have the
same interface and semantics that the accelerator modules implemented in
C. For example, if collections.defaultdict were to have a pure Python
implementation it would need to make use of positional-only parameters
to match the interface of its C counterpart.

Rationale

We propose to introduce positional-only parameters as a new syntax to
the Python language.

The new syntax will enable library authors to further control how their
API can be called. It will allow designating which parameters must be
called as positional-only, while preventing them from being called as
keyword arguments.

Previously, (informational) PEP 457 defined the syntax, but with a much
more vague scope. This PEP takes the original proposal a step further by
justifying the syntax and providing an implementation for the / syntax
in function definitions.

Performance

In addition to the aforementioned benefits, the parsing and handling of
positional-only arguments is faster. This performance benefit can be
demonstrated in this thread about converting keyword arguments to
positional: [12]. Due to this speedup, there has been a recent trend
towards moving builtins away from keyword arguments: recently,
backwards-incompatible changes were made to disallow keyword arguments
to bool, float, list, int, tuple.

Maintainability

Providing a way to specify positional-only parameters in Python will
make it easier to maintain pure Python implementations of C modules.
Additionally, library authors defining functions will have the choice
for choosing positional-only parameters if they determine that passing a
keyword argument provides no additional clarity.

This is a well discussed, recurring topic on the Python mailing lists:

-   September 2018: Anders Hovmöller: [Python-ideas] Positional-only
    parameters
-   February 2017: Victor Stinner: [Python-ideas] Positional-only
    parameters, discussion continued in March
-   February 2017:[13]
-   March 2012:[14]
-   May 2007: George Sakkis: [Python-ideas] Positional only arguments
-   May 2006: Benji York: [Python-Dev] Positional-only Arguments

Logical ordering

Positional-only parameters also have the (minor) benefit of enforcing
some logical order when calling interfaces that make use of them. For
example, the range function takes all its parameters positionally and
disallows forms like:

    range(stop=5, start=0, step=2)
    range(stop=5, step=2, start=0)
    range(step=2, start=0, stop=5)
    range(step=2, stop=5, start=0)

at the price of disallowing the use of keyword arguments for the
(unique) intended order:

    range(start=0, stop=5, step=2)

Compatibility for Pure Python and C Modules

Another critical motivation for positional-only parameters is PEP 399:
Pure Python/C Accelerator Module Compatibility Requirements. This PEP
states that:

  This PEP requires that in these instances that the C code must pass
  the test suite used for the pure Python code to act as much as a
  drop-in replacement as reasonably possible

If the C code is implemented using the existing capabilities to
implement positional-only parameters using the argument clinic, and
related machinery, it is not possible for the pure Python counterpart to
match the provided interface and requirements. This creates a disparity
between the interfaces of some functions and classes in the CPython
standard library and other Python implementations. For example:

    $ python3 # CPython 3.7.2
    >>> import binascii; binascii.crc32(data=b'data')
    TypeError: crc32() takes no keyword arguments

    $ pypy3 # PyPy 6.0.0
    >>>> import binascii; binascii.crc32(data=b'data')
<PHONE_NUMBER>

Other Python implementations can reproduce the CPython APIs manually,
but this goes against the spirit of PEP 399 to avoid duplication of
effort by mandating that all modules added to Python's standard library
must have a pure Python implementation with the same interface and
semantics.

Consistency in Subclasses

Another scenario where positional-only parameters provide benefit occurs
when a subclass overrides a method of the base class and changes the
name of parameters that are intended to be positional:

    class Base:
        def meth(self, arg: int) -> str:
            ...

    class Sub(Base):
        def meth(self, other_arg: int) -> str:
            ...

    def func(x: Base):
        x.meth(arg=12)

    func(Sub())  # Runtime error

This situation could be considered a Liskov violation — the subclass
cannot be used in a context when an instance of the base class is
expected. Renaming arguments when overloading methods can happen when
the subclass has reasons to use a different choice for the parameter
name that is more appropriate for the specific domain of the subclass
(e.g., when subclassing Mapping to implement a DNS lookup cache, the
derived class may not want to use the generic argument names ‘key’ and
‘value’ but rather ‘host’ and ‘address’). Having this function
definition with positional-only parameters can avoid this problem
because users will not be able to call the interface using keyword
arguments. In general, designing for subclassing usually involves
anticipating code that hasn't been written yet and over which the author
has no control. Having measures that can facilitate the evolution of
interfaces in a backwards-compatible would be useful for library
authors.

Optimizations

A final argument in favor of positional-only parameters is that they
allow some new optimizations like the ones already present in the
argument clinic due to the fact that parameters are expected to be
passed in strict order. For example, CPython's internal METH_FASTCALL
calling convention has been recently specialized for functions with
positional-only parameters to eliminate the cost for handling empty
keywords. Similar performance improvements can be applied when creating
the evaluation frame of Python functions thanks to positional-only
parameters.

Specification

Syntax and Semantics

From the "ten-thousand foot view", eliding *args and **kwargs for
illustration, the grammar for a function definition would look like:

    def name(positional_or_keyword_parameters, *, keyword_only_parameters):

Building on that example, the new syntax for function definitions would
look like:

    def name(positional_only_parameters, /, positional_or_keyword_parameters,
             *, keyword_only_parameters):

The following would apply:

-   All parameters left of the / are treated as positional-only.
-   If / is not specified in the function definition, that function does
    not accept any positional-only arguments.
-   The logic around optional values for positional-only parameters
    remains the same as for positional-or-keyword parameters.
-   Once a positional-only parameter is specified with a default, the
    following positional-only and positional-or-keyword parameters need
    to have defaults as well.
-   Positional-only parameters which do not have default values are
    required positional-only parameters.

Therefore, the following would be valid function definitions:

    def name(p1, p2, /, p_or_kw, *, kw):
    def name(p1, p2=None, /, p_or_kw=None, *, kw):
    def name(p1, p2=None, /, *, kw):
    def name(p1, p2=None, /):
    def name(p1, p2, /, p_or_kw):
    def name(p1, p2, /):

Just like today, the following would be valid function definitions:

    def name(p_or_kw, *, kw):
    def name(*, kw):

While the following would be invalid:

    def name(p1, p2=None, /, p_or_kw, *, kw):
    def name(p1=None, p2, /, p_or_kw=None, *, kw):
    def name(p1=None, p2, /):

Full Grammar Specification

A simplified view of the proposed grammar specification is:

    typedargslist:
      tfpdef ['=' test] (',' tfpdef ['=' test])* ',' '/' [','  # and so on

    varargslist:
      vfpdef ['=' test] (',' vfpdef ['=' test])* ',' '/' [','  # and so on

Based on the reference implementation in this PEP, the new rule for
typedarglist would be:

    typedargslist: (tfpdef ['=' test] (',' tfpdef ['=' test])* ',' '/' [',' [tfpdef ['=' test] (',' tfpdef ['=' test])* [',' [
            '*' [tfpdef] (',' tfpdef ['=' test])* [',' ['**' tfpdef [',']]]
          | '**' tfpdef [',']]]
      | '*' [tfpdef] (',' tfpdef ['=' test])* [',' ['**' tfpdef [',']]]
      | '**' tfpdef [',']] ] )| (
       tfpdef ['=' test] (',' tfpdef ['=' test])* [',' [
            '*' [tfpdef] (',' tfpdef ['=' test])* [',' ['**' tfpdef [',']]]
          | '**' tfpdef [',']]]
     | '*' [tfpdef] (',' tfpdef ['=' test])* [',' ['**' tfpdef [',']]]
     | '**' tfpdef [','])

and for varargslist would be:

    varargslist: vfpdef ['=' test ](',' vfpdef ['=' test])* ',' '/' [',' [ (vfpdef ['=' test] (',' vfpdef ['=' test])* [',' [
            '*' [vfpdef] (',' vfpdef ['=' test])* [',' ['**' vfpdef [',']]]
          | '**' vfpdef [',']]]
      | '*' [vfpdef] (',' vfpdef ['=' test])* [',' ['**' vfpdef [',']]]
      | '**' vfpdef [',']) ]] | (vfpdef ['=' test] (',' vfpdef ['=' test])* [',' [
            '*' [vfpdef] (',' vfpdef ['=' test])* [',' ['**' vfpdef [',']]]
          | '**' vfpdef [',']]]
      | '*' [vfpdef] (',' vfpdef ['=' test])* [',' ['**' vfpdef [',']]]
      | '**' vfpdef [',']
    )

Semantic Corner Case

The following is an interesting corollary of the specification. Consider
this function definition:

    def foo(name, **kwds):
        return 'name' in kwds

There is no possible call that will make it return True. For example:

    >>> foo(1, **{'name': 2})
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: foo() got multiple values for argument 'name'
    >>>

But using / we can support this:

    def foo(name, /, **kwds):
        return 'name' in kwds

Now the above call will return True.

In other words, the names of positional-only parameters can be used in
**kwds without ambiguity. (As another example, this benefits the
signatures of dict() and dict.update().)

Origin of "/" as a Separator

Using / as a separator was initially proposed by Guido van Rossum in
2012[15] :

  Alternative proposal: how about using '/' ? It's kind of the opposite
  of '*' which means "keyword argument", and '/' is not a new character.

How To Teach This

Introducing a dedicated syntax to mark positional-only parameters is
closely analogous to existing keyword-only arguments. Teaching these
concepts together may simplify how to teach the possible function
definitions a user may encounter or design.

This PEP recommends adding a new subsection to the Python documentation,
in the section "More on Defining Functions", where the rest of the
argument types are discussed. The following paragraphs serve as a draft
for these additions. They will introduce the notation for both
positional-only and keyword-only parameters. It is not intended to be
exhaustive, nor should it be considered the final version to be
incorporated into the documentation.

------------------------------------------------------------------------

By default, arguments may be passed to a Python function either by
position or explicitly by keyword. For readability and performance, it
makes sense to restrict the way arguments can be passed so that a
developer need only look at the function definition to determine if
items are passed by position, by position or keyword, or by keyword.

A function definition may look like:

    def f(pos1, pos2, /, pos_or_kwd, *, kwd1, kwd2):
          -----------    ----------     ----------
            |             |                  |
            |        Positional or keyword   |
            |                                - Keyword only
             -- Positional only

where / and * are optional. If used, these symbols indicate the kind of
parameter by how the arguments may be passed to the function:
positional-only, positional-or-keyword, and keyword-only. Keyword
parameters are also referred to as named parameters.

Positional-or-Keyword Arguments

If / and * are not present in the function definition, arguments may be
passed to a function by position or by keyword.

Positional-Only Parameters

Looking at this in a bit more detail, it is possible to mark certain
parameters as positional-only. If positional-only, the parameters' order
matters, and the parameters cannot be passed by keyword. Positional-only
parameters would be placed before a / (forward-slash). The / is used to
logically separate the positional-only parameters from the rest of the
parameters. If there is no / in the function definition, there are no
positional-only parameters.

Parameters following the / may be positional-or-keyword or keyword-only.

Keyword-Only Arguments

To mark parameters as keyword-only, indicating the parameters must be
passed by keyword argument, place an * in the arguments list just before
the first keyword-only parameter.

Function Examples

Consider the following example function definitions paying close
attention to the markers / and *:

    >>> def standard_arg(arg):
    ...     print(arg)
    ...
    >>> def pos_only_arg(arg, /):
    ...     print(arg)
    ...
    >>> def kwd_only_arg(*, arg):
    ...     print(arg)
    ...
    >>> def combined_example(pos_only, /, standard, *, kwd_only):
    ...     print(pos_only, standard, kwd_only)

The first function definition standard_arg, the most familiar form,
places no restrictions on the calling convention and arguments may be
passed by position or keyword:

    >>> standard_arg(2)
    2

    >>> standard_arg(arg=2)
    2

The second function pos_only_arg is restricted to only use positional
parameters as there is a / in the function definition:

    >>> pos_only_arg(1)
    1

    >>> pos_only_arg(arg=1)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: pos_only_arg() got an unexpected keyword argument 'arg'

The third function kwd_only_args only allows keyword arguments as
indicated by a * in the function definition:

    >>> kwd_only_arg(3)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: kwd_only_arg() takes 0 positional arguments but 1 was given

    >>> kwd_only_arg(arg=3)
    3

And the last uses all three calling conventions in the same function
definition:

    >>> combined_example(1, 2, 3)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: combined_example() takes 2 positional arguments but 3 were given

    >>> combined_example(1, 2, kwd_only=3)
    1 2 3

    >>> combined_example(1, standard=2, kwd_only=3)
    1 2 3

    >>> combined_example(pos_only=1, standard=2, kwd_only=3)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: combined_example() got an unexpected keyword argument 'pos_only'

Recap

The use case will determine which parameters to use in the function
definition:

    def f(pos1, pos2, /, pos_or_kwd, *, kwd1, kwd2):

As guidance:

-   Use positional-only if names do not matter or have no meaning, and
    there are only a few arguments which will always be passed in the
    same order.
-   Use keyword-only when names have meaning and the function definition
    is more understandable by being explicit with names.

Reference Implementation

An initial implementation that passes the CPython test suite is
available for evaluation[16].

The benefits of this implementations are speed of handling
positional-only parameters, consistency with the implementation of
keyword-only parameters (PEP 3102), and a simpler implementation of all
the tools and modules that would be impacted by this change.

Rejected Ideas

Do Nothing

Always an option — the status quo. While this was considered, the
aforementioned benefits are worth the addition to the language.

Decorators

It has been suggested on python-ideas[17] to provide a decorator written
in Python for this feature.

This approach has the benefit of not polluting function definition with
additional syntax. However, we have decided to reject this idea because:

-   It introduces an asymmetry with how parameter behavior is declared.
-   It makes it difficult for static analyzers and type checkers to
    safely identify positional-only parameters. They would need to query
    the AST for the list of decorators and identify the correct one by
    name or with extra heuristics, while keyword-only parameters are
    exposed directly in the AST. In order for tools to correctly
    identify positional-only parameters, they would need to execute the
    module to access any metadata the decorator is setting.
-   Any error with the declaration will be reported only at runtime.
-   It may be more difficult to identify positional-only parameters in
    long function definitions, as it forces the user to count them to
    know which is the last one that is impacted by the decorator.
-   The / syntax has already been introduced for C functions. This
    inconsistency will make it more challenging to implement any tools
    and modules that deal with this syntax — including but not limited
    to, the argument clinic, the inspect module and the ast module.
-   The decorator implementation would likely impose a runtime
    performance cost, particularly when compared to adding support
    directly to the interpreter.

Per-Argument Marker

A per-argument marker is another language-intrinsic option. The approach
adds a token to each of the parameters to indicate they are
positional-only and requires those parameters to be placed together.
Example:

    def (.arg1, .arg2, arg3):

Note the dot (i.e., .) on .arg1 and .arg2. While this approach may be
easier to read, it has been rejected because / as an explicit marker is
congruent with * for keyword-only arguments and is less error-prone.

It should be noted that some libraries already use leading underscore
[18] to conventionally indicate parameters as positional-only.

Using "__" as a Per-Argument Marker

Some libraries and applications (like mypy or jinja) use names prepended
with a double underscore (i.e., __) as a convention to indicate
positional-only parameters. We have rejected the idea of introducing __
as a new syntax because:

-   It is a backwards-incompatible change.
-   It is not symmetric with how the keyword-only parameters are
    currently declared.
-   Querying the AST for positional-only parameters would require
    checking the normal arguments and inspecting their names, whereas
    keyword-only parameters have a property associated with them
    (FunctionDef.args.kwonlyargs).
-   Every parameter would need to be inspected to know when
    positional-only arguments end.
-   The marker is more verbose, forcing marking every positional-only
    parameter.
-   It clashes with other uses of the double underscore prefix like
    invoking name mangling in classes.

Group Positional-Only Parameters With Parentheses

Tuple parameter unpacking is a Python 2 feature which allows the use of
a tuple as a parameter in a function definition. It allows a sequence
argument to be unpacked automatically. An example is:

    def fxn(a, (b, c), d):
        pass

Tuple argument unpacking was removed in Python 3 (PEP 3113). There has
been a proposition to reuse this syntax to implement positional-only
parameters. We have rejected this syntax for indicating positional only
parameters for several reasons:

-   The syntax is asymmetric with respect to how keyword-only parameters
    are declared.
-   Python 2 uses this syntax which could raise confusion regarding the
    behavior of this syntax. This would be surprising to users porting
    Python 2 codebases that were using this feature.
-   This syntax is very similar to tuple literals. This can raise
    additional confusion because it can be confused with a tuple
    declaration.

After Separator Proposal

Marking positional-parameters after the / was another idea considered.
However, we were unable to find an approach which would modify the
arguments after the marker. Otherwise, would force the parameters before
the marker to be positional-only as well. For example:

    def (x, y, /, z):

If we define that / marks z as positional-only, it would not be possible
to specify x and y as keyword arguments. Finding a way to work around
this limitation would add confusion given that at the moment keyword
arguments cannot be followed by positional arguments. Therefore, / would
make both the preceding and following parameters positional-only.

Thanks

Credit for some of the content of this PEP is contained in Larry
Hastings’s PEP 457.

Credit for the use of / as the separator between positional-only and
positional-or-keyword parameters go to Guido van Rossum, in a proposal
from 2012.[19]

Credit for discussion about the simplification of the grammar goes to
Braulio Valdivieso.

Copyright

This document has been placed in the public domain.

[1] http://docs.python.org/3/library/functions.html#func-range

[2] http://docs.python.org/3/library/stdtypes.html#dict

[3] https://docs.scipy.org/doc/numpy/reference/ufuncs.html#available-ufuncs

[4] https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.gammaln.html

[5] https://bugs.python.org/issue21314

[6] http://docs.python.org/3/library/curses.html#curses.window.border

[7] http://docs.python.org/3/library/os.html#os.sendfile

[8] http://docs.python.org/3/library/functions.html#func-range

[9] http://docs.python.org/3/library/curses.html#curses.window.addch

[10] https://docs.scipy.org/doc/numpy/reference/ufuncs.html#available-ufuncs

[11] https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.gammaln.html

[12] https://mail.python.org/pipermail/python-ideas/2016-January/037874.html

[13] https://mail.python.org/pipermail/python-ideas/2017-February/044888.html

[14] Guido van Rossum, posting to python-ideas, March 2012:
https://mail.python.org/pipermail/python-ideas/2012-March/014364.html
and
https://mail.python.org/pipermail/python-ideas/2012-March/014378.html
and
https://mail.python.org/pipermail/python-ideas/2012-March/014417.html

[15] Guido van Rossum, posting to python-ideas, March 2012:
https://mail.python.org/pipermail/python-ideas/2012-March/014364.html
and
https://mail.python.org/pipermail/python-ideas/2012-March/014378.html
and
https://mail.python.org/pipermail/python-ideas/2012-March/014417.html

[16] https://github.com/pablogsal/cpython_positional_only

[17] https://mail.python.org/pipermail/python-ideas/2017-February/044888.html

[18] https://mail.python.org/pipermail/python-ideas/2018-September/053319.html

[19] Guido van Rossum, posting to python-ideas, March 2012:
https://mail.python.org/pipermail/python-ideas/2012-March/014364.html
and
https://mail.python.org/pipermail/python-ideas/2012-March/014378.html
and
https://mail.python.org/pipermail/python-ideas/2012-March/014417.html

license: Public Domain


text:
PEP: 359 Title: The "make" Statement Version: $Revision$ Last-Modified:
$Date$ Author: Steven Bethard<EMAIL_ADDRESS>Status:
Withdrawn Type: Standards Track Content-Type: text/x-rst Created:
05-Apr-2006 Python-Version: 2.6 Post-History: 05-Apr-2006, 06-Apr-2006,
13-Apr-2006

Abstract

This PEP proposes a generalization of the class-declaration syntax, the
make statement. The proposed syntax and semantics parallel the syntax
for class definition, and so:

    make <callable> <name> <tuple>:
        <block>

is translated into the assignment:

    <name> = <callable>("<name>", <tuple>, <namespace>)

where <namespace> is the dict created by executing <block>. This is
mostly syntactic sugar for:

    class <name> <tuple>:
        __metaclass__ = <callable>
        <block>

and is intended to help more clearly express the intent of the statement
when something other than a class is being created. Of course, other
syntax for such a statement is possible, but it is hoped that by keeping
a strong parallel to the class statement, an understanding of how
classes and metaclasses work will translate into an understanding of how
the make-statement works as well.

The PEP is based on a suggestion[1] from Michele Simionato on the
python-dev list.

Withdrawal Notice

This PEP was withdrawn at Guido's request[2]. Guido didn't like it, and
in particular didn't like how the property use-case puts the instance
methods of a property at a different level than other instance methods
and requires fixed names for the property functions.

Motivation

Class statements provide two nice facilities to Python:

(1) They execute a block of statements and provide the resulting
    bindings as a dict to the metaclass.
(2) They encourage DRY (don't repeat yourself) by allowing the class
    being created to know the name it is being assigned.

Thus in a simple class statement like:

    class C(object):
        x = 1
        def foo(self):
            return 'bar'

the metaclass (type) gets called with something like:

    C = type('C', (object,), {'x':1, 'foo':<function foo at ...>})

The class statement is just syntactic sugar for the above assignment
statement, but clearly a very useful sort of syntactic sugar. It avoids
not only the repetition of C, but also simplifies the creation of the
dict by allowing it to be expressed as a series of statements.

Historically, type instances (a.k.a. class objects) have been the only
objects blessed with this sort of syntactic support. The make statement
aims to extend this support to other sorts of objects where such syntax
would also be useful.

Example: simple namespaces

Let's say I have some attributes in a module that I access like:

    mod.thematic_roletype
    mod.opinion_roletype

    mod.text_format
    mod.html_format

and since "Namespaces are one honking great idea", I'd like to be able
to access these attributes instead as:

    mod.roletypes.thematic
    mod.roletypes.opinion

    mod.format.text
    mod.format.html

I currently have two main options:

(1) Turn the module into a package, turn roletypes and format into
    submodules, and move the attributes to the submodules.
(2) Create roletypes and format classes, and move the attributes to the
    classes.

The former is a fair chunk of refactoring work, and produces two tiny
modules without much content. The latter keeps the attributes local to
the module, but creates classes when there is no intention of ever
creating instances of those classes.

In situations like this, it would be nice to simply be able to declare a
"namespace" to hold the few attributes. With the new make statement, I
could introduce my new namespaces with something like:

    make namespace roletypes:
        thematic = ...
        opinion = ...

    make namespace format:
        text = ...
        html = ...

and keep my attributes local to the module without making classes that
are never intended to be instantiated. One definition of namespace that
would make this work is:

    class namespace(object):
        def __init__(self, name, args, kwargs):
            self.__dict__.update(kwargs)

Given this definition, at the end of the make-statements above,
roletypes and format would be namespace instances.

Example: GUI objects

In GUI toolkits, objects like frames and panels are often associated
with attributes and functions. With the make-statement, code that looks
something like:

    root = Tkinter.Tk()
    frame = Tkinter.Frame(root)
    frame.pack()
    def say_hi():
        print "hi there, everyone!"
    hi_there = Tkinter.Button(frame, text="Hello", command=say_hi)
    hi_there.pack(side=Tkinter.LEFT)
    root.mainloop()

could be rewritten to group the Button's function with its declaration:

    root = Tkinter.Tk()
    frame = Tkinter.Frame(root)
    frame.pack()
    make Tkinter.Button hi_there(frame):
        text = "Hello"
        def command():
            print "hi there, everyone!"
    hi_there.pack(side=Tkinter.LEFT)
    root.mainloop()

Example: custom descriptors

Since descriptors are used to customize access to an attribute, it's
often useful to know the name of that attribute. Current Python doesn't
give an easy way to find this name and so a lot of custom descriptors,
like Ian Bicking's setonce descriptor[3], have to hack around this
somehow. With the make-statement, you could create a setonce attribute
like:

    class A(object):
        ...
        make setonce x:
            "A's x attribute"
        ...

where the setonce descriptor would be defined like:

    class setonce(object):

        def __init__(self, name, args, kwargs):
            self._name = '_setonce_attr_%s' % name
            self.__doc__ = kwargs.pop('__doc__', None)

        def __get__(self, obj, type=None):
            if obj is None:
                return self
            return getattr(obj, self._name)

        def __set__(self, obj, value):
            try:
                getattr(obj, self._name)
            except AttributeError:
                setattr(obj, self._name, value)
            else:
                raise AttributeError("Attribute already set")

        def set(self, obj, value):
            setattr(obj, self._name, value)

        def __delete__(self, obj):
            delattr(obj, self._name)

Note that unlike the original implementation, the private attribute name
is stable since it uses the name of the descriptor, and therefore
instances of class A are pickleable.

Example: property namespaces

Python's property type takes three function arguments and a docstring
argument which, though relevant only to the property, must be declared
before it and then passed as arguments to the property call, e.g.:

    class C(object):
        ...
        def get_x(self):
            ...
        def set_x(self):
            ...
        x = property(get_x, set_x, "the x of the frobulation")

This issue has been brought up before, and Guido[4] and others[5] have
briefly mused over alternate property syntaxes to make declaring
properties easier. With the make-statement, the following syntax could
be supported:

    class C(object):
        ...
        make block_property x:
            '''The x of the frobulation'''
            def fget(self):
                ...
            def fset(self):
                ...

with the following definition of block_property:

    def block_property(name, args, block_dict):
        fget = block_dict.pop('fget', None)
        fset = block_dict.pop('fset', None)
        fdel = block_dict.pop('fdel', None)
        doc = block_dict.pop('__doc__', None)
        assert not block_dict
        return property(fget, fset, fdel, doc)

Example: interfaces

Guido[6] and others have occasionally suggested introducing interfaces
into python. Most suggestions have offered syntax along the lines of:

    interface IFoo:
        """Foo blah blah"""

        def fumble(name, count):
            """docstring"""

but since there is currently no way in Python to declare an interface in
this manner, most implementations of Python interfaces use class objects
instead, e.g. Zope's:

    class IFoo(Interface):
        """Foo blah blah"""

        def fumble(name, count):
            """docstring"""

With the new make-statement, these interfaces could instead be declared
as:

    make Interface IFoo:
        """Foo blah blah"""

        def fumble(name, count):
            """docstring"""

which makes the intent (that this is an interface, not a class) much
clearer.

Specification

Python will translate a make-statement:

    make <callable> <name> <tuple>:
        <block>

into the assignment:

    <name> = <callable>("<name>", <tuple>, <namespace>)

where <namespace> is the dict created by executing <block>. The <tuple>
expression is optional; if not present, an empty tuple will be assumed.

A patch is available implementing these semantics[7].

The make-statement introduces a new keyword, make. Thus in Python 2.6,
the make-statement will have to be enabled using
from __future__ import make_statement.

Open Issues

Keyword

Does the make keyword break too much code? Originally, the make
statement used the keyword create (a suggestion due to Alyssa Coghlan).
However, investigations into the standard library[8] and Zope+Plone
code[9] revealed that create would break a lot more code, so make was
adopted as the keyword instead. However, there are still a few instances
where make would break code. Is there a better keyword for the
statement?

Some possible keywords and their counts in the standard library (plus
some installed packages):

-   make - 2 (both in tests)
-   create - 19 (including existing function in imaplib)
-   build - 83 (including existing class in distutils.command.build)
-   construct - 0
-   produce - 0

The make-statement as an alternate constructor

Currently, there are not many functions which have the signature
(name, args, kwargs). That means that something like:

    make dict params:
        x = 1
        y = 2

is currently impossible because the dict constructor has a different
signature. Does this sort of thing need to be supported? One suggestion,
by Carl Banks, would be to add a __make__ magic method that if found
would be called instead of __call__. For types, the __make__ method
would be identical to __call__ and thus unnecessary, but dicts could
support the make-statement by defining a __make__ method on the dict
type that looks something like:

    def __make__(cls, name, args, kwargs):
        return cls(**kwargs)

Of course, rather than adding another magic method, the dict type could
just grow a classmethod something like dict.fromblock that could be used
like:

    make dict.fromblock params:
        x = 1
        y = 2

So the question is, will many types want to use the make-statement as an
alternate constructor? And if so, does that alternate constructor need
to have the same name as the original constructor?

Customizing the dict in which the block is executed

Should users of the make-statement be able to determine in which dict
object the code is executed? This would allow the make-statement to be
used in situations where a normal dict object would not suffice, e.g. if
order and repeated names must be allowed. Allowing this sort of
customization could allow XML to be written without repeating element
names, and with nesting of make-statements corresponding to nesting of
XML elements:

    make Element html:
        make Element body:
            text('before first h1')
            make Element h1:
                attrib(style='first')
                text('first h1')
                tail('after first h1')
            make Element h1:
                attrib(style='second')
                text('second h1')
                tail('after second h1')

If the make-statement tried to get the dict in which to execute its
block by calling the callable's __make_dict__ method, the following code
would allow the make-statement to be used as above:

    class Element(object):

        class __make_dict__(dict):

            def __init__(self, *args, **kwargs):
                self._super = super(Element.__make_dict__, self)
                self._super.__init__(*args, **kwargs)
                self.elements = []
                self.text = None
                self.tail = None
                self.attrib = {}

            def __getitem__(self, name):
                try:
                    return self._super.__getitem__(name)
                except KeyError:
                    if name in ['attrib', 'text', 'tail']:
                        return getattr(self, 'set_%s' % name)
                    else:
                        return globals()[name]

            def __setitem__(self, name, value):
                self._super.__setitem__(name, value)
                self.elements.append(value)

            def set_attrib(self, **kwargs):
                self.attrib = kwargs

            def set_text(self, text):
                self.text = text

            def set_tail(self, text):
                self.tail = text

        def __new__(cls, name, args, edict):
            get_element = etree.ElementTree.Element
            result = get_element(name, attrib=edict.attrib)
            result.text = edict.text
            result.tail = edict.tail
            for element in edict.elements:
                result.append(element)
            return result

Note, however, that the code to support this is somewhat fragile --it
has to magically populate the namespace with attrib, text and tail, and
it assumes that every name binding inside the make statement body is
creating an Element. As it stands, this code would break with the
introduction of a simple for-loop to any one of the make-statement
bodies, because the for-loop would bind a name to a non-Element object.
This could be worked around by adding some sort of isinstance check or
attribute examination, but this still results in a somewhat fragile
solution.

It has also been pointed out that the with-statement can provide
equivalent nesting with a much more explicit syntax:

    with Element('html') as html:
        with Element('body') as body:
            body.text = 'before first h1'
            with Element('h1', style='first') as h1:
                h1.text = 'first h1'
                h1.tail = 'after first h1'
            with Element('h1', style='second') as h1:
                h1.text = 'second h1'
                h1.tail = 'after second h1'

And if the repetition of the element names here is too much of a DRY
violation, it is also possible to eliminate all as-clauses except for
the first by adding a few methods to Element.[10]

So are there real use-cases for executing the block in a dict of a
different type? And if so, should the make-statement be extended to
support them?

Optional Extensions

Remove the make keyword

It might be possible to remove the make keyword so that such statements
would begin with the callable being called, e.g.:

    namespace ns:
        badger = 42
        def spam():
            ...

    interface C(...):
        ...

However, almost all other Python statements begin with a keyword, and
removing the keyword would make it harder to look up this construct in
the documentation. Additionally, this would add some complexity in the
grammar and so far I (Steven Bethard) have not been able to implement
the feature without the keyword.

Removing __metaclass__ in Python 3000

As a side-effect of its generality, the make-statement mostly eliminates
the need for the __metaclass__ attribute in class objects. Thus in
Python 3000, instead of:

    class <name> <bases-tuple>:
        __metaclass__ = <metaclass>
        <block>

metaclasses could be supported by using the metaclass as the callable in
a make-statement:

    make <metaclass> <name> <bases-tuple>:
        <block>

Removing the __metaclass__ hook would simplify the BUILD_CLASS opcode a
bit.

Removing class statements in Python 3000

In the most extreme application of make-statements, the class statement
itself could be deprecated in favor of make type statements.

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] Michele Simionato's original suggestion
(https://mail.python.org/pipermail/python-dev/2005-October/057435.html)

[2] Guido requests withdrawal
(https://mail.python.org/pipermail/python-3000/2006-April/000936.html)

[3] Ian Bicking's setonce descriptor
(http://blog.ianbicking.org/easy-readonly-attributes.html)

[4] Guido ponders property syntax
(https://mail.python.org/pipermail/python-dev/2005-October/057404.html)

[5] Namespace-based property recipe
(http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/442418)

[6] Python interfaces
(http://www.artima.com/weblogs/viewpost.jsp?thread=86641)

[7] Make Statement patch
(http://ucsu.colorado.edu/~bethard/py/make_statement.patch)

[8] Instances of create in the stdlib
(https://mail.python.org/pipermail/python-list/2006-April/335159.html)

[9] Instances of create in Zope+Plone
(https://mail.python.org/pipermail/python-list/2006-April/335284.html)

[10] Eliminate as-clauses in with-statement XML
(https://mail.python.org/pipermail/python-list/2006-April/336774.html)


text:
PEP: 3138 Title: String representation in Python 3000 Version:
$Revision$ Last-Modified: $Date$ Author: Atsuo Ishimoto <ishimoto at
gembook.org> Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 05-May-2008 Python-Version: 3.0 Post-History:
05-May-2008, 05-Jun-2008

Abstract

This PEP proposes a new string representation form for Python 3000. In
Python prior to Python 3000, the repr() built-in function converted
arbitrary objects to printable ASCII strings for debugging and logging.
For Python 3000, a wider range of characters, based on the Unicode
standard, should be considered 'printable'.

Motivation

The current repr() converts 8-bit strings to ASCII using following
algorithm.

-   Convert CR, LF, TAB and '\' to '\r', '\n', '\t', '\\'.
-   Convert other non-printable characters(0x00-0x1f, 0x7f) and
    non-ASCII characters (>= 0x80) to '\xXX'.
-   Backslash-escape quote characters (apostrophe, ') and add the quote
    character at the beginning and the end.

For Unicode strings, the following additional conversions are done.

-   Convert leading surrogate pair characters without trailing character
    (0xd800-0xdbff, but not followed by 0xdc00-0xdfff) to '\uXXXX'.
-   Convert 16-bit characters (>= 0x100) to '\uXXXX'.
-   Convert 21-bit characters (>= 0x10000) and surrogate pair characters
    to '\U00xxxxxx'.

This algorithm converts any string to printable ASCII, and repr() is
used as a handy and safe way to print strings for debugging or for
logging. Although all non-ASCII characters are escaped, this does not
matter when most of the string's characters are ASCII. But for other
languages, such as Japanese where most characters in a string are not
ASCII, this is very inconvenient.

We can use print(aJapaneseString) to get a readable string, but we don't
have a similar workaround for printing strings from collections such as
lists or tuples. print(listOfJapaneseStrings) uses repr() to build the
string to be printed, so the resulting strings are always hex-escaped.
Or when open(japaneseFilename) raises an exception, the error message is
something like
IOError: [Errno 2] No such file or directory: '\u65e5\u672c\u8a9e',
which isn't helpful.

Python 3000 has a lot of nice features for non-Latin users such as
non-ASCII identifiers, so it would be helpful if Python could also
progress in a similar way for printable output.

Some users might be concerned that such output will mess up their
console if they print binary data like images. But this is unlikely to
happen in practice because bytes and strings are different types in
Python 3000, so printing an image to the console won't mess it up.

This issue was once discussed by Hye-Shik Chang[1], but was rejected.

Specification

-   Add a new function to the Python C API
    int Py_UNICODE_ISPRINTABLE (Py_UNICODE ch). This function returns 0
    if repr() should escape the Unicode character ch; otherwise it
    returns 1. Characters that should be escaped are defined in the
    Unicode character database as:
    -   Cc (Other, Control)
    -   Cf (Other, Format)
    -   Cs (Other, Surrogate)
    -   Co (Other, Private Use)
    -   Cn (Other, Not Assigned)
    -   Zl (Separator, Line), refers to LINE SEPARATOR ('\u2028').
    -   Zp (Separator, Paragraph), refers to PARAGRAPH SEPARATOR
        ('\u2029').
    -   Zs (Separator, Space) other than ASCII space ('\x20').
        Characters in this category should be escaped to avoid
        ambiguity.
-   The algorithm to build repr() strings should be changed to:
    -   Convert CR, LF, TAB and '\' to '\r', '\n', '\t', '\\'.
    -   Convert non-printable ASCII characters (0x00-0x1f, 0x7f) to
        '\xXX'.
    -   Convert leading surrogate pair characters without trailing
        character (0xd800-0xdbff, but not followed by 0xdc00-0xdfff) to
        '\uXXXX'.
    -   Convert non-printable characters (Py_UNICODE_ISPRINTABLE()
        returns
        0)  to '\xXX', '\uXXXX' or '\U00xxxxxx'.
    -   Backslash-escape quote characters (apostrophe, 0x27) and add a
        quote character at the beginning and the end.
-   Set the Unicode error-handler for sys.stderr to 'backslashreplace'
    by default.
-   Add a new function to the Python C API
    PyObject *PyObject_ASCII (PyObject *o). This function converts any
    python object to a string using PyObject_Repr() and then hex-escapes
    all non-ASCII characters. PyObject_ASCII() generates the same string
    as PyObject_Repr() in Python 2.
-   Add a new built-in function, ascii(). This function converts any
    python object to a string using repr() and then hex-escapes all
    non-ASCII characters. ascii() generates the same string as repr() in
    Python 2.
-   Add a '%a' string format operator. '%a' converts any python object
    to a string using repr() and then hex-escapes all non-ASCII
    characters. The '%a' format operator generates the same string as
    '%r' in Python 2. Also, add '!a' conversion flags to the
    string.format() method and add '%A' operator to the
    PyUnicode_FromFormat(). They convert any object to an ASCII string
    as '%a' string format operator.
-   Add an isprintable() method to the string type. str.isprintable()
    returns False if repr() would escape any character in the string;
    otherwise returns True. The isprintable() method calls the
    Py_UNICODE_ISPRINTABLE() function internally.

Rationale

The repr() in Python 3000 should be Unicode, not ASCII based, just like
Python 3000 strings. Also, conversion should not be affected by the
locale setting, because the locale is not necessarily the same as the
output device's locale. For example, it is common for a daemon process
to be invoked in an ASCII setting, but writes UTF-8 to its log files.
Also, web applications might want to report the error information in
more readable form based on the HTML page's encoding.

Characters not supported by the user's console could be hex-escaped on
printing, by the Unicode encoder's error-handler. If the error-handler
of the output file is 'backslashreplace', such characters are
hex-escaped without raising UnicodeEncodeError. For example, if the
default encoding is ASCII, print('Hello ¢') will print 'Hello \xa2'. If
the encoding is ISO-8859-1, 'Hello ¢' will be printed.

The default error-handler for sys.stdout is 'strict'. Other applications
reading the output might not understand hex-escaped characters, so
unsupported characters should be trapped when writing. If unsupported
characters must be escaped, the error-handler should be changed
explicitly. Unlike sys.stdout, sys.stderr doesn't raise
UnicodeEncodingError by default, because the default error-handler is
'backslashreplace'. So printing error messages containing non-ASCII
characters to sys.stderr will not raise an exception. Also, information
about uncaught exceptions (exception object, traceback) is printed by
the interpreter without raising exceptions.

Alternate Solutions

To help debugging in non-Latin languages without changing repr(), other
suggestions were made.

-   Supply a tool to print lists or dicts.

    Strings to be printed for debugging are not only contained by lists
    or dicts, but also in many other types of object. File objects
    contain a file name in Unicode, exception objects contain a message
    in Unicode, etc. These strings should be printed in readable form
    when repr()ed. It is unlikely to be possible to implement a tool to
    print all possible object types.

-   Use sys.displayhook and sys.excepthook.

    For interactive sessions, we can write hooks to restore hex escaped
    characters to the original characters. But these hooks are called
    only when printing the result of evaluating an expression entered in
    an interactive Python session, and don't work for the print()
    function, for non-interactive sessions or for
    logging.debug("%r", ...), etc.

-   Subclass sys.stdout and sys.stderr.

    It is difficult to implement a subclass to restore hex-escaped
    characters since there isn't enough information left by the time
    it's a string to undo the escaping correctly in all cases. For
    example, print("\\"+"u0041") should be printed as '\u0041', not 'A'.
    But there is no chance to tell file objects apart.

-   Make the encoding used by unicode_repr() adjustable, and make the
    existing repr() the default.

    With adjustable repr(), the result of using repr() is unpredictable
    and would make it impossible to write correct code involving repr().
    And if current repr() is the default, then the old convention
    remains intact and users may expect ASCII strings as the result of
    repr(). Third party applications or libraries could be confused when
    a custom repr() function is used.

Backwards Compatibility

Changing repr() may break some existing code, especially testing code.
Five of Python's regression tests fail with this modification. If you
need repr() strings without non-ASCII character as Python 2, you can use
the following function. :

    def repr_ascii(obj):
        return str(repr(obj).encode("ASCII", "backslashreplace"), "ASCII")

For logging or for debugging, the following code can raise
UnicodeEncodeError. :

    log = open("logfile", "w")
    log.write(repr(data))     # UnicodeEncodeError will be raised
                              # if data contains unsupported characters.

To avoid exceptions being raised, you can explicitly specify the
error-handler. :

    log = open("logfile", "w", errors="backslashreplace")
    log.write(repr(data))  # Unsupported characters will be escaped.

For a console that uses a Unicode-based encoding, for example,
en_US.utf8 or de_DE.utf8, the backslashreplace trick doesn't work and
all printable characters are not escaped. This will cause a problem of
similarly drawing characters in Western, Greek and Cyrillic languages.
These languages use similar (but different) alphabets (descended from a
common ancestor) and contain letters that look similar but have
different character codes. For example, it is hard to distinguish Latin
'a', 'e' and 'o' from Cyrillic 'а', 'е' and 'о'. (The visual
representation, of course, very much depends on the fonts used but
usually these letters are almost indistinguishable.) To avoid the
problem, the user can adjust the terminal encoding to get a result
suitable for their environment.

Rejected Proposals

-   Add encoding and errors arguments to the builtin print() function,
    with defaults of sys.getfilesystemencoding() and 'backslashreplace'.

    Complicated to implement, and in general, this is not seen as a good
    idea.[2]

-   Use character names to escape characters, instead of hex character
    codes. For example, repr('\u03b1') can be converted to
    "\N{GREEK SMALL LETTER ALPHA}".

    Using character names can be very verbose compared to hex-escape.
    e.g., repr("\ufbf9") is converted to
    "\N{ARABIC LIGATURE UIGHUR KIRGHIZ YEH WITH HAMZA ABOVE WITH ALEF MAKSURA ISOLATED FORM}".

-   Default error-handler of sys.stdout should be 'backslashreplace'.

    Stuff written to stdout might be consumed by another program that
    might misinterpret the \ escapes. For interactive sessions, it is
    possible to make the 'backslashreplace' error-handler the default,
    but this may add confusion of the kind "it works in interactive mode
    but not when redirecting to a file".

Implementation

The author wrote a patch in http://bugs.python.org/issue2630; this was
committed to the Python 3.0 branch in revision 64138 on 06-11-2008.

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] Multibyte string on string::string_print
(http://bugs.python.org/issue479898)

[2] [Python-3000] Displaying strings containing unicode escapes
(https://mail.python.org/pipermail/python-3000/2008-April/013366.html)


text:
PEP: 316 Title: Programming by Contract for Python Version: $Revision$
Last-Modified: $Date$ Author: Terence Way<EMAIL_ADDRESS>Status:
Deferred Type: Standards Track Content-Type: text/x-rst Created:
02-May-2003 Post-History:

Abstract

This submission describes programming by contract for Python. Eiffel's
Design By Contract(tm) is perhaps the most popular use of programming
contracts[1].

Programming contracts extends the language to include invariant
expressions for classes and modules, and pre- and post-condition
expressions for functions and methods.

These expressions (contracts) are similar to assertions: they must be
true or the program is stopped, and run-time checking of the contracts
is typically only enabled while debugging. Contracts are higher-level
than straight assertions and are typically included in documentation.

Motivation

Python already has assertions, why add extra stuff to the language to
support something like contracts? The two best reasons are 1) better,
more accurate documentation, and 2) easier testing.

Complex modules and classes never seem to be documented quite right. The
documentation provided may be enough to convince a programmer to use a
particular module or class over another, but the programmer almost
always has to read the source code when the real debugging starts.

Contracts extend the excellent example provided by the doctest
module[2]. Documentation is readable by programmers, yet has executable
tests embedded in it.

Testing code with contracts is easier too. Comprehensive contracts are
equivalent to unit tests[3]. Tests exercise the full range of
pre-conditions, and fail if the post-conditions are triggered.
Theoretically, a correctly specified function can be tested completely
randomly.

So why add this to the language? Why not have several different
implementations, or let programmers implement their own assertions? The
answer is the behavior of contracts under inheritance.

Suppose Alice and Bob use different assertions packages. If Alice
produces a class library protected by assertions, Bob cannot derive
classes from Alice's library and expect proper checking of
post-conditions and invariants. If they both use the same assertions
package, then Bob can override Alice's methods yet still test against
Alice's contract assertions. The natural place to find this assertions
system is in the language's run-time library.

Specification

The docstring of any module or class can include invariant contracts
marked off with a line that starts with the keyword inv followed by a
colon (:). Whitespace at the start of the line and around the colon is
ignored. The colon is either immediately followed by a single expression
on the same line, or by a series of expressions on following lines
indented past the inv keyword. The normal Python rules about implicit
and explicit line continuations are followed here. Any number of
invariant contracts can be in a docstring.

Some examples:

    # state enumeration
    START, CONNECTING, CONNECTED, CLOSING, CLOSED = range(5)

    class conn:

        """A network connection

        inv: self.state in [START, CLOSED,       # closed states
                            CONNECTING, CLOSING, # transition states
                            CONNECTED]

        inv: 0 <= self.seqno < 256
        """

    class circbuf:

        """A circular buffer.

        inv:
            # there can be from 0 to max items on the buffer
            0 <= self.len <= len(self.buf)

            # g is a valid index into buf
            0 <= self.g < len(self.buf)

            # p is also a valid index into buf
            0 <= self.p < len(self.buf)

            # there are len items between get and put
            (self.p - self.g) % len(self.buf) == \
                  self.len % len(self.buf)
        """

Module invariants must be true after the module is loaded, and at the
entry and exit of every public function within the module.

Class invariants must be true after the __init__ function returns, at
the entry of the __del__ function, and at the entry and exit of every
other public method of the class. Class invariants must use the self
variable to access instance variables.

A method or function is public if its name doesn't start with an
underscore (_), unless it starts and ends with '__' (two underscores).

The docstring of any function or method can have pre-conditions
documented with the keyword pre following the same rules above.
Post-conditions are documented with the keyword post optionally followed
by a list of variables. The variables are in the same scope as the body
of the function or method. This list declares the variables that the
function/method is allowed to modify.

An example:

    class circbuf:

        def __init__(self, leng):
            """Construct an empty circular buffer.

            pre: leng > 0
            post[self]:
                self.is_empty()
                len(self.buf) == leng
            """

A double-colon (::) can be used instead of a single colon (:) to support
docstrings written using reStructuredText[4]. For example, the following
two docstrings describe the same contract:

    """pre: leng > 0"""
    """pre:: leng > 0"""

Expressions in pre- and post-conditions are defined in the module
namespace -- they have access to nearly all the variables that the
function can access, except closure variables.

The contract expressions in post-conditions have access to two
additional variables: __old__ which is filled with shallow copies of
values declared in the variable list immediately following the post
keyword, and __return__ which is bound to the return value of the
function or method.

An example:

    class circbuf:

        def get(self):
            """Pull an entry from a non-empty circular buffer.

            pre: not self.is_empty()
            post[self.g, self.len]:
                __return__ == self.buf[__old__.self.g]
                self.len == __old__.self.len - 1
            """

All contract expressions have access to some additional convenience
functions. To make evaluating the truth of sequences easier, two
functions forall and exists are defined as:

    def forall(a, fn = bool):
        """Return True only if all elements in a are true.

        >>> forall([])
        1
        >>> even = lambda x: x % 2 == 0
        >>> forall([2, 4, 6, 8], even)
        1
        >>> forall('this is a test'.split(), lambda x: len(x) == 4)
        0
        """

    def exists(a, fn = bool):
        """Returns True if there is at least one true value in a.

        >>> exists([])
        0
        >>> exists('this is a test'.split(), lambda x: len(x) == 4)
        1
        """

An example:

    def sort(a):
        """Sort a list.

        pre: isinstance(a, type(list))
        post[a]:
            # array size is unchanged
            len(a) == len(__old__.a)

            # array is ordered
            forall([a[i] >= a[i-1] for i in range(1, len(a))])

            # all the old elements are still in the array
            forall(__old__.a, lambda e: __old__.a.count(e) == a.count(e))
        """

To make evaluating conditions easier, the function implies is defined.
With two arguments, this is similar to the logical implies (=>)
operator. With three arguments, this is similar to C's conditional
expression (x?a:b). This is defined as:

    implies(False, a) => True
    implies(True, a) => a
    implies(False, a, b) => b
    implies(True, a, b) => a

On entry to a function, the function's pre-conditions are checked. An
assertion error is raised if any pre-condition is false. If the function
is public, then the class or module's invariants are also checked.
Copies of variables declared in the post are saved, the function is
called, and if the function exits without raising an exception, the
post-conditions are checked.

Exceptions

Class/module invariants are checked even if a function or method exits
by signalling an exception (post-conditions are not).

All failed contracts raise exceptions which are subclasses of the
ContractViolationError exception, which is in turn a subclass of the
AssertionError exception. Failed pre-conditions raise a
PreconditionViolationError exception. Failed post-conditions raise a
PostconditionViolationError exception, and failed invariants raise a
InvariantViolationError exception.

The class hierarchy:

    AssertionError
        ContractViolationError
            PreconditionViolationError
            PostconditionViolationError
            InvariantViolationError
            InvalidPreconditionError

The InvalidPreconditionError is raised when pre-conditions are illegally
strengthened, see the next section on Inheritance.

Example:

    try:
        some_func()
    except contract.PreconditionViolationError:
        # failed pre-condition, ok
        pass

Inheritance

A class's invariants include all the invariants for all super-classes
(class invariants are ANDed with super-class invariants). These
invariants are checked in method-resolution order.

A method's post-conditions also include all overridden post-conditions
(method post-conditions are ANDed with all overridden method
post-conditions).

An overridden method's pre-conditions can be ignored if the overriding
method's pre-conditions are met. However, if the overriding method's
pre-conditions fail, all of the overridden method's pre-conditions must
also fail. If not, a separate exception is raised, the
InvalidPreconditionError. This supports weakening pre-conditions.

A somewhat contrived example:

    class SimpleMailClient:

        def send(self, msg, dest):
            """Sends a message to a destination:

            pre: self.is_open() # we must have an open connection
            """

        def recv(self):
            """Gets the next unread mail message.

            Returns None if no message is available.

            pre: self.is_open() # we must have an open connection
            post: __return__ is None or isinstance(__return__, Message)
            """

     class ComplexMailClient(SimpleMailClient):
        def send(self, msg, dest):
            """Sends a message to a destination.

            The message is sent immediately if currently connected.
            Otherwise, the message is queued locally until a
            connection is made.

            pre: True # weakens the pre-condition from SimpleMailClient
            """

        def recv(self):
            """Gets the next unread mail message.

            Waits until a message is available.

            pre: True # can always be called
            post: isinstance(__return__, Message)
            """

Because pre-conditions can only be weakened, a ComplexMailClient can
replace a SimpleMailClient with no fear of breaking existing code.

Rationale

Except for the following differences, programming-by-contract for Python
mirrors the Eiffel DBC specification[5].

Embedding contracts in docstrings is patterned after the doctest module.
It removes the need for extra syntax, ensures that programs with
contracts are backwards-compatible, and no further work is necessary to
have the contracts included in the docs.

The keywords pre, post, and inv were chosen instead of the Eiffel-style
REQUIRE, ENSURE, and INVARIANT because they're shorter, more in line
with mathematical notation, and for a more subtle reason: the word
'require' implies caller responsibilities, while 'ensure' implies
provider guarantees. Yet pre-conditions can fail through no fault of the
caller when using multiple inheritance, and post-conditions can fail
through no fault of the function when using multiple threads.

Loop invariants as used in Eiffel are unsupported. They're a pain to
implement, and not part of the documentation anyway.

The variable names __old__ and __return__ were picked to avoid conflicts
with the return keyword and to stay consistent with Python naming
conventions: they're public and provided by the Python implementation.

Having variable declarations after a post keyword describes exactly what
the function or method is allowed to modify. This removes the need for
the NoChange syntax in Eiffel, and makes the implementation of __old__
much easier. It also is more in line with Z schemas[6], which are
divided into two parts: declaring what changes followed by limiting the
changes.

Shallow copies of variables for the __old__ value prevent an
implementation of contract programming from slowing down a system too
much. If a function changes values that wouldn't be caught by a shallow
copy, it can declare the changes like so:

    post[self, self.obj, self.obj.p]

The forall, exists, and implies functions were added after spending some
time documenting existing functions with contracts. These capture a
majority of common specification idioms. It might seem that defining
implies as a function might not work (the arguments are evaluated
whether needed or not, in contrast with other boolean operators), but it
works for contracts since there should be no side-effects for any
expression in a contract.

Reference Implementation

A reference implementation is available[7]. It replaces existing
functions with new functions that do contract checking, by directly
changing the class' or module's namespace.

Other implementations exist that either hack __getattr__[8] or use
__metaclass__[9].

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 End:

[1] Design By Contract is a registered trademark of Eiffel Software Inc.
(http://archive.eiffel.com/doc/manuals/technology/contract/)

[2] http://docs.python.org/library/doctest.html doctest -- Test
docstrings represent reality

[3] Extreme Programming Explained, Kent Beck, ISBN 0-201-61641-6

[4] ReStructuredText (http://docutils.sourceforge.net/rst.html)

[5] Object-oriented Software Construction, Bertrand Meyer, ISBN
0-13-629031-0

[6] The Z Notation, Second Edition, J.M. Spivey ISBN 0-13-978529-9

[7] Implementation described in this document.
(http://www.wayforward.net/pycontract/)

[8] Design by Contract for Python, R. Plosch IEEE Proceedings of the
Joint Asia Pacific Software Engineering Conference (APSEC97/ICSC97),
Hong Kong, December 2-5, 1997
(http://www.swe.uni-linz.ac.at/publications/abstract/TR-SE-97.24.html)

[9] PyDBC -- Design by Contract for Python 2.2+, Daniel Arbuckle
(http://www.nongnu.org/pydbc/)


text:
PEP: 464 Title: Removal of the PyPI Mirror Authenticity API Version:
$Revision$ Last-Modified: $Date$ Author: Donald Stufft
<EMAIL_ADDRESS>BDFL-Delegate: Richard Jones<EMAIL_ADDRESS>Discussions-To<EMAIL_ADDRESS>Status: Final Type: Process
Topic: Packaging Content-Type: text/x-rst Created: 02-Mar-2014
Post-History: 04-Mar-2014 Replaces: 381 Resolution:
https://mail.python.org/pipermail/distutils-sig/2014-March/024027.html

Abstract

This PEP proposes the deprecation and removal of the PyPI Mirror
Authenticity API, this includes the /serverkey URL and all of the URLs
under /serversig.

Rationale

The PyPI mirroring infrastructure (defined in PEP 381) provides a means
to mirror the content of PyPI used by the automatic installers, and as a
component of that, it provides a method for verifying the authenticity
of the mirrored content.

This PEP proposes the removal of this API due to:

-   There are no known implementations that utilize this API, this
    includes pip and setuptools.
-   Because this API uses DSA it is vulnerable to leaking the private
    key if there is any bias in the random nonce.
-   This API solves one small corner of the trust problem, however the
    problem itself is much larger and it would be better to have a fully
    fledged system, such as The Update Framework <458>, instead.

Due to the issues it has and the lack of use it is the opinion of this
PEP that it does not provide any practical benefit to justify the
additional complexity.

Plan for Deprecation & Removal

Immediately upon the acceptance of this PEP the Mirror Authenticity API
will be considered deprecated and mirroring agents and installation
tools should stop accessing it.

Instead of actually removing it from the current code base (PyPI 1.0)
the current work to replace PyPI 1.0 with a new code base (PyPI 2.0)
will simply not implement this API. This would cause the API to be
"removed" when the switch from 1.0 to 2.0 occurs.

If PyPI 2.0 has not been deployed in place of PyPI 1.0 by Sept 01 2014
then this PEP will be implemented in the PyPI 1.0 code base instead (by
removing the associated code).

No changes will be required in the installers, however PEP 381 compliant
mirroring clients, such as bandersnatch and pep381client will need to be
updated to no longer attempt to mirror the /serversig URLs.

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End: