text:
PEP: 397 Title: Python launcher for Windows Author: Mark Hammond
<EMAIL_ADDRESS>Martin von Löwis<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type: text/x-rst Created:
15-Mar-2011 Python-Version: 3.3 Post-History: 21-Jul-2011, 17-May-2011,
15-Mar-2011 Resolution:
https://mail.python.org/pipermail/python-dev/2012-June/120505.html

Abstract

This PEP describes a Python launcher for the Windows platform. A Python
launcher is a single executable which uses a number of heuristics to
locate a Python executable and launch it with a specified command line.

Rationale

Windows provides "file associations" so an executable can be associated
with an extension, allowing for scripts to be executed directly in some
contexts (eg., double-clicking the file in Windows Explorer.) Until now,
a strategy of "last installed Python wins" has been used and while not
ideal, has generally been workable due to the conservative changes in
Python 2.x releases. As Python 3.x scripts are often syntactically
incompatible with Python 2.x scripts, a different strategy must be used
to allow files with a '.py' extension to use a different executable
based on the Python version the script targets. This will be done by
borrowing the existing practices of another operating system - scripts
will be able to nominate the version of Python they need by way of a
"shebang" line, as described below.

Unix-like operating systems (referred to simply as "Unix" in this PEP)
allow scripts to be executed as if they were executable images by
examining the script for a "shebang" line which specifies the actual
executable to be used to run the script. This is described in detail in
the evecve(2) man page[1] and while user documentation will be created
for this feature, for the purposes of this PEP that man page describes a
valid shebang line.

Additionally, these operating systems provide symbolic-links to Python
executables in well-known directories. For example, many systems will
have a link /usr/bin/python which references a particular version of
Python installed under the operating-system. These symbolic links allow
Python to be executed without regard for where Python it actually
installed on the machine (eg., without requiring the path where Python
is actually installed to be referenced in the shebang line or in the
PATH.) PEP 394 'The "python" command on Unix-Like Systems' describes
additional conventions for more fine-grained specification of a
particular Python version.

These 2 facilities combined allow for a portable and somewhat
predictable way of both starting Python interactively and for allowing
Python scripts to execute. This PEP describes an implementation of a
launcher which can offer the same benefits for Python on the Windows
platform and therefore allows the launcher to be the executable
associated with '.py' files to support multiple Python versions
concurrently.

While this PEP offers the ability to use a shebang line which should
work on both Windows and Unix, this is not the primary motivation for
this PEP - the primary motivation is to allow a specific version to be
specified without inventing new syntax or conventions to describe it.

Specification

This PEP specifies features of the launcher; a prototype implementation
is provided in[2] which will be distributed together with the Windows
installer of Python, but will also be available separately (but released
along with the Python installer). New features may be added to the
launcher as long as the features prescribed here continue to work.

Installation

The launcher comes in 2 versions - one which is a console program and
one which is a "windows" (ie., GUI) program. These 2 launchers
correspond to the 'python.exe' and 'pythonw.exe' executables which
currently ship with Python. The console launcher will be named 'py.exe'
and the Windows one named 'pyw.exe'. The "windows" (ie., GUI) version of
the launcher will attempt to locate and launch pythonw.exe even if a
virtual shebang line nominates simply "python" - in fact, the trailing
'w' notation is not supported in the virtual shebang line at all.

The launcher is installed into the Windows directory (see discussion
below) if installed by a privileged user. The stand-alone installer asks
for an alternative location of the installer, and adds that location to
the user's PATH.

The installation in the Windows directory is a 32-bit executable (see
discussion); the standalone installer may also offer to install 64-bit
versions of the launcher.

The launcher installation is registered in
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\CurrentVersion\SharedDLLs with a
reference counter. It contains a version resource matching the version
number of the pythonXY.dll with which it is distributed. Independent
installations will overwrite older version of the launcher with newer
versions. Stand-alone releases use a release level of 0x10 in FIELD3 of
the CPython release on which they are based.

Once installed, the "console" version of the launcher is associated with
.py files and the "windows" version associated with .pyw files.

The launcher is not tied to a specific version of Python - eg., a
launcher distributed with Python 3.3 should be capable of locating and
executing any Python 2.x and Python 3.x version. However, the launcher
binaries have a version resource that is the same as the version
resource in the Python binaries that they are released with.

Python Script Launching

The launcher is restricted to launching Python scripts. It is not
intended as a general-purpose script launcher or shebang processor.

The launcher supports the syntax of shebang lines as described in[3],
including all restrictions listed.

The launcher supports shebang lines referring to Python executables with
any of the (regex) prefixes "/usr/bin/", "/usr/local/bin" and
"/usr/bin/env *", as well as binaries specified without

For example, a shebang line of '#! /usr/bin/python' should work even
though there is unlikely to be an executable in the relative Windows
directory "\usr\bin". This means that many scripts can use a single
shebang line and be likely to work on both Unix and Windows without
modification.

The launcher will support fully-qualified paths to executables. While
this will make the script inherently non-portable, it is a feature
offered by Unix and would be useful for Windows users in some cases.

The launcher will be capable of supporting implementations other than
CPython, such as jython and IronPython, but given both the absence of
common links on Unix (such as "/usr/bin/jython") and the inability for
the launcher to automatically locate the installation location of these
implementations on Windows, the launcher will support this via
customization options. Scripts taking advantage of this will not be
portable (as these customization options must be set to reflect the
configuration of the machine on which the launcher is running) but this
ability is nonetheless considered worthwhile.

On Unix, the user can control which specific version of Python is used
by adjusting the links in /usr/bin to point to the desired version. As
the launcher on Windows will not use Windows links, customization
options (exposed via both environment variables and INI files) will be
used to override the semantics for determining what version of Python
will be used. For example, while a shebang line of "/usr/bin/python2"
will automatically locate a Python 2.x implementation, an environment
variable can override exactly which Python 2.x implementation will be
chosen. Similarly for "/usr/bin/python" and "/usr/bin/python3". This is
specified in detail later in this PEP.

Shebang line parsing

If the first command-line argument does not start with a dash ('-')
character, an attempt will be made to open that argument as a file and
parsed for a shebang line according to the rules in[4]:

    #! interpreter [optional-arg]

Once parsed, the command will be categorized according to the following
rules:

-   If the command starts with the definition of a customized command
    followed by a whitespace character (including a newline), the
    customized command will be used. See below for a description of
    customized commands.
-   The launcher will define a set of prefixes which are considered Unix
    compatible commands to launch Python, namely "/usr/bin/python",
    "/usr/local/bin/python", "/usr/bin/env python", and "python". If a
    command starts with one of these strings will be treated as a
    'virtual command' and the rules described in Python Version
    Qualifiers (below) will be used to locate the executable to use.
-   Otherwise the command is assumed to be directly ready to execute -
    ie. a fully-qualified path (or a reference to an executable on the
    PATH) optionally followed by arguments. The contents of the string
    will not be parsed - it will be passed directly to the Windows
    CreateProcess function after appending the name of the script and
    the launcher command-line arguments. This means that the rules used
    by CreateProcess will be used, including how relative path names and
    executable references without extensions are treated. Notably, the
    Windows command processor will not be used, so special rules used by
    the command processor (such as automatic appending of extensions
    other than '.exe', support for batch files, etc) will not be used.

The use of 'virtual' shebang lines is encouraged as this should allow
for portable shebang lines to be specified which work on multiple
operating systems and different installations of the same operating
system.

If the first argument can not be opened as a file or if no valid shebang
line can be found, the launcher will act as if a shebang line of
'#!python' was found - ie., a default Python interpreter will be located
and the arguments passed to that. However, if a valid shebang line is
found but the process specified by that line can not be started, the
default interpreter will not be started - the error to create the
specified child process will cause the launcher to display an
appropriate message and terminate with a specific exit code.

Configuration file

Two .ini files will be searched by the launcher - py.ini in the current
user's "application data" directory (i.e. the directory returned by
calling the Windows function SHGetFolderPath with CSIDL_LOCAL_APPDATA,
%USERPROFILE%\AppData\Local on Vista+,
%USERPROFILE%\Local Settings\Application Data on XP) and py.ini in the
same directory as the launcher. The same .ini files are used for both
the 'console' version of the launcher (i.e. py.exe) and for the
'windows' version (i.e. pyw.exe)

Customization specified in the "application directory" will have
precedence over the one next to the executable, so a user, who may not
have write access to the .ini file next to the launcher, can override
commands in that global .ini file)

Virtual commands in shebang lines

Virtual Commands are shebang lines which start with strings which would
be expected to work on Unix platforms - examples include
'/usr/bin/python', '/usr/bin/env python' and 'python'. Optionally, the
virtual command may be suffixed with a version qualifier (see below),
such as '/usr/bin/python2' or '/usr/bin/python3.2'. The command executed
is based on the rules described in Python Version Qualifiers below.

Customized Commands

The launcher will support the ability to define "Customized Commands" in
a Windows .ini file (ie, a file which can be parsed by the Windows
function GetPrivateProfileString). A section called '[commands]' can be
created with key names defining the virtual command and the value
specifying the actual command-line to be used for this virtual command.

For example, if an INI file has the contents:

    [commands]
    vpython=c:\bin\vpython.exe -foo

Then a shebang line of '#! vpython' in a script named 'doit.py' will
result in the launcher using the command-line
c:\bin\vpython.exe -foo doit.py

The precise details about the names, locations and search order of the
.ini files is in the launcher documentation[5]

Python Version Qualifiers

Some of the features described allow an optional Python version
qualifier to be used.

A version qualifier starts with a major version number and can
optionally be followed by a period ('.') and a minor version specifier.
If the minor qualifier is specified, it may optionally be followed by
"-32" to indicate the 32bit implementation of that version be used. Note
that no "-64" qualifier is necessary as this is the default
implementation (see below).

On 64bit Windows with both 32bit and 64bit implementations of the same
(major.minor) Python version installed, the 64bit version will always be
preferred. This will be true for both 32bit and 64bit implementations of
the launcher - a 32bit launcher will prefer to execute a 64bit Python
installation of the specified version if available. This is so the
behavior of the launcher can be predicted knowing only what versions are
installed on the PC and without regard to the order in which they were
installed (ie, without knowing whether a 32 or 64bit version of Python
and corresponding launcher was installed last). As noted above, an
optional "-32" suffix can be used on a version specifier to change this
behaviour.

If no version qualifiers are found in a command, the environment
variable PY_PYTHON can be set to specify the default version qualifier -
the default value is "2". Note this value could specify just a major
version (e.g. "2") or a major.minor qualifier (e.g. "2.6"), or even
major.minor-32.

If no minor version qualifiers are found, the environment variable
PY_PYTHON{major} (where {major} is the current major version qualifier
as determined above) can be set to specify the full version. If no such
option is found, the launcher will enumerate the installed Python
versions and use the latest minor release found for the major version,
which is likely, although not guaranteed, to be the most recently
installed version in that family.

In addition to environment variables, the same settings can be
configured in the .INI file used by the launcher. The section in the INI
file is called [defaults] and the key name will be the same as the
environment variables without the leading PY_ prefix (and note that the
key names in the INI file are case insensitive.) The contents of an
environment variable will override things specified in the INI file.

Command-line handling

Only the first command-line argument will be checked for a shebang line
and only if that argument does not start with a '-'.

If the only command-line argument is "-h" or "--help", the launcher will
print a small banner and command-line usage, then pass the argument to
the default Python. This will cause help for the launcher being printed
followed by help for Python itself. The output from the launcher will
clearly indicate the extended help information is coming from the
launcher and not Python.

As a concession to interactively launching Python, the launcher will
support the first command-line argument optionally being a dash ("-")
followed by a version qualifier, as described above, to nominate a
specific version be used. For example, while "py.exe" may locate and
launch the latest Python 2.x implementation installed, a command-line
such as "py.exe -3" could specify the latest Python 3.x implementation
be launched, while "py.exe -2.6-32" could specify a 32bit implementation
Python 2.6 be located and launched. If a Python 2.x implementation is
desired to be launched with the -3 flag, the command-line would need to
be similar to "py.exe -2 -3" (or the specific version of Python could
obviously be launched manually without use of this launcher.) Note that
this feature can not be used with shebang processing as the file scanned
for a shebang line and this argument must both be the first argument and
therefore are mutually exclusive.

All other arguments will be passed untouched to the child Python
process.

Process Launching

The launcher offers some conveniences for Python developers working
interactively - for example, starting the launcher with no command-line
arguments will launch the default Python with no command-line arguments.
Further, command-line arguments will be supported to allow a specific
Python version to be launched interactively - however, these
conveniences must not detract from the primary purpose of launching
scripts and must be easy to avoid if desired.

The launcher creates a subprocess to start the actual interpreter. See
Discussion below for the rationale.

Discussion

It may be surprising that the launcher is installed into the Windows
directory, and not the System32 directory. The reason is that the
System32 directory is not on the Path of a 32-bit process running on a
64-bit system. However, the Windows directory is always on the path.

The launcher that is installed into the Windows directory is a 32-bit
executable so that the 32-bit CPython installer can provide the same
binary for both 32-bit and 64-bit Windows installations.

Ideally, the launcher process would execute Python directly inside the
same process, primarily so the parent of the launcher process could
terminate the launcher and have the Python interpreter terminate. If the
launcher executes Python as a sub-process and the parent of the launcher
terminates the launcher, the Python process will be unaffected.

However, there are a number of practical problems associated with this
approach. Windows does not support the execv* family of Unix functions,
so this could only be done by the launcher dynamically loading the
Python DLL, but this would have a number of side-effects. The most
serious side effect of this is that the value of sys.executable would
refer to the launcher instead of the Python implementation. Many Python
scripts use the value of sys.executable to launch child processes, and
these scripts may fail to work as expected if the launcher is used.
Consider a "parent" script with a shebang line of '#! /usr/bin/python3'
which attempts to launch a child script (with no shebang) via
sys.executable - currently the child is launched using the exact same
version running the parent script. If sys.executable referred to the
launcher the child would be likely executed using a Python 2.x version
and would be likely to fail with a SyntaxError.

Another hurdle is the support for alternative Python implementations
using the "customized commands" feature described above, where loading
the command dynamically into a running executable is not possible.

The final hurdle is the rules above regarding 64bit and 32bit programs
-a 32bit launcher would be unable to load the 64bit version of Python
and vice-versa.

Given these considerations, the launcher will execute its command in a
child process, remaining alive while the child process is executing,
then terminate with the same exit code as returned by the child. To
address concerns regarding the termination of the launcher not killing
the child, the Win32 Job API will be used to arrange so that the child
process is automatically killed when the parent is terminated (although
children of that child process will continue as is the case now.) As
this Windows API is available in Windows XP and later, this launcher
will not work on Windows 2000 or earlier.

References

Copyright

This document has been placed in the public domain.

[1] http://linux.die.net/man/2/execve

[2] https://bitbucket.org/vinay.sajip/pylauncher

[3] http://linux.die.net/man/2/execve

[4] http://linux.die.net/man/2/execve

[5] https://bitbucket.org/vinay.sajip/pylauncher/src/tip/Doc/launcher.rst


text:
PEP: 304 Title: Controlling Generation of Bytecode Files Version:
$Revision$ Last-Modified: $Date$ Author: Skip Montanaro Status:
Withdrawn Type: Standards Track Content-Type: text/x-rst Created:
22-Jan-2003 Post-History: 27-Jan-2003, 31-Jan-2003, 17-Jun-2005

Historical Note

While this original PEP was withdrawn, a variant of this feature was
eventually implemented for Python 3.8 in
https://bugs.python.org/issue33499

Several of the issues and concerns originally raised in this PEP were
resolved by other changes in the intervening years:

-   the introduction of isolated mode to handle potential security
    concerns
-   the switch to importlib, a fully import-hook based import system
    implementation
-   PEP 3147's change in the bytecode cache layout to use __pycache__
    subdirectories, including the source_to_cache(path) and
    cache_to_source(path) APIs that allow the interpreter to
    automatically handle the redirection to a separate cache directory

Abstract

This PEP outlines a mechanism for controlling the generation and
location of compiled Python bytecode files. This idea originally arose
as a patch request[1] and evolved into a discussion thread on the
python-dev mailing list[2]. The introduction of an environment variable
will allow people installing Python or Python-based third-party packages
to control whether or not bytecode files should be generated at
installation time, and if so, where they should be written. It will also
allow users to control whether or not bytecode files should be generated
at application run-time, and if so, where they should be written.

Proposal

Add a new environment variable, PYTHONBYTECODEBASE, to the mix of
environment variables which Python understands. PYTHONBYTECODEBASE is
interpreted as follows:

-   If not defined, Python bytecode is generated in exactly the same way
    as is currently done. sys.bytecodebase is set to the root directory
    (either / on Unix and Mac OSX or the root directory of the startup
    (installation???) drive -- typically C:\ -- on Windows).

-   If defined and it refers to an existing directory to which the user
    has write permission, sys.bytecodebase is set to that directory and
    bytecode files are written into a directory structure rooted at that
    location.

-   If defined but empty, sys.bytecodebase is set to None and generation
    of bytecode files is suppressed altogether.

-   If defined and one of the following is true:

    -   it does not refer to a directory,
    -   it refers to a directory, but not one for which the user has
        write permission

    a warning is displayed, sys.bytecodebase is set to None and
    generation of bytecode files is suppressed altogether.

After startup initialization, all runtime references are to
sys.bytecodebase, not the PYTHONBYTECODEBASE environment variable.
sys.path is not modified.

From the above, we see sys.bytecodebase can only take on two valid types
of values: None or a string referring to a valid directory on the
system.

During import, this extension works as follows:

-   The normal search for a module is conducted. The search order is
    roughly: dynamically loaded extension module, Python source file,
    Python bytecode file. The only time this mechanism comes into play
    is if a Python source file is found.
-   Once we've found a source module, an attempt to read a byte-compiled
    file in the same directory is made. (This is the same as before.)
-   If no byte-compiled file is found, an attempt to read a
    byte-compiled file from the augmented directory is made.
-   If bytecode generation is required, the generated bytecode is
    written to the augmented directory if possible.

Note that this PEP is explicitly not about providing module-by-module or
directory-by-directory control over the disposition of bytecode files.

Glossary

-   "bytecode base" refers to the current setting of sys.bytecodebase.
-   "augmented directory" refers to the directory formed from the
    bytecode base and the directory name of the source file.
-   PYTHONBYTECODEBASE refers to the environment variable when necessary
    to distinguish it from "bytecode base".

Locating bytecode files

When the interpreter is searching for a module, it will use sys.path as
usual. However, when a possible bytecode file is considered, an extra
probe for a bytecode file may be made. First, a check is made for the
bytecode file using the directory in sys.path which holds the source
file (the current behavior). If a valid bytecode file is not found there
(either one does not exist or exists but is out-of-date) and the
bytecode base is not None, a second probe is made using the directory in
sys.path prefixed appropriately by the bytecode base.

Writing bytecode files

When the bytecode base is not None, a new bytecode file is written to
the appropriate augmented directory, never directly to a directory in
sys.path.

Defining augmented directories

Conceptually, the augmented directory for a bytecode file is the
directory in which the source file exists prefixed by the bytecode base.
In a Unix environment this would be:

    pcb = os.path.abspath(sys.bytecodebase)
    if sourcefile[0] == os.sep: sourcefile = sourcefile[1:]
    augdir = os.path.join(pcb, os.path.dirname(sourcefile))

On Windows, which does not have a single-rooted directory tree, the
drive letter of the directory containing the source file is treated as a
directory component after removing the trailing colon. The augmented
directory is thus derived as :

    pcb = os.path.abspath(sys.bytecodebase)
    drive, base = os.path.splitdrive(os.path.dirname(sourcefile))
    drive = drive[:-1]
    if base[0] == "\\": base = base[1:]
    augdir = os.path.join(pcb, drive, base)

Fixing the location of the bytecode base

During program startup, the value of the PYTHONBYTECODEBASE environment
variable is made absolute, checked for validity and added to the sys
module, effectively:

    pcb = os.path.abspath(os.environ["PYTHONBYTECODEBASE"])
    probe = os.path.join(pcb, "foo")
    try:
        open(probe, "w")
    except IOError:
        sys.bytecodebase = None
    else:
        os.unlink(probe)
        sys.bytecodebase = pcb

This allows the user to specify the bytecode base as a relative path,
but not have it subject to changes to the current working directory
during program execution. (I can't imagine you'd want it to move around
during program execution.)

There is nothing special about sys.bytecodebase. The user may change it
at runtime if desired, but normally it will not be modified.

Rationale

In many environments it is not possible for non-root users to write into
directories containing Python source files. Most of the time, this is
not a problem as Python source is generally byte compiled during
installation. However, there are situations where bytecode files are
either missing or need to be updated. If the directory containing the
source file is not writable by the current user a performance penalty is
incurred each time a program importing the module is run.[3] Warning
messages may also be generated in certain circumstances. If the
directory is writable, nearly simultaneous attempts to write the
bytecode file by two separate processes may occur, resulting in file
corruption.[4]

In environments with RAM disks available, it may be desirable for
performance reasons to write bytecode files to a directory on such a
disk. Similarly, in environments where Python source code resides on
network file systems, it may be desirable to cache bytecode files on
local disks.

Alternatives

The only other alternative proposed so far[5] seems to be to add a -R
flag to the interpreter to disable writing bytecode files altogether.
This proposal subsumes that. Adding a command-line option is certainly
possible, but is probably not sufficient, as the interpreter's command
line is not readily available during installation (early during program
startup???).

Issues

-   Interpretation of a module's __file__ attribute. I believe the
    __file__ attribute of a module should reflect the true location of
    the bytecode file. If people want to locate a module's source code,
    they should use imp.find_module(module).
-   Security - What if root has PYTHONBYTECODEBASE set? Yes, this can
    present a security risk, but so can many other things the root user
    does. The root user should probably not set PYTHONBYTECODEBASE
    except possibly during installation. Still, perhaps this problem can
    be minimized. When running as root the interpreter should check to
    see if PYTHONBYTECODEBASE refers to a directory which is writable by
    anyone other than root. If so, it could raise an exception or
    warning and set sys.bytecodebase to None. Or, see the next item.
-   More security - What if PYTHONBYTECODEBASE refers to a general
    directory (say, /tmp)? In this case, perhaps loading of a
    preexisting bytecode file should occur only if the file is owned by
    the current user or root. (Does this matter on Windows?)
-   The interaction of this PEP with import hooks has not been
    considered yet. In fact, the best way to implement this idea might
    be as an import hook. See PEP 302.
-   In the current (pre-PEP 304) environment, it is safe to delete a
    source file after the corresponding bytecode file has been created,
    since they reside in the same directory. With PEP 304 as currently
    defined, this is not the case. A bytecode file in the augmented
    directory is only considered when the source file is present and it
    thus never considered when looking for module files ending in
    ".pyc". I think this behavior may have to change.

Examples

In the examples which follow, the urllib source code resides in
/usr/lib/python2.3/urllib.py and /usr/lib/python2.3 is in sys.path but
is not writable by the current user.

-   The bytecode base is /tmp. /usr/lib/python2.3/urllib.pyc exists and
    is valid. When urllib is imported, the contents of
    /usr/lib/python2.3/urllib.pyc are used. The augmented directory is
    not consulted. No other bytecode file is generated.
-   The bytecode base is /tmp. /usr/lib/python2.3/urllib.pyc exists, but
    is out-of-date. When urllib is imported, the generated bytecode file
    is written to urllib.pyc in the augmented directory which has the
    value /tmp/usr/lib/python2.3. Intermediate directories will be
    created as needed.
-   The bytecode base is None. No urllib.pyc file is found. When urllib
    is imported, no bytecode file is written.
-   The bytecode base is /tmp. No urllib.pyc file is found. When urllib
    is imported, the generated bytecode file is written to the augmented
    directory which has the value /tmp/usr/lib/python2.3. Intermediate
    directories will be created as needed.
-   At startup, PYTHONBYTECODEBASE is /tmp/foobar, which does not exist.
    A warning is emitted, sys.bytecodebase is set to None and no
    bytecode files are written during program execution unless
    sys.bytecodebase is later changed to refer to a valid, writable
    directory.
-   At startup, PYTHONBYTECODEBASE is set to /, which exists, but is not
    writable by the current user. A warning is emitted, sys.bytecodebase
    is set to None and no bytecode files are written during program
    execution unless sys.bytecodebase is later changed to refer to a
    valid, writable directory. Note that even though the augmented
    directory constructed for a particular bytecode file may be writable
    by the current user, what counts is that the bytecode base directory
    itself is writable.
-   At startup PYTHONBYTECODEBASE is set to the empty string.
    sys.bytecodebase is set to None. No warning is generated, however.
    If no urllib.pyc file is found when urllib is imported, no bytecode
    file is written.

In the Windows examples which follow, the urllib source code resides in
C:\PYTHON22\urllib.py. C:\PYTHON22 is in sys.path but is not writable by
the current user.

-   The bytecode base is set to C:\TEMP. C:\PYTHON22\urllib.pyc exists
    and is valid. When urllib is imported, the contents of
    C:\PYTHON22\urllib.pyc are used. The augmented directory is not
    consulted.
-   The bytecode base is set to C:\TEMP. C:\PYTHON22\urllib.pyc exists,
    but is out-of-date. When urllib is imported, a new bytecode file is
    written to the augmented directory which has the value
    C:\TEMP\C\PYTHON22. Intermediate directories will be created as
    needed.
-   At startup PYTHONBYTECODEBASE is set to TEMP and the current working
    directory at application startup is H:\NET. The potential bytecode
    base is thus H:\NET\TEMP. If this directory exists and is writable
    by the current user, sys.bytecodebase will be set to that value. If
    not, a warning will be emitted and sys.bytecodebase will be set to
    None.
-   The bytecode base is C:\TEMP. No urllib.pyc file is found. When
    urllib is imported, the generated bytecode file is written to the
    augmented directory which has the value C:\TEMP\C\PYTHON22.
    Intermediate directories will be created as needed.

Implementation

See the patch on Sourceforge.[6]

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 End:

[1] patch 602345, Option for not writing py.[co] files, Klose
(https://bugs.python.org/issue602345)

[2] python-dev thread, Disable writing .py[co], Norwitz
(https://mail.python.org/pipermail/python-dev/2003-January/032270.html)

[3] Debian bug report, Mailman is writing to /usr in cron, Wegner
(http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=96111)

[4] python-dev thread, Parallel pyc construction, Dubois
(https://mail.python.org/pipermail/python-dev/2003-January/032060.html)

[5] patch 602345, Option for not writing py.[co] files, Klose
(https://bugs.python.org/issue602345)

[6] patch 677103, PYTHONBYTECODEBASE patch (PEP 304), Montanaro
(https://bugs.python.org/issue677103)

license: Public Domain


text:
PEP: 299 Title: Special __main__() function in modules Version:
$Revision$ Last-Modified: $Date$ Author: Jeff Epler
<EMAIL_ADDRESS>Status: Rejected Type: Standards Track
Content-Type: text/x-rst Created: 12-Aug-2002 Python-Version: 2.3
Post-History: 29-Mar-2006

Abstract

Many Python modules are also intended to be callable as standalone
scripts. This PEP proposes that a special function called __main__()
should serve this purpose.

Motivation

There should be one simple and universal idiom for invoking a module as
a standalone script.

The semi-standard idiom:

    if __name__ == '__main__':
        perform "standalone" functionality

is unclear to programmers of languages like C and C++. It also does not
permit invocation of the standalone function when the module is
imported. The variant:

    if __name__ == '__main__':
        main_function()

is sometimes seen, but there exists no standard name for the function,
and because arguments are taken from sys.argv it is not possible to pass
specific arguments without changing the argument list seen by all other
modules. (Imagine a threaded Python program, with two threads wishing to
invoke the standalone functionality of different modules with different
argument lists)

Proposal

The standard name of the 'main function' should be __main__. When a
module is invoked on the command line, such as:

    python mymodule.py

then the module behaves as though the following lines existed at the end
of the module (except that the attribute __sys may not be used or
assumed to exist elsewhere in the script):

    if globals().has_key("__main__"):
        import sys as __sys
        __sys.exit(__main__(__sys.argv))

Other modules may execute:

    import mymodule mymodule.__main__(['mymodule', ...])

It is up to mymodule to document thread-safety issues or other issues
which might restrict use of __main__. (Other issues might include use of
mutually exclusive GUI modules, non-sharable resources like hardware
devices, reassignment of sys.stdin/stdout, etc)

Implementation

In modules/main.c, the block near line 385 (after the
PyRun_AnyFileExFlags call) will be changed so that the above code (or
its C equivalent) is executed.

Open Issues

-   Should the return value from __main__ be treated as the exit value?

    Yes. Many __main__ will naturally return None, which sys.exit
    translates into a "success" return code. In those that return a
    numeric result, it behaves just like the argument to sys.exit() or
    the return value from C's main().

-   Should the argument list to __main__ include argv[0], or just the
    "real" arguments argv[1:]?

    argv[0] is included for symmetry with sys.argv and easy transition
    to the new standard idiom.

Rejection

In a short discussion on python-dev[1], two major backwards
compatibility problems were brought up and Guido pronounced that he
doesn't like the idea anyway as it's "not worth the change (in docs,
user habits, etc.) and there's nothing particularly broken."

References

Copyright

This document has been placed in the public domain.

[1] Georg Brandl, "What about PEP 299",
https://mail.python.org/pipermail/python-dev/2006-March/062951.html


text:
PEP: 341 Title: Unifying try-except and try-finally Version: $Revision$
Last-Modified: $Date$ Author: Georg Brandl<EMAIL_ADDRESS>Status:
Final Type: Standards Track Content-Type: text/x-rst Created:
04-May-2005 Python-Version: 2.5 Post-History:

Abstract

This PEP proposes a change in the syntax and semantics of try statements
to allow combined try-except-finally blocks. This means in short that it
would be valid to write:

    try:
        <do something>
    except Exception:
        <handle the error>
    finally:
        <cleanup>

Rationale/Proposal

There are many use cases for the try-except statement and for the
try-finally statement per se; however, often one needs to catch
exceptions and execute some cleanup code afterwards. It is slightly
annoying and not very intelligible that one has to write:

    f = None
    try:
        try:
            f = open(filename)
            text = f.read()
        except IOError:
            print 'An error occurred'
    finally:
        if f:
            f.close()

So it is proposed that a construction like this:

    try:
        <suite 1>
    except Ex1:
        <suite 2>
    <more except: clauses>
    else:
        <suite 3>
    finally:
        <suite 4>

be exactly the same as the legacy:

    try:
        try:
            <suite 1>
        except Ex1:
            <suite 2>
        <more except: clauses>
        else:
            <suite 3>
    finally:
        <suite 4>

This is backwards compatible, and every try statement that is legal
today would continue to work.

Changes to the grammar

The grammar for the try statement, which is currently:

    try_stmt: ('try' ':' suite (except_clause ':' suite)+
            ['else' ':' suite] | 'try' ':' suite 'finally' ':' suite)

would have to become:

    try_stmt: 'try' ':' suite
            (
                (except_clause ':' suite)+
                ['else' ':' suite]
                ['finally' ':' suite]
            |
                'finally' ':' suite
            )

Implementation

As the PEP author currently does not have sufficient knowledge of the
CPython implementation, he is unfortunately not able to deliver one.
Thomas Lee has submitted a patch[1].

However, according to Guido, it should be a piece of cake to
implement[2] -- at least for a core hacker.

This patch was committed 17 December 2005, SVN revision 41740[3].

References

Copyright

This document has been placed in the public domain.

[1] https://bugs.python.org/issue1355913

[2] https://mail.python.org/pipermail/python-dev/2005-May/053319.html

[3] https://mail.python.org/pipermail/python-checkins/2005-December/048457.html


text:
PEP: 363 Title: Syntax For Dynamic Attribute Access Author: Ben North
<EMAIL_ADDRESS>Status: Rejected Type: Standards Track Created:
29-Jan-2007 Post-History: 12-Feb-2007

See 363-rejection for more information.

Abstract

Dynamic attribute access is currently possible using the "getattr" and
"setattr" builtins. The present PEP suggests a new syntax to make such
access easier, allowing the coder for example to write:

    x.('foo_%d' % n) += 1

    z = y.('foo_%d' % n).('bar_%s' % s)

instead of:

    attr_name = 'foo_%d' % n
    setattr(x, attr_name, getattr(x, attr_name) + 1)

    z = getattr(getattr(y, 'foo_%d' % n), 'bar_%s' % s)

Rationale

Dictionary access and indexing both have a friendly invocation syntax:
instead of x.__getitem__(12) the coder can write x[12]. This also allows
the use of subscripted elements in an augmented assignment, as in "x[12]
+= 1". The present proposal brings this ease-of-use to dynamic attribute
access too.

Attribute access is currently possible in two ways:

-   When the attribute name is known at code-writing time, the ".NAME"
    trailer can be used, as in:

        x.foo = 42
        y.bar += 100

-   When the attribute name is computed dynamically at run-time, the
    "getattr" and "setattr" builtins must be used:

        x = getattr(y, 'foo_%d' % n)
        setattr(z, 'bar_%s' % s, 99)

    The "getattr" builtin also allows the coder to specify a default
    value to be returned in the event that the object does not have an
    attribute of the given name:

        x = getattr(y, 'foo_%d' % n, 0)

This PEP describes a new syntax for dynamic attribute access
---"x.(expr)" --- with examples given in the Abstract above.

(The new syntax could also allow the provision of a default value in the
"get" case, as in:

    x = y.('foo_%d' % n, None)

This 2-argument form of dynamic attribute access would not be permitted
as the target of an (augmented or normal) assignment. The "Discussion"
section below includes opinions specifically on the 2-argument
extension.)

Finally, the new syntax can be used with the "del" statement, as in:

    del x.(attr_name)

Impact On Existing Code

The proposed new syntax is not currently valid, so no existing
well-formed programs have their meaning altered by this proposal.

Across all "*.py" files in the 2.5 distribution, there are around 600
uses of "getattr", "setattr" or "delattr". They break down as follows
(figures have some room for error because they were arrived at by
partially-manual inspection):

    c.300 uses of plain "getattr(x, attr_name)", which could be
          replaced with the new syntax;

    c.150 uses of the 3-argument form, i.e., with the default
          value; these could be replaced with the 2-argument form
          of the new syntax (the cases break down into c.125 cases
          where the attribute name is a literal string, and c.25
          where it's only known at run-time);

    c.5   uses of the 2-argument form with a literal string
          attribute name, which I think could be replaced with the
          standard "x.attribute" syntax;

    c.120 uses of setattr, of which 15 use getattr to find the
          new value; all could be replaced with the new syntax,
          the 15 where getattr is also involved would show a
          particular increase in clarity;

    c.5   uses which would have to stay as "getattr" because they
          are calls of a variable named "getattr" whose default
          value is the builtin "getattr";

    c.5   uses of the 2-argument form, inside a try/except block
          which catches AttributeError and uses a default value
          instead; these could use 2-argument form of the new
          syntax;

    c.10  uses of "delattr", which could use the new syntax.

As examples, the line:

    setattr(self, attr, change_root(self.root, getattr(self, attr)))

from Lib/distutils/command/install.py could be rewritten:

    self.(attr) = change_root(self.root, self.(attr))

and the line:

    setattr(self, method_name, getattr(self.metadata, method_name))

from Lib/distutils/dist.py could be rewritten:

    self.(method_name) = self.metadata.(method_name)

Performance Impact

Initial pystone measurements are inconclusive, but suggest there may be
a performance penalty of around 1% in the pystones score with the
patched version. One suggestion is that this is because the longer main
loop in ceval.c hurts the cache behaviour, but this has not been
confirmed.

On the other hand, measurements suggest a speed-up of around 40--45% for
dynamic attribute access.

Error Cases

Only strings are permitted as attribute names, so for instance the
following error is produced:

    >>> x.(99) = 8
       Traceback (most recent call last):
         File "<stdin>", line 1, in <module>
       TypeError: attribute name must be string, not 'int'

This is handled by the existing PyObject_GetAttr function.

Draft Implementation

A draft implementation adds a new alternative to the "trailer" clause in
Grammar/Grammar; a new AST type, "DynamicAttribute" in Python.asdl, with
accompanying changes to symtable.c, ast.c, and compile.c, and three new
opcodes (load/store/del) with accompanying changes to opcode.h and
ceval.c. The patch consists of c.180 additional lines in the core code,
and c.100 additional lines of tests. It is available as sourceforge
patch #1657573[1].

Mailing Lists Discussion

Initial posting of this PEP in draft form was to python-ideas on
20070209[2], and the response was generally positive. The PEP was then
posted to python-dev on 20070212[3], and an interesting discussion
ensued. A brief summary:

Initially, there was reasonable (but not unanimous) support for the
idea, although the precise choice of syntax had a more mixed reception.
Several people thought the "." would be too easily overlooked, with the
result that the syntax could be confused with a method/function call. A
few alternative syntaxes were suggested:

    obj.(foo)
    obj.[foo]
    obj.{foo}
    obj{foo}
    obj.*foo
    obj->foo
    obj<-foo
    obj@[foo]
    obj.[[foo]]

with "obj.[foo]" emerging as the preferred one. In this initial
discussion, the two-argument form was universally disliked, so it was to
be taken out of the PEP.

Discussion then took a step back to whether this particular feature
provided enough benefit to justify new syntax. As well as requiring
coders to become familiar with the new syntax, there would also be the
problem of backward compatibility --- code using the new syntax would
not run on older pythons.

Instead of new syntax, a new "wrapper class" was proposed, with the
following specification / conceptual implementation suggested by Martin
von Löwis:

    class attrs:
       def __init__(self, obj):
         self.obj = obj
       def __getitem__(self, name):
         return getattr(self.obj, name)
       def __setitem__(self, name, value):
         return setattr(self.obj, name, value)
       def __delitem__(self, name):
         return delattr(self, name)
       def __contains__(self, name):
         return hasattr(self, name)

This was considered a cleaner and more elegant solution to the original
problem. (Another suggestion was a mixin class providing
dictionary-style access to an object's attributes.)

The decision was made that the present PEP did not meet the burden of
proof for the introduction of new syntax, a view which had been put
forward by some from the beginning of the discussion. The wrapper class
idea was left open as a possibility for a future PEP.

References

Copyright

This document has been placed in the public domain.

[1] Sourceforge patch #1657573
http://sourceforge.net/tracker/index.php?func=detail&aid=1657573&group_id=5470&atid=305470

[2] https://mail.python.org/pipermail/python-ideas/2007-February/000210.html
and following posts

[3] https://mail.python.org/pipermail/python-dev/2007-February/070939.html
and following posts


text:
PEP: 406 Title: Improved Encapsulation of Import State Version:
$Revision$ Last-Modified: $Date$ Author: Alyssa Coghlan
<EMAIL_ADDRESS>Greg Slodkowicz<EMAIL_ADDRESS>Status:
Withdrawn Type: Standards Track Content-Type: text/x-rst Created:
04-Jul-2011 Python-Version: 3.4 Post-History: 31-Jul-2011, 13-Nov-2011,
04-Dec-2011

Abstract

This PEP proposes the introduction of a new 'ImportEngine' class as part
of importlib which would encapsulate all state related to importing
modules into a single object. Creating new instances of this object
would then provide an alternative to completely replacing the built-in
implementation of the import statement, by overriding the __import__()
function. To work with the builtin import functionality and importing
via import engine objects, this PEP proposes a context management based
approach to temporarily replacing the global import state.

The PEP also proposes inclusion of a GlobalImportEngine subclass and a
globally accessible instance of that class, which "writes through" to
the process global state. This provides a backwards compatible bridge
between the proposed encapsulated API and the legacy process global
state, and allows straightforward support for related state updates
(e.g. selectively invalidating path cache entries when sys.path is
modified).

PEP Withdrawal

The import system has seen substantial changes since this PEP was
originally written, as part of PEP 420 in Python 3.3 and PEP 451 in
Python 3.4.

While providing an encapsulation of the import state is still highly
desirable, it is better tackled in a new PEP using PEP 451 as a
foundation, and permitting only the use of PEP 451 compatible finders
and loaders (as those avoid many of the issues of direct manipulation of
global state associated with the previous loader API).

Rationale

Currently, most state related to the import system is stored as module
level attributes in the sys module. The one exception is the import
lock, which is not accessible directly, but only via the related
functions in the imp module. The current process global import state
comprises:

-   sys.modules
-   sys.path
-   sys.path_hooks
-   sys.meta_path
-   sys.path_importer_cache
-   the import lock (imp.lock_held()/acquire_lock()/release_lock())

Isolating this state would allow multiple import states to be
conveniently stored within a process. Placing the import functionality
in a self-contained object would also allow subclassing to add
additional features (e.g. module import notifications or fine-grained
control over which modules can be imported). The engine would also be
subclassed to make it possible to use the import engine API to interact
with the existing process-global state.

The namespace PEPs (especially PEP 402) raise a potential need for
additional process global state, in order to correctly update package
paths as sys.path is modified.

Finally, providing a coherent object for all this state makes it
feasible to also provide context management features that allow the
import state to be temporarily substituted.

Proposal

We propose introducing an ImportEngine class to encapsulate import
functionality. This includes an __import__() method which can be used as
an alternative to the built-in __import__() when desired and also an
import_module() method, equivalent to importlib.import_module()[1].

Since there are global import state invariants that are assumed and
should be maintained, we introduce a GlobalImportState class with an
interface identical to ImportEngine but directly accessing the current
global import state. This can be easily implemented using class
properties.

Specification

ImportEngine API

The proposed extension consists of the following objects:

importlib.engine.ImportEngine

  from_engine(self, other)

    Create a new import object from another ImportEngine instance. The
    new object is initialised with a copy of the state in other. When
    called on importlib engine.sysengine, from_engine() can be used to
    create an ImportEngine object with a copy of the global import
    state.

  __import__(self, name, globals={}, locals={}, fromlist=[], level=0)

    Reimplementation of the builtin __import__() function. The import of
    a module will proceed using the state stored in the ImportEngine
    instance rather than the global import state. For full documentation
    of __import__ functionality, see[2] . __import__() from ImportEngine
    and its subclasses can be used to customise the behaviour of the
    import statement by replacing __builtin__.__import__ with
    ImportEngine().__import__.

  import_module(name, package=None)

    A reimplementation of importlib.import_module() which uses the
    import state stored in the ImportEngine instance. See[3] for a full
    reference.

  modules, path, path_hooks, meta_path, path_importer_cache

    Instance-specific versions of their process global sys equivalents

importlib.engine.GlobalImportEngine(ImportEngine)

  Convenience class to provide engine-like access to the global state.
  Provides __import__(), import_module() and from_engine() methods like
  ImportEngine but writes through to the global state in sys.

To support various namespace package mechanisms, when sys.path is
altered, tools like pkgutil.extend_path should be used to also modify
other parts of the import state (in this case, package __path__
attributes). The path importer cache should also be invalidated when a
variety of changes are made.

The ImportEngine API will provide convenience methods that automatically
make related import state updates as part of a single operation.

Global variables

importlib.engine.sysengine

  A precreated instance of GlobalImportEngine. Intended for use by
  importers and loaders that have been updated to accept optional engine
  parameters and with ImportEngine.from_engine(sysengine) to start with
  a copy of the process global import state.

No changes to finder/loader interfaces

Rather than attempting to update the PEP 302 APIs to accept additional
state, this PEP proposes that ImportEngine support the content
management protocol (similar to the context substitution mechanisms in
the decimal module).

The context management mechanism for ImportEngine would:

-   On entry:
    -   Acquire the import lock
    -   Substitute the global import state with the import engine's own
        state
-   On exit:
    -   Restore the previous global import state
    -   Release the import lock

The precise API for this is TBD (but will probably use a distinct
context management object, along the lines of that created by
decimal.localcontext).

Open Issues

API design for falling back to global import state

The current proposal relies on the from_engine() API to fall back to the
global import state. It may be desirable to offer a variant that instead
falls back to the global import state dynamically.

However, one big advantage of starting with an "as isolated as possible"
design is that it becomes possible to experiment with subclasses that
blur the boundaries between the engine instance state and the process
global state in various ways.

Builtin and extension modules must be process global

Due to platform limitations, only one copy of each builtin and extension
module can readily exist in each process. Accordingly, it is impossible
for each ImportEngine instance to load such modules independently.

The simplest solution is for ImportEngine to refuse to load such
modules, raising ImportError. GlobalImportEngine would be able to load
them normally.

ImportEngine will still return such modules from a prepopulated module
cache - it's only loading them directly which causes problems.

Scope of substitution

Related to the previous open issue is the question of what state to
substitute when using the context management API. It is currently the
case that replacing sys.modules can be unreliable due to cached
references and there's the underlying fact that having independent
copies of some modules is simply impossible due to platform limitations.

As part of this PEP, it will be necessary to document explicitly:

-   Which parts of the global import state can be substituted (and
    declare code which caches references to that state without dealing
    with the substitution case buggy)
-   Which parts must be modified in-place (and hence are not substituted
    by the ImportEngine context management API, or otherwise scoped to
    ImportEngine instances)

Reference Implementation

A reference implementation[4] for an earlier draft of this PEP, based on
Brett Cannon's importlib has been developed by Greg Slodkowicz as part
of the 2011 Google Summer of Code. Note that the current implementation
avoids modifying existing code, and hence duplicates a lot of things
unnecessarily. An actual implementation would just modify any such
affected code in place.

That earlier draft of the PEP proposed change the PEP 302 APIs to
support passing in an optional engine instance. This had the (serious)
downside of not correctly affecting further imports from the imported
module, hence the change to the context management based proposal for
substituting the global state.

References

Copyright

This document has been placed in the public domain.

[1] Importlib documentation, Cannon
(http://docs.python.org/dev/library/importlib)

[2] __import__() builtin function, The Python Standard Library
documentation (http://docs.python.org/library/functions.html#__import__)

[3] Importlib documentation, Cannon
(http://docs.python.org/dev/library/importlib)

[4] Reference implementation
(https://bitbucket.org/jergosh/gsoc_import_engine/src/default/Lib/importlib/engine.py)

authors:
Alyssa Coghlan
Greg Slodkowicz


text:
PEP: 225 Title: Elementwise/Objectwise Operators Author: Huaiyu Zhu
<EMAIL_ADDRESS>Gregory Lielens<EMAIL_ADDRESS>Status: Rejected Type: Standards Track Created: 19-Sep-2000
Python-Version: 2.1 Post-History:

The approach in the later PEP 465 was eventually accepted in lieu of
this PEP. The Rejected Ideas
<465#rejected-alternatives-to-adding-a-new-operator> of that PEP
explains the rationale in more detail.

Introduction

This PEP describes a proposal to add new operators to Python which are
useful for distinguishing elementwise and objectwise operations, and
summarizes discussions in the news group comp.lang.python on this topic.
See Credits and Archives section at end. Issues discussed here include:

-   Background.
-   Description of proposed operators and implementation issues.
-   Analysis of alternatives to new operators.
-   Analysis of alternative forms.
-   Compatibility issues
-   Description of wider extensions and other related ideas.

A substantial portion of this PEP describes ideas that do not go into
the proposed extension. They are presented because the extension is
essentially syntactic sugar, so its adoption must be weighed against
various possible alternatives. While many alternatives may be better in
some aspects, the current proposal appears to be overall advantageous.

The issues concerning elementwise-objectwise operations extends to wider
areas than numerical computation. This document also describes how the
current proposal may be integrated with more general future extensions.

Background

Python provides six binary infix math operators: + - * / % ** hereafter
generically represented by op. They can be overloaded with new semantics
for user-defined classes. However, for objects composed of homogeneous
elements, such as arrays, vectors and matrices in numerical computation,
there are two essentially distinct flavors of semantics. The objectwise
operations treat these objects as points in multidimensional spaces. The
elementwise operations treat them as collections of individual elements.
These two flavors of operations are often intermixed in the same
formulas, thereby requiring syntactical distinction.

Many numerical computation languages provide two sets of math operators.
For example, in MatLab, the ordinary op is used for objectwise operation
while .op is used for elementwise operation. In R, op stands for
elementwise operation while %op% stands for objectwise operation.

In Python, there are other methods of representation, some of which
already used by available numerical packages, such as:

-   function: mul(a,b)
-   method: a.mul(b)
-   casting: a.E*b

In several aspects these are not as adequate as infix operators. More
details will be shown later, but the key points are:

-   Readability: Even for moderately complicated formulas, infix
    operators are much cleaner than alternatives.
-   Familiarity: Users are familiar with ordinary math operators.
-   Implementation: New infix operators will not unduly clutter Python
    syntax. They will greatly ease the implementation of numerical
    packages.

While it is possible to assign current math operators to one flavor of
semantics, there is simply not enough infix operators to overload for
the other flavor. It is also impossible to maintain visual symmetry
between these two flavors if one of them does not contain symbols for
ordinary math operators.

Proposed extension

-   Six new binary infix operators ~+ ~- ~* ~/ ~% ~** are added to core
    Python. They parallel the existing operators + - * / % **.
-   Six augmented assignment operators ~+= ~-= ~*= ~/= ~%= ~**= are
    added to core Python. They parallel the operators += -= *= /= %= **=
    available in Python 2.0.
-   Operator ~op retains the syntactical properties of operator op,
    including precedence.
-   Operator ~op retains the semantical properties of operator op on
    built-in number types.
-   Operator ~op raise syntax error on non-number builtin types. This is
    temporary until the proper behavior can be agreed upon.
-   These operators are overloadable in classes with names that prepend
    t (for tilde) to names of ordinary math operators. For example,
    __tadd__ and __rtadd__ work for ~+ just as __add__ and __radd__ work
    for +.
-   As with existing operators, the __r*__() methods are invoked when
    the left operand does not provide the appropriate method.

It is intended that one set of op or ~op is used for elementwise
operations, the other for objectwise operations, but it is not specified
which version of operators stands for elementwise or objectwise
operations, leaving the decision to applications.

The proposed implementation is to patch several files relating to the
tokenizer, parser, grammar and compiler to duplicate the functionality
of corresponding existing operators as necessary. All new semantics are
to be implemented in the classes that overload them.

The symbol ~ is already used in Python as the unary bitwise not
operator. Currently it is not allowed for binary operators. The new
operators are completely backward compatible.

Prototype Implementation

Greg Lielens implemented the infix ~op as a patch against Python 2.0b1
source[1].

To allow ~ to be part of binary operators, the tokenizer would treat ~+
as one token. This means that currently valid expression ~+1 would be
tokenized as ~+ 1 instead of ~ + 1. The parser would then treat ~+ as
composite of ~ +. The effect is invisible to applications.

Notes about current patch:

-   It does not include ~op= operators yet.
-   The ~op behaves the same as op on lists, instead of raising
    exceptions.

These should be fixed when the final version of this proposal is ready.

-   It reserves xor as an infix operator with the semantics equivalent
    to:

        def __xor__(a, b):
            if not b: return a
            elif not a: return b
            else: 0

This preserves true value as much as possible, otherwise preserve left
hand side value if possible.

This is done so that bitwise operators could be regarded as elementwise
logical operators in the future (see below).

Alternatives to adding new operators

The discussions on comp.lang.python and python-dev mailing list explored
many alternatives. Some of the leading alternatives are listed here,
using the multiplication operator as an example.

1.  Use function mul(a,b).

    Advantage:

    -   No need for new operators.

    Disadvantage:

    -   Prefix forms are cumbersome for composite formulas.
    -   Unfamiliar to the intended users.
    -   Too verbose for the intended users.
    -   Unable to use natural precedence rules.

2.  Use method call a.mul(b).

    Advantage:

    -   No need for new operators.

    Disadvantage:

    -   Asymmetric for both operands.
    -   Unfamiliar to the intended users.
    -   Too verbose for the intended users.
    -   Unable to use natural precedence rules.

3.  Use shadow classes. For matrix class define a shadow array class
    accessible through a method .E, so that for matrices a and b, a.E*b
    would be a matrix object that is elementwise_mul(a,b).

    Likewise define a shadow matrix class for arrays accessible through
    a method .M so that for arrays a and b, a.M*b would be an array that
    is matrixwise_mul(a,b).

    Advantage:

    -   No need for new operators.
    -   Benefits of infix operators with correct precedence rules.
    -   Clean formulas in applications.

    Disadvantage:

    -   Hard to maintain in current Python because ordinary numbers
        cannot have user defined class methods; i.e. a.E*b will fail if
        a is a pure number.
    -   Difficult to implement, as this will interfere with existing
        method calls, like .T for transpose, etc.
    -   Runtime overhead of object creation and method lookup.
    -   The shadowing class cannot replace a true class, because it does
        not return its own type. So there need to be a M class with
        shadow E class, and an E class with shadow M class.
    -   Unnatural to mathematicians.

4.  Implement matrixwise and elementwise classes with easy casting to
    the other class. So matrixwise operations for arrays would be like
    a.M*b.M and elementwise operations for matrices would be like
    a.E*b.E. For error detection a.E*b.M would raise exceptions.

    Advantage:

    -   No need for new operators.
    -   Similar to infix notation with correct precedence rules.

    Disadvantage:

    -   Similar difficulty due to lack of user-methods for pure numbers.
    -   Runtime overhead of object creation and method lookup.
    -   More cluttered formulas.
    -   Switching of flavor of objects to facilitate operators becomes
        persistent. This introduces long range context dependencies in
        application code that would be extremely hard to maintain.

5.  Using mini parser to parse formulas written in arbitrary extension
    placed in quoted strings.

    Advantage:

    -   Pure Python, without new operators

    Disadvantage:

    -   The actual syntax is within the quoted string, which does not
        resolve the problem itself.
    -   Introducing zones of special syntax.
    -   Demanding on the mini-parser.

6.  Introducing a single operator, such as @, for matrix multiplication.

    Advantage:

    -   Introduces less operators

    Disadvantage:

    -   The distinctions for operators like + - ** are equally
        important. Their meaning in matrix or array-oriented packages
        would be reversed (see below).
    -   The new operator occupies a special character.
    -   This does not work well with more general object-element issues.

Among these alternatives, the first and second are used in current
applications to some extent, but found inadequate. The third is the most
favorite for applications, but it will incur huge implementation
complexity. The fourth would make applications codes very
context-sensitive and hard to maintain. These two alternatives also
share significant implementational difficulties due to current
type/class split. The fifth appears to create more problems than it
would solve. The sixth does not cover the same range of applications.

Alternative forms of infix operators

Two major forms and several minor variants of new infix operators were
discussed:

-   Bracketed form:

        (op)
        [op]
        {op}
        <op>
        :op:
        ~op~
        %op%

-   Meta character form:

        .op
        @op
        ~op

    Alternatively the meta character is put after the operator.

-   Less consistent variations of these themes. These are considered
    unfavorably. For completeness some are listed here:

    -   Use @/ and /@ for left and right division
    -   Use [*] and (*) for outer and inner products
    -   Use a single operator @ for multiplication.

-   Use __call__ to simulate multiplication:

        a(b) or (a)(b)

Criteria for choosing among the representations include:

-   No syntactical ambiguities with existing operators.
-   Higher readability in actual formulas. This makes the bracketed
    forms unfavorable. See examples below.
-   Visually similar to existing math operators.
-   Syntactically simple, without blocking possible future extensions.

With these criteria the overall winner in bracket form appear to be
{op}. A clear winner in the meta character form is ~op. Comparing these
it appears that ~op is the favorite among them all.

Some analysis are as follows:

-   The .op form is ambiguous: 1.+a would be different from 1 .+a.
-   The bracket type operators are most favorable when standing alone,
    but not in formulas, as they interfere with visual parsing of
    parentheses for precedence and function argument. This is so for
    (op) and [op], and somewhat less so for {op} and <op>.
-   The <op> form has the potential to be confused with < > and =.
-   The @op is not favored because @ is visually heavy (dense, more like
    a letter): a@+b is more readily read as a@ + b than a @+ b.
-   For choosing meta-characters: Most of existing ASCII symbols have
    already been used. The only three unused are @ $ ?.

Semantics of new operators

There are convincing arguments for using either set of operators as
objectwise or elementwise. Some of them are listed here:

1.  op for element, ~op for object
    -   Consistent with current multiarray interface of Numeric package.
    -   Consistent with some other languages.
    -   Perception that elementwise operations are more natural.
    -   Perception that elementwise operations are used more frequently
2.  op for object, ~op for element
    -   Consistent with current linear algebra interface of MatPy
        package.
    -   Consistent with some other languages.
    -   Perception that objectwise operations are more natural.
    -   Perception that objectwise operations are used more frequently.
    -   Consistent with the current behavior of operators on lists.
    -   Allow ~ to be a general elementwise meta-character in future
        extensions.

It is generally agreed upon that

-   There is no absolute reason to favor one or the other.
-   It is easy to cast from one representation to another in a sizable
    chunk of code, so the other flavor of operators is always minority.
-   There are other semantic differences that favor existence of
    array-oriented and matrix-oriented packages, even if their operators
    are unified.
-   Whatever the decision is taken, codes using existing interfaces
    should not be broken for a very long time.

Therefore, not much is lost, and much flexibility retained, if the
semantic flavors of these two sets of operators are not dictated by the
core language. The application packages are responsible for making the
most suitable choice. This is already the case for NumPy and MatPy which
use opposite semantics. Adding new operators will not break this. See
also observation after subsection 2 in the Examples below.

The issue of numerical precision was raised, but if the semantics is
left to the applications, the actual precisions should also go there.

Examples

Following are examples of the actual formulas that will appear using
various operators or other representations described above.

1.  The matrix inversion formula:

    -   Using op for object and ~op for element:

            b = a.I - a.I * u / (c.I + v/a*u) * v / a

            b = a.I - a.I * u * (c.I + v*a.I*u).I * v * a.I

    -   Using op for element and ~op for object:

            b = a.I @- a.I @* u @/ (c.I @+ v@/a@*u) @* v @/ a

            b = a.I ~- a.I ~* u ~/ (c.I ~+ v~/a~*u) ~* v ~/ a

            b = a.I (-) a.I (*) u (/) (c.I (+) v(/)a(*)u) (*) v (/) a

            b = a.I [-] a.I [*] u [/] (c.I [+] v[/]a[*]u) [*] v [/] a

            b = a.I <-> a.I <*> u </> (c.I <+> v</>a<*>u) <*> v </> a

            b = a.I {-} a.I {*} u {/} (c.I {+} v{/}a{*}u) {*} v {/} a

    Observation: For linear algebra using op for object is preferable.

    Observation: The ~op type operators look better than (op) type in
    complicated formulas.

    -   using named operators:

            b = a.I @sub a.I @mul u @div (c.I @add v @div a @mul u) @mul v @div a

            b = a.I ~sub a.I ~mul u ~div (c.I ~add v ~div a ~mul u) ~mul v ~div a

    Observation: Named operators are not suitable for math formulas.

2.  Plotting a 3d graph

    -   Using op for object and ~op for element:

            z = sin(x~**2 ~+ y~**2);    plot(x,y,z)

    -   Using op for element and ~op for object:

            z = sin(x**2 + y**2);   plot(x,y,z)

    Observation: Elementwise operations with broadcasting allows much
    more efficient implementation than MatLab.

    Observation: It is useful to have two related classes with the
    semantics of op and ~op swapped. Using these the ~op operators would
    only need to appear in chunks of code where the other flavor
    dominates, while maintaining consistent semantics of the code.

3.  Using + and - with automatic broadcasting:

        a = b - c;  d = a.T*a

    Observation: This would silently produce hard-to-trace bugs if one
    of b or c is row vector while the other is column vector.

Miscellaneous issues

-   Need for the ~+ ~- operators. The objectwise + - are important
    because they provide important sanity checks as per linear algebra.
    The elementwise + - are important because they allow broadcasting
    that are very efficient in applications.

-   Left division (solve). For matrix, a*x is not necessarily equal to
    x*a. The solution of a*x==b, denoted x=solve(a,b), is therefore
    different from the solution of x*a==b, denoted x=div(b,a). There are
    discussions about finding a new symbol for solve. [Background:
    MatLab use b/a for div(b,a) and a\b for solve(a,b).]

    It is recognized that Python provides a better solution without
    requiring a new symbol: the inverse method .I can be made to be
    delayed so that a.I*b and b*a.I are equivalent to Matlab's a\b and
    b/a. The implementation is quite simple and the resulting
    application code clean.

-   Power operator. Python's use of a**b as pow(a,b) has two perceived
    disadvantages:

    -   Most mathematicians are more familiar with a^b for this purpose.
    -   It results in long augmented assignment operator ~**=.

    However, this issue is distinct from the main issue here.

-   Additional multiplication operators. Several forms of
    multiplications are used in (multi-)linear algebra. Most can be seen
    as variations of multiplication in linear algebra sense (such as
    Kronecker product). But two forms appear to be more fundamental:
    outer product and inner product. However, their specification
    includes indices, which can be either

    -   associated with the operator, or
    -   associated with the objects.

    The latter (the Einstein notation) is used extensively on paper, and
    is also the easier one to implement. By implementing a
    tensor-with-indices class, a general form of multiplication would
    cover both outer and inner products, and specialize to linear
    algebra multiplication as well. The index rule can be defined as
    class methods, like:

        a = b.i(1,2,-1,-2) * c.i(4,-2,3,-1)   # a_ijkl = b_ijmn c_lnkm

    Therefore, one objectwise multiplication is sufficient.

-   Bitwise operators.

    -   The proposed new math operators use the symbol ~ that is bitwise
        not operator. This poses no compatibility problem but somewhat
        complicates implementation.
    -   The symbol ^ might be better used for pow than bitwise xor. But
        this depends on the future of bitwise operators. It does not
        immediately impact on the proposed math operator.
    -   The symbol | was suggested to be used for matrix solve. But the
        new solution of using delayed .I is better in several ways.
    -   The current proposal fits in a larger and more general extension
        that will remove the need for special bitwise operators. (See
        elementization below.)

-   Alternative to special operator names used in definition,

        def "+"(a, b)      in place of       def __add__(a, b)

    This appears to require greater syntactical change, and would only
    be useful when arbitrary additional operators are allowed.

Impact on general elementization

The distinction between objectwise and elementwise operations are
meaningful in other contexts as well, where an object can be
conceptually regarded as a collection of elements. It is important that
the current proposal does not preclude possible future extensions.

One general future extension is to use ~ as a meta operator to
elementize a given operator. Several examples are listed here:

1.  Bitwise operators. Currently Python assigns six operators to bitwise
    operations: and (&), or (|), xor (^), complement (~), left shift
    (<<) and right shift (>>), with their own precedence levels.

    Among them, the & | ^ ~ operators can be regarded as elementwise
    versions of lattice operators applied to integers regarded as bit
    strings.:

        5 and 6                # 6
        5 or 6                 # 5

        5 ~and 6               # 4
        5 ~or 6                # 7

    These can be regarded as general elementwise lattice operators, not
    restricted to bits in integers.

    In order to have named operators for xor ~xor, it is necessary to
    make xor a reserved word.

2.  List arithmetics.:

        [1, 2] + [3, 4]        # [1, 2, 3, 4]
        [1, 2] ~+ [3, 4]       # [4, 6]

        ['a', 'b'] * 2         # ['a', 'b', 'a', 'b']
        'ab' * 2               # 'abab'

        ['a', 'b'] ~* 2        # ['aa', 'bb']
        [1, 2] ~* 2            # [2, 4]

    It is also consistent to Cartesian product:

        [1,2]*[3,4]            # [(1,3),(1,4),(2,3),(2,4)]

3.  List comprehension.:

        a = [1, 2]; b = [3, 4]
        ~f(a,b)                # [f(x,y) for x, y in zip(a,b)]
        ~f(a*b)                # [f(x,y) for x in a for y in b]
        a ~+ b                 # [x + y for x, y in zip(a,b)]

4.  Tuple generation (the zip function in Python 2.0):

        [1, 2, 3], [4, 5, 6]   # ([1,2, 3], [4, 5, 6])
        [1, 2, 3]~,[4, 5, 6]   # [(1,4), (2, 5), (3,6)]

5.  Using ~ as generic elementwise meta-character to replace map:

        ~f(a, b)               # map(f, a, b)
        ~~f(a, b)              # map(lambda *x:map(f, *x), a, b)

    More generally,:

        def ~f(*x): return map(f, *x)
        def ~~f(*x): return map(~f, *x)
        ...

6.  Elementwise format operator (with broadcasting):

        a = [1,2,3,4,5]
        print ["%5d "] ~% a
        a = [[1,2],[3,4]]
        print ["%5d "] ~~% a

7.  Rich comparison:

        [1, 2, 3]  ~< [3, 2, 1]  # [1, 0, 0]
        [1, 2, 3] ~== [3, 2, 1]  # [0, 1, 0]

8.  Rich indexing:

        [a, b, c, d] ~[2, 3, 1]  # [c, d, b]

9.  Tuple flattening:

        a = (1,2);  b = (3,4)
        f(~a, ~b)                # f(1,2,3,4)

10. Copy operator:

        a ~= b                   # a = b.copy()

  There can be specific levels of deep copy:

      a ~~= b                  # a = b.copy(2)

Notes

1.  There are probably many other similar situations. This general
    approach seems well suited for most of them, in place of several
    separated extensions for each of them (parallel and cross iteration,
    list comprehension, rich comparison, etc).
2.  The semantics of elementwise depends on applications. For example,
    an element of matrix is two levels down from the list-of-list point
    of view. This requires more fundamental change than the current
    proposal. In any case, the current proposal will not negatively
    impact on future possibilities of this nature.

Note that this section describes a type of future extensions that is
consistent with current proposal, but may present additional
compatibility or other problems. They are not tied to the current
proposal.

Impact on named operators

The discussions made it generally clear that infix operators is a scarce
resource in Python, not only in numerical computation, but in other
fields as well. Several proposals and ideas were put forward that would
allow infix operators be introduced in ways similar to named functions.
We show here that the current extension does not negatively impact on
future extensions in this regard.

1.  Named infix operators.

    Choose a meta character, say @, so that for any identifier opname,
    the combination @opname would be a binary infix operator, and:

        a @opname b == opname(a,b)

    Other representations mentioned include:

        .name ~name~ :name: (.name) %name%

    and similar variations. The pure bracket based operators cannot be
    used this way.

    This requires a change in the parser to recognize @opname, and parse
    it into the same structure as a function call. The precedence of all
    these operators would have to be fixed at one level, so the
    implementation would be different from additional math operators
    which keep the precedence of existing math operators.

    The current proposed extension do not limit possible future
    extensions of such form in any way.

2.  More general symbolic operators.

    One additional form of future extension is to use meta character and
    operator symbols (symbols that cannot be used in syntactical
    structures other than operators). Suppose @ is the meta character.
    Then:

        a + b,    a @+ b,    a @@+ b,  a @+- b

    would all be operators with a hierarchy of precedence, defined by:

        def "+"(a, b)
        def "@+"(a, b)
        def "@@+"(a, b)
        def "@+-"(a, b)

    One advantage compared with named operators is greater flexibility
    for precedences based on either the meta character or the ordinary
    operator symbols. This also allows operator composition. The
    disadvantage is that they are more like line noise. In any case the
    current proposal does not impact its future possibility.

    These kinds of future extensions may not be necessary when Unicode
    becomes generally available.

    Note that this section discusses compatibility of the proposed
    extension with possible future extensions. The desirability or
    compatibility of these other extensions themselves are specifically
    not considered here.

Credits and archives

The discussions mostly happened in July to August of 2000 on news group
comp.lang.python and the mailing list python-dev. There are altogether
several hundred postings, most can be retrieved from these two pages
(and searching word "operator"):

  http://www.python.org/pipermail/python-list/2000-July/
  http://www.python.org/pipermail/python-list/2000-August/

The names of contributors are too numerous to mention here, suffice to
say that a large proportion of ideas discussed here are not our own.

Several key postings (from our point of view) that may help to navigate
the discussions include:

  http://www.python.org/pipermail/python-list/2000-July/108893.html
  http://www.python.org/pipermail/python-list/2000-July/108777.html
  http://www.python.org/pipermail/python-list/2000-July/108848.html
  http://www.python.org/pipermail/python-list/2000-July/109237.html
  http://www.python.org/pipermail/python-list/2000-July/109250.html
  http://www.python.org/pipermail/python-list/2000-July/109310.html
  http://www.python.org/pipermail/python-list/2000-July/109448.html
  http://www.python.org/pipermail/python-list/2000-July/109491.html
  http://www.python.org/pipermail/python-list/2000-July/109537.html
  http://www.python.org/pipermail/python-list/2000-July/109607.html
  http://www.python.org/pipermail/python-list/2000-July/109709.html
  http://www.python.org/pipermail/python-list/2000-July/109804.html
  http://www.python.org/pipermail/python-list/2000-July/109857.html
  http://www.python.org/pipermail/python-list/2000-July/110061.html
  http://www.python.org/pipermail/python-list/2000-July/110208.html
  http://www.python.org/pipermail/python-list/2000-August/111427.html
  http://www.python.org/pipermail/python-list/2000-August/111558.html
  http://www.python.org/pipermail/python-list/2000-August/112551.html
  http://www.python.org/pipermail/python-list/2000-August/112606.html
  http://www.python.org/pipermail/python-list/2000-August/112758.html

  http://www.python.org/pipermail/python-dev/2000-July/013243.html
  http://www.python.org/pipermail/python-dev/2000-July/013364.html
  http://www.python.org/pipermail/python-dev/2000-August/014940.html

These are earlier drafts of this PEP:

  http://www.python.org/pipermail/python-list/2000-August/111785.html
  http://www.python.org/pipermail/python-list/2000-August/112529.html
  http://www.python.org/pipermail/python-dev/2000-August/014906.html

There is an alternative PEP (officially, PEP 211) by Greg Wilson, titled
"Adding New Linear Algebra Operators to Python".

Its first (and current) version is at:

  http://www.python.org/pipermail/python-dev/2000-August/014876.html PEP
  211

Additional References

[1] http://MatPy.sourceforge.net/Misc/index.html