text:
PEP: 594 Title: Removing dead batteries from the standard library
Author: Christian Heimes<EMAIL_ADDRESS>Brett Cannon
<EMAIL_ADDRESS>Discussions-To: https://discuss.python.org/t/13508
Status: Final Type: Standards Track Created: 20-May-2019 Python-Version:
3.11 Post-History: 21-May-2019, 04-Feb-2022 Resolution:
https://discuss.python.org/t/13508/22

Abstract

This PEP proposed a list of standard library modules to be removed from
the standard library. The modules are mostly historic data formats (e.g.
Commodore and SUN file formats), APIs and operating systems that have
been superseded a long time ago (e.g. Mac OS 9), or modules that have
security implications and better alternatives (e.g. password and login).

The PEP follows in the footsteps of other PEPS like PEP 3108. The
Standard Library Reorganization proposal removed a bunch of modules from
Python 3.0. In 2007, the PEP referred to maintenance burden as:

  "Over the years, certain modules have become a heavy burden upon
  python-dev to maintain. In situations like this, it is better for the
  module to be given to the community to maintain to free python-dev to
  focus more on language support and other modules in the standard
  library that do not take up an undue amount of time and effort."

The withdrawn PEP 206 from 2000 expresses issues with the Python
standard library in an unvarnished and forthright manner:

  "[...] the standard library modules aren't always the best choices for
  a job. Some library modules were quick hacks (e.g. calendar,
  commands), some were designed poorly and are now near-impossible to
  fix (cgi), and some have been rendered obsolete by other, more
  complete modules [...]."

Rationale

Back in the early days of Python, the interpreter came with a large set
of useful modules. This was often referred to as "batteries included"
philosophy and was one of the cornerstones to Python's success story.
Users didn't have to figure out how to download and install separate
packages in order to write a simple web server or parse email.

Times have changed. With the introduction of PyPI (née Cheeseshop),
setuptools, and later pip, it became simple and straightforward to
download and install packages. Nowadays Python has a rich and vibrant
ecosystem of third-party packages. It's pretty much standard to either
install packages from PyPI or use one of the many Python or Linux
distributions.

On the other hand, Python's standard library is piling up with cruft,
unnecessary duplication of functionality, and dispensable features. This
is undesirable for several reasons.

-   Any additional module increases the maintenance cost for the Python
    core development team. The team has limited resources, reduced
    maintenance cost frees development time for other improvements.
-   Modules in the standard library are generally favored and seen as
    the de facto solution for a problem. A majority of users only pick
    third-party modules to replace a stdlib module, when they have a
    compelling reason, e.g. lxml instead of xml. The removal of an
    unmaintained stdlib module increases the chances of a
    community-contributed module to become widely used.
-   A lean and mean standard library benefits platforms with limited
    resources like devices with just a few hundred kilobyte of storage
    (e.g. BBC Micro:bit). Python on mobile platforms like BeeWare or
    WebAssembly (e.g. pyodide) also benefit from reduced download size.

The modules in this PEP have been selected for deprecation because their
removal is either least controversial or most beneficial. For example,
least controversial are 30-year-old multimedia formats like the sunau
audio format, which was used on SPARC and NeXT workstations in the late
1980s. The crypt module has fundamental flaws that are better solved
outside the standard library.

This PEP also designates some modules as not scheduled for removal. Some
modules have been deprecated for several releases or seem unnecessary at
first glance. However it is beneficial to keep the modules in the
standard library, mostly for environments where installing a package
from PyPI is not an option. This can be corporate environments or
classrooms where external code is not permitted without legal approval.

-   The usage of FTP is declining, but some files are still provided
    over the FTP protocol or hosters offer FTP to upload content.
    Therefore, ftplib is going to stay.
-   The optparse and getopt modules are widely used. They are mature
    modules with very low maintenance overhead.
-   According to David Beazley[1] the wave module is easy to teach to
    kids and can make crazy sounds. Making a computer generate sounds is
    a powerful and highly motivating exercise for a nine-year-old
    aspiring developer. It's a fun battery to keep.

Deprecation schedule

3.11

Starting with Python 3.11, deprecated modules will start issuing
DeprecationWarning. The estimated EOL of Python 3.10, the last version
without the warning, is October 2026.

3.12

There should be no specific change compared to Python 3.11. This is the
last version of Python with the deprecated modules, with an estimated
EOL of October 2028.

3.13

All modules deprecated by this PEP are removed from the main branch of
the CPython repository and are no longer distributed as part of Python.

Deprecated modules

The modules are grouped as data encoding, multimedia, network, OS
interface, and misc modules. The majority of modules are for old data
formats or old APIs. Some others are rarely useful and have better
replacements on PyPI, e.g. Pillow for image processing or NumPy-based
projects to deal with audio processing.

Some module deprecations proposed by PEP 3108 for 3.0 and PEP 206 for
2.0. The added in column illustrates, when a module was originally
designed and added to the standard library. The has maintainer column
refers to the expert index, a list of domain experts and maintainers in
the DevGuide.

Data encoding modules

uu and the uu encoding

The uu module provides uuencode format, an old binary encoding format
for email from 1980. The uu format has been replaced by MIME. The uu
codec is provided by the binascii module. There's also
encodings/uu_codec.py which is a codec for the same encoding; it should
also be deprecated.

xdrlib

The xdrlib module supports the Sun External Data Representation
Standard. XDR is an old binary serialization format from 1987. These
days it's rarely used outside specialized domains like NFS.

Multimedia modules

aifc

The aifc module provides support for reading and writing AIFF and AIFF-C
files. The Audio Interchange File Format is an old audio format from
1988 based on Amiga IFF. It was most commonly used on the Apple
Macintosh. These days only few specialized application use AIFF.

A user disclosed[2] that the post production film industry makes heavy
use of the AIFC file format. The usage of the aifc module in closed
source and internal software was unknown prior to the first posting of
this PEP. This may be a compelling argument to keep the aifc module in
the standard library. The file format is stable and the module does not
require much maintenance. The strategic benefits for Python may outmatch
the burden.

audioop

The audioop module contains helper functions to manipulate raw audio
data and adaptive differential pulse-code modulated audio data. The
module is implemented in C without any additional dependencies. The
aifc, sunau, and wave modules depend on audioop for some operations.

The byteswap operation in the wave module can be substituted with little
extra work. In case aifc is not deprecated as well, a reduced version of
the audioop module is converted into a private implementation detail,
e.g. _audioop with byteswap, alaw2lin, ulaw2lin, lin2alaw, lin2ulaw, and
lin2adpcm.

chunk

The chunk module provides support for reading and writing Electronic
Arts' Interchange File Format. IFF is an old audio file format
originally introduced for Commodore and Amiga. The format is no longer
relevant.

imghdr

The imghdr module is a simple tool to guess the image file format from
the first 32 bytes of a file or buffer. It supports only a limited
number of formats and neither returns resolution nor color depth.

ossaudiodev

The ossaudiodev module provides support for Open Sound System, an
interface to sound playback and capture devices. OSS was initially free
software, but later support for newer sound devices and improvements
were proprietary. Linux community abandoned OSS in favor of ALSA[3].
Some operating systems like OpenBSD and NetBSD provide an incomplete[4]
emulation of OSS.

To best of my knowledge, FreeBSD is the only widespread operating system
that uses Open Sound System as of today. The ossaudiodev hasn't seen any
improvements or new features since 2003. All commits since 2003 are
project-wide code cleanups and a couple of bug fixes. It would be
beneficial for both FreeBSD community and core development, if the
module would be maintained and distributed by people that care for it
and use it.

The standard library used to have more audio-related modules. The other
audio device interfaces (audiodev, linuxaudiodev, sunaudiodev) were
removed in 2007 as part of the PEP 3108 stdlib re-organization.

sndhdr

The sndhdr module is similar to the imghdr module but for audio formats.
It guesses file format, channels, frame rate, and sample widths from the
first 512 bytes of a file or buffer. The module only supports AU, AIFF,
HCOM, VOC, WAV, and other ancient formats.

sunau

The sunau module provides support for Sun AU sound format. It's yet
another old, obsolete file format.

Networking modules

asynchat

The asynchat module is built on top of asyncore and has been deprecated
since Python 3.6.

asyncore

The asyncore module was the first module for asynchronous socket service
clients and servers. It has been replaced by asyncio and is deprecated
since Python 3.6.

The asyncore module is also used in stdlib tests. The tests for ftplib,
logging, smptd, smtplib, and ssl are partly based on asyncore. These
tests must be updated to use asyncio or threading.

cgi

The cgi module is a support module for Common Gateway Interface (CGI)
scripts. CGI is deemed as inefficient because every incoming request is
handled in a new process. PEP 206 considers the module as:

  "[...] designed poorly and are now near-impossible to fix (cgi) [...]"

Replacements for the various parts of cgi which are not directly related
to executing code are:

-   parse with urllib.parse.parse_qs (parse is just a wrapper)
-   parse_header with email.message.Message (see example below)
-   parse_multipart with email.message.Message (same MIME RFCs)
-   FieldStorage/MiniFieldStorage has no direct replacement, but can
    typically be replaced by using multipart (for POST and PUT requests)
    or urllib.parse.parse_qsl (for GET and HEAD requests)
-   valid_boundary (undocumented) with re.compile("^[ -~]{0,200}[!-~]$")

As an explicit example of how close parse_header and
email.message.Message are:

    >>> from cgi import parse_header
    >>> from email.message import Message
    >>> parse_header(h)
    ('application/json', {'charset': 'utf8'})
    >>> m = Message()
    >>> m['content-type'] = h
    >>> m.get_params()
    [('application/json', ''), ('charset', 'utf8')]
    >>> m.get_param('charset')
    'utf8'

cgitb

The cgitb module is a helper for the cgi module for configurable
tracebacks.

The cgitb module is not used by any major Python web framework (Django,
Pyramid, Plone, Flask, CherryPy, or Bottle). Only Paste uses it in an
optional debugging middleware.

smtpd

The smtpd module provides a simple implementation of a SMTP mail server.
The module documentation marks the module as deprecated and recommends
aiosmtpd instead. The deprecation message was added in releases 3.4.7,
3.5.4, and 3.6.1.

nntplib

The nntplib module implements the client side of the Network News
Transfer Protocol (nntp). News groups used to be a dominant platform for
online discussions. Over the last two decades, news has been slowly but
steadily replaced with mailing lists and web-based discussion platforms.
Twisted is also planning to deprecate NNTP support and pynntp hasn't
seen any activity since 2014. This is a good indicator that the public
interest in NNTP support is declining.

The nntplib tests have been the cause of additional work in the recent
past. Python only contains the client side of NNTP, so the tests connect
to external news servers. The servers are sometimes unavailable, too
slow, or do not work correctly over IPv6. The situation causes flaky
test runs on buildbots.

telnetlib

The telnetlib module provides a Telnet class that implements the Telnet
protocol.

Operating system interface

crypt

The crypt module implements password hashing based on the crypt(3)
function from libcrypt or libxcrypt on Unix-like platforms. The
algorithms are mostly old, of poor quality and insecure. Users are
discouraged from using them.

-   The module is not available on Windows. Cross-platform applications
    need an alternative implementation anyway.
-   Only DES encryption is guaranteed to be available. DES has an
    extremely limited key space of 2**56.
-   MD5, salted SHA256, salted SHA512, and Blowfish are optional
    extensions. SSHA256 and SSHA512 are glibc extensions. Blowfish
    (bcrypt) is the only algorithm that is still secure. However it's in
    glibc and therefore not commonly available on Linux.
-   Depending on the platform, the crypt module is not thread safe. Only
    implementations with crypt_r(3) are thread safe.
-   The module was never useful to interact with system user and
    password databases. On BSD, macOS, and Linux, all user
    authentication and password modification operations must go through
    PAM (pluggable authentication module); see the spwd deprecation.

nis

The nis module provides NIS/YP support. Network Information Service /
Yellow Pages is an old and deprecated directory service protocol
developed by Sun Microsystems. Its designed successor NIS+ from 1992
never took off. For a long time, libc's Name Service Switch, LDAP, and
Kerberos/GSSAPI have been considered a more powerful and more secure
replacement for NIS.

spwd

The spwd module provides direct access to Unix shadow password database
using non-standard APIs.

In general, it's a bad idea to use spwd. It circumvents system security
policies, does not use the PAM stack, and is only compatible with local
user accounts, because it ignores NSS. The use of the spwd module for
access control must be considered a security bug, as it bypasses PAM's
access control.

Furthermore, the spwd module uses the shadow(3) APIs. Functions like
getspnam(3) access the /etc/shadow file directly. This is dangerous and
even forbidden for confined services on systems with a security engine
like SELinux or AppArmor.

Misc modules

mailcap

The mailcap package reads mail capability files to assist in handling a
file attachment in an email. In most modern operating systems the email
client itself handles reacting to file attachments. Operating systems
also have their own way to register handling files by their file name
extension. Finally, the module has CVE-2015-20107 filed against it while
having no maintainer to help fix it.

msilib

The msilib package is a Windows-only package. It supports the creation
of Microsoft Installers (MSI). The package also exposes additional APIs
to create cabinet files (CAB). The module is used to facilitate
distutils to create MSI installers with the bdist_msi command. In the
past it was used to create CPython's official Windows installer, too.

Microsoft is slowly moving away from MSI in favor of Windows 10 Apps
(AppX) as a new deployment model[5].

pipes

The pipes module provides helpers to pipe the input of one command into
the output of another command. The module is built on top of os.popen.
Users are encouraged to use the subprocess module instead.

Modules to keep

Some modules were originally proposed for deprecation but are no longer
listed as such in this PEP.

  -----------------------------------------------------------------------
  Module            Deprecated in     Replacement
  ----------------- ----------------- -----------------------------------
  colorsys          -                 colormath, colour, colorspacious,
                                      Pillow

  fileinput         -                 argparse

  getopt            -                 argparse, optparse

  optparse          3.2               argparse

  wave              -
  -----------------------------------------------------------------------

  : Table 2: Withdrawn deprecations

colorsys

The colorsys module defines color conversion functions between RGB, YIQ,
HSL, and HSV coordinate systems.

Walter Dörwald, Petr Viktorin, and others requested to keep colorsys.
The module is useful to convert CSS colors between coordinate systems.
The implementation is simple, mature, and does not impose maintenance
overhead on core development.

The PyPI packages colormath, colour, and colorspacious provide more and
advanced features. The Pillow library is better suited to transform
images between color systems.

fileinput

The fileinput module implements helpers to iterate over a list of files
from sys.argv. The module predates the optparse and argparse modules.
The same functionality can be implemented with the argparse module.

Several core developers expressed their interest to keep the module in
the standard library, as it is handy for quick scripts.

getopt

The getopt module mimics C's getopt() option parser.

Although users are encouraged to use argparse instead, the getopt module
is still widely used. The module is small, simple, and handy for C
developers to write simple Python scripts.

optparse

The optparse module is the predecessor of the argparse module.

Although it has been deprecated for many years, it's still too widely
used to remove it.

wave

The wave module provides support for the WAV sound format.

The module is not deprecated, because the WAV format is still relevant
these days. The wave module is also used in education, e.g. to show kids
how to make noise with a computer.

The module uses one simple function from the audioop module to perform
byte swapping between little and big endian formats. Before 24 bit WAV
support was added, byte swap used to be implemented with the array
module. To remove wave's dependency on audioop, the byte swap function
could be either be moved to another module (e.g. operator) or the array
module could gain support for 24-bit (3-byte) arrays.

Discussions

-   Elana Hashman and Alyssa Coghlan suggested to keep the getopt
    module.
-   Berker Peksag proposed to deprecate and remove msilib.
-   Brett Cannon recommended to delay active deprecation warnings and
    removal of modules like imp until Python 3.10. Version 3.8 will be
    released shortly before Python 2 reaches end-of-life. A delay
    reduced churn for users that migrate from Python 2 to 3.8.
-   At one point, distutils was mentioned in the same sentence as this
    PEP. To avoid lengthy discussion and delay of the PEP, I decided
    against dealing with distutils. Deprecation of the distutils package
    will be handled by another PEP.
-   Multiple people (Gregory P. Smith, David Beazley, Alyssa Coghlan,
    ...) convinced me to keep the wave module.[6]
-   Gregory P. Smith proposed to deprecate nntplib.[7]
-   Andrew Svetlov mentioned the socketserver module is questionable.
    However it's used to implement http.server and xmlrpc.server. The
    stdlib doesn't have a replacement for the servers, yet.

Rejected ideas

Creating/maintaining a separate repo for the deprecated modules

It was previously proposed to create a separate repository containing
the deprecated modules packaged for installation. One of the PEP authors
went so far as to create a demo repository. In the end, though, it was
decided that the added workload to create and maintain such a repo
officially wasn't justified, as the source code will continue to be
available in the CPython repository for people to vendor as necessary.
Similar work has also not been done when previous modules were
deprecated and removed, and it seemingly wasn't an undue burden on the
community.

Update history

Update 1

-   Deprecate parser module
-   Keep fileinput module
-   Elaborate why crypt and spwd are dangerous and bad
-   Improve sections for cgitb, colorsys, nntplib, and smtpd modules
-   The colorsys, crypt, imghdr, sndhdr, and spwd sections now list
    suitable substitutions
-   Mention that socketserver is going to stay for http.server and
    xmlrpc.server
-   The future maintenance section now states that the deprecated
    modules may be adopted by Python community members

Update 2

-   Keep colorsys module
-   Add experts
-   Redirect discussions to discuss.python.org
-   Deprecate telnetlib
-   Deprecate compat32 policy of email package
-   Add creation year to overview table
-   Mention PEP 206 and PEP 3108
-   Update sections for aifc, audioop, cgi, and wave.

Update 3

-   Keep the legacy email API modules. Internal deprecations will be
    handled separately.

Update 4

-   Add Brett as a co-author.
-   Retarget the PEP for Python 3.11.
-   Examples of how to replace the relevant parts of cgi (thanks Martijn
    Pieters).

References

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

[1] https://web.archive.org/web/20220420154535/https://twitter.com/dabeaz/status/1130278844479545351

[2] https://mail.python.org/pipermail/python-dev/2019-May/157634.html

[3] https://en.wikipedia.org/wiki/Open_Sound_System#History

[4] https://man.openbsd.org/ossaudio

[5] https://web.archive.org/web/20221206204209/https://blogs.msmvps.com/installsite/blog/2015/05/03/the-future-of-windows-installer-msi-in-the-light-of-windows-10-and-the-universal-windows-platform/

[6] https://twitter.com/ChristianHeimes/status/1130257799475335169

[7] https://twitter.com/ChristianHeimes/status/1130257799475335169


text:
PEP: 8000 Title: Python Language Governance Proposal Overview Author:
Barry Warsaw<EMAIL_ADDRESS>Status: Final Type: Informational Topic:
Governance Content-Type: text/x-rst Created: 24-Aug-2018

Abstract

This PEP provides an overview of the selection process for a new model
of Python language governance in the wake of Guido's retirement. Once
the governance model is selected, it will be codified in PEP 13.

Here is a list of PEPs related to the governance model selection
process. PEPs in the lower 8000s describe the general process for
selecting a governance model.

-   PEP 8001 - Python Governance Voting Process

    This PEP describes how the vote for the new governance model will be
    conducted. It outlines the voting method, timeline, criteria for
    participation, and explicit list of eligible voters.

-   PEP 8002 - Open Source Governance Survey

    Surveys will be conducted of governance models for similar open
    source and free software projects, and summaries of these models
    will be outlined in this PEP. These surveys will serve as useful
    barometers for how such projects can be successfully governed, and
    may serve as inspiration for Python's own governance model. Python
    is unique, so it's expected that it will have its own spin on
    governance, rather than directly adopting any of those surveyed.

PEPs in the 801Xs describe the actual proposals for Python governance.
It is expected that these PEPs will cover the broad scope of governance,
and that differences in details (such as the size of a governing
council) will be covered in the same PEP, rather than in potentially
vote-splitting individual PEPs.

-   PEP 8010 - The Technical Leader Governance Model

    This PEP proposes a continuation of the singular technical project
    leader model. Also within scope is whether an advisory council aids
    or supports the BDFL. This PEP does not name either the next BDFL,
    nor members of such an advisory council. For that, see PEP 13.

-   PEP 8011 - Python Governance Model Lead by Trio of Pythonistas

    This PEP describes a new model of Python governance lead by a Trio
    of Pythonistas (TOP). It describes the role and responsibilities of
    the Trio. This PEP does not name members of the Trio. For that, see
    PEP 13.

-   PEP 8012 - The Community Governance Model

    This is a placeholder PEP for a new model of Python governance based
    on consensus and voting, without the role of a centralized singular
    leader or a governing council. It describes how, when, and why votes
    are conducted for decisions affecting the Python language. It also
    describes the criteria for voting eligibility.

-   PEP 8013 - The External Governance Model

    This PEP describes a new model of Python governance based on an
    external council who are responsible for ensuring good process.
    Elected by the core development team, this council may reject
    proposals that are not sufficiently detailed, do not consider all
    affected users, or are not appropriate for the upcoming release.
    This PEP does not name members of such a council. For that, see PEP
    13.

-   PEP 8014 - The Commons Governance Model

    This PEP describes a new model of Python governance based on a
    council of elders who are responsible for ensuring a PEP is
    supported by a sufficient majority of the Python community before
    being accepted. Unlike some of the other governance PEPs it
    explicitly does not specify who has voting rights and what a
    majority vote consists of. In stead this is determined by the
    council of elders on a case-by-case basis.

-   PEP 8015 - Organization of the Python community

    This PEP formalizes the current organization of the Python community
    and proposes 3 main changes: formalize the existing concept of
    "Python teams"; give more autonomy to Python teams; replace the BDFL
    (Guido van Rossum) with a new "Python board" of 3 members which has
    limited roles, mostly decide how a PEP is approved (or rejected).

-   PEP 8016 - The Steering Council Model

    This PEP proposes a model of Python governance based around a
    steering council. The council has broad authority, which they seek
    to exercise as rarely as possible; instead, they use this power to
    establish standard processes, like those proposed in the other
    801x-series PEPs. This follows the general philosophy that it's
    better to split up large changes into a series of small changes that
    can be reviewed independently: instead of trying to do everything in
    one PEP, we focus on providing a minimal-but-solid foundation for
    further governance decisions.

Additional governance models may be added before the final selection.

Copyright

This document has been placed in the public domain.

license: Public Domain


text:
PEP: 592 Title: Adding "Yank" Support to the Simple API Author: Donald
Stufft<EMAIL_ADDRESS>BDFL-Delegate: Paul Moore
<EMAIL_ADDRESS>Discussions-To: https://discuss.python.org/t/1629
Status: Final Type: Standards Track Topic: Packaging Content-Type:
text/x-rst Created: 07-May-2019 Resolution:
https://discuss.python.org/t/pep-592-support-for-yanked-files-in-the-simple-repository-api/1629/30

Abstract

This PEP proposes adding the ability to mark a particular file download
on a simple repository as "yanked". Yanking a file allows authors to
effectively delete a file, without breaking things for people who have
pinned to exactly a specific version.

It also changes to the canonical source for the simple repository API to
the Simple Repository API reference document.

Motivation

Whenever a project detects that a particular release on PyPI might be
broken, they oftentimes will want to prevent further users from
inadvertently using that version. However, the obvious solution of
deleting the existing file from a repository will break users who have
followed best practices and pinned to a specific version of the project.

This leaves projects in a catch-22 situation where new projects may be
pulling down this known broken version, but if they do anything to
prevent that they'll break projects that are already using it.

By allowing the ability to "yank" a file, but still make it available
for users who are explicitly asking for it, this allows projects to
mitigate the worst of the breakage while still keeping things working
for projects who have otherwise worked around or didn't hit the
underlying issues.

One of the main scenarios where this may happen, is when dropping
support for a particular version of Python. The python-requires metadata
allows for dropping support for a version of Python in a way that is not
disruptive to users who are still using that Python. However, a common
mistake is to either omit or forget to update that bit of metadata. When
that mistake has been made, a project really only has three options:

-   Prevent that version from being installed through some mechanism
    (currently, the only mechanism is by deleting the release entirely).
-   Re-release the version that worked as a higher version number, and
    then re-release the version that dropped support as an even higher
    version number with the correct metadata.
-   Do nothing, and document that people using that older Python have to
    manually exclude that release.

With this PEP, projects can choose the first option, but with a
mechanism that is less likely to break the world for people who are
currently successfully using said project.

Specification

Links in the simple repository MAY have a data-yanked attribute which
may have no value, or may have an arbitrary string as a value. The
presence of a data-yanked attribute SHOULD be interpreted as indicating
that the file pointed to by this particular link has been "Yanked", and
should not generally be selected by an installer, except under specific
scenarios.

The value of the data-yanked attribute, if present, is an arbitrary
string that represents the reason for why the file has been yanked.
Tools that process the simple repository API MAY surface this string to
end users.

The yanked attribute is not immutable once set, and may be rescinded in
the future (and once rescinded, may be reset as well). Thus API users
MUST be able to cope with a yanked file being "unyanked" (and even
yanked again).

Installers

The desirable experience for users is that once a file is yanked, when a
human being is currently trying to directly install a yanked file, that
it fails as if that file had been deleted. However, when a human did
that a while ago, and now a computer is just continuing to mechanically
follow the original order to install the now yanked file, then it acts
as if it had not been yanked.

An installer MUST ignore yanked releases, if the selection constraints
can be satisfied with a non-yanked version, and MAY refuse to use a
yanked release even if it means that the request cannot be satisfied at
all. An implementation SHOULD choose a policy that follows the spirit of
the intention above, and that prevents "new" dependencies on yanked
releases/files.

What this means is left up to the specific installer, to decide how to
best fit into the overall usage of their installer. However, there are
two suggested approaches to take:

1.  Yanked files are always ignored, unless they are the only file that
    matches a version specifier that "pins" to an exact version using
    either == (without any modifiers that make it a range, such as .*)
    or ===. Matching this version specifier should otherwise be done as
    per PEP 440 for things like local versions, zero padding, etc.
2.  Yanked files are always ignored, unless they are the only file that
    matches what a lock file (such as Pipfile.lock or poetry.lock)
    specifies to be installed. In this case, a yanked file SHOULD not be
    used when creating or updating a lock file from some input file or
    command.

Regardless of the specific strategy that an installer chooses for
deciding when to install yanked files, an installer SHOULD emit a
warning when it does decide to install a yanked file. That warning MAY
utilize the value of the data-yanked attribute (if it has a value) to
provide more specific feedback to the user about why that file had been
yanked.

Mirrors

Mirrors can generally treat yanked files one of two ways:

1.  They may choose to omit them from their simple repository API
    completely, providing a view over the repository that shows only
    "active", unyanked files.
2.  They may choose to include yanked files, and additionally mirror the
    data-yanked attribute as well.

Mirrors MUST NOT mirror a yanked file without also mirroring the
data-yanked attribute for it.

Rejected Ideas

A previous, undocumented, version of the simple repository API had
version specific pages, like /simple/<project>/<version>/. If we were to
add those back, the yanked files could only appear on those pages and
not on the version-less page at all. However this would drastically
reduce the cache-ability of the simple API and would directly impact our
ability to scale it out to handle all of the incoming traffic.

A previous iteration of this PEP had the data-yanked attribute act as a
boolean value. However it was decided that allowing a string both
simplified the implementation, and provided additional generalized
functionality to allow projects to provide a mechanism to indicate why
they were yanking a release.

Another suggestion was to reserve some syntax in the arbitrary string to
allow us to evolve the standard in the future if we ever need to.
However, given we can add additional attributes in the future, this idea
has been rejected, favoring instead to use additional attributes if the
need ever arose.

Warehouse/PyPI Implementation Notes

While this PEP implements yanking at the file level, that is largely due
to the shape the simple repository API takes, not a specific decision
made by this PEP.

In Warehouse, the user experience will be implemented in terms of
yanking or unyanking an entire release, rather than as an operation on
individual files, which will then be exposed via the API as individual
files being yanked.

Other repository implementations may choose to expose this capability in
a different way, or not expose it at all.

Journal Handling

Whenever a release has been yanked, an entry will be recorded in the
journal using one of the following string patterns:

-   yank release
-   unyank release

In both cases, the standard journal structure will indicate which
release of which project has been yanked or unyanked.

Copyright

This document has been placed in the public domain.


text:
PEP: 346 Title: User Defined ("with") Statements Version: $Revision$
Last-Modified: $Date$ Author: Alyssa Coghlan<EMAIL_ADDRESS>Status: Withdrawn Type: Standards Track Content-Type: text/x-rst
Created: 06-May-2005 Python-Version: 2.5 Post-History:

Abstract

This PEP is a combination of PEP 310's "Reliable Acquisition/Release
Pairs" with the "Anonymous Block Statements" of Guido's PEP 340. This
PEP aims to take the good parts of PEP 340, blend them with parts of PEP
310 and rearrange the lot into an elegant whole. It borrows from various
other PEPs in order to paint a complete picture, and is intended to
stand on its own.

Author's Note

During the discussion of PEP 340, I maintained drafts of this PEP as PEP
3XX on my own website (since I didn't have CVS access to update a
submitted PEP fast enough to track the activity on python-dev).

Since the first draft of this PEP, Guido wrote PEP 343 as a simplified
version of PEP 340. PEP 343 (at the time of writing) uses the exact same
semantics for the new statements as this PEP, but uses a slightly
different mechanism to allow generators to be used to write statement
templates. However, Guido has indicated that he intends to accept a new
PEP being written by Raymond Hettinger that will integrate PEP 288 and
PEP 325, and will permit a generator decorator like the one described in
this PEP to be used to write statement templates for PEP 343. The other
difference was the choice of keyword ('with' versus 'do') and Guido has
stated he will organise a vote on that in the context of PEP 343.

Accordingly, the version of this PEP submitted for archiving on
python.org is to be WITHDRAWN immediately after submission. PEP 343 and
the combined generator enhancement PEP will cover the important ideas.

Introduction

This PEP proposes that Python's ability to reliably manage resources be
enhanced by the introduction of a new with statement that allows
factoring out of arbitrary try/finally and some try/except/else
boilerplate. The new construct is called a 'user defined statement', and
the associated class definitions are called 'statement templates'.

The above is the main point of the PEP. However, if that was all it
said, then PEP 310 would be sufficient and this PEP would be essentially
redundant. Instead, this PEP recommends additional enhancements that
make it natural to write these statement templates using appropriately
decorated generators. A side effect of those enhancements is that it
becomes important to appropriately deal with the management of resources
inside generators.

This is quite similar to PEP 343, but the exceptions that occur are
re-raised inside the generators frame, and the issue of generator
finalisation needs to be addressed as a result. The template generator
decorator suggested by this PEP also creates reusable templates, rather
than the single use templates of PEP 340.

In comparison to PEP 340, this PEP eliminates the ability to suppress
exceptions, and makes the user defined statement a non-looping
construct. The other main difference is the use of a decorator to turn
generators into statement templates, and the incorporation of ideas for
addressing iterator finalisation.

If all that seems like an ambitious operation. . . well, Guido was the
one to set the bar that high when he wrote PEP 340 :)

Relationship with other PEPs

This PEP competes directly with PEP 310, PEP 340 and PEP 343, as those
PEPs all describe alternative mechanisms for handling deterministic
resource management.

It does not compete with PEP 342 which splits off PEP 340's enhancements
related to passing data into iterators. The associated changes to the
for loop semantics would be combined with the iterator finalisation
changes suggested in this PEP. User defined statements would not be
affected.

Neither does this PEP compete with the generator enhancements described
in PEP 288. While this PEP proposes the ability to inject exceptions
into generator frames, it is an internal implementation detail, and does
not require making that ability publicly available to Python code. PEP
288 is, in part, about making that implementation detail easily
accessible.

This PEP would, however, make the generator resource release support
described in PEP 325 redundant - iterators which require finalisation
should provide an appropriate implementation of the statement template
protocol.

User defined statements

To steal the motivating example from PEP 310, correct handling of a
synchronisation lock currently looks like this:

    the_lock.acquire()
    try:
        # Code here executes with the lock held
    finally:
        the_lock.release()

Like PEP 310, this PEP proposes that such code be able to be written as:

    with the_lock:
        # Code here executes with the lock held

These user defined statements are primarily designed to allow easy
factoring of try blocks that are not easily converted to functions. This
is most commonly the case when the exception handling pattern is
consistent, but the body of the try block changes. With a user-defined
statement, it is straightforward to factor out the exception handling
into a statement template, with the body of the try clause provided
inline in the user code.

The term 'user defined statement' reflects the fact that the meaning of
a with statement is governed primarily by the statement template used,
and programmers are free to create their own statement templates, just
as they are free to create their own iterators for use in for loops.

Usage syntax for user defined statements

The proposed syntax is simple:

    with EXPR1 [as VAR1]:
        BLOCK1

Semantics for user defined statements

    the_stmt = EXPR1
    stmt_enter = getattr(the_stmt, "__enter__", None)
    stmt_exit = getattr(the_stmt, "__exit__", None)
    if stmt_enter is None or stmt_exit is None:
        raise TypeError("Statement template required")

    VAR1 = stmt_enter() # Omit 'VAR1 =' if no 'as' clause
    exc = (None, None, None)
    try:
        try:
            BLOCK1
        except:
            exc = sys.exc_info()
            raise
    finally:
        stmt_exit(*exc)

Other than VAR1, none of the local variables shown above will be visible
to user code. Like the iteration variable in a for loop, VAR1 is visible
in both BLOCK1 and code following the user defined statement.

Note that the statement template can only react to exceptions, it cannot
suppress them. See Rejected Options for an explanation as to why.

Statement template protocol: __enter__

The __enter__() method takes no arguments, and if it raises an
exception, BLOCK1 is never executed. If this happens, the __exit__()
method is not called. The value returned by this method is assigned to
VAR1 if the as clause is used. Object's with no other value to return
should generally return self rather than None to permit in-place
creation in the with statement.

Statement templates should use this method to set up the conditions that
are to exist during execution of the statement (e.g. acquisition of a
synchronisation lock).

Statement templates which are not always usable (e.g. closed file
objects) should raise a RuntimeError if an attempt is made to call
__enter__() when the template is not in a valid state.

Statement template protocol: __exit__

The __exit__() method accepts three arguments which correspond to the
three "arguments" to the raise statement: type, value, and traceback.
All arguments are always supplied, and will be set to None if no
exception occurred. This method will be called exactly once by the with
statement machinery if the __enter__() method completes successfully.

Statement templates perform their exception handling in this method. If
the first argument is None, it indicates non-exceptional completion of
BLOCK1 - execution either reached the end of block, or early completion
was forced using a return, break or continue statement. Otherwise, the
three arguments reflect the exception that terminated BLOCK1.

Any exceptions raised by the __exit__() method are propagated to the
scope containing the with statement. If the user code in BLOCK1 also
raised an exception, that exception would be lost, and replaced by the
one raised by the __exit__() method.

Factoring out arbitrary exception handling

Consider the following exception handling arrangement:

    SETUP_BLOCK
    try:
        try:
            TRY_BLOCK
        except exc_type1, exc:
            EXCEPT_BLOCK1
        except exc_type2, exc:
            EXCEPT_BLOCK2
        except:
            EXCEPT_BLOCK3
        else:
            ELSE_BLOCK
    finally:
        FINALLY_BLOCK

It can be roughly translated to a statement template as follows:

    class my_template(object):

        def __init__(self, *args):
            # Any required arguments (e.g. a file name)
            # get stored in member variables
            # The various BLOCK's will need updating to reflect
            # that.

        def __enter__(self):
            SETUP_BLOCK

        def __exit__(self, exc_type, value, traceback):
            try:
                try:
                    if exc_type is not None:
                        raise exc_type, value, traceback
                except exc_type1, exc:
                    EXCEPT_BLOCK1
                except exc_type2, exc:
                    EXCEPT_BLOCK2
                except:
                    EXCEPT_BLOCK3
                else:
                    ELSE_BLOCK
            finally:
                FINALLY_BLOCK

Which can then be used as:

    with my_template(*args):
        TRY_BLOCK

However, there are two important semantic differences between this code
and the original try statement.

Firstly, in the original try statement, if a break, return or continue
statement is encountered in TRY_BLOCK, only FINALLY_BLOCK will be
executed as the statement completes. With the statement template,
ELSE_BLOCK will also execute, as these statements are treated like any
other non-exceptional block termination. For use cases where it matters,
this is likely to be a good thing (see transaction in the Examples), as
this hole where neither the except nor the else clause gets executed is
easy to forget when writing exception handlers.

Secondly, the statement template will not suppress any exceptions. If,
for example, the original code suppressed the exc_type1 and exc_type2
exceptions, then this would still need to be done inline in the user
code:

    try:
        with my_template(*args):
            TRY_BLOCK
    except (exc_type1, exc_type2):
        pass

However, even in these cases where the suppression of exceptions needs
to be made explicit, the amount of boilerplate repeated at the calling
site is significantly reduced (See Rejected Options for further
discussion of this behaviour).

In general, not all of the clauses will be needed. For resource handling
(like files or synchronisation locks), it is possible to simply execute
the code that would have been part of FINALLY_BLOCK in the __exit__()
method. This can be seen in the following implementation that makes
synchronisation locks into statement templates as mentioned at the
beginning of this section:

    # New methods of synchronisation lock objects

    def __enter__(self):
        self.acquire()
        return self

    def __exit__(self, *exc_info):
        self.release()

Generators

With their ability to suspend execution, and return control to the
calling frame, generators are natural candidates for writing statement
templates. Adding user defined statements to the language does not
require the generator changes described in this section, thus making
this PEP an obvious candidate for a phased implementation (with
statements in phase 1, generator integration in phase 2). The suggested
generator updates allow arbitrary exception handling to be factored out
like this:

    @statement_template
    def my_template(*arguments):
        SETUP_BLOCK
        try:
            try:
                yield
            except exc_type1, exc:
                EXCEPT_BLOCK1
            except exc_type2, exc:
                EXCEPT_BLOCK2
            except:
                EXCEPT_BLOCK3
            else:
                ELSE_BLOCK
        finally:
            FINALLY_BLOCK

Notice that, unlike the class based version, none of the blocks need to
be modified, as shared values are local variables of the generator's
internal frame, including the arguments passed in by the invoking code.
The semantic differences noted earlier (all non-exceptional block
termination triggers the else clause, and the template is unable to
suppress exceptions) still apply.

Default value for yield

When creating a statement template with a generator, the yield statement
will often be used solely to return control to the body of the user
defined statement, rather than to return a useful value.

Accordingly, if this PEP is accepted, yield, like return, will supply a
default value of None (i.e. yield and yield None will become equivalent
statements).

This same change is being suggested in PEP 342. Obviously, it would only
need to be implemented once if both PEPs were accepted :)

Template generator decorator: statement_template

As with PEP 343, a new decorator is suggested that wraps a generator in
an object with the appropriate statement template semantics. Unlike PEP
343, the templates suggested here are reusable, as the generator is
instantiated anew in each call to __enter__(). Additionally, any
exceptions that occur in BLOCK1 are re-raised in the generator's
internal frame:

    class template_generator_wrapper(object):

        def __init__(self, func, func_args, func_kwds):
             self.func = func
             self.args = func_args
             self.kwds = func_kwds
             self.gen = None

        def __enter__(self):
            if self.gen is not None:
                raise RuntimeError("Enter called without exit!")
            self.gen = self.func(*self.args, **self.kwds)
            try:
                return self.gen.next()
            except StopIteration:
                raise RuntimeError("Generator didn't yield")

        def __exit__(self, *exc_info):
            if self.gen is None:
                raise RuntimeError("Exit called without enter!")
            try:
                try:
                    if exc_info[0] is not None:
                        self.gen._inject_exception(*exc_info)
                    else:
                        self.gen.next()
                except StopIteration:
                    pass
                else:
                    raise RuntimeError("Generator didn't stop")
            finally:
                self.gen = None

    def statement_template(func):
        def factory(*args, **kwds):
            return template_generator_wrapper(func, args, kwds)
        return factory

Template generator wrapper: __enter__() method

The template generator wrapper has an __enter__() method that creates a
new instance of the contained generator, and then invokes next() once.
It will raise a RuntimeError if the last generator instance has not been
cleaned up, or if the generator terminates instead of yielding a value.

Template generator wrapper: __exit__() method

The template generator wrapper has an __exit__() method that simply
invokes next() on the generator if no exception is passed in. If an
exception is passed in, it is re-raised in the contained generator at
the point of the last yield statement.

In either case, the generator wrapper will raise a RuntimeError if the
internal frame does not terminate as a result of the operation. The
__exit__() method will always clean up the reference to the used
generator instance, permitting __enter__() to be called again.

A StopIteration raised by the body of the user defined statement may be
inadvertently suppressed inside the __exit__() method, but this is
unimportant, as the originally raised exception still propagates
correctly.

Injecting exceptions into generators

To implement the __exit__() method of the template generator wrapper, it
is necessary to inject exceptions into the internal frame of the
generator. This is new implementation level behaviour that has no
current Python equivalent.

The injection mechanism (referred to as _inject_exception in this PEP)
raises an exception in the generator's frame with the specified type,
value and traceback information. This means that the exception looks
like the original if it is allowed to propagate.

For the purposes of this PEP, there is no need to make this capability
available outside the Python implementation code.

Generator finalisation

To support resource management in template generators, this PEP will
eliminate the restriction on yield statements inside the try block of a
try/finally statement. Accordingly, generators which require the use of
a file or some such object can ensure the object is managed correctly
through the use of try/finally or with statements.

This restriction will likely need to be lifted globally - it would be
difficult to restrict it so that it was only permitted inside generators
used to define statement templates. Accordingly, this PEP includes
suggestions designed to ensure generators which are not used as
statement templates are still finalised appropriately.

Generator finalisation: TerminateIteration exception

A new exception is proposed:

    class TerminateIteration(Exception): pass

The new exception is injected into a generator in order to request
finalisation. It should not be suppressed by well-behaved code.

Generator finalisation: __del__() method

To ensure a generator is finalised eventually (within the limits of
Python's garbage collection), generators will acquire a __del__() method
with the following semantics:

    def __del__(self):
        try:
            self._inject_exception(TerminateIteration, None, None)
        except TerminateIteration:
            pass

Deterministic generator finalisation

There is a simple way to provide deterministic finalisation of
generators - give them appropriate __enter__() and __exit__() methods:

    def __enter__(self):
        return self

    def __exit__(self, *exc_info):
        try:
            self._inject_exception(TerminateIteration, None, None)
        except TerminateIteration:
            pass

Then any generator can be finalised promptly by wrapping the relevant
for loop inside a with statement:

    with all_lines(filenames) as lines:
        for line in lines:
            print lines

(See the Examples for the definition of all_lines, and the reason it
requires prompt finalisation)

Compare the above example to the usage of file objects:

    with open(filename) as f:
        for line in f:
            print f

Generators as user defined statement templates

When used to implement a user defined statement, a generator should
yield only once on a given control path. The result of that yield will
then be provided as the result of the generator's __enter__() method.
Having a single yield on each control path ensures that the internal
frame will terminate when the generator's __exit__() method is called.
Multiple yield statements on a single control path will result in a
RuntimeError being raised by the __exit__() method when the internal
frame fails to terminate correctly. Such an error indicates a bug in the
statement template.

To respond to exceptions, or to clean up resources, it is sufficient to
wrap the yield statement in an appropriately constructed try statement.
If execution resumes after the yield without an exception, the generator
knows that the body of the do statement completed without incident.

Examples

1.  A template for ensuring that a lock, acquired at the start of a
    block, is released when the block is left:

        # New methods on synchronisation locks
            def __enter__(self):
                self.acquire()
                return self

            def __exit__(self, *exc_info):
                lock.release()

    Used as follows:

        with myLock:
            # Code here executes with myLock held.  The lock is
            # guaranteed to be released when the block is left (even
            # if via return or by an uncaught exception).

2.  A template for opening a file that ensures the file is closed when
    the block is left:

        # New methods on file objects
            def __enter__(self):
                if self.closed:
                    raise RuntimeError, "Cannot reopen closed file handle"
                return self

            def __exit__(self, *args):
                self.close()

    Used as follows:

        with open("/etc/passwd") as f:
            for line in f:
                print line.rstrip()

3.  A template for committing or rolling back a database transaction:

        def transaction(db):
            try:
                yield
            except:
                db.rollback()
            else:
                db.commit()

    Used as follows:

        with transaction(the_db):
            make_table(the_db)
            add_data(the_db)
            # Getting to here automatically triggers a commit
            # Any exception automatically triggers a rollback

4.  It is possible to nest blocks and combine templates:

        @statement_template
        def lock_opening(lock, filename, mode="r"):
            with lock:
                with open(filename, mode) as f:
                    yield f

    Used as follows:

        with lock_opening(myLock, "/etc/passwd") as f:
            for line in f:
                print line.rstrip()

5.  Redirect stdout temporarily:

        @statement_template
        def redirected_stdout(new_stdout):
            save_stdout = sys.stdout
            try:
                sys.stdout = new_stdout
                yield
            finally:
                sys.stdout = save_stdout

    Used as follows:

        with open(filename, "w") as f:
            with redirected_stdout(f):
                print "Hello world"

6.  A variant on open() that also returns an error condition:

        @statement_template
        def open_w_error(filename, mode="r"):
            try:
                f = open(filename, mode)
            except IOError, err:
                yield None, err
            else:
                try:
                    yield f, None
                finally:
                    f.close()

    Used as follows:

        do open_w_error("/etc/passwd", "a") as f, err:
            if err:
                print "IOError:", err
            else:
                f.write("guido::0:0::/:/bin/sh\n")

7.  Find the first file with a specific header:

        for name in filenames:
            with open(name) as f:
                if f.read(2) == 0xFEB0:
                    break

8.  Find the first item you can handle, holding a lock for the entire
    loop, or just for each iteration:

        with lock:
            for item in items:
                if handle(item):
                    break

        for item in items:
            with lock:
                if handle(item):
                    break

9.  Hold a lock while inside a generator, but release it when returning
    control to the outer scope:

        @statement_template
        def released(lock):
            lock.release()
            try:
                yield
            finally:
                lock.acquire()

    Used as follows:

        with lock:
            for item in items:
                with released(lock):
                    yield item

10. Read the lines from a collection of files (e.g. processing multiple
    configuration sources):

        def all_lines(filenames):
            for name in filenames:
                with open(name) as f:
                    for line in f:
                        yield line

    Used as follows:

        with all_lines(filenames) as lines:
            for line in lines:
                update_config(line)

11. Not all uses need to involve resource management:

        @statement_template
        def tag(*args, **kwds):
            name = cgi.escape(args[0])
            if kwds:
                kwd_pairs = ["%s=%s" % cgi.escape(key), cgi.escape(value)
                             for key, value in kwds]
                print '<%s %s>' % name, " ".join(kwd_pairs)
            else:
                print '<%s>' % name
            yield
            print '</%s>' % name

    Used as follows:

        with tag('html'):
            with tag('head'):
               with tag('title'):
                  print 'A web page'
            with tag('body'):
               for par in pars:
                  with tag('p'):
                     print par
               with tag('a', href="http://www.python.org"):
                   print "Not a dead parrot!"

12. From PEP 343, another useful example would be an operation that
    blocks signals. The use could be like this:

        from signal import blocked_signals

        with blocked_signals():
            # code executed without worrying about signals

    An optional argument might be a list of signals to be blocked; by
    default all signals are blocked. The implementation is left as an
    exercise to the reader.

13. Another use for this feature is for Decimal contexts:

        # New methods on decimal Context objects

        def __enter__(self):
            if self._old_context is not None:
                raise RuntimeError("Already suspending other Context")
            self._old_context = getcontext()
            setcontext(self)

        def __exit__(self, *args):
            setcontext(self._old_context)
            self._old_context = None

    Used as follows:

        with decimal.Context(precision=28):
           # Code here executes with the given context
           # The context always reverts after this statement

Open Issues

None, as this PEP has been withdrawn.

Rejected Options

Having the basic construct be a looping construct

The major issue with this idea, as illustrated by PEP 340's block
statements, is that it causes problems with factoring try statements
that are inside loops, and contain break and continue statements (as
these statements would then apply to the block construct, instead of the
original loop). As a key goal is to be able to factor out arbitrary
exception handling (other than suppression) into statement templates,
this is a definite problem.

There is also an understandability problem, as can be seen in the
Examples. In the example showing acquisition of a lock either for an
entire loop, or for each iteration of the loop, if the user defined
statement was itself a loop, moving it from outside the for loop to
inside the for loop would have major semantic implications, beyond those
one would expect.

Finally, with a looping construct, there are significant problems with
TOOWTDI, as it is frequently unclear whether a particular situation
should be handled with a conventional for loop or the new looping
construct. With the current PEP, there is no such problem - for loops
continue to be used for iteration, and the new do statements are used to
factor out exception handling.

Another issue, specifically with PEP 340's anonymous block statements,
is that they make it quite difficult to write statement templates
directly (i.e. not using a generator). This problem is addressed by the
current proposal, as can be seen by the relative simplicity of the
various class based implementations of statement templates in the
Examples.

Allowing statement templates to suppress exceptions

Earlier versions of this PEP gave statement templates the ability to
suppress exceptions. The BDFL expressed concern over the associated
complexity, and I agreed after reading an article by Raymond Chen about
the evils of hiding flow control inside macros in C code[1].

Removing the suppression ability eliminated a whole lot of complexity
from both the explanation and implementation of user defined statements,
further supporting it as the correct choice. Older versions of the PEP
had to jump through some horrible hoops to avoid inadvertently
suppressing exceptions in __exit__() methods - that issue does not exist
with the current suggested semantics.

There was one example (auto_retry) that actually used the ability to
suppress exceptions. This use case, while not quite as elegant, has
significantly more obvious control flow when written out in full in the
user code:

    def attempts(num_tries):
        return reversed(xrange(num_tries))

    for retry in attempts(3):
        try:
            make_attempt()
        except IOError:
            if not retry:
                raise

For what it's worth, the perverse could still write this as:

    for attempt in auto_retry(3, IOError):
        try:
            with attempt:
                make_attempt()
        except FailedAttempt:
            pass

To protect the innocent, the code to actually support that is not
included here.

Differentiating between non-exceptional exits

Earlier versions of this PEP allowed statement templates to distinguish
between exiting the block normally, and exiting via a return, break or
continue statement. The BDFL flirted with a similar idea in PEP 343 and
its associated discussion. This added significant complexity to the
description of the semantics, and it required each and every statement
template to decide whether or not those statements should be treated
like exceptions, or like a normal mechanism for exiting the block.

This template-by-template decision process raised great potential for
confusion - consider if one database connector provided a transaction
template that treated early exits like an exception, whereas a second
connector treated them as normal block termination.

Accordingly, this PEP now uses the simplest solution - early exits
appear identical to normal block termination as far as the statement
template is concerned.

Not injecting raised exceptions into generators

PEP 343 suggests simply invoking next() unconditionally on generators
used to define statement templates. This means the template generators
end up looking rather unintuitive, and the retention of the ban against
yielding inside try/finally means that Python's exception handling
capabilities cannot be used to deal with management of multiple
resources.

The alternative which this PEP advocates (injecting raised exceptions
into the generator frame), means that multiple resources can be managed
elegantly as shown by lock_opening in the Examples

Making all generators statement templates

Separating the template object from the generator itself makes it
possible to have reusable generator templates. That is, the following
code will work correctly if this PEP is accepted:

    open_it = lock_opening(parrot_lock, "dead_parrot.txt")

    with open_it as f:
        # use the file for a while

    with open_it as f:
        # use the file again

The second benefit is that iterator generators and template generators
are very different things - the decorator keeps that distinction clear,
and prevents one being used where the other is required.

Finally, requiring the decorator allows the native methods of generator
objects to be used to implement generator finalisation.

Using do as the keyword

do was an alternative keyword proposed during the PEP 340 discussion. It
reads well with appropriately named functions, but it reads poorly when
used with methods, or with objects that provide native statement
template support.

When do was first suggested, the BDFL had rejected PEP 310's with
keyword, based on a desire to use it for a Pascal/Delphi style with
statement. Since then, the BDFL has retracted this objection, as he no
longer intends to provide such a statement. This change of heart was
apparently based on the C# developers reasons for not providing the
feature[2].

Not having a keyword

This is an interesting option, and can be made to read quite well.
However, it's awkward to look up in the documentation for new users, and
strikes some as being too magical. Accordingly, this PEP goes with a
keyword based suggestion.

Enhancing try statements

This suggestion involves give bare try statements a signature similar to
that proposed for with statements.

I think that trying to write a with statement as an enhanced try
statement makes as much sense as trying to write a for loop as an
enhanced while loop. That is, while the semantics of the former can be
explained as a particular way of using the latter, the former is not an
instance of the latter. The additional semantics added around the more
fundamental statement result in a new construct, and the two different
statements shouldn't be confused.

This can be seen by the fact that the 'enhanced' try statement still
needs to be explained in terms of a 'non-enhanced' try statement. If
it's something different, it makes more sense to give it a different
name.

Having the template protocol directly reflect try statements

One suggestion was to have separate methods in the protocol to cover
different parts of the structure of a generalised try statement. Using
the terms try, except, else and finally, we would have something like:

    class my_template(object):

        def __init__(self, *args):
            # Any required arguments (e.g. a file name)
            # get stored in member variables
            # The various BLOCK's will need to updated to reflect
            # that.

        def __try__(self):
            SETUP_BLOCK

        def __except__(self, exc, value, traceback):
            if isinstance(exc, exc_type1):
                EXCEPT_BLOCK1
            if isinstance(exc, exc_type2):
                EXCEPT_BLOCK2
            else:
                EXCEPT_BLOCK3

        def __else__(self):
            ELSE_BLOCK

        def __finally__(self):
            FINALLY_BLOCK

Aside from preferring the addition of two method slots rather than four,
I consider it significantly easier to be able to simply reproduce a
slightly modified version of the original try statement code in the
__exit__() method (as shown in Factoring out arbitrary exception
handling), rather than have to split the functionality amongst several
different methods (or figure out which method to use if not all clauses
are used by the template).

To make this discussion less theoretical, here is the transaction
example implemented using both the two method and the four method
protocols instead of a generator. Both implementations guarantee a
commit if a break, return or continue statement is encountered (as does
the generator-based implementation in the Examples section):

    class transaction_2method(object):

        def __init__(self, db):
            self.db = db

        def __enter__(self):
            pass

        def __exit__(self, exc_type, *exc_details):
            if exc_type is None:
                self.db.commit()
            else:
                self.db.rollback()

    class transaction_4method(object):

        def __init__(self, db):
            self.db = db
            self.commit = False

        def __try__(self):
            self.commit = True

        def __except__(self, exc_type, exc_value, traceback):
            self.db.rollback()
            self.commit = False

        def __else__(self):
            pass

        def __finally__(self):
            if self.commit:
                self.db.commit()
                self.commit = False

There are two more minor points, relating to the specific method names
in the suggestion. The name of the __try__() method is misleading, as
SETUP_BLOCK executes before the try statement is entered, and the name
of the __else__() method is unclear in isolation, as numerous other
Python statements include an else clause.

Iterator finalisation (WITHDRAWN)

The ability to use user defined statements inside generators is likely
to increase the need for deterministic finalisation of iterators, as
resource management is pushed inside the generators, rather than being
handled externally as is currently the case.

The PEP currently suggests handling this by making all generators
statement templates, and using with statements to handle finalisation.
However, earlier versions of this PEP suggested the following, more
complex, solution, that allowed the author of a generator to flag the
need for finalisation, and have for loops deal with it automatically. It
is included here as a long, detailed rejected option.

Iterator protocol addition: __finish__

An optional new method for iterators is proposed, called __finish__().
It takes no arguments, and should not return anything.

The __finish__ method is expected to clean up all resources the iterator
has open. Iterators with a __finish__() method are called 'finishable
iterators' for the remainder of the PEP.

Best effort finalisation

A finishable iterator should ensure that it provides a __del__ method
that also performs finalisation (e.g. by invoking the __finish__()
method). This allows Python to still make a best effort at finalisation
in the event that deterministic finalisation is not applied to the
iterator.

Deterministic finalisation

If the iterator used in a for loop has a __finish__() method, the
enhanced for loop semantics will guarantee that that method will be
executed, regardless of the means of exiting the loop. This is important
for iterator generators that utilise user defined statements or the now
permitted try/finally statements, or for new iterators that rely on
timely finalisation to release allocated resources (e.g. releasing a
thread or database connection back into a pool).

for loop syntax

No changes are suggested to for loop syntax. This is just to define the
statement parts needed for the description of the semantics:

    for VAR1 in EXPR1:
        BLOCK1
    else:
        BLOCK2

Updated for loop semantics

When the target iterator does not have a __finish__() method, a for loop
will execute as follows (i.e. no change from the status quo):

    itr = iter(EXPR1)
    exhausted = False
    while True:
        try:
            VAR1 = itr.next()
        except StopIteration:
            exhausted = True
            break
        BLOCK1
    if exhausted:
        BLOCK2

When the target iterator has a __finish__() method, a for loop will
execute as follows:

    itr = iter(EXPR1)
    exhausted = False
    try:
        while True:
            try:
                VAR1 = itr.next()
            except StopIteration:
                exhausted = True
                break
            BLOCK1
        if exhausted:
            BLOCK2
    finally:
        itr.__finish__()

The implementation will need to take some care to avoid incurring the
try/finally overhead when the iterator does not have a __finish__()
method.

Generator iterator finalisation: __finish__() method

When enabled with the appropriate decorator, generators will have a
__finish__() method that raises TerminateIteration in the internal
frame:

    def __finish__(self):
        try:
            self._inject_exception(TerminateIteration)
        except TerminateIteration:
            pass

A decorator (e.g. needs_finish()) is required to enable this feature, so
that existing generators (which are not expecting finalisation) continue
to work as expected.

Partial iteration of finishable iterators

Partial iteration of a finishable iterator is possible, although it
requires some care to ensure the iterator is still finalised promptly
(it was made finishable for a reason!). First, we need a class to enable
partial iteration of a finishable iterator by hiding the iterator's
__finish__() method from the for loop:

    class partial_iter(object):

        def __init__(self, iterable):
            self.iter = iter(iterable)

        def __iter__(self):
            return self

        def next(self):
            return self.itr.next()

Secondly, an appropriate statement template is needed to ensure the
iterator is finished eventually:

    @statement_template
    def finishing(iterable):
          itr = iter(iterable)
          itr_finish = getattr(itr, "__finish__", None)
          if itr_finish is None:
              yield itr
          else:
              try:
                  yield partial_iter(itr)
              finally:
                  itr_finish()

This can then be used as follows:

    do finishing(finishable_itr) as itr:
        for header_item in itr:
            if end_of_header(header_item):
                break
            # process header item
        for body_item in itr:
            # process body item

Note that none of the above is needed for an iterator that is not
finishable - without a __finish__() method, it will not be promptly
finalised by the for loop, and hence inherently allows partial
iteration. Allowing partial iteration of non-finishable iterators as the
default behaviour is a key element in keeping this addition to the
iterator protocol backwards compatible.

Acknowledgements

The acknowledgements section for PEP 340 applies, since this text grew
out of the discussion of that PEP, but additional thanks go to Michael
Hudson, Paul Moore and Guido van Rossum for writing PEP 310 and PEP 340
in the first place, and to (in no meaningful order) Fredrik Lundh,
Phillip J. Eby, Steven Bethard, Josiah Carlson, Greg Ewing, Tim Delaney
and Arnold deVos for prompting particular ideas that made their way into
this text.

References

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 End:

[1] A rant against flow control macros
(http://blogs.msdn.com/oldnewthing/archive/2005/01/06/347666.aspx)

[2] Why doesn't C# have a 'with' statement?
(http://msdn.microsoft.com/vcsharp/programming/language/ask/withstatement/)


text:
PEP: 494 Title: Python 3.6 Release Schedule Version: $Revision$
Last-Modified: $Date$ Author: Ned Deily<EMAIL_ADDRESS>Status: Final
Type: Informational Topic: Release Content-Type: text/x-rst Created:
30-May-2015 Python-Version: 3.6

Abstract

This document describes the development and release schedule for Python
3.6. The schedule primarily concerns itself with PEP-sized items.

Release Manager and Crew

-   3.6 Release Manager: Ned Deily
-   Windows installers: Steve Dower
-   Mac installers: Ned Deily
-   Documentation: Julien Palard, Georg Brandl

3.6 Lifespan

3.6 will receive bugfix updates approximately every 3 months for about
24 months. Sometime after the release of 3.7.0 final, a final 3.6 bugfix
update will be released. After that, it is expected that security
updates (source only) will be released as needed until 5 years after the
release of 3.6 final, so until approximately 2021-12.

As of 2021-12-23, 3.6 has reached the end-of-life phase of its release
cycle. 3.6.15 was the final security release. The code base for 3.6 is
now frozen and no further updates will be provided nor issues of any
kind will be accepted on the bug tracker.

Release Schedule

3.6.0 schedule

-   3.6 development begins: 2015-05-24
-   3.6.0 alpha 1: 2016-05-17
-   3.6.0 alpha 2: 2016-06-13
-   3.6.0 alpha 3: 2016-07-11
-   3.6.0 alpha 4: 2016-08-15
-   3.6.0 beta 1: 2016-09-12 (No new features beyond this point.)
-   3.6.0 beta 2: 2016-10-10
-   3.6.0 beta 3: 2016-10-31
-   3.6.0 beta 4: 2016-11-21
-   3.6.0 candidate 1: 2016-12-06
-   3.6.0 candidate 2: 2016-12-16
-   3.6.0 final: 2016-12-23

3.6.1 schedule (first bugfix release)

-   3.6.1 candidate: 2017-03-05
-   3.6.1 final: 2017-03-21

3.6.2 schedule

-   3.6.2 candidate 1: 2017-06-17
-   3.6.2 candidate 2: 2017-07-07
-   3.6.2 final: 2017-07-17

3.6.3 schedule

-   3.6.3 candidate: 2017-09-19
-   3.6.3 final: 2017-10-03

3.6.4 schedule

-   3.6.4 candidate: 2017-12-05
-   3.6.4 final: 2017-12-19

3.6.5 schedule

-   3.6.5 candidate: 2018-03-13
-   3.6.5 final: 2018-03-28

3.6.6 schedule

-   3.6.6 candidate: 2018-06-12
-   3.6.6 final: 2018-06-27

3.6.7 schedule

-   3.6.7 candidate: 2018-09-26
-   3.6.7 candidate 2: 2018-10-13
-   3.6.7 final: 2018-10-20

3.6.8 schedule (last bugfix release)

Last binary releases

-   3.6.8 candidate: 2018-12-11
-   3.6.8 final: 2018-12-24

3.6.9 schedule (first security-only release)

Source only

-   3.6.9 candidate 1: 2019-06-18
-   3.6.9 final: 2019-07-02

3.6.10 schedule

-   3.6.10 candidate 1: 2019-12-11
-   3.6.10 final: 2019-12-18

3.6.11 schedule

-   3.6.11 candidate 1: 2020-06-15
-   3.6.11 final: 2020-06-27

3.6.12 schedule

-   3.6.12 final: 2020-08-17

3.6.13 schedule

-   3.6.13 final: 2021-02-15

3.6.14 schedule

-   3.6.14 final: 2021-06-28

3.6.15 schedule (last security-only release)

-   3.6.15 final: 2021-09-04

Features for 3.6

Implemented changes for 3.6 (as of 3.6.0 beta 1):

-   PEP 468, Preserving Keyword Argument Order
-   PEP 487, Simpler customization of class creation
-   PEP 495, Local Time Disambiguation
-   PEP 498, Literal String Formatting
-   PEP 506, Adding A Secrets Module To The Standard Library
-   PEP 509, Add a private version to dict
-   PEP 515, Underscores in Numeric Literals
-   PEP 519, Adding a file system path protocol
-   PEP 520, Preserving Class Attribute Definition Order
-   PEP 523, Adding a frame evaluation API to CPython
-   PEP 524, Make os.urandom() blocking on Linux (during system startup)
-   PEP 525, Asynchronous Generators (provisional)
-   PEP 526, Syntax for Variable Annotations (provisional)
-   PEP 528, Change Windows console encoding to UTF-8 (provisional)
-   PEP 529, Change Windows filesystem encoding to UTF-8 (provisional)
-   PEP 530, Asynchronous Comprehensions

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:


text:
PEP: 305 Title: CSV File API Version: $Revision$ Last-Modified: $Date$
Author: Kevin Altis<EMAIL_ADDRESS>Dave Cole
<EMAIL_ADDRESS>Andrew McNamara
<EMAIL_ADDRESS>Skip Montanaro<EMAIL_ADDRESS>Cliff
Wells<EMAIL_ADDRESS>Discussions-To<EMAIL_ADDRESS>Status: Final Type: Standards Track Content-Type: text/x-rst Created:
26-Jan-2003 Python-Version: 2.3 Post-History: 31-Jan-2003, 13-Feb-2003

Abstract

The Comma Separated Values (CSV) file format is the most common import
and export format for spreadsheets and databases. Although many CSV
files are simple to parse, the format is not formally defined by a
stable specification and is subtle enough that parsing lines of a CSV
file with something like line.split(",") is eventually bound to fail.
This PEP defines an API for reading and writing CSV files. It is
accompanied by a corresponding module which implements the API.

To Do (Notes for the Interested and Ambitious)

-   Better motivation for the choice of passing a file object to the
    constructors. See
    https://mail.python.org/pipermail/csv/2003-January/000179.html
-   Unicode. ugh.

Application Domain

This PEP is about doing one thing well: parsing tabular data which may
use a variety of field separators, quoting characters, quote escape
mechanisms and line endings. The authors intend the proposed module to
solve this one parsing problem efficiently. The authors do not intend to
address any of these related topics:

-   data interpretation (is a field containing the string "10" supposed
    to be a string, a float or an int? is it a number in base 10, base
    16 or base 2? is a number in quotes a number or a string?)
-   locale-specific data representation (should the number 1.23 be
    written as "1.23" or "1,23" or "1 23"?) -- this may eventually be
    addressed.
-   fixed width tabular data - can already be parsed reliably.

Rationale

Often, CSV files are formatted simply enough that you can get by reading
them line-by-line and splitting on the commas which delimit the fields.
This is especially true if all the data being read is numeric. This
approach may work for a while, then come back to bite you in the butt
when somebody puts something unexpected in the data like a comma. As you
dig into the problem you may eventually come to the conclusion that you
can solve the problem using regular expressions. This will work for a
while, then break mysteriously one day. The problem grows, so you dig
deeper and eventually realize that you need a purpose-built parser for
the format.

CSV formats are not well-defined and different implementations have a
number of subtle corner cases. It has been suggested that the "V" in the
acronym stands for "Vague" instead of "Values". Different delimiters and
quoting characters are just the start. Some programs generate whitespace
after each delimiter which is not part of the following field. Others
quote embedded quoting characters by doubling them, others by prefixing
them with an escape character. The list of weird ways to do things can
seem endless.

All this variability means it is difficult for programmers to reliably
parse CSV files from many sources or generate CSV files designed to be
fed to specific external programs without a thorough understanding of
those sources and programs. This PEP and the software which accompany it
attempt to make the process less fragile.

Existing Modules

This problem has been tackled before. At least three modules currently
available in the Python community enable programmers to read and write
CSV files:

-   Object Craft's CSV module[1]
-   Cliff Wells' Python-DSV module[2]
-   Laurence Tratt's ASV module[3]

Each has a different API, making it somewhat difficult for programmers
to switch between them. More of a problem may be that they interpret
some of the CSV corner cases differently, so even after surmounting the
differences between the different module APIs, the programmer has to
also deal with semantic differences between the packages.

Module Interface

This PEP supports three basic APIs, one to read and parse CSV files, one
to write them, and one to identify different CSV dialects to the readers
and writers.

Reading CSV Files

CSV readers are created with the reader factory function:

    obj = reader(iterable [, dialect='excel']
                 [optional keyword args])

A reader object is an iterator which takes an iterable object returning
lines as the sole required parameter. If it supports a binary mode (file
objects do), the iterable argument to the reader function must have been
opened in binary mode. This gives the reader object full control over
the interpretation of the file's contents. The optional dialect
parameter is discussed below. The reader function also accepts several
optional keyword arguments which define specific format settings for the
parser (see the section "Formatting Parameters"). Readers are typically
used as follows:

    csvreader = csv.reader(file("some.csv"))
    for row in csvreader:
        process(row)

Each row returned by a reader object is a list of strings or Unicode
objects.

When both a dialect parameter and individual formatting parameters are
passed to the constructor, first the dialect is queried for formatting
parameters, then individual formatting parameters are examined.

Writing CSV Files

Creating writers is similar:

    obj = writer(fileobj [, dialect='excel'],
                 [optional keyword args])

A writer object is a wrapper around a file-like object opened for
writing in binary mode (if such a distinction is made). It accepts the
same optional keyword parameters as the reader constructor.

Writers are typically used as follows:

    csvwriter = csv.writer(file("some.csv", "w"))
    for row in someiterable:
        csvwriter.writerow(row)

To generate a set of field names as the first row of the CSV file, the
programmer must explicitly write it, e.g.:

    csvwriter = csv.writer(file("some.csv", "w"), fieldnames=names)
    csvwriter.write(names)
    for row in someiterable:
        csvwriter.write(row)

or arrange for it to be the first row in the iterable being written.

Managing Different Dialects

Because CSV is a somewhat ill-defined format, there are plenty of ways
one CSV file can differ from another, yet contain exactly the same data.
Many tools which can import or export tabular data allow the user to
indicate the field delimiter, quote character, line terminator, and
other characteristics of the file. These can be fairly easily
determined, but are still mildly annoying to figure out, and make for
fairly long function calls when specified individually.

To try and minimize the difficulty of figuring out and specifying a
bunch of formatting parameters, reader and writer objects support a
dialect argument which is just a convenient handle on a group of these
lower level parameters. When a dialect is given as a string it
identifies one of the dialects known to the module via its registration
functions, otherwise it must be an instance of the Dialect class as
described below.

Dialects will generally be named after applications or organizations
which define specific sets of format constraints. Two dialects are
defined in the module as of this writing, "excel", which describes the
default format constraints for CSV file export by Excel 97 and Excel
2000, and "excel-tab", which is the same as "excel" but specifies an
ASCII TAB character as the field delimiter.

Dialects are implemented as attribute only classes to enable users to
construct variant dialects by subclassing. The "excel" dialect is a
subclass of Dialect and is defined as follows:

    class Dialect:
        # placeholders
        delimiter = None
        quotechar = None
        escapechar = None
        doublequote = None
        skipinitialspace = None
        lineterminator = None
        quoting = None

    class excel(Dialect):
        delimiter = ','
        quotechar = '"'
        doublequote = True
        skipinitialspace = False
        lineterminator = '\r\n'
        quoting = QUOTE_MINIMAL

The "excel-tab" dialect is defined as:

    class exceltsv(excel):
        delimiter = '\t'

(For a description of the individual formatting parameters see the
section "Formatting Parameters".)

To enable string references to specific dialects, the module defines
several functions:

    dialect = get_dialect(name)
    names = list_dialects()
    register_dialect(name, dialect)
    unregister_dialect(name)

get_dialect() returns the dialect instance associated with the given
name. list_dialects() returns a list of all registered dialect names.
register_dialects() associates a string name with a dialect class.
unregister_dialect() deletes a name/dialect association.

Formatting Parameters

In addition to the dialect argument, both the reader and writer
constructors take several specific formatting parameters, specified as
keyword parameters. The formatting parameters understood are:

-   quotechar specifies a one-character string to use as the quoting
    character. It defaults to '"'. Setting this to None has the same
    effect as setting quoting to csv.QUOTE_NONE.
-   delimiter specifies a one-character string to use as the field
    separator. It defaults to ','.
-   escapechar specifies a one-character string used to escape the
    delimiter when quotechar is set to None.
-   skipinitialspace specifies how to interpret whitespace which
    immediately follows a delimiter. It defaults to False, which means
    that whitespace immediately following a delimiter is part of the
    following field.
-   lineterminator specifies the character sequence which should
    terminate rows.
-   quoting controls when quotes should be generated by the writer. It
    can take on any of the following module constants:
    -   csv.QUOTE_MINIMAL means only when required, for example, when a
        field contains either the quotechar or the delimiter
    -   csv.QUOTE_ALL means that quotes are always placed around fields.
    -   csv.QUOTE_NONNUMERIC means that quotes are always placed around
        nonnumeric fields.
    -   csv.QUOTE_NONE means that quotes are never placed around fields.
-   doublequote controls the handling of quotes inside fields. When True
    two consecutive quotes are interpreted as one during read, and when
    writing, each quote is written as two quotes.

When processing a dialect setting and one or more of the other optional
parameters, the dialect parameter is processed before the individual
formatting parameters. This makes it easy to choose a dialect, then
override one or more of the settings without defining a new dialect
class. For example, if a CSV file was generated by Excel 2000 using
single quotes as the quote character and a colon as the delimiter, you
could create a reader like:

    csvreader = csv.reader(file("some.csv"), dialect="excel",
                           quotechar="'", delimiter=':')

Other details of how Excel generates CSV files would be handled
automatically because of the reference to the "excel" dialect.

Reader Objects

Reader objects are iterables whose next() method returns a sequence of
strings, one string per field in the row.

Writer Objects

Writer objects have two methods, writerow() and writerows(). The former
accepts an iterable (typically a list) of fields which are to be written
to the output. The latter accepts a list of iterables and calls
writerow() for each.

Implementation

There is a sample implementation available.[4] The goal is for it to
efficiently implement the API described in the PEP. It is heavily based
on the Object Craft csv module.[5]

Testing

The sample implementation[6] includes a set of test cases.

Issues

1.  Should a parameter control how consecutive delimiters are
    interpreted? Our thought is "no". Consecutive delimiters should
    always denote an empty field.

2.  What about Unicode? Is it sufficient to pass a file object gotten
    from codecs.open()? For example:

        csvreader = csv.reader(codecs.open("some.csv", "r", "cp1252"))

        csvwriter = csv.writer(codecs.open("some.csv", "w", "utf-8"))

    In the first example, text would be assumed to be encoded as cp1252.
    Should the system be aggressive in converting to Unicode or should
    Unicode strings only be returned if necessary?

    In the second example, the file will take care of automatically
    encoding Unicode strings as utf-8 before writing to disk.

    Note: As of this writing, the csv module doesn't handle Unicode
    data.

3.  What about alternate escape conventions? If the dialect in use
    includes an escapechar parameter which is not None and the quoting
    parameter is set to QUOTE_NONE, delimiters appearing within fields
    will be prefixed by the escape character when writing and are
    expected to be prefixed by the escape character when reading.

4.  Should there be a "fully quoted" mode for writing? What about "fully
    quoted except for numeric values"? Both are implemented (QUOTE_ALL
    and QUOTE_NONNUMERIC, respectively).

5.  What about end-of-line? If I generate a CSV file on a Unix system,
    will Excel properly recognize the LF-only line terminators? Files
    must be opened for reading or writing as appropriate using binary
    mode. Specify the lineterminator sequence as '\r\n'. The resulting
    file will be written correctly.

6.  What about an option to generate dicts from the reader and accept
    dicts by the writer? See the DictReader and DictWriter classes in
    csv.py.

7.  Are quote character and delimiters limited to single characters? For
    the time being, yes.

8.  How should rows of different lengths be handled? Interpretation of
    the data is the application's job. There is no such thing as a
    "short row" or a "long row" at this level.

References

There are many references to other CSV-related projects on the Web. A
few are included here.

Copyright

This document has been placed in the public domain.

  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 End:

[1] csv module, Object Craft
(http://www.object-craft.com.au/projects/csv)

[2] Python-DSV module, Wells
(http://sourceforge.net/projects/python-dsv/)

[3] ASV module, Tratt (http://tratt.net/laurie/python/asv/)

[4] csv module, Python Sandbox
(http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/nondist/sandbox/csv/)

[5] csv module, Object Craft
(http://www.object-craft.com.au/projects/csv)

[6] csv module, Python Sandbox
(http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/nondist/sandbox/csv/)