Python Documentation contents¶

What’s New in Python¶

The “What’s New in Python” series of essays takes tours through the most important changes between major Python versions. They are a “must read” for anyone wishing to stay up-to-date after a new release.

What’s New In Python 3.2¶

Author:	Raymond Hettinger
Release:	3.2.2
Date:	August 02, 2015

This article explains the new features in Python 3.2 as compared to 3.1. It focuses on a few highlights and gives a few examples. For full details, see the Misc/NEWS file.

See also

PEP 392 - Python 3.2 Release Schedule

PEP 384: Defining a Stable ABI¶

In the past, extension modules built for one Python version were often not usable with other Python versions. Particularly on Windows, every feature release of Python required rebuilding all extension modules that one wanted to use. This requirement was the result of the free access to Python interpreter internals that extension modules could use.

With Python 3.2, an alternative approach becomes available: extension modules which restrict themselves to a limited API (by defining Py_LIMITED_API) cannot use many of the internals, but are constrained to a set of API functions that are promised to be stable for several releases. As a consequence, extension modules built for 3.2 in that mode will also work with 3.3, 3.4, and so on. Extension modules that make use of details of memory structures can still be built, but will need to be recompiled for every feature release.

See also

PEP 384 - Defining a Stable ABI: PEP written by Martin von Löwis.

PEP 389: Argparse Command Line Parsing Module¶

A new module for command line parsing, argparse, was introduced to overcome the limitations of optparse which did not provide support for positional arguments (not just options), subcommands, required options and other common patterns of specifying and validating options.

This module has already had widespread success in the community as a third-party module. Being more fully featured than its predecessor, the argparse module is now the preferred module for command-line processing. The older module is still being kept available because of the substantial amount of legacy code that depends on it.

Here’s an annotated example parser showing features like limiting results to a set of choices, specifying a metavar in the help screen, validating that one or more positional arguments is present, and making a required option:

import argparse
parser = argparse.ArgumentParser(
            description = 'Manage servers',         # main description for help
            epilog = 'Tested on Solaris and Linux') # displayed after help
parser.add_argument('action',                       # argument name
            choices = ['deploy', 'start', 'stop'],  # three allowed values
            help = 'action on each target')         # help msg
parser.add_argument('targets',
            metavar = 'HOSTNAME',                   # var name used in help msg
            nargs = '+',                            # require one or more targets
            help = 'url for target machines')       # help msg explanation
parser.add_argument('-u', '--user',                 # -u or --user option
            required = True,                        # make it a required argument
            help = 'login as user')

Example of calling the parser on a command string:

>>> cmd  = 'deploy sneezy.example.com sleepy.example.com -u skycaptain'
>>> result = parser.parse_args(cmd.split())
>>> result.action
'deploy'
>>> result.targets
['sneezy.example.com', 'sleepy.example.com']
>>> result.user
'skycaptain'

Example of the parser’s automatically generated help:

>>> parser.parse_args('-h'.split())

usage: manage_cloud.py [-h] -u USER
                       {deploy,start,stop} HOSTNAME [HOSTNAME ...]

Manage servers

positional arguments:
  {deploy,start,stop}   action on each target
  HOSTNAME              url for target machines

optional arguments:
  -h, --help            show this help message and exit
  -u USER, --user USER  login as user

Tested on Solaris and Linux

An especially nice argparse feature is the ability to define subparsers, each with their own argument patterns and help displays:

import argparse
parser = argparse.ArgumentParser(prog='HELM')
subparsers = parser.add_subparsers()

parser_l = subparsers.add_parser('launch', help='Launch Control')   # first subgroup
parser_l.add_argument('-m', '--missiles', action='store_true')
parser_l.add_argument('-t', '--torpedos', action='store_true')

parser_m = subparsers.add_parser('move', help='Move Vessel',        # second subgroup
                                 aliases=('steer', 'turn'))         # equivalent names
parser_m.add_argument('-c', '--course', type=int, required=True)
parser_m.add_argument('-s', '--speed', type=int, default=0)

$ ./helm.py --help                         # top level help (launch and move)
$ ./helm.py launch --help                  # help for launch options
$ ./helm.py launch --missiles              # set missiles=True and torpedos=False
$ ./helm.py steer --course 180 --speed 5   # set movement parameters

See also

PEP 389 - New Command Line Parsing Module: PEP written by Steven Bethard.

Upgrading optparse code for details on the differences from optparse.

PEP 391: Dictionary Based Configuration for Logging¶

The logging module provided two kinds of configuration, one style with function calls for each option or another style driven by an external file saved in a ConfigParser format. Those options did not provide the flexibility to create configurations from JSON or YAML files, nor did they support incremental configuration, which is needed for specifying logger options from a command line.

To support a more flexible style, the module now offers logging.config.dictConfig() for specifying logging configuration with plain Python dictionaries. The configuration options include formatters, handlers, filters, and loggers. Here’s a working example of a configuration dictionary:

{"version": 1,
 "formatters": {"brief": {"format": "%(levelname)-8s: %(name)-15s: %(message)s"},
                "full": {"format": "%(asctime)s %(name)-15s %(levelname)-8s %(message)s"}
                },
 "handlers": {"console": {
                   "class": "logging.StreamHandler",
                   "formatter": "brief",
                   "level": "INFO",
                   "stream": "ext://sys.stdout"},
              "console_priority": {
                   "class": "logging.StreamHandler",
                   "formatter": "full",
                   "level": "ERROR",
                   "stream": "ext://sys.stderr"}
              },
 "root": {"level": "DEBUG", "handlers": ["console", "console_priority"]}}

If that dictionary is stored in a file called conf.json, it can be loaded and called with code like this:

>>> import json, logging.config
>>> with open('conf.json') as f:
        conf = json.load(f)
>>> logging.config.dictConfig(conf)
>>> logging.info("Transaction completed normally")
INFO    : root           : Transaction completed normally
>>> logging.critical("Abnormal termination")
2011-02-17 11:14:36,694 root            CRITICAL Abnormal termination

See also

PEP 391 - Dictionary Based Configuration for Logging: PEP written by Vinay Sajip.

PEP 3148: The `concurrent.futures` module¶

Code for creating and managing concurrency is being collected in a new top-level namespace, concurrent. Its first member is a futures package which provides a uniform high-level interface for managing threads and processes.

The design for concurrent.futures was inspired by java.util.concurrent.package. In that model, a running call and its result are represented by a Future object that abstracts features common to threads, processes, and remote procedure calls. That object supports status checks (running or done), timeouts, cancellations, adding callbacks, and access to results or exceptions.

The primary offering of the new module is a pair of executor classes for launching and managing calls. The goal of the executors is to make it easier to use existing tools for making parallel calls. They save the effort needed to setup a pool of resources, launch the calls, create a results queue, add time-out handling, and limit the total number of threads, processes, or remote procedure calls.

Ideally, each application should share a single executor across multiple components so that process and thread limits can be centrally managed. This solves the design challenge that arises when each component has its own competing strategy for resource management.

Both classes share a common interface with three methods: submit() for scheduling a callable and returning a Future object; map() for scheduling many asynchronous calls at a time, and shutdown() for freeing resources. The class is a context manager and can be used in a with statement to assure that resources are automatically released when currently pending futures are done executing.

A simple of example of ThreadPoolExecutor is a launch of four parallel threads for copying files:

import concurrent.futures, shutil
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as e:
    e.submit(shutil.copy, 'src1.txt', 'dest1.txt')
    e.submit(shutil.copy, 'src2.txt', 'dest2.txt')
    e.submit(shutil.copy, 'src3.txt', 'dest3.txt')
    e.submit(shutil.copy, 'src4.txt', 'dest4.txt')

See also

PEP 3148 - Futures – Execute Computations Asynchronously: PEP written by Brian Quinlan.

Code for Threaded Parallel URL reads, an example using threads to fetch multiple web pages in parallel.

Code for computing prime numbers in parallel, an example demonstrating ProcessPoolExecutor.

PEP 3147: PYC Repository Directories¶

Python’s scheme for caching bytecode in .pyc files did not work well in environments with multiple Python interpreters. If one interpreter encountered a cached file created by another interpreter, it would recompile the source and overwrite the cached file, thus losing the benefits of caching.

The issue of “pyc fights” has become more pronounced as it has become commonplace for Linux distributions to ship with multiple versions of Python. These conflicts also arise with CPython alternatives such as Unladen Swallow.

To solve this problem, Python’s import machinery has been extended to use distinct filenames for each interpreter. Instead of Python 3.2 and Python 3.3 and Unladen Swallow each competing for a file called “mymodule.pyc”, they will now look for “mymodule.cpython-32.pyc”, “mymodule.cpython-33.pyc”, and “mymodule.unladen10.pyc”. And to prevent all of these new files from cluttering source directories, the pyc files are now collected in a “__pycache__” directory stored under the package directory.

Aside from the filenames and target directories, the new scheme has a few aspects that are visible to the programmer:

Imported modules now have a __cached__ attribute which stores the name of the actual file that was imported:

>>> import collections
>>> collections.__cached__
'c:/py32/lib/__pycache__/collections.cpython-32.pyc'

The tag that is unique to each interpreter is accessible from the imp module:
```
>>> import imp
>>> imp.get_tag()
'cpython-32'
```

Scripts that try to deduce source filename from the imported file now need to be smarter. It is no longer sufficient to simply strip the “c” from a ”.pyc” filename. Instead, use the new functions in the imp module:

>>> imp.source_from_cache('c:/py32/lib/__pycache__/collections.cpython-32.pyc')
'c:/py32/lib/collections.py'
>>> imp.cache_from_source('c:/py32/lib/collections.py')
'c:/py32/lib/__pycache__/collections.cpython-32.pyc'

The py_compile and compileall modules have been updated to reflect the new naming convention and target directory. The command-line invocation of compileall has new options: -i for specifying a list of files and directories to compile and -b which causes bytecode files to be written to their legacy location rather than __pycache__.
The importlib.abc module has been updated with new abstract base classes for loading bytecode files. The obsolete ABCs, PyLoader and PyPycLoader, have been deprecated (instructions on how to stay Python 3.1 compatible are included with the documentation).

See also

PEP 3147 - PYC Repository Directories: PEP written by Barry Warsaw.

PEP 3149: ABI Version Tagged .so Files¶

The PYC repository directory allows multiple bytecode cache files to be co-located. This PEP implements a similar mechanism for shared object files by giving them a common directory and distinct names for each version.

The common directory is “pyshared” and the file names are made distinct by identifying the Python implementation (such as CPython, PyPy, Jython, etc.), the major and minor version numbers, and optional build flags (such as “d” for debug, “m” for pymalloc, “u” for wide-unicode). For an arbitrary package “foo”, you may see these files when the distribution package is installed:

/usr/share/pyshared/foo.cpython-32m.so
/usr/share/pyshared/foo.cpython-33md.so

In Python itself, the tags are accessible from functions in the sysconfig module:

>>> import sysconfig
>>> sysconfig.get_config_var('SOABI')    # find the version tag
'cpython-32mu'
>>> sysconfig.get_config_var('SO')       # find the full filename extension
'.cpython-32mu.so'

See also

PEP 3149 - ABI Version Tagged .so Files: PEP written by Barry Warsaw.

PEP 3333: Python Web Server Gateway Interface v1.0.1¶

This informational PEP clarifies how bytes/text issues are to be handled by the WSGI protocol. The challenge is that string handling in Python 3 is most conveniently handled with the str type even though the HTTP protocol is itself bytes oriented.

The PEP differentiates so-called native strings that are used for request/response headers and metadata versus byte strings which are used for the bodies of requests and responses.

The native strings are always of type str but are restricted to code points between U+0000 through U+00FF which are translatable to bytes using Latin-1 encoding. These strings are used for the keys and values in the environment dictionary and for response headers and statuses in the start_response() function. They must follow RFC 2616 with respect to encoding. That is, they must either be ISO-8859-1 characters or use RFC 2047 MIME encoding.

For developers porting WSGI applications from Python 2, here are the salient points:

If the app already used strings for headers in Python 2, no change is needed.
If instead, the app encoded output headers or decoded input headers, then the headers will need to be re-encoded to Latin-1. For example, an output header encoded in utf-8 was using h.encode('utf-8') now needs to convert from bytes to native strings using h.encode('utf-8').decode('latin-1').
Values yielded by an application or sent using the write() method must be byte strings. The start_response() function and environ must use native strings. The two cannot be mixed.

For server implementers writing CGI-to-WSGI pathways or other CGI-style protocols, the users must to be able access the environment using native strings even though the underlying platform may have a different convention. To bridge this gap, the wsgiref module has a new function, wsgiref.handlers.read_environ() for transcoding CGI variables from os.environ into native strings and returning a new dictionary.

See also

PEP 3333 - Python Web Server Gateway Interface v1.0.1: PEP written by Phillip Eby.

Other Language Changes¶

Some smaller changes made to the core Python language are:

String formatting for format() and str.format() gained new capabilities for the format character #. Previously, for integers in binary, octal, or hexadecimal, it caused the output to be prefixed with ‘0b’, ‘0o’, or ‘0x’ respectively. Now it can also handle floats, complex, and Decimal, causing the output to always have a decimal point even when no digits follow it.
```
>>> format(20, '#o')
'0o24'
>>> format(12.34, '#5.0f')
'  12.'
```
(Suggested by Mark Dickinson and implemented by Eric Smith in issue 7094.)

There is also a new str.format_map() method that extends the capabilities of the existing str.format() method by accepting arbitrary mapping objects. This new method makes it possible to use string formatting with any of Python’s many dictionary-like objects such as defaultdict, Shelf, ConfigParser, or dbm. It is also useful with custom dict subclasses that normalize keys before look-up or that supply a __missing__() method for unknown keys:

>>> import shelve
>>> d = shelve.open('tmp.shl')
>>> 'The {project_name} status is {status} as of {date}'.format_map(d)
'The testing project status is green as of February 15, 2011'

>>> class LowerCasedDict(dict):
        def __getitem__(self, key):
            return dict.__getitem__(self, key.lower())
>>> lcd = LowerCasedDict(part='widgets', quantity=10)
>>> 'There are {QUANTITY} {Part} in stock'.format_map(lcd)
'There are 10 widgets in stock'

>>> class PlaceholderDict(dict):
        def __missing__(self, key):
            return '<{}>'.format(key)
>>> 'Hello {name}, welcome to {location}'.format_map(PlaceholderDict())
'Hello <name>, welcome to <location>'

(Suggested by Raymond Hettinger and implemented by Eric Smith in issue 6081.)

The interpreter can now be started with a quiet option, -q, to prevent the copyright and version information from being displayed in the interactive mode. The option can be introspected using the sys.flags attribute:
```
$ python -q
>>> sys.flags
sys.flags(debug=0, division_warning=0, inspect=0, interactive=0,
optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0,
ignore_environment=0, verbose=0, bytes_warning=0, quiet=1)
```
(Contributed by Marcin Wojdyr in issue 1772833).
The hasattr() function works by calling getattr() and detecting whether an exception is raised. This technique allows it to detect methods created dynamically by __getattr__() or __getattribute__() which would otherwise be absent from the class dictionary. Formerly, hasattr would catch any exception, possibly masking genuine errors. Now, hasattr has been tightened to only catch AttributeError and let other exceptions pass through:
```
>>> class A:
        @property
        def f(self):
            return 1 // 0

>>> a = A()
>>> hasattr(a, 'f')
Traceback (most recent call last):
  ...
ZeroDivisionError: integer division or modulo by zero
```
(Discovered by Yury Selivanov and fixed by Benjamin Peterson; issue 9666.)
The str() of a float or complex number is now the same as its repr(). Previously, the str() form was shorter but that just caused confusion and is no longer needed now that the shortest possible repr() is displayed by default:
```
>>> import math
>>> repr(math.pi)
'3.141592653589793'
>>> str(math.pi)
'3.141592653589793'
```
(Proposed and implemented by Mark Dickinson; issue 9337.)
memoryview objects now have a release() method and they also now support the context manager protocol. This allows timely release of any resources that were acquired when requesting a buffer from the original object.
```
>>> with memoryview(b'abcdefgh') as v:
        print(v.tolist())
[97, 98, 99, 100, 101, 102, 103, 104]
```
(Added by Antoine Pitrou; issue 9757.)
Previously it was illegal to delete a name from the local namespace if it occurs as a free variable in a nested block:
```
def outer(x):
    def inner():
       return x
    inner()
    del x
```
This is now allowed. Remember that the target of an except clause is cleared, so this code which used to work with Python 2.6, raised a SyntaxError with Python 3.1 and now works again:
```
def f():
    def print_error():
       print(e)
    try:
       something
    except Exception as e:
       print_error()
       # implicit "del e" here
```
(See issue 4617.)
The internal structsequence tool now creates subclasses of tuple. This means that C structures like those returned by os.stat(), time.gmtime(), and sys.version_info now work like a named tuple and now work with functions and methods that expect a tuple as an argument. This is a big step forward in making the C structures as flexible as their pure Python counterparts:
```
>>> isinstance(sys.version_info, tuple)
True
>>> 'Version %d.%d.%d %s(%d)' % sys.version_info
'Version 3.2.0 final(0)'
```
(Suggested by Arfrever Frehtes Taifersar Arahesis and implemented by Benjamin Peterson in issue 8413.)
Warnings are now easier to control using the PYTHONWARNINGS environment variable as an alternative to using -W at the command line:
```
$ export PYTHONWARNINGS='ignore::RuntimeWarning::,once::UnicodeWarning::'
```
(Suggested by Barry Warsaw and implemented by Philip Jenvey in issue 7301.)
A new warning category, ResourceWarning, has been added. It is emitted when potential issues with resource consumption or cleanup are detected. It is silenced by default in normal release builds but can be enabled through the means provided by the warnings module, or on the command line.

A ResourceWarning is issued at interpreter shutdown if the gc.garbage list isn’t empty, and if gc.DEBUG_UNCOLLECTABLE is set, all uncollectable objects are printed. This is meant to make the programmer aware that their code contains object finalization issues.

A ResourceWarning is also issued when a file object is destroyed without having been explicitly closed. While the deallocator for such object ensures it closes the underlying operating system resource (usually, a file descriptor), the delay in deallocating the object could produce various issues, especially under Windows. Here is an example of enabling the warning from the command line:
```
$ python -q -Wdefault
>>> f = open("foo", "wb")
>>> del f
__main__:1: ResourceWarning: unclosed file <_io.BufferedWriter name='foo'>
```
(Added by Antoine Pitrou and Georg Brandl in issue 10093 and issue 477863.)
range objects now support index and count methods. This is part of an effort to make more objects fully implement the collections.Sequence abstract base class. As a result, the language will have a more uniform API. In addition, range objects now support slicing and negative indices, even with values larger than sys.maxsize. This makes range more interoperable with lists:
```
>>> range(0, 100, 2).count(10)
1
>>> range(0, 100, 2).index(10)
5
>>> range(0, 100, 2)[5]
10
>>> range(0, 100, 2)[0:5]
range(0, 10, 2)
```
(Contributed by Daniel Stutzbach in issue 9213, by Alexander Belopolsky in issue 2690, and by Nick Coghlan in issue 10889.)
The callable() builtin function from Py2.x was resurrected. It provides a concise, readable alternative to using an abstract base class in an expression like isinstance(x, collections.Callable):
```
>>> callable(max)
True
>>> callable(20)
False
```
(See issue 10518.)
Python’s import mechanism can now load modules installed in directories with non-ASCII characters in the path name. This solved an aggravating problem with home directories for users with non-ASCII characters in their usernames.

(Required extensive work by Victor Stinner in issue 9425.)

New, Improved, and Deprecated Modules¶

Python’s standard library has undergone significant maintenance efforts and quality improvements.

The biggest news for Python 3.2 is that the email package, mailbox module, and nntplib modules now work correctly with the bytes/text model in Python 3. For the first time, there is correct handling of messages with mixed encodings.

Throughout the standard library, there has been more careful attention to encodings and text versus bytes issues. In particular, interactions with the operating system are now better able to exchange non-ASCII data using the Windows MBCS encoding, locale-aware encodings, or UTF-8.

Another significant win is the addition of substantially better support for SSL connections and security certificates.

In addition, more classes now implement a context manager to support convenient and reliable resource clean-up using a with statement.

email¶

The usability of the email package in Python 3 has been mostly fixed by the extensive efforts of R. David Murray. The problem was that emails are typically read and stored in the form of bytes rather than str text, and they may contain multiple encodings within a single email. So, the email package had to be extended to parse and generate email messages in bytes format.

New functions message_from_bytes() and message_from_binary_file(), and new classes BytesFeedParser and BytesParser allow binary message data to be parsed into model objects.
Given bytes input to the model, get_payload() will by default decode a message body that has a Content-Transfer-Encoding of 8bit using the charset specified in the MIME headers and return the resulting string.
Given bytes input to the model, Generator will convert message bodies that have a Content-Transfer-Encoding of 8bit to instead have a 7bit Content-Transfer-Encoding.

Headers with unencoded non-ASCII bytes are deemed to be RFC 2047-encoded using the unknown-8bit character set.
A new class BytesGenerator produces bytes as output, preserving any unchanged non-ASCII data that was present in the input used to build the model, including message bodies with a Content-Transfer-Encoding of 8bit.
The smtplib SMTP class now accepts a byte string for the msg argument to the sendmail() method, and a new method, send_message() accepts a Message object and can optionally obtain the from_addr and to_addrs addresses directly from the object.

(Proposed and implemented by R. David Murray, issue 4661 and issue 10321.)

elementtree¶

The xml.etree.ElementTree package and its xml.etree.cElementTree counterpart have been updated to version 1.3.

Several new and useful functions and methods have been added:

xml.etree.ElementTree.fromstringlist() which builds an XML document from a sequence of fragments
xml.etree.ElementTree.register_namespace() for registering a global namespace prefix
xml.etree.ElementTree.tostringlist() for string representation including all sublists
xml.etree.ElementTree.Element.extend() for appending a sequence of zero or more elements
xml.etree.ElementTree.Element.iterfind() searches an element and subelements
xml.etree.ElementTree.Element.itertext() creates a text iterator over an element and its subelements
xml.etree.ElementTree.TreeBuilder.end() closes the current element
xml.etree.ElementTree.TreeBuilder.doctype() handles a doctype declaration

Two methods have been deprecated:

xml.etree.ElementTree.getchildren() use list(elem) instead.
xml.etree.ElementTree.getiterator() use Element.iter instead.

For details of the update, see Introducing ElementTree on Fredrik Lundh’s website.

(Contributed by Florent Xicluna and Fredrik Lundh, issue 6472.)

functools¶

The functools module includes a new decorator for caching function calls. functools.lru_cache() can save repeated queries to an external resource whenever the results are expected to be the same.

For example, adding a caching decorator to a database query function can save database accesses for popular searches:
```
>>> import functools
>>> @functools.lru_cache(maxsize=300)
>>> def get_phone_number(name):
        c = conn.cursor()
        c.execute('SELECT phonenumber FROM phonelist WHERE name=?', (name,))
        return c.fetchone()[0]
```
```
>>> for name in user_requests:
        get_phone_number(name)        # cached lookup
```
To help with choosing an effective cache size, the wrapped function is instrumented for tracking cache statistics:
```
>>> get_phone_number.cache_info()
CacheInfo(hits=4805, misses=980, maxsize=300, currsize=300)
```
If the phonelist table gets updated, the outdated contents of the cache can be cleared with:
```
>>> get_phone_number.cache_clear()
```
(Contributed by Raymond Hettinger and incorporating design ideas from Jim Baker, Miki Tebeka, and Nick Coghlan; see recipe 498245, recipe 577479, issue 10586, and issue 10593.)
The functools.wraps() decorator now adds a __wrapped__ attribute pointing to the original callable function. This allows wrapped functions to be introspected. It also copies __annotations__ if defined. And now it also gracefully skips over missing attributes such as __doc__ which might not be defined for the wrapped callable.

In the above example, the cache can be removed by recovering the original function:
```
>>> get_phone_number = get_phone_number.__wrapped__    # uncached function
```
(By Nick Coghlan and Terrence Cole; issue 9567, issue 3445, and issue 8814.)

To help write classes with rich comparison methods, a new decorator functools.total_ordering() will use a existing equality and inequality methods to fill in the remaining methods.

For example, supplying __eq__ and __lt__ will enable total_ordering() to fill-in __le__, __gt__ and __ge__:

@total_ordering
class Student:
    def __eq__(self, other):
        return ((self.lastname.lower(), self.firstname.lower()) ==
                (other.lastname.lower(), other.firstname.lower()))
    def __lt__(self, other):
        return ((self.lastname.lower(), self.firstname.lower()) <
                (other.lastname.lower(), other.firstname.lower()))

With the total_ordering decorator, the remaining comparison methods are filled in automatically.

(Contributed by Raymond Hettinger.)

To aid in porting programs from Python 2, the functools.cmp_to_key() function converts an old-style comparison function to modern key function:
```
>>> # locale-aware sort order
>>> sorted(iterable, key=cmp_to_key(locale.strcoll))
```
For sorting examples and a brief sorting tutorial, see the Sorting HowTo tutorial.

(Contributed by Raymond Hettinger.)

itertools¶

The itertools module has a new accumulate() function modeled on APL’s scan operator and Numpy’s accumulate function:
```
>>> from itertools import accumulate
>>> list(accumulate([8, 2, 50]))
[8, 10, 60]
```
```
>>> prob_dist = [0.1, 0.4, 0.2, 0.3]
>>> list(accumulate(prob_dist))      # cumulative probability distribution
[0.1, 0.5, 0.7, 1.0]
```
For an example using accumulate(), see the examples for the random module.

(Contributed by Raymond Hettinger and incorporating design suggestions from Mark Dickinson.)

collections¶

The collections.Counter class now has two forms of in-place subtraction, the existing -= operator for saturating subtraction and the new subtract() method for regular subtraction. The former is suitable for multisets which only have positive counts, and the latter is more suitable for use cases that allow negative counts:
```
>>> tally = Counter(dogs=5, cat=3)
>>> tally -= Counter(dogs=2, cats=8)    # saturating subtraction
>>> tally
Counter({'dogs': 3})
```
```
>>> tally = Counter(dogs=5, cats=3)
>>> tally.subtract(dogs=2, cats=8)      # regular subtraction
>>> tally
Counter({'dogs': 3, 'cats': -5})
```
(Contributed by Raymond Hettinger.)
The collections.OrderedDict class has a new method move_to_end() which takes an existing key and moves it to either the first or last position in the ordered sequence.

The default is to move an item to the last position. This is equivalent of renewing an entry with od[k] = od.pop(k).

A fast move-to-end operation is useful for resequencing entries. For example, an ordered dictionary can be used to track order of access by aging entries from the oldest to the most recently accessed.
```
>>> d = OrderedDict.fromkeys(['a', 'b', 'X', 'd', 'e'])
>>> list(d)
['a', 'b', 'X', 'd', 'e']
>>> d.move_to_end('X')
>>> list(d)
['a', 'b', 'd', 'e', 'X']
```
(Contributed by Raymond Hettinger.)
The collections.deque class grew two new methods count() and reverse() that make them more substitutable for list objects:
```
>>> d = deque('simsalabim')
>>> d.count('s')
2
>>> d.reverse()
>>> d
deque(['m', 'i', 'b', 'a', 'l', 'a', 's', 'm', 'i', 's'])
```
(Contributed by Raymond Hettinger.)

threading¶

The threading module has a new Barrier synchronization class for making multiple threads wait until all of them have reached a common barrier point. Barriers are useful for making sure that a task with multiple preconditions does not run until all of the predecessor tasks are complete.

Barriers can work with an arbitrary number of threads. This is a generalization of a Rendezvous which is defined for only two threads.

Implemented as a two-phase cyclic barrier, Barrier objects are suitable for use in loops. The separate filling and draining phases assure that all threads get released (drained) before any one of them can loop back and re-enter the barrier. The barrier fully resets after each cycle.

Example of using barriers:

from threading import Barrier, Thread

def get_votes(site):
    ballots = conduct_election(site)
    all_polls_closed.wait()        # do not count until all polls are closed
    totals = summarize(ballots)
    publish(site, totals)

all_polls_closed = Barrier(len(sites))
for site in sites:
    Thread(target=get_votes, args=(site,)).start()

In this example, the barrier enforces a rule that votes cannot be counted at any polling site until all polls are closed. Notice how a solution with a barrier is similar to one with threading.Thread.join(), but the threads stay alive and continue to do work (summarizing ballots) after the barrier point is crossed.

If any of the predecessor tasks can hang or be delayed, a barrier can be created with an optional timeout parameter. Then if the timeout period elapses before all the predecessor tasks reach the barrier point, all waiting threads are released and a BrokenBarrierError exception is raised:

def get_votes(site):
    ballots = conduct_election(site)
    try:
        all_polls_closed.wait(timeout = midnight - time.now())
    except BrokenBarrierError:
        lockbox = seal_ballots(ballots)
        queue.put(lockbox)
    else:
        totals = summarize(ballots)
        publish(site, totals)

In this example, the barrier enforces a more robust rule. If some election sites do not finish before midnight, the barrier times-out and the ballots are sealed and deposited in a queue for later handling.

See Barrier Synchronization Patterns for more examples of how barriers can be used in parallel computing. Also, there is a simple but thorough explanation of barriers in The Little Book of Semaphores, section 3.6.

(Contributed by Kristján Valur Jónsson with an API review by Jeffrey Yasskin in issue 8777.)

datetime and time¶

The datetime module has a new type timezone that implements the tzinfo interface by returning a fixed UTC offset and timezone name. This makes it easier to create timezone-aware datetime objects:

>>> from datetime import datetime, timezone

>>> datetime.now(timezone.utc)
datetime.datetime(2010, 12, 8, 21, 4, 2, 923754, tzinfo=datetime.timezone.utc)

>>> datetime.strptime("01/01/2000 12:00 +0000", "%m/%d/%Y %H:%M %z")
datetime.datetime(2000, 1, 1, 12, 0, tzinfo=datetime.timezone.utc)

Also, timedelta objects can now be multiplied by float and divided by float and int objects. And timedelta objects can now divide one another.
The datetime.date.strftime() method is no longer restricted to years after 1900. The new supported year range is from 1000 to 9999 inclusive.
Whenever a two-digit year is used in a time tuple, the interpretation has been governed by time.accept2dyear. The default is True which means that for a two-digit year, the century is guessed according to the POSIX rules governing the %y strptime format.

Starting with Py3.2, use of the century guessing heuristic will emit a DeprecationWarning. Instead, it is recommended that time.accept2dyear be set to False so that large date ranges can be used without guesswork:
```
>>> import time, warnings
>>> warnings.resetwarnings()      # remove the default warning filters

>>> time.accept2dyear = True      # guess whether 11 means 11 or 2011
>>> time.asctime((11, 1, 1, 12, 34, 56, 4, 1, 0))
Warning (from warnings module):
  ...
DeprecationWarning: Century info guessed for a 2-digit year.
'Fri Jan  1 12:34:56 2011'

>>> time.accept2dyear = False     # use the full range of allowable dates
>>> time.asctime((11, 1, 1, 12, 34, 56, 4, 1, 0))
'Fri Jan  1 12:34:56 11'
```
Several functions now have significantly expanded date ranges. When time.accept2dyear is false, the time.asctime() function will accept any year that fits in a C int, while the time.mktime() and time.strftime() functions will accept the full range supported by the corresponding operating system functions.

(Contributed by Alexander Belopolsky and Victor Stinner in issue 1289118, issue 5094, issue 6641, issue 2706, issue 1777412, issue 8013, and issue 10827.)

math¶

The math module has been updated with six new functions inspired by the C99 standard.

The isfinite() function provides a reliable and fast way to detect special values. It returns True for regular numbers and False for Nan or Infinity:

>>> [isfinite(x) for x in (123, 4.56, float('Nan'), float('Inf'))]
[True, True, False, False]

The expm1() function computes e**x-1 for small values of x without incurring the loss of precision that usually accompanies the subtraction of nearly equal quantities:

>>> expm1(0.013671875)   # more accurate way to compute e**x-1 for a small x
0.013765762467652909

The erf() function computes a probability integral or Gaussian error function. The complementary error function, erfc(), is 1 - erf(x):

>>> erf(1.0/sqrt(2.0))   # portion of normal distribution within 1 standard deviation
0.682689492137086
>>> erfc(1.0/sqrt(2.0))  # portion of normal distribution outside 1 standard deviation
0.31731050786291404
>>> erf(1.0/sqrt(2.0)) + erfc(1.0/sqrt(2.0))
1.0

The gamma() function is a continuous extension of the factorial function. See http://en.wikipedia.org/wiki/Gamma_function for details. Because the function is related to factorials, it grows large even for small values of x, so there is also a lgamma() function for computing the natural logarithm of the gamma function:

>>> gamma(7.0)           # six factorial
720.0
>>> lgamma(801.0)        # log(800 factorial)
4551.950730698041

(Contributed by Mark Dickinson.)

abc¶

The abc module now supports abstractclassmethod() and abstractstaticmethod().

These tools make it possible to define an abstract base class that requires a particular classmethod() or staticmethod() to be implemented:

class Temperature(metaclass=abc.ABCMeta):
    @abc.abstractclassmethod
    def from_fahrenheit(cls, t):
        ...
    @abc.abstractclassmethod
    def from_celsius(cls, t):
        ...

(Patch submitted by Daniel Urban; issue 5867.)

io¶

The io.BytesIO has a new method, getbuffer(), which provides functionality similar to memoryview(). It creates an editable view of the data without making a copy. The buffer’s random access and support for slice notation are well-suited to in-place editing:

>>> REC_LEN, LOC_START, LOC_LEN = 34, 7, 11

>>> def change_location(buffer, record_number, location):
        start = record_number * REC_LEN + LOC_START
        buffer[start: start+LOC_LEN] = location

>>> import io

>>> byte_stream = io.BytesIO(
    b'G3805  storeroom  Main chassis    '
    b'X7899  shipping   Reserve cog     '
    b'L6988  receiving  Primary sprocket'
)
>>> buffer = byte_stream.getbuffer()
>>> change_location(buffer, 1, b'warehouse  ')
>>> change_location(buffer, 0, b'showroom   ')
>>> print(byte_stream.getvalue())
b'G3805  showroom   Main chassis    '
b'X7899  warehouse  Reserve cog     '
b'L6988  receiving  Primary sprocket'

(Contributed by Antoine Pitrou in issue 5506.)

reprlib¶

When writing a __repr__() method for a custom container, it is easy to forget to handle the case where a member refers back to the container itself. Python’s builtin objects such as list and set handle self-reference by displaying ”...” in the recursive part of the representation string.

To help write such __repr__() methods, the reprlib module has a new decorator, recursive_repr(), for detecting recursive calls to __repr__() and substituting a placeholder string instead:

>>> class MyList(list):
        @recursive_repr()
        def __repr__(self):
            return '<' + '|'.join(map(repr, self)) + '>'

>>> m = MyList('abc')
>>> m.append(m)
>>> m.append('x')
>>> print(m)
<'a'|'b'|'c'|...|'x'>

(Contributed by Raymond Hettinger in issue 9826 and issue 9840.)

logging¶

In addition to dictionary-based configuration described above, the logging package has many other improvements.

The logging documentation has been augmented by a basic tutorial, an advanced tutorial, and a cookbook of logging recipes. These documents are the fastest way to learn about logging.

The logging.basicConfig() set-up function gained a style argument to support three different types of string formatting. It defaults to “%” for traditional %-formatting, can be set to “{” for the new str.format() style, or can be set to “$” for the shell-style formatting provided by string.Template. The following three configurations are equivalent:

>>> from logging import basicConfig
>>> basicConfig(style='%', format="%(name)s -> %(levelname)s: %(message)s")
>>> basicConfig(style='{', format="{name} -> {levelname} {message}")
>>> basicConfig(style='$', format="$name -> $levelname: $message")

If no configuration is set-up before a logging event occurs, there is now a default configuration using a StreamHandler directed to sys.stderr for events of WARNING level or higher. Formerly, an event occurring before a configuration was set-up would either raise an exception or silently drop the event depending on the value of logging.raiseExceptions. The new default handler is stored in logging.lastResort.

The use of filters has been simplified. Instead of creating a Filter object, the predicate can be any Python callable that returns True or False.

There were a number of other improvements that add flexibility and simplify configuration. See the module documentation for a full listing of changes in Python 3.2.

csv¶

The csv module now supports a new dialect, unix_dialect, which applies quoting for all fields and a traditional Unix style with '\n' as the line terminator. The registered dialect name is unix.

The csv.DictWriter has a new method, writeheader() for writing-out an initial row to document the field names:

>>> import csv, sys
>>> w = csv.DictWriter(sys.stdout, ['name', 'dept'], dialect='unix')
>>> w.writeheader()
"name","dept"
>>> w.writerows([
        {'name': 'tom', 'dept': 'accounting'},
        {'name': 'susan', 'dept': 'Salesl'}])
"tom","accounting"
"susan","sales"

(New dialect suggested by Jay Talbot in issue 5975, and the new method suggested by Ed Abraham in issue 1537721.)

contextlib¶

There is a new and slightly mind-blowing tool ContextDecorator that is helpful for creating a context manager that does double duty as a function decorator.

As a convenience, this new functionality is used by contextmanager() so that no extra effort is needed to support both roles.

The basic idea is that both context managers and function decorators can be used for pre-action and post-action wrappers. Context managers wrap a group of statements using a with statement, and function decorators wrap a group of statements enclosed in a function. So, occasionally there is a need to write a pre-action or post-action wrapper that can be used in either role.

For example, it is sometimes useful to wrap functions or groups of statements with a logger that can track the time of entry and time of exit. Rather than writing both a function decorator and a context manager for the task, the contextmanager() provides both capabilities in a single definition:

from contextlib import contextmanager
import logging

logging.basicConfig(level=logging.INFO)

@contextmanager
def track_entry_and_exit(name):
    logging.info('Entering: {}'.format(name))
    yield
    logging.info('Exiting: {}'.format(name))

Formerly, this would have only been usable as a context manager:

with track_entry_and_exit('widget loader'):
    print('Some time consuming activity goes here')
    load_widget()

Now, it can be used as a decorator as well:

@track_entry_and_exit('widget loader')
def activity():
    print('Some time consuming activity goes here')
    load_widget()

Trying to fulfill two roles at once places some limitations on the technique. Context managers normally have the flexibility to return an argument usable by a with statement, but there is no parallel for function decorators.

In the above example, there is not a clean way for the track_entry_and_exit context manager to return a logging instance for use in the body of enclosed statements.

(Contributed by Michael Foord in issue 9110.)

decimal and fractions¶

Mark Dickinson crafted an elegant and efficient scheme for assuring that different numeric datatypes will have the same hash value whenever their actual values are equal (issue 8188):

assert hash(Fraction(3, 2)) == hash(1.5) == \
       hash(Decimal("1.5")) == hash(complex(1.5, 0))

Some of the hashing details are exposed through a new attribute, sys.hash_info, which describes the bit width of the hash value, the prime modulus, the hash values for infinity and nan, and the multiplier used for the imaginary part of a number:

>>> sys.hash_info
sys.hash_info(width=64, modulus=2305843009213693951, inf=314159, nan=0, imag=1000003)

An early decision to limit the inter-operability of various numeric types has been relaxed. It is still unsupported (and ill-advised) to have implicit mixing in arithmetic expressions such as Decimal('1.1') + float('1.1') because the latter loses information in the process of constructing the binary float. However, since existing floating point value can be converted losslessly to either a decimal or rational representation, it makes sense to add them to the constructor and to support mixed-type comparisons.

The decimal.Decimal constructor now accepts float objects directly so there in no longer a need to use the from_float() method (issue 8257).
Mixed type comparisons are now fully supported so that Decimal objects can be directly compared with float and fractions.Fraction (issue 2531 and issue 8188).

Similar changes were made to fractions.Fraction so that the from_float() and from_decimal() methods are no longer needed (issue 8294):

>>> Decimal(1.1)
Decimal('1.100000000000000088817841970012523233890533447265625')
>>> Fraction(1.1)
Fraction(2476979795053773, 2251799813685248)

Another useful change for the decimal module is that the Context.clamp attribute is now public. This is useful in creating contexts that correspond to the decimal interchange formats specified in IEEE 754 (see issue 8540).

(Contributed by Mark Dickinson and Raymond Hettinger.)

ftp¶

The ftplib.FTP class now supports the context manager protocol to unconditionally consume socket.error exceptions and to close the FTP connection when done:

>>> from ftplib import FTP
>>> with FTP("ftp1.at.proftpd.org") as ftp:
        ftp.login()
        ftp.dir()

'230 Anonymous login ok, restrictions apply.'
dr-xr-xr-x   9 ftp      ftp           154 May  6 10:43 .
dr-xr-xr-x   9 ftp      ftp           154 May  6 10:43 ..
dr-xr-xr-x   5 ftp      ftp          4096 May  6 10:43 CentOS
dr-xr-xr-x   3 ftp      ftp            18 Jul 10  2008 Fedora

Other file-like objects such as mmap.mmap and fileinput.input() also grew auto-closing context managers:

with fileinput.input(files=('log1.txt', 'log2.txt')) as f:
    for line in f:
        process(line)

(Contributed by Tarek Ziadé and Giampaolo Rodolà in issue 4972, and by Georg Brandl in issue 8046 and issue 1286.)

The FTP_TLS class now accepts a context parameter, which is a ssl.SSLContext object allowing bundling SSL configuration options, certificates and private keys into a single (potentially long-lived) structure.

(Contributed by Giampaolo Rodolà; issue 8806.)

popen¶

The os.popen() and subprocess.Popen() functions now support with statements for auto-closing of the file descriptors.

(Contributed by Antoine Pitrou and Brian Curtin in issue 7461 and issue 10554.)

select¶

The select module now exposes a new, constant attribute, PIPE_BUF, which gives the minimum number of bytes which are guaranteed not to block when select.select() says a pipe is ready for writing.

>>> import select
>>> select.PIPE_BUF
512

(Available on Unix systems. Patch by Sébastien Sablé in issue 9862)

gzip and zipfile¶

gzip.GzipFile now implements the io.BufferedIOBase abstract base class (except for truncate()). It also has a peek() method and supports unseekable as well as zero-padded file objects.

The gzip module also gains the compress() and decompress() functions for easier in-memory compression and decompression. Keep in mind that text needs to be encoded as bytes before compressing and decompressing:

>>> s = 'Three shall be the number thou shalt count, '
>>> s += 'and the number of the counting shall be three'
>>> b = s.encode()                        # convert to utf-8
>>> len(b)
89
>>> c = gzip.compress(b)
>>> len(c)
77
>>> gzip.decompress(c).decode()[:42]      # decompress and convert to text
'Three shall be the number thou shalt count,'

(Contributed by Anand B. Pillai in issue 3488; and by Antoine Pitrou, Nir Aides and Brian Curtin in issue 9962, issue 1675951, issue 7471 and issue 2846.)

Also, the zipfile.ZipExtFile class was reworked internally to represent files stored inside an archive. The new implementation is significantly faster and can be wrapped in a io.BufferedReader object for more speedups. It also solves an issue where interleaved calls to read and readline gave the wrong results.

(Patch submitted by Nir Aides in issue 7610.)

tarfile¶

The TarFile class can now be used as a context manager. In addition, its add() method has a new option, filter, that controls which files are added to the archive and allows the file metadata to be edited.

The new filter option replaces the older, less flexible exclude parameter which is now deprecated. If specified, the optional filter parameter needs to be a keyword argument. The user-supplied filter function accepts a TarInfo object and returns an updated TarInfo object, or if it wants the file to be excluded, the function can return None:

>>> import tarfile, glob

>>> def myfilter(tarinfo):
       if tarinfo.isfile():             # only save real files
            tarinfo.uname = 'monty'     # redact the user name
            return tarinfo

>>> with tarfile.open(name='myarchive.tar.gz', mode='w:gz') as tf:
        for filename in glob.glob('*.txt'):
            tf.add(filename, filter=myfilter)
        tf.list()
-rw-r--r-- monty/501        902 2011-01-26 17:59:11 annotations.txt
-rw-r--r-- monty/501        123 2011-01-26 17:59:11 general_questions.txt
-rw-r--r-- monty/501       3514 2011-01-26 17:59:11 prion.txt
-rw-r--r-- monty/501        124 2011-01-26 17:59:11 py_todo.txt
-rw-r--r-- monty/501       1399 2011-01-26 17:59:11 semaphore_notes.txt

(Proposed by Tarek Ziadé and implemented by Lars Gustäbel in issue 6856.)

hashlib¶

The hashlib module has two new constant attributes listing the hashing algorithms guaranteed to be present in all implementations and those available on the current implementation:

>>> import hashlib

>>> hashlib.algorithms_guaranteed
{'sha1', 'sha224', 'sha384', 'sha256', 'sha512', 'md5'}

>>> hashlib.algorithms_available
{'md2', 'SHA256', 'SHA512', 'dsaWithSHA', 'mdc2', 'SHA224', 'MD4', 'sha256',
'sha512', 'ripemd160', 'SHA1', 'MDC2', 'SHA', 'SHA384', 'MD2',
'ecdsa-with-SHA1','md4', 'md5', 'sha1', 'DSA-SHA', 'sha224',
'dsaEncryption', 'DSA', 'RIPEMD160', 'sha', 'MD5', 'sha384'}

(Suggested by Carl Chenet in issue 7418.)

ast¶

The ast module has a wonderful a general-purpose tool for safely evaluating expression strings using the Python literal syntax. The ast.literal_eval() function serves as a secure alternative to the builtin eval() function which is easily abused. Python 3.2 adds bytes and set literals to the list of supported types: strings, bytes, numbers, tuples, lists, dicts, sets, booleans, and None.

>>> from ast import literal_eval

>>> request = "{'req': 3, 'func': 'pow', 'args': (2, 0.5)}"
>>> literal_eval(request)
{'args': (2, 0.5), 'req': 3, 'func': 'pow'}

>>> request = "os.system('do something harmful')"
>>> literal_eval(request)
Traceback (most recent call last):
  ...
ValueError: malformed node or string: <_ast.Call object at 0x101739a10>

(Implemented by Benjamin Peterson and Georg Brandl.)

os¶

Different operating systems use various encodings for filenames and environment variables. The os module provides two new functions, fsencode() and fsdecode(), for encoding and decoding filenames:

>>> filename = 'Sehenswürdigkeiten'
>>> os.fsencode(filename)
b'Sehensw\xc3\xbcrdigkeiten'

Some operating systems allow direct access to encoded bytes in the environment. If so, the os.supports_bytes_environ constant will be true.

For direct access to encoded environment variables (if available), use the new os.getenvb() function or use os.environb which is a bytes version of os.environ.

(Contributed by Victor Stinner.)

shutil¶

The shutil.copytree() function has two new options:

ignore_dangling_symlinks: when symlinks=False so that the function copies a file pointed to by a symlink, not the symlink itself. This option will silence the error raised if the file doesn’t exist.
copy_function: is a callable that will be used to copy files. shutil.copy2() is used by default.

(Contributed by Tarek Ziadé.)

In addition, the shutil module now supports archiving operations for zipfiles, uncompressed tarfiles, gzipped tarfiles, and bzipped tarfiles. And there are functions for registering additional archiving file formats (such as xz compressed tarfiles or custom formats).

The principal functions are make_archive() and unpack_archive(). By default, both operate on the current directory (which can be set by os.chdir()) and on any sub-directories. The archive filename needs to be specified with a full pathname. The archiving step is non-destructive (the original files are left unchanged).

>>> import shutil, pprint

>>> os.chdir('mydata')                               # change to the source directory
>>> f = shutil.make_archive('/var/backup/mydata',
                            'zip')                   # archive the current directory
>>> f                                                # show the name of archive
'/var/backup/mydata.zip'
>>> os.chdir('tmp')                                  # change to an unpacking
>>> shutil.unpack_archive('/var/backup/mydata.zip')  # recover the data

>>> pprint.pprint(shutil.get_archive_formats())      # display known formats
[('bztar', "bzip2'ed tar-file"),
 ('gztar', "gzip'ed tar-file"),
 ('tar', 'uncompressed tar file'),
 ('zip', 'ZIP file')]

>>> shutil.register_archive_format(                  # register a new archive format
        name = 'xz',
        function = xz.compress,                      # callable archiving function
        extra_args = [('level', 8)],                 # arguments to the function
        description = 'xz compression'
)

(Contributed by Tarek Ziadé.)

sqlite3¶

The sqlite3 module was updated to pysqlite version 2.6.0. It has two new capabilities.

The sqlite3.Connection.in_transit attribute is true if there is an active transaction for uncommitted changes.
The sqlite3.Connection.enable_load_extension() and sqlite3.Connection.load_extension() methods allows you to load SQLite extensions from ”.so” files. One well-known extension is the fulltext-search extension distributed with SQLite.

(Contributed by R. David Murray and Shashwat Anand; issue 8845.)

html¶

A new html module was introduced with only a single function, escape(), which is used for escaping reserved characters from HTML markup:

>>> import html
>>> html.escape('x > 2 && x < 7')
'x &gt; 2 &amp;&amp; x &lt; 7'

socket¶

The socket module has two new improvements.

Socket objects now have a detach() method which puts the socket into closed state without actually closing the underlying file descriptor. The latter can then be reused for other purposes. (Added by Antoine Pitrou; issue 8524.)
socket.create_connection() now supports the context manager protocol to unconditionally consume socket.error exceptions and to close the socket when done. (Contributed by Giampaolo Rodolà; issue 9794.)

ssl¶

The ssl module added a number of features to satisfy common requirements for secure (encrypted, authenticated) internet connections:

A new class, SSLContext, serves as a container for persistent SSL data, such as protocol settings, certificates, private keys, and various other options. It includes a wrap_socket() for creating an SSL socket from an SSL context.
A new function, ssl.match_hostname(), supports server identity verification for higher-level protocols by implementing the rules of HTTPS (from RFC 2818) which are also suitable for other protocols.
The ssl.wrap_socket() constructor function now takes a ciphers argument. The ciphers string lists the allowed encryption algorithms using the format described in the OpenSSL documentation.
When linked against recent versions of OpenSSL, the ssl module now supports the Server Name Indication extension to the TLS protocol, allowing multiple “virtual hosts” using different certificates on a single IP port. This extension is only supported in client mode, and is activated by passing the server_hostname argument to ssl.SSLContext.wrap_socket().
Various options have been added to the ssl module, such as OP_NO_SSLv2 which disables the insecure and obsolete SSLv2 protocol.
The extension now loads all the OpenSSL ciphers and digest algorithms. If some SSL certificates cannot be verified, they are reported as an “unknown algorithm” error.
The version of OpenSSL being used is now accessible using the module attributes ssl.OPENSSL_VERSION (a string), ssl.OPENSSL_VERSION_INFO (a 5-tuple), and ssl.OPENSSL_VERSION_NUMBER (an integer).

(Contributed by Antoine Pitrou in issue 8850, issue 1589, issue 8322, issue 5639, issue 4870, issue 8484, and issue 8321.)

nntp¶

The nntplib module has a revamped implementation with better bytes and text semantics as well as more practical APIs. These improvements break compatibility with the nntplib version in Python 3.1, which was partly dysfunctional in itself.

Support for secure connections through both implicit (using nntplib.NNTP_SSL) and explicit (using nntplib.NNTP.starttls()) TLS has also been added.

(Contributed by Antoine Pitrou in issue 9360 and Andrew Vant in issue 1926.)

certificates¶

http.client.HTTPSConnection, urllib.request.HTTPSHandler and urllib.request.urlopen() now take optional arguments to allow for server certificate checking against a set of Certificate Authorities, as recommended in public uses of HTTPS.

(Added by Antoine Pitrou, issue 9003.)

imaplib¶

Support for explicit TLS on standard IMAP4 connections has been added through the new imaplib.IMAP4.starttls method.

(Contributed by Lorenzo M. Catucci and Antoine Pitrou, issue 4471.)

http.client¶

There were a number of small API improvements in the http.client module. The old-style HTTP 0.9 simple responses are no longer supported and the strict parameter is deprecated in all classes.

The HTTPConnection and HTTPSConnection classes now have a source_address parameter for a (host, port) tuple indicating where the HTTP connection is made from.

Support for certificate checking and HTTPS virtual hosts were added to HTTPSConnection.

The request() method on connection objects allowed an optional body argument so that a file object could be used to supply the content of the request. Conveniently, the body argument now also accepts an iterable object so long as it includes an explicit Content-Length header. This extended interface is much more flexible than before.

To establish an HTTPS connection through a proxy server, there is a new set_tunnel() method that sets the host and port for HTTP Connect tunneling.

To match the behavior of http.server, the HTTP client library now also encodes headers with ISO-8859-1 (Latin-1) encoding. It was already doing that for incoming headers, so now the behavior is consistent for both incoming and outgoing traffic. (See work by Armin Ronacher in issue 10980.)

unittest¶

The unittest module has a number of improvements supporting test discovery for packages, easier experimentation at the interactive prompt, new testcase methods, improved diagnostic messages for test failures, and better method names.

The command-line call python -m unittest can now accept file paths instead of module names for running specific tests (issue 10620). The new test discovery can find tests within packages, locating any test importable from the top-level directory. The top-level directory can be specified with the -t option, a pattern for matching files with -p, and a directory to start discovery with -s:
```
$ python -m unittest discover -s my_proj_dir -p _test.py
```
(Contributed by Michael Foord.)
Experimentation at the interactive prompt is now easier because the unittest.case.TestCase class can now be instantiated without arguments:
```
>>> TestCase().assertEqual(pow(2, 3), 8)
```
(Contributed by Michael Foord.)
The unittest module has two new methods, assertWarns() and assertWarnsRegex() to verify that a given warning type is triggered by the code under test:
```
with self.assertWarns(DeprecationWarning):
    legacy_function('XYZ')
```
(Contributed by Antoine Pitrou, issue 9754.)

Another new method, assertCountEqual() is used to compare two iterables to determine if their element counts are equal (whether the same elements are present with the same number of occurrences regardless of order):
```
def test_anagram(self):
    self.assertCountEqual('algorithm', 'logarithm')
```
(Contributed by Raymond Hettinger.)
A principal feature of the unittest module is an effort to produce meaningful diagnostics when a test fails. When possible, the failure is recorded along with a diff of the output. This is especially helpful for analyzing log files of failed test runs. However, since diffs can sometime be voluminous, there is a new maxDiff attribute that sets maximum length of diffs displayed.
In addition, the method names in the module have undergone a number of clean-ups.

For example, assertRegex() is the new name for assertRegexpMatches() which was misnamed because the test uses re.search(), not re.match(). Other methods using regular expressions are now named using short form “Regex” in preference to “Regexp” – this matches the names used in other unittest implementations, matches Python’s old name for the re module, and it has unambiguous camel-casing.

(Contributed by Raymond Hettinger and implemented by Ezio Melotti.)

To improve consistency, some long-standing method aliases are being deprecated in favor of the preferred names:

Old Name Preferred Name

assert_() assertTrue()

assertEquals() assertEqual()

assertNotEquals() assertNotEqual()

assertAlmostEquals() assertAlmostEqual()

assertNotAlmostEquals() assertNotAlmostEqual()

Old Name	Preferred Name
`assert_()`	`assertTrue()`
`assertEquals()`	`assertEqual()`
`assertNotEquals()`	`assertNotEqual()`
`assertAlmostEquals()`	`assertAlmostEqual()`
`assertNotAlmostEquals()`	`assertNotAlmostEqual()`

Likewise, the TestCase.fail* methods deprecated in Python 3.1 are expected to be removed in Python 3.3. Also see the Deprecated aliases section in the unittest documentation.

(Contributed by Ezio Melotti; issue 9424.)

The assertDictContainsSubset() method was deprecated because it was misimplemented with the arguments in the wrong order. This created hard-to-debug optical illusions where tests like TestCase().assertDictContainsSubset({'a':1, 'b':2}, {'a':1}) would fail.

(Contributed by Raymond Hettinger.)

random¶

The integer methods in the random module now do a better job of producing uniform distributions. Previously, they computed selections with int(n*random()) which had a slight bias whenever n was not a power of two. Now, multiple selections are made from a range up to the next power of two and a selection is kept only when it falls within the range 0 <= x < n. The functions and methods affected are randrange(), randint(), choice(), shuffle() and sample().

(Contributed by Raymond Hettinger; issue 9025.)

poplib¶

POP3_SSL class now accepts a context parameter, which is a ssl.SSLContext object allowing bundling SSL configuration options, certificates and private keys into a single (potentially long-lived) structure.

(Contributed by Giampaolo Rodolà; issue 8807.)

asyncore¶

asyncore.dispatcher now provides a handle_accepted() method returning a (sock, addr) pair which is called when a connection has actually been established with a new remote endpoint. This is supposed to be used as a replacement for old handle_accept() and avoids the user to call accept() directly.

(Contributed by Giampaolo Rodolà; issue 6706.)

tempfile¶

The tempfile module has a new context manager, TemporaryDirectory which provides easy deterministic cleanup of temporary directories:

with tempfile.TemporaryDirectory() as tmpdirname:
    print('created temporary dir:', tmpdirname)

(Contributed by Neil Schemenauer and Nick Coghlan; issue 5178.)

inspect¶

The inspect module has a new function getgeneratorstate() to easily identify the current state of a generator-iterator:

>>> from inspect import getgeneratorstate
>>> def gen():
        yield 'demo'
>>> g = gen()
>>> getgeneratorstate(g)
'GEN_CREATED'
>>> next(g)
'demo'
>>> getgeneratorstate(g)
'GEN_SUSPENDED'
>>> next(g, None)
>>> getgeneratorstate(g)
'GEN_CLOSED'

(Contributed by Rodolpho Eckhardt and Nick Coghlan, issue 10220.)

To support lookups without the possibility of activating a dynamic attribute, the inspect module has a new function, getattr_static(). Unlike hasattr(), this is a true read-only search, guaranteed not to change state while it is searching:

>>> class A:
        @property
        def f(self):
            print('Running')
            return 10

>>> a = A()
>>> getattr(a, 'f')
Running
10
>>> inspect.getattr_static(a, 'f')
<property object at 0x1022bd788>

(Contributed by Michael Foord.)

pydoc¶

The pydoc module now provides a much-improved Web server interface, as well as a new command-line option -b to automatically open a browser window to display that server:

$ pydoc3.2 -b

(Contributed by Ron Adam; issue 2001.)

dis¶

The dis module gained two new functions for inspecting code, code_info() and show_code(). Both provide detailed code object information for the supplied function, method, source code string or code object. The former returns a string and the latter prints it:

>>> import dis, random
>>> dis.show_code(random.choice)
Name:              choice
Filename:          /Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/random.py
Argument count:    2
Kw-only arguments: 0
Number of locals:  3
Stack size:        11
Flags:             OPTIMIZED, NEWLOCALS, NOFREE
Constants:
   0: 'Choose a random element from a non-empty sequence.'
   1: 'Cannot choose from an empty sequence'
Names:
   0: _randbelow
   1: len
   2: ValueError
   3: IndexError
Variable names:
   0: self
   1: seq
   2: i

In addition, the dis() function now accepts string arguments so that the common idiom dis(compile(s, '', 'eval')) can be shortened to dis(s):

>>> dis('3*x+1 if x%2==1 else x//2')
  1           0 LOAD_NAME                0 (x)
              3 LOAD_CONST               0 (2)
              6 BINARY_MODULO
              7 LOAD_CONST               1 (1)
             10 COMPARE_OP               2 (==)
             13 POP_JUMP_IF_FALSE       28
             16 LOAD_CONST               2 (3)
             19 LOAD_NAME                0 (x)
             22 BINARY_MULTIPLY
             23 LOAD_CONST               1 (1)
             26 BINARY_ADD
             27 RETURN_VALUE
        >>   28 LOAD_NAME                0 (x)
             31 LOAD_CONST               0 (2)
             34 BINARY_FLOOR_DIVIDE
             35 RETURN_VALUE

Taken together, these improvements make it easier to explore how CPython is implemented and to see for yourself what the language syntax does under-the-hood.

(Contributed by Nick Coghlan in issue 9147.)

dbm¶

All database modules now support the get() and setdefault() methods.

(Suggested by Ray Allen in issue 9523.)

ctypes¶

A new type, ctypes.c_ssize_t represents the C ssize_t datatype.

site¶

The site module has three new functions useful for reporting on the details of a given Python installation.

getsitepackages() lists all global site-packages directories.
getuserbase() reports on the user’s base directory where data can be stored.
getusersitepackages() reveals the user-specific site-packages directory path.

>>> import site
>>> site.getsitepackages()
['/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages',
 '/Library/Frameworks/Python.framework/Versions/3.2/lib/site-python',
 '/Library/Python/3.2/site-packages']
>>> site.getuserbase()
'/Users/raymondhettinger/Library/Python/3.2'
>>> site.getusersitepackages()
'/Users/raymondhettinger/Library/Python/3.2/lib/python/site-packages'

Conveniently, some of site’s functionality is accessible directly from the command-line:

$ python -m site --user-base
/Users/raymondhettinger/.local
$ python -m site --user-site
/Users/raymondhettinger/.local/lib/python3.2/site-packages

(Contributed by Tarek Ziadé in issue 6693.)

sysconfig¶

The new sysconfig module makes it straightforward to discover installation paths and configuration variables that vary across platforms and installations.

The module offers access simple access functions for platform and version information:

get_platform() returning values like linux-i586 or macosx-10.6-ppc.
get_python_version() returns a Python version string such as “3.2”.

It also provides access to the paths and variables corresponding to one of seven named schemes used by distutils. Those include posix_prefix, posix_home, posix_user, nt, nt_user, os2, os2_home:

get_paths() makes a dictionary containing installation paths for the current installation scheme.
get_config_vars() returns a dictionary of platform specific variables.

There is also a convenient command-line interface:

C:\Python32>python -m sysconfig
Platform: "win32"
Python version: "3.2"
Current installation scheme: "nt"

Paths:
        data = "C:\Python32"
        include = "C:\Python32\Include"
        platinclude = "C:\Python32\Include"
        platlib = "C:\Python32\Lib\site-packages"
        platstdlib = "C:\Python32\Lib"
        purelib = "C:\Python32\Lib\site-packages"
        scripts = "C:\Python32\Scripts"
        stdlib = "C:\Python32\Lib"

Variables:
        BINDIR = "C:\Python32"
        BINLIBDEST = "C:\Python32\Lib"
        EXE = ".exe"
        INCLUDEPY = "C:\Python32\Include"
        LIBDEST = "C:\Python32\Lib"
        SO = ".pyd"
        VERSION = "32"
        abiflags = ""
        base = "C:\Python32"
        exec_prefix = "C:\Python32"
        platbase = "C:\Python32"
        prefix = "C:\Python32"
        projectbase = "C:\Python32"
        py_version = "3.2"
        py_version_nodot = "32"
        py_version_short = "3.2"
        srcdir = "C:\Python32"
        userbase = "C:\Documents and Settings\Raymond\Application Data\Python"

(Moved out of Distutils by Tarek Ziadé.)

pdb¶

The pdb debugger module gained a number of usability improvements:

pdb.py now has a -c option that executes commands as given in a .pdbrc script file.
A .pdbrc script file can contain continue and next commands that continue debugging.
The Pdb class constructor now accepts a nosigint argument.
New commands: l(list), ll(long list) and source for listing source code.
New commands: display and undisplay for showing or hiding the value of an expression if it has changed.
New command: interact for starting an interactive interpreter containing the global and local names found in the current scope.
Breakpoints can be cleared by breakpoint number.

(Contributed by Georg Brandl, Antonio Cuni and Ilya Sandler.)

configparser¶

The configparser module was modified to improve usability and predictability of the default parser and its supported INI syntax. The old ConfigParser class was removed in favor of SafeConfigParser which has in turn been renamed to ConfigParser. Support for inline comments is now turned off by default and section or option duplicates are not allowed in a single configuration source.

Config parsers gained a new API based on the mapping protocol:

>>> parser = ConfigParser()
>>> parser.read_string("""
[DEFAULT]
location = upper left
visible = yes
editable = no
color = blue

[main]
title = Main Menu
color = green

[options]
title = Options
""")
>>> parser['main']['color']
'green'
>>> parser['main']['editable']
'no'
>>> section = parser['options']
>>> section['title']
'Options'
>>> section['title'] = 'Options (editable: %(editable)s)'
>>> section['title']
'Options (editable: no)'

The new API is implemented on top of the classical API, so custom parser subclasses should be able to use it without modifications.

The INI file structure accepted by config parsers can now be customized. Users can specify alternative option/value delimiters and comment prefixes, change the name of the DEFAULT section or switch the interpolation syntax.

There is support for pluggable interpolation including an additional interpolation handler ExtendedInterpolation:

>>> parser = ConfigParser(interpolation=ExtendedInterpolation())
>>> parser.read_dict({'buildout': {'directory': '/home/ambv/zope9'},
                      'custom': {'prefix': '/usr/local'}})
>>> parser.read_string("""
    [buildout]
    parts =
      zope9
      instance
    find-links =
      ${buildout:directory}/downloads/dist

    [zope9]
    recipe = plone.recipe.zope9install
    location = /opt/zope

    [instance]
    recipe = plone.recipe.zope9instance
    zope9-location = ${zope9:location}
    zope-conf = ${custom:prefix}/etc/zope.conf
    """)
>>> parser['buildout']['find-links']
'\n/home/ambv/zope9/downloads/dist'
>>> parser['instance']['zope-conf']
'/usr/local/etc/zope.conf'
>>> instance = parser['instance']
>>> instance['zope-conf']
'/usr/local/etc/zope.conf'
>>> instance['zope9-location']
'/opt/zope'

A number of smaller features were also introduced, like support for specifying encoding in read operations, specifying fallback values for get-functions, or reading directly from dictionaries and strings.

(All changes contributed by Łukasz Langa.)

urllib.parse¶

A number of usability improvements were made for the urllib.parse module.

The urlparse() function now supports IPv6 addresses as described in RFC 2732:

>>> import urllib.parse
>>> urllib.parse.urlparse('http://[dead:beef:cafe:5417:affe:8FA3:deaf:feed]/foo/')
ParseResult(scheme='http',
            netloc='[dead:beef:cafe:5417:affe:8FA3:deaf:feed]',
            path='/foo/',
            params='',
            query='',
            fragment='')

The urldefrag() function now returns a named tuple:

>>> r = urllib.parse.urldefrag('http://python.org/about/#target')
>>> r
DefragResult(url='http://python.org/about/', fragment='target')
>>> r[0]
'http://python.org/about/'
>>> r.fragment
'target'

And, the urlencode() function is now much more flexible, accepting either a string or bytes type for the query argument. If it is a string, then the safe, encoding, and error parameters are sent to quote_plus() for encoding:

>>> urllib.parse.urlencode([
         ('type', 'telenovela'),
         ('name', '¿Dónde Está Elisa?')],
         encoding='latin-1')
'type=telenovela&name=%BFD%F3nde+Est%E1+Elisa%3F'

As detailed in Parsing ASCII Encoded Bytes, all the urllib.parse functions now accept ASCII-encoded byte strings as input, so long as they are not mixed with regular strings. If ASCII-encoded byte strings are given as parameters, the return types will also be an ASCII-encoded byte strings:

>>> urllib.parse.urlparse(b'http://www.python.org:80/about/')
ParseResultBytes(scheme=b'http', netloc=b'www.python.org:80',
                 path=b'/about/', params=b'', query=b'', fragment=b'')

(Work by Nick Coghlan, Dan Mahn, and Senthil Kumaran in issue 2987, issue 5468, and issue 9873.)

mailbox¶

Thanks to a concerted effort by R. David Murray, the mailbox module has been fixed for Python 3.2. The challenge was that mailbox had been originally designed with a text interface, but email messages are best represented with bytes because various parts of a message may have different encodings.

The solution harnessed the email package’s binary support for parsing arbitrary email messages. In addition, the solution required a number of API changes.

As expected, the add() method for mailbox.Mailbox objects now accepts binary input.

StringIO and text file input are deprecated. Also, string input will fail early if non-ASCII characters are used. Previously it would fail when the email was processed in a later step.

There is also support for binary output. The get_file() method now returns a file in the binary mode (where it used to incorrectly set the file to text-mode). There is also a new get_bytes() method that returns a bytes representation of a message corresponding to a given key.

It is still possible to get non-binary output using the old API’s get_string() method, but that approach is not very useful. Instead, it is best to extract messages from a Message object or to load them from binary input.

(Contributed by R. David Murray, with efforts from Steffen Daode Nurpmeso and an initial patch by Victor Stinner in issue 9124.)

turtledemo¶

The demonstration code for the turtle module was moved from the Demo directory to main library. It includes over a dozen sample scripts with lively displays. Being on sys.path, it can now be run directly from the command-line:

$ python -m turtledemo

(Moved from the Demo directory by Alexander Belopolsky in issue 10199.)

Multi-threading¶

The mechanism for serializing execution of concurrently running Python threads (generally known as the GIL or Global Interpreter Lock) has been rewritten. Among the objectives were more predictable switching intervals and reduced overhead due to lock contention and the number of ensuing system calls. The notion of a “check interval” to allow thread switches has been abandoned and replaced by an absolute duration expressed in seconds. This parameter is tunable through sys.setswitchinterval(). It currently defaults to 5 milliseconds.

Additional details about the implementation can be read from a python-dev mailing-list message (however, “priority requests” as exposed in this message have not been kept for inclusion).

(Contributed by Antoine Pitrou.)
Regular and recursive locks now accept an optional timeout argument to their acquire() method. (Contributed by Antoine Pitrou; issue 7316.)
Similarly, threading.Semaphore.acquire() also gained a timeout argument. (Contributed by Torsten Landschoff; issue 850728.)
Regular and recursive lock acquisitions can now be interrupted by signals on platforms using Pthreads. This means that Python programs that deadlock while acquiring locks can be successfully killed by repeatedly sending SIGINT to the process (by pressing Ctrl+C in most shells). (Contributed by Reid Kleckner; issue 8844.)

Optimizations¶

A number of small performance enhancements have been added:

Python’s peephole optimizer now recognizes patterns such x in {1, 2, 3} as being a test for membership in a set of constants. The optimizer recasts the set as a frozenset and stores the pre-built constant.

Now that the speed penalty is gone, it is practical to start writing membership tests using set-notation. This style is both semantically clear and operationally fast:
```
extension = name.rpartition('.')[2]
if extension in {'xml', 'html', 'xhtml', 'css'}:
    handle(name)
```
(Patch and additional tests contributed by Dave Malcolm; issue 6690).
Serializing and unserializing data using the pickle module is now several times faster.

(Contributed by Alexandre Vassalotti, Antoine Pitrou and the Unladen Swallow team in issue 9410 and issue 3873.)
The Timsort algorithm used in list.sort() and sorted() now runs faster and uses less memory when called with a key function. Previously, every element of a list was wrapped with a temporary object that remembered the key value associated with each element. Now, two arrays of keys and values are sorted in parallel. This saves the memory consumed by the sort wrappers, and it saves time lost to delegating comparisons.

(Patch by Daniel Stutzbach in issue 9915.)
JSON decoding performance is improved and memory consumption is reduced whenever the same string is repeated for multiple keys. Also, JSON encoding now uses the C speedups when the sort_keys argument is true.

(Contributed by Antoine Pitrou in issue 7451 and by Raymond Hettinger and Antoine Pitrou in issue 10314.)
Recursive locks (created with the threading.RLock() API) now benefit from a C implementation which makes them as fast as regular locks, and between 10x and 15x faster than their previous pure Python implementation.

(Contributed by Antoine Pitrou; issue 3001.)
The fast-search algorithm in stringlib is now used by the split(), splitlines() and replace() methods on bytes, bytearray and str objects. Likewise, the algorithm is also used by rfind(), rindex(), rsplit() and rpartition().

(Patch by Florent Xicluna in issue 7622 and issue 7462.)
String to integer conversions now work two “digits” at a time, reducing the number of division and modulo operations.

(issue 6713 by Gawain Bolton, Mark Dickinson, and Victor Stinner.)

There were several other minor optimizations. Set differencing now runs faster when one operand is much larger than the other (patch by Andress Bennetts in issue 8685). The array.repeat() method has a faster implementation (issue 1569291 by Alexander Belopolsky). The BaseHTTPRequestHandler has more efficient buffering (issue 3709 by Andrew Schaaf). The operator.attrgetter() function has been sped-up (issue 10160 by Christos Georgiou). And ConfigParser loads multi-line arguments a bit faster (issue 7113 by Łukasz Langa).

Unicode¶

Python has been updated to Unicode 6.0.0. The update to the standard adds over 2,000 new characters including emoji symbols which are important for mobile phones.

In addition, the updated standard has altered the character properties for two Kannada characters (U+0CF1, U+0CF2) and one New Tai Lue numeric character (U+19DA), making the former eligible for use in identifiers while disqualifying the latter. For more information, see Unicode Character Database Changes.

Codecs¶

Support was added for cp720 Arabic DOS encoding (issue 1616979).

MBCS encoding no longer ignores the error handler argument. In the default strict mode, it raises an UnicodeDecodeError when it encounters an undecodable byte sequence and an UnicodeEncodeError for an unencodable character.

The MBCS codec supports 'strict' and 'ignore' error handlers for decoding, and 'strict' and 'replace' for encoding.

To emulate Python3.1 MBCS encoding, select the 'ignore' handler for decoding and the 'replace' handler for encoding.

On Mac OS X, Python decodes command line arguments with 'utf-8' rather than the locale encoding.

By default, tarfile uses 'utf-8' encoding on Windows (instead of 'mbcs') and the 'surrogateescape' error handler on all operating systems.

Documentation¶

The documentation continues to be improved.

A table of quick links has been added to the top of lengthy sections such as 内置函数. In the case of itertools, the links are accompanied by tables of cheatsheet-style summaries to provide an overview and memory jog without having to read all of the docs.
In some cases, the pure Python source code can be a helpful adjunct to the documentation, so now many modules now feature quick links to the latest version of the source code. For example, the functools module documentation has a quick link at the top labeled:

Source code Lib/functools.py.

(Contributed by Raymond Hettinger; see rationale.)
The docs now contain more examples and recipes. In particular, re module has an extensive section, Regular Expression Examples. Likewise, the itertools module continues to be updated with new Itertools Recipes.
The datetime module now has an auxiliary implementation in pure Python. No functionality was changed. This just provides an easier-to-read alternate implementation.

(Contributed by Alexander Belopolsky in issue 9528.)
The unmaintained Demo directory has been removed. Some demos were integrated into the documentation, some were moved to the Tools/demo directory, and others were removed altogether.

(Contributed by Georg Brandl in issue 7962.)

IDLE¶

The format menu now has an option to clean source files by stripping trailing whitespace.

(Contributed by Raymond Hettinger; issue 5150.)
IDLE on Mac OS X now works with both Carbon AquaTk and Cocoa AquaTk.

(Contributed by Kevin Walzer, Ned Deily, and Ronald Oussoren; issue 6075.)

Code Repository¶

In addition to the existing Subversion code repository at http://svn.python.org there is now a Mercurial repository at http://hg.python.org/.

After the 3.2 release, there are plans to switch to Mercurial as the primary repository. This distributed version control system should make it easier for members of the community to create and share external changesets. See PEP 385 for details.

To learn the new version control system, see the tutorial by Joel Spolsky or the Guide to Mercurial Workflows.

Build and C API Changes¶

Changes to Python’s build process and to the C API include:

The idle, pydoc and 2to3 scripts are now installed with a version-specific suffix on make altinstall (issue 10679).
The C functions that access the Unicode Database now accept and return characters from the full Unicode range, even on narrow unicode builds (Py_UNICODE_TOLOWER, Py_UNICODE_ISDECIMAL, and others). A visible difference in Python is that unicodedata.numeric() now returns the correct value for large code points, and repr() may consider more characters as printable.

(Reported by Bupjoe Lee and fixed by Amaury Forgeot D’Arc; issue 5127.)
Computed gotos are now enabled by default on supported compilers (which are detected by the configure script). They can still be disabled selectively by specifying --without-computed-gotos.

(Contributed by Antoine Pitrou; issue 9203.)
The option --with-wctype-functions was removed. The built-in unicode database is now used for all functions.

(Contributed by Amaury Forgeot D’Arc; issue 9210.)
Hash values are now values of a new type, Py_hash_t, which is defined to be the same size as a pointer. Previously they were of type long, which on some 64-bit operating systems is still only 32 bits long. As a result of this fix, set and dict can now hold more than 2**32 entries on builds with 64-bit pointers (previously, they could grow to that size but their performance degraded catastrophically).

(Suggested by Raymond Hettinger and implemented by Benjamin Peterson; issue 9778.)
A new macro Py_VA_COPY copies the state of the variable argument list. It is equivalent to C99 va_copy but available on all Python platforms (issue 2443).
A new C API function PySys_SetArgvEx() allows an embedded interpreter to set sys.argv without also modifying sys.path (issue 5753).
PyEval_CallObject is now only available in macro form. The function declaration, which was kept for backwards compatibility reasons, is now removed – the macro was introduced in 1997 (issue 8276).
There is a new function PyLong_AsLongLongAndOverflow() which is analogous to PyLong_AsLongAndOverflow(). They both serve to convert Python int into a native fixed-width type while providing detection of cases where the conversion won’t fit (issue 7767).
The PyUnicode_CompareWithASCIIString() function now returns not equal if the Python string is NUL terminated.
There is a new function PyErr_NewExceptionWithDoc() that is like PyErr_NewException() but allows a docstring to be specified. This lets C exceptions have the same self-documenting capabilities as their pure Python counterparts (issue 7033).
When compiled with the --with-valgrind option, the pymalloc allocator will be automatically disabled when running under Valgrind. This gives improved memory leak detection when running under Valgrind, while taking advantage of pymalloc at other times (issue 2422).
Removed the O? format from the PyArg_Parse functions. The format is no longer used and it had never been documented (issue 8837).

There were a number of other small changes to the C-API. See the Misc/NEWS file for a complete list.

Also, there were a number of updates to the Mac OS X build, see Mac/BuildScript/README.txt for details. For users running a 32/64-bit build, there is a known problem with the default Tcl/Tk on Mac OS X 10.6. Accordingly, we recommend installing an updated alternative such as ActiveState Tcl/Tk 8.5.9. See http://www.python.org/download/mac/tcltk/ for additional details.

Porting to Python 3.2¶

This section lists previously described changes and other bugfixes that may require changes to your code:

The configparser module has a number of clean-ups. The major change is to replace the old ConfigParser class with long-standing preferred alternative SafeConfigParser. In addition there are a number of smaller incompatibilities:
- The interpolation syntax is now validated on get() and set() operations. In the default interpolation scheme, only two tokens with percent signs are valid: %(name)s and %%, the latter being an escaped percent sign.
- The set() and add_section() methods now verify that values are actual strings. Formerly, unsupported types could be introduced unintentionally.
- Duplicate sections or options from a single source now raise either DuplicateSectionError or DuplicateOptionError. Formerly, duplicates would silently overwrite a previous entry.
- Inline comments are now disabled by default so now the ; character can be safely used in values.
- Comments now can be indented. Consequently, for ; or # to appear at the start of a line in multiline values, it has to be interpolated. This keeps comment prefix characters in values from being mistaken as comments.
- "" is now a valid value and is no longer automatically converted to an empty string. For empty strings, use "option =" in a line.
The nntplib module was reworked extensively, meaning that its APIs are often incompatible with the 3.1 APIs.
bytearray objects can no longer be used as filenames; instead, they should be converted to bytes.
The array.tostring() and array.fromstring() have been renamed to array.tobytes() and array.frombytes() for clarity. The old names have been deprecated. (See issue 8990.)
PyArg_Parse*() functions:
- “t#” format has been removed: use “s#” or “s*” instead
- “w” and “w#” formats has been removed: use “w*” instead
The PyCObject type, deprecated in 3.1, has been removed. To wrap opaque C pointers in Python objects, the PyCapsule API should be used instead; the new type has a well-defined interface for passing typing safety information and a less complicated signature for calling a destructor.
The sys.setfilesystemencoding() function was removed because it had a flawed design.
The random.seed() function and method now salt string seeds with an sha512 hash function. To access the previous version of seed in order to reproduce Python 3.1 sequences, set the version argument to 1, random.seed(s, version=1).
The previously deprecated string.maketrans() function has been removed in favor of the static methods bytes.maketrans() and bytearray.maketrans(). This change solves the confusion around which types were supported by the string module. Now, str, bytes, and bytearray each have their own maketrans and translate methods with intermediate translation tables of the appropriate type.

(Contributed by Georg Brandl; issue 5675.)
The previously deprecated contextlib.nested() function has been removed in favor of a plain with statement which can accept multiple context managers. The latter technique is faster (because it is built-in), and it does a better job finalizing multiple context managers when one of them raises an exception:
```
with open('mylog.txt') as infile, open('a.out', 'w') as outfile:
    for line in infile:
        if '<critical>' in line:
            outfile.write(line)
```
(Contributed by Georg Brandl and Mattias Brändström; appspot issue 53094.)
struct.pack() now only allows bytes for the s string pack code. Formerly, it would accept text arguments and implicitly encode them to bytes using UTF-8. This was problematic because it made assumptions about the correct encoding and because a variable-length encoding can fail when writing to fixed length segment of a structure.

Code such as struct.pack('<6sHHBBB', 'GIF87a', x, y) should be rewritten with to use bytes instead of text, struct.pack('<6sHHBBB', b'GIF87a', x, y).

(Discovered by David Beazley and fixed by Victor Stinner; issue 10783.)
The xml.etree.ElementTree class now raises an xml.etree.ElementTree.ParseError when a parse fails. Previously it raised a xml.parsers.expat.ExpatError.
The new, longer str() value on floats may break doctests which rely on the old output format.
In subprocess.Popen, the default value for close_fds is now True under Unix; under Windows, it is True if the three standard streams are set to None, False otherwise. Previously, close_fds was always False by default, which produced difficult to solve bugs or race conditions when open file descriptors would leak into the child process.
Support for legacy HTTP 0.9 has been removed from urllib.request and http.client. Such support is still present on the server side (in http.server).

(Contributed by Antoine Pitrou, issue 10711.)
SSL sockets in timeout mode now raise socket.timeout when a timeout occurs, rather than a generic SSLError.

(Contributed by Antoine Pitrou, issue 10272.)
The misleading functions PyEval_AcquireLock() and PyEval_ReleaseLock() have been officially deprecated. The thread-state aware APIs (such as PyEval_SaveThread() and PyEval_RestoreThread()) should be used instead.
Due to security risks, asyncore.handle_accept() has been deprecated, and a new function, asyncore.handle_accepted(), was added to replace it.

(Contributed by Giampaolo Rodola in issue 6706.)
Due to the new GIL implementation, PyEval_InitThreads() cannot be called before Py_Initialize() anymore.

What’s New In Python 3.1¶

Author:	Raymond Hettinger
Release:	3.2.2
Date:	August 02, 2015

This article explains the new features in Python 3.1, compared to 3.0.

PEP 372: Ordered Dictionaries¶

Regular Python dictionaries iterate over key/value pairs in arbitrary order. Over the years, a number of authors have written alternative implementations that remember the order that the keys were originally inserted. Based on the experiences from those implementations, a new collections.OrderedDict class has been introduced.

The OrderedDict API is substantially the same as regular dictionaries but will iterate over keys and values in a guaranteed order depending on when a key was first inserted. If a new entry overwrites an existing entry, the original insertion position is left unchanged. Deleting an entry and reinserting it will move it to the end.

The standard library now supports use of ordered dictionaries in several modules. The configparser module uses them by default. This lets configuration files be read, modified, and then written back in their original order. The _asdict() method for collections.namedtuple() now returns an ordered dictionary with the values appearing in the same order as the underlying tuple indicies. The json module is being built-out with an object_pairs_hook to allow OrderedDicts to be built by the decoder. Support was also added for third-party tools like PyYAML.

See also

PEP 372 - Ordered Dictionaries: PEP written by Armin Ronacher and Raymond Hettinger. Implementation written by Raymond Hettinger.

PEP 378: Format Specifier for Thousands Separator¶

The built-in format() function and the str.format() method use a mini-language that now includes a simple, non-locale aware way to format a number with a thousands separator. That provides a way to humanize a program’s output, improving its professional appearance and readability:

>>> format(1234567, ',d')
'1,234,567'
>>> format(1234567.89, ',.2f')
'1,234,567.89'
>>> format(12345.6 + 8901234.12j, ',f')
'12,345.600000+8,901,234.120000j'
>>> format(Decimal('1234567.89'), ',f')
'1,234,567.89'

The supported types are int, float, complex and decimal.Decimal.

Discussions are underway about how to specify alternative separators like dots, spaces, apostrophes, or underscores. Locale-aware applications should use the existing n format specifier which already has some support for thousands separators.

See also

PEP 378 - Format Specifier for Thousands Separator: PEP written by Raymond Hettinger and implemented by Eric Smith and Mark Dickinson.

Other Language Changes¶

Some smaller changes made to the core Python language are:

Directories and zip archives containing a __main__.py file can now be executed directly by passing their name to the interpreter. The directory/zipfile is automatically inserted as the first entry in sys.path. (Suggestion and initial patch by Andy Chu; revised patch by Phillip J. Eby and Nick Coghlan; issue 1739468.)
The int() type gained a bit_length method that returns the number of bits necessary to represent its argument in binary:
```
>>> n = 37
>>> bin(37)
'0b100101'
>>> n.bit_length()
6
>>> n = 2**123-1
>>> n.bit_length()
123
>>> (n+1).bit_length()
124
```
(Contributed by Fredrik Johansson, Victor Stinner, Raymond Hettinger, and Mark Dickinson; issue 3439.)
The fields in format() strings can now be automatically numbered:
```
>>> 'Sir {} of {}'.format('Gallahad', 'Camelot')
'Sir Gallahad of Camelot'
```
Formerly, the string would have required numbered fields such as: 'Sir {0} of {1}'.

(Contributed by Eric Smith; issue 5237.)
The string.maketrans() function is deprecated and is replaced by new static methods, bytes.maketrans() and bytearray.maketrans(). This change solves the confusion around which types were supported by the string module. Now, str, bytes, and bytearray each have their own maketrans and translate methods with intermediate translation tables of the appropriate type.

(Contributed by Georg Brandl; issue 5675.)
The syntax of the with statement now allows multiple context managers in a single statement:
```
>>> with open('mylog.txt') as infile, open('a.out', 'w') as outfile:
...     for line in infile:
...         if '<critical>' in line:
...             outfile.write(line)
```
With the new syntax, the contextlib.nested() function is no longer needed and is now deprecated.

(Contributed by Georg Brandl and Mattias Brändström; appspot issue 53094.)
round(x, n) now returns an integer if x is an integer. Previously it returned a float:
```
>>> round(1123, -2)
1100
```
(Contributed by Mark Dickinson; issue 4707.)
Python now uses David Gay’s algorithm for finding the shortest floating point representation that doesn’t change its value. This should help mitigate some of the confusion surrounding binary floating point numbers.

The significance is easily seen with a number like 1.1 which does not have an exact equivalent in binary floating point. Since there is no exact equivalent, an expression like float('1.1') evaluates to the nearest representable value which is 0x1.199999999999ap+0 in hex or 1.100000000000000088817841970012523233890533447265625 in decimal. That nearest value was and still is used in subsequent floating point calculations.

What is new is how the number gets displayed. Formerly, Python used a simple approach. The value of repr(1.1) was computed as format(1.1, '.17g') which evaluated to '1.1000000000000001'. The advantage of using 17 digits was that it relied on IEEE-754 guarantees to assure that eval(repr(1.1)) would round-trip exactly to its original value. The disadvantage is that many people found the output to be confusing (mistaking intrinsic limitations of binary floating point representation as being a problem with Python itself).

The new algorithm for repr(1.1) is smarter and returns '1.1'. Effectively, it searches all equivalent string representations (ones that get stored with the same underlying float value) and returns the shortest representation.

The new algorithm tends to emit cleaner representations when possible, but it does not change the underlying values. So, it is still the case that 1.1 + 2.2 != 3.3 even though the representations may suggest otherwise.

The new algorithm depends on certain features in the underlying floating point implementation. If the required features are not found, the old algorithm will continue to be used. Also, the text pickle protocols assure cross-platform portability by using the old algorithm.

(Contributed by Eric Smith and Mark Dickinson; issue 1580)

New, Improved, and Deprecated Modules¶

Added a collections.Counter class to support convenient counting of unique items in a sequence or iterable:
```
>>> Counter(['red', 'blue', 'red', 'green', 'blue', 'blue'])
Counter({'blue': 3, 'red': 2, 'green': 1})
```
(Contributed by Raymond Hettinger; issue 1696199.)
Added a new module, tkinter.ttk for access to the Tk themed widget set. The basic idea of ttk is to separate, to the extent possible, the code implementing a widget’s behavior from the code implementing its appearance.

(Contributed by Guilherme Polo; issue 2983.)

The gzip.GzipFile and bz2.BZ2File classes now support the context manager protocol:

>>> # Automatically close file after writing
>>> with gzip.GzipFile(filename, "wb") as f:
...     f.write(b"xxx")

(Contributed by Antoine Pitrou.)

The decimal module now supports methods for creating a decimal object from a binary float. The conversion is exact but can sometimes be surprising:
```
>>> Decimal.from_float(1.1)
Decimal('1.100000000000000088817841970012523233890533447265625')
```
The long decimal result shows the actual binary fraction being stored for 1.1. The fraction has many digits because 1.1 cannot be exactly represented in binary.

(Contributed by Raymond Hettinger and Mark Dickinson.)
The itertools module grew two new functions. The itertools.combinations_with_replacement() function is one of four for generating combinatorics including permutations and Cartesian products. The itertools.compress() function mimics its namesake from APL. Also, the existing itertools.count() function now has an optional step argument and can accept any type of counting sequence including fractions.Fraction and decimal.Decimal:
```
>>> [p+q for p,q in combinations_with_replacement('LOVE', 2)]
['LL', 'LO', 'LV', 'LE', 'OO', 'OV', 'OE', 'VV', 'VE', 'EE']

>>> list(compress(data=range(10), selectors=[0,0,1,1,0,1,0,1,0,0]))
[2, 3, 5, 7]

>>> c = count(start=Fraction(1,2), step=Fraction(1,6))
>>> [next(c), next(c), next(c), next(c)]
[Fraction(1, 2), Fraction(2, 3), Fraction(5, 6), Fraction(1, 1)]
```
(Contributed by Raymond Hettinger.)

collections.namedtuple() now supports a keyword argument rename which lets invalid fieldnames be automatically converted to positional names in the form _0, _1, etc. This is useful when the field names are being created by an external source such as a CSV header, SQL field list, or user input:

>>> query = input()
SELECT region, dept, count(*) FROM main GROUPBY region, dept

>>> cursor.execute(query)
>>> query_fields = [desc[0] for desc in cursor.description]
>>> UserQuery = namedtuple('UserQuery', query_fields, rename=True)
>>> pprint.pprint([UserQuery(*row) for row in cursor])
[UserQuery(region='South', dept='Shipping', _2=185),
 UserQuery(region='North', dept='Accounting', _2=37),
 UserQuery(region='West', dept='Sales', _2=419)]

(Contributed by Raymond Hettinger; issue 1818.)

The re.sub(), re.subn() and re.split() functions now accept a flags parameter.

(Contributed by Gregory Smith.)
The logging module now implements a simple logging.NullHandler class for applications that are not using logging but are calling library code that does. Setting-up a null handler will suppress spurious warnings such as “No handlers could be found for logger foo”:
```
>>> h = logging.NullHandler()
>>> logging.getLogger("foo").addHandler(h)
```
(Contributed by Vinay Sajip; issue 4384).
The runpy module which supports the -m command line switch now supports the execution of packages by looking for and executing a __main__ submodule when a package name is supplied.

(Contributed by Andi Vajda; issue 4195.)
The pdb module can now access and display source code loaded via zipimport (or any other conformant PEP 302 loader).

(Contributed by Alexander Belopolsky; issue 4201.)
functools.partial objects can now be pickled.

(Suggested by Antoine Pitrou and Jesse Noller. Implemented by Jack Diederich; issue 5228.)

Add pydoc help topics for symbols so that help('@') works as expected in the interactive environment.

(Contributed by David Laban; issue 4739.)
The unittest module now supports skipping individual tests or classes of tests. And it supports marking a test as a expected failure, a test that is known to be broken, but shouldn’t be counted as a failure on a TestResult:
```
class TestGizmo(unittest.TestCase):

    @unittest.skipUnless(sys.platform.startswith("win"), "requires Windows")
    def test_gizmo_on_windows(self):
        ...

    @unittest.expectedFailure
    def test_gimzo_without_required_library(self):
        ...
```
Also, tests for exceptions have been builtout to work with context managers using the with statement:
```
def test_division_by_zero(self):
    with self.assertRaises(ZeroDivisionError):
        x / 0
```
In addition, several new assertion methods were added including assertSetEqual(), assertDictEqual(), assertDictContainsSubset(), assertListEqual(), assertTupleEqual(), assertSequenceEqual(), assertRaisesRegexp(), assertIsNone(), and assertIsNotNone().

(Contributed by Benjamin Peterson and Antoine Pitrou.)
The io module has three new constants for the seek() method SEEK_SET, SEEK_CUR, and SEEK_END.

The sys.version_info tuple is now a named tuple:

>>> sys.version_info
sys.version_info(major=3, minor=1, micro=0, releaselevel='alpha', serial=2)

(Contributed by Ross Light; issue 4285.)

The nntplib and imaplib modules now support IPv6.

(Contributed by Derek Morr; issue 1655 and issue 1664.)
The pickle module has been adapted for better interoperability with Python 2.x when used with protocol 2 or lower. The reorganization of the standard library changed the formal reference for many objects. For example, __builtin__.set in Python 2 is called builtins.set in Python 3. This change confounded efforts to share data between different versions of Python. But now when protocol 2 or lower is selected, the pickler will automatically use the old Python 2 names for both loading and dumping. This remapping is turned-on by default but can be disabled with the fix_imports option:
```
>>> s = {1, 2, 3}
>>> pickle.dumps(s, protocol=0)
b'c__builtin__\nset\np0\n((lp1\nL1L\naL2L\naL3L\natp2\nRp3\n.'
>>> pickle.dumps(s, protocol=0, fix_imports=False)
b'cbuiltins\nset\np0\n((lp1\nL1L\naL2L\naL3L\natp2\nRp3\n.'
```
An unfortunate but unavoidable side-effect of this change is that protocol 2 pickles produced by Python 3.1 won’t be readable with Python 3.0. The latest pickle protocol, protocol 3, should be used when migrating data between Python 3.x implementations, as it doesn’t attempt to remain compatible with Python 2.x.

(Contributed by Alexandre Vassalotti and Antoine Pitrou, issue 6137.)
A new module, importlib was added. It provides a complete, portable, pure Python reference implementation of the import statement and its counterpart, the __import__() function. It represents a substantial step forward in documenting and defining the actions that take place during imports.

(Contributed by Brett Cannon.)

Optimizations¶

Major performance enhancements have been added:

The new I/O library (as defined in PEP 3116) was mostly written in Python and quickly proved to be a problematic bottleneck in Python 3.0. In Python 3.1, the I/O library has been entirely rewritten in C and is 2 to 20 times faster depending on the task at hand. The pure Python version is still available for experimentation purposes through the _pyio module.

(Contributed by Amaury Forgeot d’Arc and Antoine Pitrou.)
Added a heuristic so that tuples and dicts containing only untrackable objects are not tracked by the garbage collector. This can reduce the size of collections and therefore the garbage collection overhead on long-running programs, depending on their particular use of datatypes.

(Contributed by Antoine Pitrou, issue 4688.)
Enabling a configure option named --with-computed-gotos on compilers that support it (notably: gcc, SunPro, icc), the bytecode evaluation loop is compiled with a new dispatch mechanism which gives speedups of up to 20%, depending on the system, the compiler, and the benchmark.

(Contributed by Antoine Pitrou along with a number of other participants, issue 4753).
The decoding of UTF-8, UTF-16 and LATIN-1 is now two to four times faster.

(Contributed by Antoine Pitrou and Amaury Forgeot d’Arc, issue 4868.)
The json module now has a C extension to substantially improve its performance. In addition, the API was modified so that json works only with str, not with bytes. That change makes the module closely match the JSON specification which is defined in terms of Unicode.

(Contributed by Bob Ippolito and converted to Py3.1 by Antoine Pitrou and Benjamin Peterson; issue 4136.)
Unpickling now interns the attribute names of pickled objects. This saves memory and allows pickles to be smaller.

(Contributed by Jake McGuire and Antoine Pitrou; issue 5084.)

IDLE¶

IDLE’s format menu now provides an option to strip trailing whitespace from a source file.

(Contributed by Roger D. Serwy; issue 5150.)

Build and C API Changes¶

Changes to Python’s build process and to the C API include:

Integers are now stored internally either in base 2**15 or in base 2**30, the base being determined at build time. Previously, they were always stored in base 2**15. Using base 2**30 gives significant performance improvements on 64-bit machines, but benchmark results on 32-bit machines have been mixed. Therefore, the default is to use base 2**30 on 64-bit machines and base 2**15 on 32-bit machines; on Unix, there’s a new configure option --enable-big-digits that can be used to override this default.

Apart from the performance improvements this change should be invisible to end users, with one exception: for testing and debugging purposes there’s a new sys.int_info that provides information about the internal format, giving the number of bits per digit and the size in bytes of the C type used to store each digit:
```
>>> import sys
>>> sys.int_info
sys.int_info(bits_per_digit=30, sizeof_digit=4)
```
(Contributed by Mark Dickinson; issue 4258.)
The PyLong_AsUnsignedLongLong() function now handles a negative pylong by raising OverflowError instead of TypeError.

(Contributed by Mark Dickinson and Lisandro Dalcrin; issue 5175.)
Deprecated PyNumber_Int(). Use PyNumber_Long() instead.

(Contributed by Mark Dickinson; issue 4910.)
Added a new PyOS_string_to_double() function to replace the deprecated functions PyOS_ascii_strtod() and PyOS_ascii_atof().

(Contributed by Mark Dickinson; issue 5914.)
Added PyCapsule as a replacement for the PyCObject API. The principal difference is that the new type has a well defined interface for passing typing safety information and a less complicated signature for calling a destructor. The old type had a problematic API and is now deprecated.

(Contributed by Larry Hastings; issue 5630.)

Porting to Python 3.1¶

This section lists previously described changes and other bugfixes that may require changes to your code:

The new floating point string representations can break existing doctests. For example:

def e():
    '''Compute the base of natural logarithms.

    >>> e()
    2.7182818284590451

    '''
    return sum(1/math.factorial(x) for x in reversed(range(30)))

doctest.testmod()

**********************************************************************
Failed example:
    e()
Expected:
    2.7182818284590451
Got:
    2.718281828459045
**********************************************************************

The automatic name remapping in the pickle module for protocol 2 or lower can make Python 3.1 pickles unreadable in Python 3.0. One solution is to use protocol 3. Another solution is to set the fix_imports option to False. See the discussion above for more details.

What’s New In Python 3.0¶

Author:	Guido van Rossum
Release:	3.2.2
Date:	August 02, 2015

This article explains the new features in Python 3.0, compared to 2.6. Python 3.0, also known as “Python 3000” or “Py3K”, is the first ever intentionally backwards incompatible Python release. There are more changes than in a typical release, and more that are important for all Python users. Nevertheless, after digesting the changes, you’ll find that Python really hasn’t changed all that much – by and large, we’re mostly fixing well-known annoyances and warts, and removing a lot of old cruft.

This article doesn’t attempt to provide a complete specification of all new features, but instead tries to give a convenient overview. For full details, you should refer to the documentation for Python 3.0, and/or the many PEPs referenced in the text. If you want to understand the complete implementation and design rationale for a particular feature, PEPs usually have more details than the regular documentation; but note that PEPs usually are not kept up-to-date once a feature has been fully implemented.

Due to time constraints this document is not as complete as it should have been. As always for a new release, the Misc/NEWS file in the source distribution contains a wealth of detailed information about every small thing that was changed.

Common Stumbling Blocks¶

This section lists those few changes that are most likely to trip you up if you’re used to Python 2.5.

Print Is A Function¶

The print statement has been replaced with a print() function, with keyword arguments to replace most of the special syntax of the old print statement (PEP 3105). Examples:

Old: print "The answer is", 2*2
New: print("The answer is", 2*2)

Old: print x,           # Trailing comma suppresses newline
New: print(x, end=" ")  # Appends a space instead of a newline

Old: print              # Prints a newline
New: print()            # You must call the function!

Old: print >>sys.stderr, "fatal error"
New: print("fatal error", file=sys.stderr)

Old: print (x, y)       # prints repr((x, y))
New: print((x, y))      # Not the same as print(x, y)!

You can also customize the separator between items, e.g.:

print("There are <", 2**32, "> possibilities!", sep="")

which produces:

There are <4294967296> possibilities!

Note:

The print() function doesn’t support the “softspace” feature of the old print statement. For example, in Python 2.x, print "A\n", "B" would write "A\nB\n"; but in Python 3.0, print("A\n", "B") writes "A\n B\n".
Initially, you’ll be finding yourself typing the old print x a lot in interactive mode. Time to retrain your fingers to type print(x) instead!
When using the 2to3 source-to-source conversion tool, all print statements are automatically converted to print() function calls, so this is mostly a non-issue for larger projects.

Views And Iterators Instead Of Lists¶

Some well-known APIs no longer return lists:

dict methods dict.keys(), dict.items() and dict.values() return “views” instead of lists. For example, this no longer works: k = d.keys(); k.sort(). Use k = sorted(d) instead (this works in Python 2.5 too and is just as efficient).
Also, the dict.iterkeys(), dict.iteritems() and dict.itervalues() methods are no longer supported.
map() and filter() return iterators. If you really need a list, a quick fix is e.g. list(map(...)), but a better fix is often to use a list comprehension (especially when the original code uses lambda), or rewriting the code so it doesn’t need a list at all. Particularly tricky is map() invoked for the side effects of the function; the correct transformation is to use a regular for loop (since creating a list would just be wasteful).
range() now behaves like xrange() used to behave, except it works with values of arbitrary size. The latter no longer exists.
zip() now returns an iterator.

Ordering Comparisons¶

Python 3.0 has simplified the rules for ordering comparisons:

The ordering comparison operators (<, <=, >=, >) raise a TypeError exception when the operands don’t have a meaningful natural ordering. Thus, expressions like 1 < '', 0 > None or len <= len are no longer valid, and e.g. None < None raises TypeError instead of returning False. A corollary is that sorting a heterogeneous list no longer makes sense – all the elements must be comparable to each other. Note that this does not apply to the == and != operators: objects of different incomparable types always compare unequal to each other.
builtin.sorted() and list.sort() no longer accept the cmp argument providing a comparison function. Use the key argument instead. N.B. the key and reverse arguments are now “keyword-only”.
The cmp() function should be treated as gone, and the __cmp__() special method is no longer supported. Use __lt__() for sorting, __eq__() with __hash__(), and other rich comparisons as needed. (If you really need the cmp() functionality, you could use the expression (a > b) - (a < b) as the equivalent for cmp(a, b).)

Integers¶

PEP 0237: Essentially, long renamed to int. That is, there is only one built-in integral type, named int; but it behaves mostly like the old long type.
PEP 0238: An expression like 1/2 returns a float. Use 1//2 to get the truncating behavior. (The latter syntax has existed for years, at least since Python 2.2.)
The sys.maxint constant was removed, since there is no longer a limit to the value of integers. However, sys.maxsize can be used as an integer larger than any practical list or string index. It conforms to the implementation’s “natural” integer size and is typically the same as sys.maxint in previous releases on the same platform (assuming the same build options).
The repr() of a long integer doesn’t include the trailing L anymore, so code that unconditionally strips that character will chop off the last digit instead. (Use str() instead.)
Octal literals are no longer of the form 0720; use 0o720 instead.

Text Vs. Data Instead Of Unicode Vs. 8-bit¶

Everything you thought you knew about binary data and Unicode has changed.

Python 3.0 uses the concepts of text and (binary) data instead of Unicode strings and 8-bit strings. All text is Unicode; however encoded Unicode is represented as binary data. The type used to hold text is str, the type used to hold data is bytes. The biggest difference with the 2.x situation is that any attempt to mix text and data in Python 3.0 raises TypeError, whereas if you were to mix Unicode and 8-bit strings in Python 2.x, it would work if the 8-bit string happened to contain only 7-bit (ASCII) bytes, but you would get UnicodeDecodeError if it contained non-ASCII values. This value-specific behavior has caused numerous sad faces over the years.
As a consequence of this change in philosophy, pretty much all code that uses Unicode, encodings or binary data most likely has to change. The change is for the better, as in the 2.x world there were numerous bugs having to do with mixing encoded and unencoded text. To be prepared in Python 2.x, start using unicode for all unencoded text, and str for binary or encoded data only. Then the 2to3 tool will do most of the work for you.
You can no longer use u"..." literals for Unicode text. However, you must use b"..." literals for binary data.
As the str and bytes types cannot be mixed, you must always explicitly convert between them. Use str.encode() to go from str to bytes, and bytes.decode() to go from bytes to str. You can also use bytes(s, encoding=...) and str(b, encoding=...), respectively.
Like str, the bytes type is immutable. There is a separate mutable type to hold buffered binary data, bytearray. Nearly all APIs that accept bytes also accept bytearray. The mutable API is based on collections.MutableSequence.
All backslashes in raw string literals are interpreted literally. This means that '\U' and '\u' escapes in raw strings are not treated specially. For example, r'\u20ac' is a string of 6 characters in Python 3.0, whereas in 2.6, ur'\u20ac' was the single “euro” character. (Of course, this change only affects raw string literals; the euro character is '\u20ac' in Python 3.0.)
The built-in basestring abstract type was removed. Use str instead. The str and bytes types don’t have functionality enough in common to warrant a shared base class. The 2to3 tool (see below) replaces every occurrence of basestring with str.
Files opened as text files (still the default mode for open()) always use an encoding to map between strings (in memory) and bytes (on disk). Binary files (opened with a b in the mode argument) always use bytes in memory. This means that if a file is opened using an incorrect mode or encoding, I/O will likely fail loudly, instead of silently producing incorrect data. It also means that even Unix users will have to specify the correct mode (text or binary) when opening a file. There is a platform-dependent default encoding, which on Unixy platforms can be set with the LANG environment variable (and sometimes also with some other platform-specific locale-related environment variables). In many cases, but not all, the system default is UTF-8; you should never count on this default. Any application reading or writing more than pure ASCII text should probably have a way to override the encoding. There is no longer any need for using the encoding-aware streams in the codecs module.
Filenames are passed to and returned from APIs as (Unicode) strings. This can present platform-specific problems because on some platforms filenames are arbitrary byte strings. (On the other hand, on Windows filenames are natively stored as Unicode.) As a work-around, most APIs (e.g. open() and many functions in the os module) that take filenames accept bytes objects as well as strings, and a few APIs have a way to ask for a bytes return value. Thus, os.listdir() returns a list of bytes instances if the argument is a bytes instance, and os.getcwdb() returns the current working directory as a bytes instance. Note that when os.listdir() returns a list of strings, filenames that cannot be decoded properly are omitted rather than raising UnicodeError.
Some system APIs like os.environ and sys.argv can also present problems when the bytes made available by the system is not interpretable using the default encoding. Setting the LANG variable and rerunning the program is probably the best approach.
PEP 3138: The repr() of a string no longer escapes non-ASCII characters. It still escapes control characters and code points with non-printable status in the Unicode standard, however.
PEP 3120: The default source encoding is now UTF-8.
PEP 3131: Non-ASCII letters are now allowed in identifiers. (However, the standard library remains ASCII-only with the exception of contributor names in comments.)
The StringIO and cStringIO modules are gone. Instead, import the io module and use io.StringIO or io.BytesIO for text and data respectively.
See also the Unicode HOWTO, which was updated for Python 3.0.

Overview Of Syntax Changes¶

This section gives a brief overview of every syntactic change in Python 3.0.

New Syntax¶

PEP 3107: Function argument and return value annotations. This provides a standardized way of annotating a function’s parameters and return value. There are no semantics attached to such annotations except that they can be introspected at runtime using the __annotations__ attribute. The intent is to encourage experimentation through metaclasses, decorators or frameworks.
PEP 3102: Keyword-only arguments. Named parameters occurring after *args in the parameter list must be specified using keyword syntax in the call. You can also use a bare * in the parameter list to indicate that you don’t accept a variable-length argument list, but you do have keyword-only arguments.
Keyword arguments are allowed after the list of base classes in a class definition. This is used by the new convention for specifying a metaclass (see next section), but can be used for other purposes as well, as long as the metaclass supports it.
PEP 3104: nonlocal statement. Using nonlocal x you can now assign directly to a variable in an outer (but non-global) scope. nonlocal is a new reserved word.
PEP 3132: Extended Iterable Unpacking. You can now write things like a, b, *rest = some_sequence. And even *rest, a = stuff. The rest object is always a (possibly empty) list; the right-hand side may be any iterable. Example:
```
(a, *rest, b) = range(5)
```
This sets a to 0, b to 4, and rest to [1, 2, 3].
Dictionary comprehensions: {k: v for k, v in stuff} means the same thing as dict(stuff) but is more flexible. (This is PEP 0274 vindicated. :-)
Set literals, e.g. {1, 2}. Note that {} is an empty dictionary; use set() for an empty set. Set comprehensions are also supported; e.g., {x for x in stuff} means the same thing as set(stuff) but is more flexible.
New octal literals, e.g. 0o720 (already in 2.6). The old octal literals (0720) are gone.
New binary literals, e.g. 0b1010 (already in 2.6), and there is a new corresponding built-in function, bin().
Bytes literals are introduced with a leading b or B, and there is a new corresponding built-in function, bytes().

Changed Syntax¶

PEP 3109 and PEP 3134: new raise statement syntax: raise [expr [from expr]]. See below.
as and with are now reserved words. (Since 2.6, actually.)
True, False, and None are reserved words. (2.6 partially enforced the restrictions on None already.)
Change from except exc, var to except exc as var. See PEP 3110.
PEP 3115: New Metaclass Syntax. Instead of:
```
class C:
    __metaclass__ = M
    ...
```
you must now use:
```
class C(metaclass=M):
    ...
```
The module-global __metaclass__ variable is no longer supported. (It was a crutch to make it easier to default to new-style classes without deriving every class from object.)
List comprehensions no longer support the syntactic form [... for var in item1, item2, ...]. Use [... for var in (item1, item2, ...)] instead. Also note that list comprehensions have different semantics: they are closer to syntactic sugar for a generator expression inside a list() constructor, and in particular the loop control variables are no longer leaked into the surrounding scope.
The ellipsis (...) can be used as an atomic expression anywhere. (Previously it was only allowed in slices.) Also, it must now be spelled as .... (Previously it could also be spelled as . . ., by a mere accident of the grammar.)

Removed Syntax¶

PEP 3113: Tuple parameter unpacking removed. You can no longer write def foo(a, (b, c)): .... Use def foo(a, b_c): b, c = b_c instead.
Removed backticks (use repr() instead).
Removed <> (use != instead).
Removed keyword: exec() is no longer a keyword; it remains as a function. (Fortunately the function syntax was also accepted in 2.x.) Also note that exec() no longer takes a stream argument; instead of exec(f) you can use exec(f.read()).
Integer literals no longer support a trailing l or L.
String literals no longer support a leading u or U.
The from module import * syntax is only allowed at the module level, no longer inside functions.
The only acceptable syntax for relative imports is from .[module] import name. All import forms not starting with . are interpreted as absolute imports. (PEP 0328)
Classic classes are gone.

Changes Already Present In Python 2.6¶

Since many users presumably make the jump straight from Python 2.5 to Python 3.0, this section reminds the reader of new features that were originally designed for Python 3.0 but that were back-ported to Python 2.6. The corresponding sections in What’s New in Python 2.6 should be consulted for longer descriptions.

PEP 343: The ‘with’ statement. The with statement is now a standard feature and no longer needs to be imported from the __future__. Also check out Writing Context Managers and The contextlib module.
PEP 366: Explicit Relative Imports From a Main Module. This enhances the usefulness of the -m option when the referenced module lives in a package.
PEP 370: Per-user site-packages Directory.
PEP 371: The multiprocessing Package.
PEP 3101: Advanced String Formatting. Note: the 2.6 description mentions the format() method for both 8-bit and Unicode strings. In 3.0, only the str type (text strings with Unicode support) supports this method; the bytes type does not. The plan is to eventually make this the only API for string formatting, and to start deprecating the % operator in Python 3.1.
PEP 3105: print As a Function. This is now a standard feature and no longer needs to be imported from __future__. More details were given above.
PEP 3110: Exception-Handling Changes. The except exc as var syntax is now standard and except exc, var is no longer supported. (Of course, the as var part is still optional.)
PEP 3112: Byte Literals. The b"..." string literal notation (and its variants like b'...', b"""...""", and br"...") now produces a literal of type bytes.
PEP 3116: New I/O Library. The io module is now the standard way of doing file I/O, and the initial values of sys.stdin, sys.stdout and sys.stderr are now instances of io.TextIOBase. The built-in open() function is now an alias for io.open() and has additional keyword arguments encoding, errors, newline and closefd. Also note that an invalid mode argument now raises ValueError, not IOError. The binary file object underlying a text file object can be accessed as f.buffer (but beware that the text object maintains a buffer of itself in order to speed up the encoding and decoding operations).
PEP 3118: Revised Buffer Protocol. The old builtin buffer() is now really gone; the new builtin memoryview() provides (mostly) similar functionality.
PEP 3119: Abstract Base Classes. The abc module and the ABCs defined in the collections module plays a somewhat more prominent role in the language now, and built-in collection types like dict and list conform to the collections.MutableMapping and collections.MutableSequence ABCs, respectively.
PEP 3127: Integer Literal Support and Syntax. As mentioned above, the new octal literal notation is the only one supported, and binary literals have been added.
PEP 3129: Class Decorators.
PEP 3141: A Type Hierarchy for Numbers. The numbers module is another new use of ABCs, defining Python’s “numeric tower”. Also note the new fractions module which implements numbers.Rational.

Library Changes¶

Due to time constraints, this document does not exhaustively cover the very extensive changes to the standard library. PEP 3108 is the reference for the major changes to the library. Here’s a capsule review:

Many old modules were removed. Some, like gopherlib (no longer used) and md5 (replaced by hashlib), were already deprecated by PEP 0004. Others were removed as a result of the removal of support for various platforms such as Irix, BeOS and Mac OS 9 (see PEP 0011). Some modules were also selected for removal in Python 3.0 due to lack of use or because a better replacement exists. See PEP 3108 for an exhaustive list.
The bsddb3 package was removed because its presence in the core standard library has proved over time to be a particular burden for the core developers due to testing instability and Berkeley DB’s release schedule. However, the package is alive and well, externally maintained at http://www.jcea.es/programacion/pybsddb.htm.
Some modules were renamed because their old name disobeyed PEP 0008, or for various other reasons. Here’s the list:

Old Name New Name

_winreg winreg

ConfigParser configparser

copy_reg copyreg

Queue queue

SocketServer socketserver

markupbase _markupbase

repr reprlib

test.test_support test.support
A common pattern in Python 2.x is to have one version of a module implemented in pure Python, with an optional accelerated version implemented as a C extension; for example, pickle and cPickle. This places the burden of importing the accelerated version and falling back on the pure Python version on each user of these modules. In Python 3.0, the accelerated versions are considered implementation details of the pure Python versions. Users should always import the standard version, which attempts to import the accelerated version and falls back to the pure Python version. The pickle / cPickle pair received this treatment. The profile module is on the list for 3.1. The StringIO module has been turned into a class in the io module.
Some related modules have been grouped into packages, and usually the submodule names have been simplified. The resulting new packages are:
- dbm (anydbm, dbhash, dbm, dumbdbm, gdbm, whichdb).
- html (HTMLParser, htmlentitydefs).
- http (httplib, BaseHTTPServer, CGIHTTPServer, SimpleHTTPServer, Cookie, cookielib).
- tkinter (all Tkinter-related modules except turtle). The target audience of turtle doesn’t really care about tkinter. Also note that as of Python 2.6, the functionality of turtle has been greatly enhanced.
- urllib (urllib, urllib2, urlparse, robotparse).
- xmlrpc (xmlrpclib, DocXMLRPCServer, SimpleXMLRPCServer).

Old Name	New Name
_winreg	winreg
ConfigParser	configparser
copy_reg	copyreg
Queue	queue
SocketServer	socketserver
markupbase	_markupbase
repr	reprlib
test.test_support	test.support

Some other changes to standard library modules, not covered by PEP 3108:

Killed sets. Use the built-in set() class.
Cleanup of the sys module: removed sys.exitfunc(), sys.exc_clear(), sys.exc_type, sys.exc_value, sys.exc_traceback. (Note that sys.last_type etc. remain.)
Cleanup of the array.array type: the read() and write() methods are gone; use fromfile() and tofile() instead. Also, the 'c' typecode for array is gone – use either 'b' for bytes or 'u' for Unicode characters.
Cleanup of the operator module: removed sequenceIncludes() and isCallable().
Cleanup of the thread module: acquire_lock() and release_lock() are gone; use acquire() and release() instead.
Cleanup of the random module: removed the jumpahead() API.
The new module is gone.
The functions os.tmpnam(), os.tempnam() and os.tmpfile() have been removed in favor of the tempfile module.
The tokenize module has been changed to work with bytes. The main entry point is now tokenize.tokenize(), instead of generate_tokens.
string.letters and its friends (string.lowercase and string.uppercase) are gone. Use string.ascii_letters etc. instead. (The reason for the removal is that string.letters and friends had locale-specific behavior, which is a bad idea for such attractively-named global “constants”.)
Renamed module __builtin__ to builtins (removing the underscores, adding an ‘s’). The __builtins__ variable found in most global namespaces is unchanged. To modify a builtin, you should use builtins, not __builtins__!

PEP 3101: A New Approach To String Formatting¶

A new system for built-in string formatting operations replaces the % string formatting operator. (However, the % operator is still supported; it will be deprecated in Python 3.1 and removed from the language at some later time.) Read PEP 3101 for the full scoop.

Changes To Exceptions¶

The APIs for raising and catching exception have been cleaned up and new powerful features added:

PEP 0352: All exceptions must be derived (directly or indirectly) from BaseException. This is the root of the exception hierarchy. This is not new as a recommendation, but the requirement to inherit from BaseException is new. (Python 2.6 still allowed classic classes to be raised, and placed no restriction on what you can catch.) As a consequence, string exceptions are finally truly and utterly dead.
Almost all exceptions should actually derive from Exception; BaseException should only be used as a base class for exceptions that should only be handled at the top level, such as SystemExit or KeyboardInterrupt. The recommended idiom for handling all exceptions except for this latter category is to use except Exception.
StandardError was removed.
Exceptions no longer behave as sequences. Use the args attribute instead.
PEP 3109: Raising exceptions. You must now use raise Exception(args) instead of raise Exception, args. Additionally, you can no longer explicitly specify a traceback; instead, if you have to do this, you can assign directly to the __traceback__ attribute (see below).
PEP 3110: Catching exceptions. You must now use except SomeException as variable instead of except SomeException, variable. Moreover, the variable is explicitly deleted when the except block is left.
PEP 3134: Exception chaining. There are two cases: implicit chaining and explicit chaining. Implicit chaining happens when an exception is raised in an except or finally handler block. This usually happens due to a bug in the handler block; we call this a secondary exception. In this case, the original exception (that was being handled) is saved as the __context__ attribute of the secondary exception. Explicit chaining is invoked with this syntax:
```
raise SecondaryException() from primary_exception
```
(where primary_exception is any expression that produces an exception object, probably an exception that was previously caught). In this case, the primary exception is stored on the __cause__ attribute of the secondary exception. The traceback printed when an unhandled exception occurs walks the chain of __cause__ and __context__ attributes and prints a separate traceback for each component of the chain, with the primary exception at the top. (Java users may recognize this behavior.)
PEP 3134: Exception objects now store their traceback as the __traceback__ attribute. This means that an exception object now contains all the information pertaining to an exception, and there are fewer reasons to use sys.exc_info() (though the latter is not removed).
A few exception messages are improved when Windows fails to load an extension module. For example, error code 193 is now %1 is not a valid Win32 application. Strings now deal with non-English locales.

Miscellaneous Other Changes¶

Operators And Special Methods¶

!= now returns the opposite of ==, unless == returns NotImplemented.
The concept of “unbound methods” has been removed from the language. When referencing a method as a class attribute, you now get a plain function object.
__getslice__(), __setslice__() and __delslice__() were killed. The syntax a[i:j] now translates to a.__getitem__(slice(i, j)) (or __setitem__() or __delitem__(), when used as an assignment or deletion target, respectively).
PEP 3114: the standard next() method has been renamed to __next__().
The __oct__() and __hex__() special methods are removed – oct() and hex() use __index__() now to convert the argument to an integer.
Removed support for __members__ and __methods__.
The function attributes named func_X have been renamed to use the __X__ form, freeing up these names in the function attribute namespace for user-defined attributes. To wit, func_closure, func_code, func_defaults, func_dict, func_doc, func_globals, func_name were renamed to __closure__, __code__, __defaults__, __dict__, __doc__, __globals__, __name__, respectively.
__nonzero__() is now __bool__().

Builtins¶

PEP 3135: New super(). You can now invoke super() without arguments and (assuming this is in a regular instance method defined inside a class statement) the right class and instance will automatically be chosen. With arguments, the behavior of super() is unchanged.
PEP 3111: raw_input() was renamed to input(). That is, the new input() function reads a line from sys.stdin and returns it with the trailing newline stripped. It raises EOFError if the input is terminated prematurely. To get the old behavior of input(), use eval(input()).
A new built-in function next() was added to call the __next__() method on an object.
The round() function rounding strategy and return type have changed. Exact halfway cases are now rounded to the nearest even result instead of away from zero. (For example, round(2.5) now returns 2 rather than 3.) round(x[, n])() now delegates to x.__round__([n]) instead of always returning a float. It generally returns an integer when called with a single argument and a value of the same type as x when called with two arguments.
Moved intern() to sys.intern().
Removed: apply(). Instead of apply(f, args) use f(*args).
Removed callable(). Instead of callable(f) you can use isinstance(f, collections.Callable). The operator.isCallable() function is also gone.
Removed coerce(). This function no longer serves a purpose now that classic classes are gone.
Removed execfile(). Instead of execfile(fn) use exec(open(fn).read()).
Removed the file type. Use open(). There are now several different kinds of streams that open can return in the io module.
Removed reduce(). Use functools.reduce() if you really need it; however, 99 percent of the time an explicit for loop is more readable.
Removed reload(). Use imp.reload().
Removed. dict.has_key() – use the in operator instead.

Build and C API Changes¶

Due to time constraints, here is a very incomplete list of changes to the C API.

Support for several platforms was dropped, including but not limited to Mac OS 9, BeOS, RISCOS, Irix, and Tru64.
PEP 3118: New Buffer API.
PEP 3121: Extension Module Initialization & Finalization.
PEP 3123: Making PyObject_HEAD conform to standard C.
No more C API support for restricted execution.
PyNumber_Coerce(), PyNumber_CoerceEx(), PyMember_Get(), and PyMember_Set() C APIs are removed.
New C API PyImport_ImportModuleNoBlock(), works like PyImport_ImportModule() but won’t block on the import lock (returning an error instead).
Renamed the boolean conversion C-level slot and method: nb_nonzero is now nb_bool.
Removed METH_OLDARGS and WITH_CYCLE_GC from the C API.

Performance¶

The net result of the 3.0 generalizations is that Python 3.0 runs the pystone benchmark around 10% slower than Python 2.5. Most likely the biggest cause is the removal of special-casing for small integers. There’s room for improvement, but it will happen after 3.0 is released!

Porting To Python 3.0¶

For porting existing Python 2.5 or 2.6 source code to Python 3.0, the best strategy is the following:

(Prerequisite:) Start with excellent test coverage.
Port to Python 2.6. This should be no more work than the average port from Python 2.x to Python 2.(x+1). Make sure all your tests pass.
(Still using 2.6:) Turn on the -3 command line switch. This enables warnings about features that will be removed (or change) in 3.0. Run your test suite again, and fix code that you get warnings about until there are no warnings left, and all your tests still pass.
Run the 2to3 source-to-source translator over your source code tree. (See 2to3 - Automated Python 2 to 3 code translation for more on this tool.) Run the result of the translation under Python 3.0. Manually fix up any remaining issues, fixing problems until all tests pass again.

It is not recommended to try to write source code that runs unchanged under both Python 2.6 and 3.0; you’d have to use a very contorted coding style, e.g. avoiding print statements, metaclasses, and much more. If you are maintaining a library that needs to support both Python 2.6 and Python 3.0, the best approach is to modify step 3 above by editing the 2.6 version of the source code and running the 2to3 translator again, rather than editing the 3.0 version of the source code.

For porting C extensions to Python 3.0, please see Porting Extension Modules to 3.0.

What’s New in Python 2.7¶

Author:	A.M. Kuchling (amk at amk.ca)
Release:	3.2.2
Date:	August 02, 2015

This article explains the new features in Python 2.7. The final release of 2.7 is currently scheduled for July 2010; the detailed schedule is described in PEP 373.

Numeric handling has been improved in many ways, for both floating-point numbers and for the Decimal class. There are some useful additions to the standard library, such as a greatly enhanced unittest module, the argparse module for parsing command-line options, convenient ordered-dictionary and Counter classes in the collections module, and many other improvements.

Python 2.7 is planned to be the last of the 2.x releases, so we worked on making it a good release for the long term. To help with porting to Python 3, several new features from the Python 3.x series have been included in 2.7.

This article doesn’t attempt to provide a complete specification of the new features, but instead provides a convenient overview. For full details, you should refer to the documentation for Python 2.7 at http://docs.python.org. If you want to understand the rationale for the design and implementation, refer to the PEP for a particular new feature or the issue on http://bugs.python.org in which a change was discussed. Whenever possible, “What’s New in Python” links to the bug/patch item for each change.

The Future for Python 2.x¶

Python 2.7 is intended to be the last major release in the 2.x series. The Python maintainers are planning to focus their future efforts on the Python 3.x series.

This means that 2.7 will remain in place for a long time, running production systems that have not been ported to Python 3.x. Two consequences of the long-term significance of 2.7 are:

It’s very likely the 2.7 release will have a longer period of maintenance compared to earlier 2.x versions. Python 2.7 will continue to be maintained while the transition to 3.x continues, and the developers are planning to support Python 2.7 with bug-fix releases beyond the typical two years.
A policy decision was made to silence warnings only of interest to developers. DeprecationWarning and its descendants are now ignored unless otherwise requested, preventing users from seeing warnings triggered by an application. This change was also made in the branch that will become Python 3.2. (Discussed on stdlib-sig and carried out in issue 7319.)

In previous releases, DeprecationWarning messages were enabled by default, providing Python developers with a clear indication of where their code may break in a future major version of Python.

However, there are increasingly many users of Python-based applications who are not directly involved in the development of those applications. DeprecationWarning messages are irrelevant to such users, making them worry about an application that’s actually working correctly and burdening application developers with responding to these concerns.

You can re-enable display of DeprecationWarning messages by running Python with the -Wdefault (short form: -Wd) switch, or by setting the PYTHONWARNINGS environment variable to "default" (or "d") before running Python. Python code can also re-enable them by calling warnings.simplefilter('default').

Python 3.1 Features¶

Much as Python 2.6 incorporated features from Python 3.0, version 2.7 incorporates some of the new features in Python 3.1. The 2.x series continues to provide tools for migrating to the 3.x series.

A partial list of 3.1 features that were backported to 2.7:

The syntax for set literals ({1,2,3} is a mutable set).
Dictionary and set comprehensions ({ i: i*2 for i in range(3)}).
Multiple context managers in a single with statement.
A new version of the io library, rewritten in C for performance.
The ordered-dictionary type described in PEP 372: Adding an Ordered Dictionary to collections.
The new "," format specifier described in PEP 378: Format Specifier for Thousands Separator.
The memoryview object.
A small subset of the importlib module, described below.
The repr() of a float x is shorter in many cases: it’s now based on the shortest decimal string that’s guaranteed to round back to x. As in previous versions of Python, it’s guaranteed that float(repr(x)) recovers x.
Float-to-string and string-to-float conversions are correctly rounded. The round() function is also now correctly rounded.
The PyCapsule type, used to provide a C API for extension modules.
The PyLong_AsLongAndOverflow() C API function.

Other new Python3-mode warnings include:

operator.isCallable() and operator.sequenceIncludes(), which are not supported in 3.x, now trigger warnings.
The -3 switch now automatically enables the -Qwarn switch that causes warnings about using classic division with integers and long integers.

PEP 372: Adding an Ordered Dictionary to collections¶

Regular Python dictionaries iterate over key/value pairs in arbitrary order. Over the years, a number of authors have written alternative implementations that remember the order that the keys were originally inserted. Based on the experiences from those implementations, 2.7 introduces a new OrderedDict class in the collections module.

The OrderedDict API provides the same interface as regular dictionaries but iterates over keys and values in a guaranteed order depending on when a key was first inserted:

>>> from collections import OrderedDict
>>> d = OrderedDict([('first', 1),
...                  ('second', 2),
...                  ('third', 3)])
>>> d.items()
[('first', 1), ('second', 2), ('third', 3)]

If a new entry overwrites an existing entry, the original insertion position is left unchanged:

>>> d['second'] = 4
>>> d.items()
[('first', 1), ('second', 4), ('third', 3)]

Deleting an entry and reinserting it will move it to the end:

>>> del d['second']
>>> d['second'] = 5
>>> d.items()
[('first', 1), ('third', 3), ('second', 5)]

The popitem() method has an optional last argument that defaults to True. If last is True, the most recently added key is returned and removed; if it’s False, the oldest key is selected:

>>> od = OrderedDict([(x,0) for x in range(20)])
>>> od.popitem()
(19, 0)
>>> od.popitem()
(18, 0)
>>> od.popitem(last=False)
(0, 0)
>>> od.popitem(last=False)
(1, 0)

Comparing two ordered dictionaries checks both the keys and values, and requires that the insertion order was the same:

>>> od1 = OrderedDict([('first', 1),
...                    ('second', 2),
...                    ('third', 3)])
>>> od2 = OrderedDict([('third', 3),
...                    ('first', 1),
...                    ('second', 2)])
>>> od1 == od2
False
>>> # Move 'third' key to the end
>>> del od2['third']; od2['third'] = 3
>>> od1 == od2
True

Comparing an OrderedDict with a regular dictionary ignores the insertion order and just compares the keys and values.

How does the OrderedDict work? It maintains a doubly-linked list of keys, appending new keys to the list as they’re inserted. A secondary dictionary maps keys to their corresponding list node, so deletion doesn’t have to traverse the entire linked list and therefore remains O(1).

The standard library now supports use of ordered dictionaries in several modules.

The ConfigParser module uses them by default, meaning that configuration files can now be read, modified, and then written back in their original order.
The _asdict() method for collections.namedtuple() now returns an ordered dictionary with the values appearing in the same order as the underlying tuple indices.
The json module’s JSONDecoder class constructor was extended with an object_pairs_hook parameter to allow OrderedDict instances to be built by the decoder. Support was also added for third-party tools like PyYAML.

See also

PEP 372 - Adding an ordered dictionary to collections: PEP written by Armin Ronacher and Raymond Hettinger; implemented by Raymond Hettinger.

PEP 378: Format Specifier for Thousands Separator¶

To make program output more readable, it can be useful to add separators to large numbers, rendering them as 18,446,744,073,709,551,616 instead of 18446744073709551616.

The fully general solution for doing this is the locale module, which can use different separators (”,” in North America, ”.” in Europe) and different grouping sizes, but locale is complicated to use and unsuitable for multi-threaded applications where different threads are producing output for different locales.

Therefore, a simple comma-grouping mechanism has been added to the mini-language used by the str.format() method. When formatting a floating-point number, simply include a comma between the width and the precision:

>>> '{:20,.2f}'.format(18446744073709551616.0)
'18,446,744,073,709,551,616.00'

When formatting an integer, include the comma after the width:

>>> '{:20,d}'.format(18446744073709551616)
'18,446,744,073,709,551,616'

This mechanism is not adaptable at all; commas are always used as the separator and the grouping is always into three-digit groups. The comma-formatting mechanism isn’t as general as the locale module, but it’s easier to use.

See also

PEP 378 - Format Specifier for Thousands Separator: PEP written by Raymond Hettinger; implemented by Eric Smith.

PEP 389: The argparse Module for Parsing Command Lines¶

The argparse module for parsing command-line arguments was added as a more powerful replacement for the optparse module.

This means Python now supports three different modules for parsing command-line arguments: getopt, optparse, and argparse. The getopt module closely resembles the C library’s getopt() function, so it remains useful if you’re writing a Python prototype that will eventually be rewritten in C. optparse becomes redundant, but there are no plans to remove it because there are many scripts still using it, and there’s no automated way to update these scripts. (Making the argparse API consistent with optparse‘s interface was discussed but rejected as too messy and difficult.)

In short, if you’re writing a new script and don’t need to worry about compatibility with earlier versions of Python, use argparse instead of optparse.

Here’s an example:

import argparse

parser = argparse.ArgumentParser(description='Command-line example.')

# Add optional switches
parser.add_argument('-v', action='store_true', dest='is_verbose',
                    help='produce verbose output')
parser.add_argument('-o', action='store', dest='output',
                    metavar='FILE',
                    help='direct output to FILE instead of stdout')
parser.add_argument('-C', action='store', type=int, dest='context',
                    metavar='NUM', default=0,
                    help='display NUM lines of added context')

# Allow any number of additional arguments.
parser.add_argument(nargs='*', action='store', dest='inputs',
                    help='input filenames (default is stdin)')

args = parser.parse_args()
print args.__dict__

Unless you override it, -h and --help switches are automatically added, and produce neatly formatted output:

-> ./python.exe argparse-example.py --help
usage: argparse-example.py [-h] [-v] [-o FILE] [-C NUM] [inputs [inputs ...]]

Command-line example.

positional arguments:
  inputs      input filenames (default is stdin)

optional arguments:
  -h, --help  show this help message and exit
  -v          produce verbose output
  -o FILE     direct output to FILE instead of stdout
  -C NUM      display NUM lines of added context

As with optparse, the command-line switches and arguments are returned as an object with attributes named by the dest parameters:

-> ./python.exe argparse-example.py -v
{'output': None,
 'is_verbose': True,
 'context': 0,
 'inputs': []}

-> ./python.exe argparse-example.py -v -o /tmp/output -C 4 file1 file2
{'output': '/tmp/output',
 'is_verbose': True,
 'context': 4,
 'inputs': ['file1', 'file2']}

argparse has much fancier validation than optparse; you can specify an exact number of arguments as an integer, 0 or more arguments by passing '*', 1 or more by passing '+', or an optional argument with '?'. A top-level parser can contain sub-parsers to define subcommands that have different sets of switches, as in svn commit, svn checkout, etc. You can specify an argument’s type as FileType, which will automatically open files for you and understands that '-' means standard input or output.

See also

argparse module documentation

Upgrading optparse code to use argparse: Part of the Python documentation, describing how to convert code that uses optparse.
PEP 389 - argparse - New Command Line Parsing Module: PEP written and implemented by Steven Bethard.

PEP 391: Dictionary-Based Configuration For Logging¶

The logging module is very flexible; applications can define a tree of logging subsystems, and each logger in this tree can filter out certain messages, format them differently, and direct messages to a varying number of handlers.

All this flexibility can require a lot of configuration. You can write Python statements to create objects and set their properties, but a complex set-up requires verbose but boring code. logging also supports a fileConfig() function that parses a file, but the file format doesn’t support configuring filters, and it’s messier to generate programmatically.

Python 2.7 adds a dictConfig() function that uses a dictionary to configure logging. There are many ways to produce a dictionary from different sources: construct one with code; parse a file containing JSON; or use a YAML parsing library if one is installed.

The following example configures two loggers, the root logger and a logger named “network”. Messages sent to the root logger will be sent to the system log using the syslog protocol, and messages to the “network” logger will be written to a network.log file that will be rotated once the log reaches 1Mb.

import logging
import logging.config

configdict = {
 'version': 1,    # Configuration schema in use; must be 1 for now
 'formatters': {
     'standard': {
         'format': ('%(asctime)s %(name)-15s '
                    '%(levelname)-8s %(message)s')}},

 'handlers': {'netlog': {'backupCount': 10,
                     'class': 'logging.handlers.RotatingFileHandler',
                     'filename': '/logs/network.log',
                     'formatter': 'standard',
                     'level': 'INFO',
                     'maxBytes': 1024*1024},
              'syslog': {'class': 'logging.handlers.SysLogHandler',
                         'formatter': 'standard',
                         'level': 'ERROR'}},

 # Specify all the subordinate loggers
 'loggers': {
             'network': {
                         'handlers': ['netlog']
             }
 },
 # Specify properties of the root logger
 'root': {
          'handlers': ['syslog']
 },
}

# Set up configuration
logging.config.dictConfig(configdict)

# As an example, log two error messages
logger = logging.getLogger('/')
logger.error('Database not found')

netlogger = logging.getLogger('network')
netlogger.error('Connection failed')

Three smaller enhancements to the logging module, all implemented by Vinay Sajip, are:

The SysLogHandler class now supports syslogging over TCP. The constructor has a socktype parameter giving the type of socket to use, either socket.SOCK_DGRAM for UDP or socket.SOCK_STREAM for TCP. The default protocol remains UDP.
Logger instances gained a getChild() method that retrieves a descendant logger using a relative path. For example, once you retrieve a logger by doing log = getLogger('app'), calling log.getChild('network.listen') is equivalent to getLogger('app.network.listen').
The LoggerAdapter class gained a isEnabledFor() method that takes a level and returns whether the underlying logger would process a message of that level of importance.

See also

PEP 391 - Dictionary-Based Configuration For Logging: PEP written and implemented by Vinay Sajip.

PEP 3106: Dictionary Views¶

The dictionary methods keys(), values(), and items() are different in Python 3.x. They return an object called a view instead of a fully materialized list.

It’s not possible to change the return values of keys(), values(), and items() in Python 2.7 because too much code would break. Instead the 3.x versions were added under the new names viewkeys(), viewvalues(), and viewitems().

>>> d = dict((i*10, chr(65+i)) for i in range(26))
>>> d
{0: 'A', 130: 'N', 10: 'B', 140: 'O', 20: ..., 250: 'Z'}
>>> d.viewkeys()
dict_keys([0, 130, 10, 140, 20, 150, 30, ..., 250])

Views can be iterated over, but the key and item views also behave like sets. The & operator performs intersection, and | performs a union:

>>> d1 = dict((i*10, chr(65+i)) for i in range(26))
>>> d2 = dict((i**.5, i) for i in range(1000))
>>> d1.viewkeys() & d2.viewkeys()
set([0.0, 10.0, 20.0, 30.0])
>>> d1.viewkeys() | range(0, 30)
set([0, 1, 130, 3, 4, 5, 6, ..., 120, 250])

The view keeps track of the dictionary and its contents change as the dictionary is modified:

>>> vk = d.viewkeys()
>>> vk
dict_keys([0, 130, 10, ..., 250])
>>> d[260] = '&'
>>> vk
dict_keys([0, 130, 260, 10, ..., 250])

However, note that you can’t add or remove keys while you’re iterating over the view:

>>> for k in vk:
...     d[k*2] = k
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: dictionary changed size during iteration

You can use the view methods in Python 2.x code, and the 2to3 converter will change them to the standard keys(), values(), and items() methods.

See also

PEP 3106 - Revamping dict.keys(), .values() and .items(): PEP written by Guido van Rossum. Backported to 2.7 by Alexandre Vassalotti; issue 1967.

PEP 3137: The memoryview Object¶

The memoryview object provides a view of another object’s memory content that matches the bytes type’s interface.

>>> import string
>>> m = memoryview(string.letters)
>>> m
<memory at 0x37f850>
>>> len(m)           # Returns length of underlying object
52
>>> m[0], m[25], m[26]   # Indexing returns one byte
('a', 'z', 'A')
>>> m2 = m[0:26]         # Slicing returns another memoryview
>>> m2
<memory at 0x37f080>

The content of the view can be converted to a string of bytes or a list of integers:

>>> m2.tobytes()
'abcdefghijklmnopqrstuvwxyz'
>>> m2.tolist()
[97, 98, 99, 100, 101, 102, 103, ... 121, 122]
>>>

memoryview objects allow modifying the underlying object if it’s a mutable object.

>>> m2[0] = 75
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: cannot modify read-only memory
>>> b = bytearray(string.letters)  # Creating a mutable object
>>> b
bytearray(b'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ')
>>> mb = memoryview(b)
>>> mb[0] = '*'         # Assign to view, changing the bytearray.
>>> b[0:5]              # The bytearray has been changed.
bytearray(b'*bcde')
>>>

See also

PEP 3137 - Immutable Bytes and Mutable Buffer: PEP written by Guido van Rossum. Implemented by Travis Oliphant, Antoine Pitrou and others. Backported to 2.7 by Antoine Pitrou; issue 2396.

Other Language Changes¶

Some smaller changes made to the core Python language are:

The syntax for set literals has been backported from Python 3.x. Curly brackets are used to surround the contents of the resulting mutable set; set literals are distinguished from dictionaries by not containing colons and values. {} continues to represent an empty dictionary; use set() for an empty set.
```
>>> {1,2,3,4,5}
set([1, 2, 3, 4, 5])
>>> set() # empty set
set([])
>>> {}    # empty dict
{}
```
Backported by Alexandre Vassalotti; issue 2335.
Dictionary and set comprehensions are another feature backported from 3.x, generalizing list/generator comprehensions to use the literal syntax for sets and dictionaries.
```
>>> {x: x*x for x in range(6)}
{0: 0, 1: 1, 2: 4, 3: 9, 4: 16, 5: 25}
>>> {('a'*x) for x in range(6)}
set(['', 'a', 'aa', 'aaa', 'aaaa', 'aaaaa'])
```
Backported by Alexandre Vassalotti; issue 2333.
The with statement can now use multiple context managers in one statement. Context managers are processed from left to right and each one is treated as beginning a new with statement. This means that:
```
with A() as a, B() as b:
    ... suite of statements ...
```
is equivalent to:
```
with A() as a:
    with B() as b:
        ... suite of statements ...
```
The contextlib.nested() function provides a very similar function, so it’s no longer necessary and has been deprecated.

(Proposed in http://codereview.appspot.com/53094; implemented by Georg Brandl.)
Conversions between floating-point numbers and strings are now correctly rounded on most platforms. These conversions occur in many different places: str() on floats and complex numbers; the float and complex constructors; numeric formatting; serializing and deserializing floats and complex numbers using the marshal, pickle and json modules; parsing of float and imaginary literals in Python code; and Decimal-to-float conversion.

Related to this, the repr() of a floating-point number x now returns a result based on the shortest decimal string that’s guaranteed to round back to x under correct rounding (with round-half-to-even rounding mode). Previously it gave a string based on rounding x to 17 decimal digits.

The rounding library responsible for this improvement works on Windows and on Unix platforms using the gcc, icc, or suncc compilers. There may be a small number of platforms where correct operation of this code cannot be guaranteed, so the code is not used on such systems. You can find out which code is being used by checking sys.float_repr_style, which will be short if the new code is in use and legacy if it isn’t.

Implemented by Eric Smith and Mark Dickinson, using David Gay’s dtoa.c library; issue 7117.
Conversions from long integers and regular integers to floating point now round differently, returning the floating-point number closest to the number. This doesn’t matter for small integers that can be converted exactly, but for large numbers that will unavoidably lose precision, Python 2.7 now approximates more closely. For example, Python 2.6 computed the following:
```
>>> n = 295147905179352891391
>>> float(n)
2.9514790517935283e+20
>>> n - long(float(n))
65535L
```
Python 2.7’s floating-point result is larger, but much closer to the true value:
```
>>> n = 295147905179352891391
>>> float(n)
2.9514790517935289e+20
>>> n - long(float(n))
-1L
```
(Implemented by Mark Dickinson; issue 3166.)

Integer division is also more accurate in its rounding behaviours. (Also implemented by Mark Dickinson; issue 1811.)
Implicit coercion for complex numbers has been removed; the interpreter will no longer ever attempt to call a __coerce__() method on complex objects. (Removed by Meador Inge and Mark Dickinson; issue 5211.)
The str.format() method now supports automatic numbering of the replacement fields. This makes using str.format() more closely resemble using %s formatting:
```
>>> '{}:{}:{}'.format(2009, 04, 'Sunday')
'2009:4:Sunday'
>>> '{}:{}:{day}'.format(2009, 4, day='Sunday')
'2009:4:Sunday'
```
The auto-numbering takes the fields from left to right, so the first {...} specifier will use the first argument to str.format(), the next specifier will use the next argument, and so on. You can’t mix auto-numbering and explicit numbering – either number all of your specifier fields or none of them – but you can mix auto-numbering and named fields, as in the second example above. (Contributed by Eric Smith; issue 5237.)

Complex numbers now correctly support usage with format(), and default to being right-aligned. Specifying a precision or comma-separation applies to both the real and imaginary parts of the number, but a specified field width and alignment is applied to the whole of the resulting 1.5+3j output. (Contributed by Eric Smith; issue 1588 and issue 7988.)

The ‘F’ format code now always formats its output using uppercase characters, so it will now produce ‘INF’ and ‘NAN’. (Contributed by Eric Smith; issue 3382.)

A low-level change: the object.__format__() method now triggers a PendingDeprecationWarning if it’s passed a format string, because the __format__() method for object converts the object to a string representation and formats that. Previously the method silently applied the format string to the string representation, but that could hide mistakes in Python code. If you’re supplying formatting information such as an alignment or precision, presumably you’re expecting the formatting to be applied in some object-specific way. (Fixed by Eric Smith; issue 7994.)
The int() and long() types gained a bit_length method that returns the number of bits necessary to represent its argument in binary:
```
>>> n = 37
>>> bin(n)
'0b100101'
>>> n.bit_length()
6
>>> n = 2**123-1
>>> n.bit_length()
123
>>> (n+1).bit_length()
124
```
(Contributed by Fredrik Johansson and Victor Stinner; issue 3439.)
The import statement will no longer try an absolute import if a relative import (e.g. from .os import sep) fails. This fixes a bug, but could possibly break certain import statements that were only working by accident. (Fixed by Meador Inge; issue 7902.)
It’s now possible for a subclass of the built-in unicode type to override the __unicode__() method. (Implemented by Victor Stinner; issue 1583863.)
The bytearray type’s translate() method now accepts None as its first argument. (Fixed by Georg Brandl; issue 4759.)
When using @classmethod and @staticmethod to wrap methods as class or static methods, the wrapper object now exposes the wrapped function as their __func__ attribute. (Contributed by Amaury Forgeot d’Arc, after a suggestion by George Sakkis; issue 5982.)
When a restricted set of attributes were set using __slots__, deleting an unset attribute would not raise AttributeError as you would expect. Fixed by Benjamin Peterson; issue 7604.)
Two new encodings are now supported: “cp720”, used primarily for Arabic text; and “cp858”, a variant of CP 850 that adds the euro symbol. (CP720 contributed by Alexander Belchenko and Amaury Forgeot d’Arc in issue 1616979; CP858 contributed by Tim Hatch in issue 8016.)
The file object will now set the filename attribute on the IOError exception when trying to open a directory on POSIX platforms (noted by Jan Kaliszewski; issue 4764), and now explicitly checks for and forbids writing to read-only file objects instead of trusting the C library to catch and report the error (fixed by Stefan Krah; issue 5677).
The Python tokenizer now translates line endings itself, so the compile() built-in function now accepts code using any line-ending convention. Additionally, it no longer requires that the code end in a newline.
Extra parentheses in function definitions are illegal in Python 3.x, meaning that you get a syntax error from def f((x)): pass. In Python3-warning mode, Python 2.7 will now warn about this odd usage. (Noted by James Lingard; issue 7362.)
It’s now possible to create weak references to old-style class objects. New-style classes were always weak-referenceable. (Fixed by Antoine Pitrou; issue 8268.)
When a module object is garbage-collected, the module’s dictionary is now only cleared if no one else is holding a reference to the dictionary (issue 7140).

Interpreter Changes¶

A new environment variable, PYTHONWARNINGS, allows controlling warnings. It should be set to a string containing warning settings, equivalent to those used with the -W switch, separated by commas. (Contributed by Brian Curtin; issue 7301.)

For example, the following setting will print warnings every time they occur, but turn warnings from the Cookie module into an error. (The exact syntax for setting an environment variable varies across operating systems and shells.)

export PYTHONWARNINGS=all,error:::Cookie:0

Optimizations¶

Several performance enhancements have been added:

A new opcode was added to perform the initial setup for with statements, looking up the __enter__() and __exit__() methods. (Contributed by Benjamin Peterson.)
The garbage collector now performs better for one common usage pattern: when many objects are being allocated without deallocating any of them. This would previously take quadratic time for garbage collection, but now the number of full garbage collections is reduced as the number of objects on the heap grows. The new logic only performs a full garbage collection pass when the middle generation has been collected 10 times and when the number of survivor objects from the middle generation exceeds 10% of the number of objects in the oldest generation. (Suggested by Martin von Löwis and implemented by Antoine Pitrou; issue 4074.)
The garbage collector tries to avoid tracking simple containers which can’t be part of a cycle. In Python 2.7, this is now true for tuples and dicts containing atomic types (such as ints, strings, etc.). Transitively, a dict containing tuples of atomic types won’t be tracked either. This helps reduce the cost of each garbage collection by decreasing the number of objects to be considered and traversed by the collector. (Contributed by Antoine Pitrou; issue 4688.)
Long integers are now stored internally either in base 2**15 or in base 2**30, the base being determined at build time. Previously, they were always stored in base 2**15. Using base 2**30 gives significant performance improvements on 64-bit machines, but benchmark results on 32-bit machines have been mixed. Therefore, the default is to use base 2**30 on 64-bit machines and base 2**15 on 32-bit machines; on Unix, there’s a new configure option --enable-big-digits that can be used to override this default.

Apart from the performance improvements this change should be invisible to end users, with one exception: for testing and debugging purposes there’s a new structseq sys.long_info that provides information about the internal format, giving the number of bits per digit and the size in bytes of the C type used to store each digit:
```
>>> import sys
>>> sys.long_info
sys.long_info(bits_per_digit=30, sizeof_digit=4)
```
(Contributed by Mark Dickinson; issue 4258.)

Another set of changes made long objects a few bytes smaller: 2 bytes smaller on 32-bit systems and 6 bytes on 64-bit. (Contributed by Mark Dickinson; issue 5260.)
The division algorithm for long integers has been made faster by tightening the inner loop, doing shifts instead of multiplications, and fixing an unnecessary extra iteration. Various benchmarks show speedups of between 50% and 150% for long integer divisions and modulo operations. (Contributed by Mark Dickinson; issue 5512.) Bitwise operations are also significantly faster (initial patch by Gregory Smith; issue 1087418).
The implementation of % checks for the left-side operand being a Python string and special-cases it; this results in a 1-3% performance increase for applications that frequently use % with strings, such as templating libraries. (Implemented by Collin Winter; issue 5176.)
List comprehensions with an if condition are compiled into faster bytecode. (Patch by Antoine Pitrou, back-ported to 2.7 by Jeffrey Yasskin; issue 4715.)
Converting an integer or long integer to a decimal string was made faster by special-casing base 10 instead of using a generalized conversion function that supports arbitrary bases. (Patch by Gawain Bolton; issue 6713.)
The split(), replace(), rindex(), rpartition(), and rsplit() methods of string-like types (strings, Unicode strings, and bytearray objects) now use a fast reverse-search algorithm instead of a character-by-character scan. This is sometimes faster by a factor of 10. (Added by Florent Xicluna; issue 7462 and issue 7622.)
The pickle and cPickle modules now automatically intern the strings used for attribute names, reducing memory usage of the objects resulting from unpickling. (Contributed by Jake McGuire; issue 5084.)
The cPickle module now special-cases dictionaries, nearly halving the time required to pickle them. (Contributed by Collin Winter; issue 5670.)

New and Improved Modules¶

As in every release, Python’s standard library received a number of enhancements and bug fixes. Here’s a partial list of the most notable changes, sorted alphabetically by module name. Consult the Misc/NEWS file in the source tree for a more complete list of changes, or look through the Subversion logs for all the details.

The bdb module’s base debugging class Bdb gained a feature for skipping modules. The constructor now takes an iterable containing glob-style patterns such as django.*; the debugger will not step into stack frames from a module that matches one of these patterns. (Contributed by Maru Newby after a suggestion by Senthil Kumaran; issue 5142.)
The binascii module now supports the buffer API, so it can be used with memoryview instances and other similar buffer objects. (Backported from 3.x by Florent Xicluna; issue 7703.)
Updated module: the bsddb module has been updated from 4.7.2devel9 to version 4.8.4 of the pybsddb package. The new version features better Python 3.x compatibility, various bug fixes, and adds several new BerkeleyDB flags and methods. (Updated by Jesús Cea Avión; issue 8156. The pybsddb changelog can be read at http://hg.jcea.es/pybsddb/file/tip/ChangeLog.)
The bz2 module’s BZ2File now supports the context management protocol, so you can write with bz2.BZ2File(...) as f:. (Contributed by Hagen Fürstenau; issue 3860.)
New class: the Counter class in the collections module is useful for tallying data. Counter instances behave mostly like dictionaries but return zero for missing keys instead of raising a KeyError:
```
>>> from collections import Counter
>>> c = Counter()
>>> for letter in 'here is a sample of english text':
...   c[letter] += 1
...
>>> c
Counter({' ': 6, 'e': 5, 's': 3, 'a': 2, 'i': 2, 'h': 2,
'l': 2, 't': 2, 'g': 1, 'f': 1, 'm': 1, 'o': 1, 'n': 1,
'p': 1, 'r': 1, 'x': 1})
>>> c['e']
5
>>> c['z']
0
```
There are three additional Counter methods. most_common() returns the N most common elements and their counts. elements() returns an iterator over the contained elements, repeating each element as many times as its count. subtract() takes an iterable and subtracts one for each element instead of adding; if the argument is a dictionary or another Counter, the counts are subtracted.
```
>>> c.most_common(5)
[(' ', 6), ('e', 5), ('s', 3), ('a', 2), ('i', 2)]
>>> c.elements() ->
   'a', 'a', ' ', ' ', ' ', ' ', ' ', ' ',
   'e', 'e', 'e', 'e', 'e', 'g', 'f', 'i', 'i',
   'h', 'h', 'm', 'l', 'l', 'o', 'n', 'p', 's',
   's', 's', 'r', 't', 't', 'x'
>>> c['e']
5
>>> c.subtract('very heavy on the letter e')
>>> c['e']    # Count is now lower
-1
```
Contributed by Raymond Hettinger; issue 1696199.

New class: OrderedDict is described in the earlier section PEP 372: Adding an Ordered Dictionary to collections.

New method: The deque data type now has a count() method that returns the number of contained elements equal to the supplied argument x, and a reverse() method that reverses the elements of the deque in-place. deque also exposes its maximum length as the read-only maxlen attribute. (Both features added by Raymond Hettinger.)

The namedtuple class now has an optional rename parameter. If rename is true, field names that are invalid because they’ve been repeated or aren’t legal Python identifiers will be renamed to legal names that are derived from the field’s position within the list of fields:
```
>>> from collections import namedtuple
>>> T = namedtuple('T', ['field1', '$illegal', 'for', 'field2'], rename=True)
>>> T._fields
('field1', '_1', '_2', 'field2')
```
(Added by Raymond Hettinger; issue 1818.)

Finally, the Mapping abstract base class now returns NotImplemented if a mapping is compared to another type that isn’t a Mapping. (Fixed by Daniel Stutzbach; issue 8729.)

Constructors for the parsing classes in the ConfigParser module now take a allow_no_value parameter, defaulting to false; if true, options without values will be allowed. For example:

>>> import ConfigParser, StringIO
>>> sample_config = """
... [mysqld]
... user = mysql
... pid-file = /var/run/mysqld/mysqld.pid
... skip-bdb
... """
>>> config = ConfigParser.RawConfigParser(allow_no_value=True)
>>> config.readfp(StringIO.StringIO(sample_config))
>>> config.get('mysqld', 'user')
'mysql'
>>> print config.get('mysqld', 'skip-bdb')
None
>>> print config.get('mysqld', 'unknown')
Traceback (most recent call last):
  ...
NoOptionError: No option 'unknown' in section: 'mysqld'

(Contributed by Mats Kindahl; issue 7005.)

Deprecated function: contextlib.nested(), which allows handling more than one context manager with a single with statement, has been deprecated, because the with statement now supports multiple context managers.
The cookielib module now ignores cookies that have an invalid version field, one that doesn’t contain an integer value. (Fixed by John J. Lee; issue 3924.)
The copy module’s deepcopy() function will now correctly copy bound instance methods. (Implemented by Robert Collins; issue 1515.)
The ctypes module now always converts None to a C NULL pointer for arguments declared as pointers. (Changed by Thomas Heller; issue 4606.) The underlying libffi library has been updated to version 3.0.9, containing various fixes for different platforms. (Updated by Matthias Klose; issue 8142.)
New method: the datetime module’s timedelta class gained a total_seconds() method that returns the number of seconds in the duration. (Contributed by Brian Quinlan; issue 5788.)
New method: the Decimal class gained a from_float() class method that performs an exact conversion of a floating-point number to a Decimal. This exact conversion strives for the closest decimal approximation to the floating-point representation’s value; the resulting decimal value will therefore still include the inaccuracy, if any. For example, Decimal.from_float(0.1) returns Decimal('0.1000000000000000055511151231257827021181583404541015625'). (Implemented by Raymond Hettinger; issue 4796.)

Comparing instances of Decimal with floating-point numbers now produces sensible results based on the numeric values of the operands. Previously such comparisons would fall back to Python’s default rules for comparing objects, which produced arbitrary results based on their type. Note that you still cannot combine Decimal and floating-point in other operations such as addition, since you should be explicitly choosing how to convert between float and Decimal. (Fixed by Mark Dickinson; issue 2531.)

The constructor for Decimal now accepts floating-point numbers (added by Raymond Hettinger; issue 8257) and non-European Unicode characters such as Arabic-Indic digits (contributed by Mark Dickinson; issue 6595).

Most of the methods of the Context class now accept integers as well as Decimal instances; the only exceptions are the canonical() and is_canonical() methods. (Patch by Juan José Conti; issue 7633.)

When using Decimal instances with a string’s format() method, the default alignment was previously left-alignment. This has been changed to right-alignment, which is more sensible for numeric types. (Changed by Mark Dickinson; issue 6857.)

Comparisons involving a signaling NaN value (or sNAN) now signal InvalidOperation instead of silently returning a true or false value depending on the comparison operator. Quiet NaN values (or NaN) are now hashable. (Fixed by Mark Dickinson; issue 7279.)
The difflib module now produces output that is more compatible with modern diff/patch tools through one small change, using a tab character instead of spaces as a separator in the header giving the filename. (Fixed by Anatoly Techtonik; issue 7585.)
The Distutils sdist command now always regenerates the MANIFEST file, since even if the MANIFEST.in or setup.py files haven’t been modified, the user might have created some new files that should be included. (Fixed by Tarek Ziadé; issue 8688.)
The doctest module’s IGNORE_EXCEPTION_DETAIL flag will now ignore the name of the module containing the exception being tested. (Patch by Lennart Regebro; issue 7490.)
The email module’s Message class will now accept a Unicode-valued payload, automatically converting the payload to the encoding specified by output_charset. (Added by R. David Murray; issue 1368247.)
The Fraction class now accepts a single float or Decimal instance, or two rational numbers, as arguments to its constructor. (Implemented by Mark Dickinson; rationals added in issue 5812, and float/decimal in issue 8294.)

Ordering comparisons (<, <=, >, >=) between fractions and complex numbers now raise a TypeError. This fixes an oversight, making the Fraction match the other numeric types.
New class: FTP_TLS in the ftplib module provides secure FTP connections using TLS encapsulation of authentication as well as subsequent control and data transfers. (Contributed by Giampaolo Rodola; issue 2054.)

The storbinary() method for binary uploads can now restart uploads thanks to an added rest parameter (patch by Pablo Mouzo; issue 6845.)
New class decorator: total_ordering() in the functools module takes a class that defines an __eq__() method and one of __lt__(), __le__(), __gt__(), or __ge__(), and generates the missing comparison methods. Since the __cmp__() method is being deprecated in Python 3.x, this decorator makes it easier to define ordered classes. (Added by Raymond Hettinger; issue 5479.)

New function: cmp_to_key() will take an old-style comparison function that expects two arguments and return a new callable that can be used as the key parameter to functions such as sorted(), min() and max(), etc. The primary intended use is to help with making code compatible with Python 3.x. (Added by Raymond Hettinger.)
New function: the gc module’s is_tracked() returns true if a given instance is tracked by the garbage collector, false otherwise. (Contributed by Antoine Pitrou; issue 4688.)
The gzip module’s GzipFile now supports the context management protocol, so you can write with gzip.GzipFile(...) as f: (contributed by Hagen Fürstenau; issue 3860), and it now implements the io.BufferedIOBase ABC, so you can wrap it with io.BufferedReader for faster processing (contributed by Nir Aides; issue 7471). It’s also now possible to override the modification time recorded in a gzipped file by providing an optional timestamp to the constructor. (Contributed by Jacques Frechet; issue 4272.)

Files in gzip format can be padded with trailing zero bytes; the gzip module will now consume these trailing bytes. (Fixed by Tadek Pietraszek and Brian Curtin; issue 2846.)
New attribute: the hashlib module now has an algorithms attribute containing a tuple naming the supported algorithms. In Python 2.7, hashlib.algorithms contains ('md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512'). (Contributed by Carl Chenet; issue 7418.)
The default HTTPResponse class used by the httplib module now supports buffering, resulting in much faster reading of HTTP responses. (Contributed by Kristján Valur Jónsson; issue 4879.)

The HTTPConnection and HTTPSConnection classes now support a source_address parameter, a (host, port) 2-tuple giving the source address that will be used for the connection. (Contributed by Eldon Ziegler; issue 3972.)
The ihooks module now supports relative imports. Note that ihooks is an older module for customizing imports, superseded by the imputil module added in Python 2.0. (Relative import support added by Neil Schemenauer.)
The imaplib module now supports IPv6 addresses. (Contributed by Derek Morr; issue 1655.)

New function: the inspect module’s getcallargs() takes a callable and its positional and keyword arguments, and figures out which of the callable’s parameters will receive each argument, returning a dictionary mapping argument names to their values. For example:

>>> from inspect import getcallargs
>>> def f(a, b=1, *pos, **named):
...     pass
>>> getcallargs(f, 1, 2, 3)
{'a': 1, 'b': 2, 'pos': (3,), 'named': {}}
>>> getcallargs(f, a=2, x=4)
{'a': 2, 'b': 1, 'pos': (), 'named': {'x': 4}}
>>> getcallargs(f)
Traceback (most recent call last):
...
TypeError: f() takes at least 1 argument (0 given)

Contributed by George Sakkis; issue 3135.

Updated module: The io library has been upgraded to the version shipped with Python 3.1. For 3.1, the I/O library was entirely rewritten in C and is 2 to 20 times faster depending on the task being performed. The original Python version was renamed to the _pyio module.

One minor resulting change: the io.TextIOBase class now has an errors attribute giving the error setting used for encoding and decoding errors (one of 'strict', 'replace', 'ignore').

The io.FileIO class now raises an OSError when passed an invalid file descriptor. (Implemented by Benjamin Peterson; issue 4991.) The truncate() method now preserves the file position; previously it would change the file position to the end of the new file. (Fixed by Pascal Chambon; issue 6939.)
New function: itertools.compress(data, selectors) takes two iterators. Elements of data are returned if the corresponding value in selectors is true:
```
itertools.compress('ABCDEF', [1,0,1,0,1,1]) =>
  A, C, E, F
```
New function: itertools.combinations_with_replacement(iter, r) returns all the possible r-length combinations of elements from the iterable iter. Unlike combinations(), individual elements can be repeated in the generated combinations:
```
itertools.combinations_with_replacement('abc', 2) =>
  ('a', 'a'), ('a', 'b'), ('a', 'c'),
  ('b', 'b'), ('b', 'c'), ('c', 'c')
```
Note that elements are treated as unique depending on their position in the input, not their actual values.

The itertools.count() function now has a step argument that allows incrementing by values other than 1. count() also now allows keyword arguments, and using non-integer values such as floats or Decimal instances. (Implemented by Raymond Hettinger; issue 5032.)

itertools.combinations() and itertools.product() previously raised ValueError for values of r larger than the input iterable. This was deemed a specification error, so they now return an empty iterator. (Fixed by Raymond Hettinger; issue 4816.)
Updated module: The json module was upgraded to version 2.0.9 of the simplejson package, which includes a C extension that makes encoding and decoding faster. (Contributed by Bob Ippolito; issue 4136.)

To support the new collections.OrderedDict type, json.load() now has an optional object_pairs_hook parameter that will be called with any object literal that decodes to a list of pairs. (Contributed by Raymond Hettinger; issue 5381.)
The mailbox module’s Maildir class now records the timestamp on the directories it reads, and only re-reads them if the modification time has subsequently changed. This improves performance by avoiding unneeded directory scans. (Fixed by A.M. Kuchling and Antoine Pitrou; issue 1607951, issue 6896.)
New functions: the math module gained erf() and erfc() for the error function and the complementary error function, expm1() which computes e**x - 1 with more precision than using exp() and subtracting 1, gamma() for the Gamma function, and lgamma() for the natural log of the Gamma function. (Contributed by Mark Dickinson and nirinA raseliarison; issue 3366.)
The multiprocessing module’s Manager* classes can now be passed a callable that will be called whenever a subprocess is started, along with a set of arguments that will be passed to the callable. (Contributed by lekma; issue 5585.)

The Pool class, which controls a pool of worker processes, now has an optional maxtasksperchild parameter. Worker processes will perform the specified number of tasks and then exit, causing the Pool to start a new worker. This is useful if tasks may leak memory or other resources, or if some tasks will cause the worker to become very large. (Contributed by Charles Cazabon; issue 6963.)
The nntplib module now supports IPv6 addresses. (Contributed by Derek Morr; issue 1664.)
New functions: the os module wraps the following POSIX system calls: getresgid() and getresuid(), which return the real, effective, and saved GIDs and UIDs; setresgid() and setresuid(), which set real, effective, and saved GIDs and UIDs to new values; initgroups(), which initialize the group access list for the current process. (GID/UID functions contributed by Travis H.; issue 6508. Support for initgroups added by Jean-Paul Calderone; issue 7333.)

The os.fork() function now re-initializes the import lock in the child process; this fixes problems on Solaris when fork() is called from a thread. (Fixed by Zsolt Cserna; issue 7242.)
In the os.path module, the normpath() and abspath() functions now preserve Unicode; if their input path is a Unicode string, the return value is also a Unicode string. (normpath() fixed by Matt Giuca in issue 5827; abspath() fixed by Ezio Melotti in issue 3426.)
The pydoc module now has help for the various symbols that Python uses. You can now do help('<<') or help('@'), for example. (Contributed by David Laban; issue 4739.)
The re module’s split(), sub(), and subn() now accept an optional flags argument, for consistency with the other functions in the module. (Added by Gregory P. Smith.)
New function: run_path() in the runpy module will execute the code at a provided path argument. path can be the path of a Python source file (example.py), a compiled bytecode file (example.pyc), a directory (./package/), or a zip archive (example.zip). If a directory or zip path is provided, it will be added to the front of sys.path and the module __main__ will be imported. It’s expected that the directory or zip contains a __main__.py; if it doesn’t, some other __main__.py might be imported from a location later in sys.path. This makes more of the machinery of runpy available to scripts that want to mimic the way Python’s command line processes an explicit path name. (Added by Nick Coghlan; issue 6816.)
New function: in the shutil module, make_archive() takes a filename, archive type (zip or tar-format), and a directory path, and creates an archive containing the directory’s contents. (Added by Tarek Ziadé.)

shutil‘s copyfile() and copytree() functions now raise a SpecialFileError exception when asked to copy a named pipe. Previously the code would treat named pipes like a regular file by opening them for reading, and this would block indefinitely. (Fixed by Antoine Pitrou; issue 3002.)
The signal module no longer re-installs the signal handler unless this is truly necessary, which fixes a bug that could make it impossible to catch the EINTR signal robustly. (Fixed by Charles-Francois Natali; issue 8354.)
New functions: in the site module, three new functions return various site- and user-specific paths. getsitepackages() returns a list containing all global site-packages directories, getusersitepackages() returns the path of the user’s site-packages directory, and getuserbase() returns the value of the USER_BASE environment variable, giving the path to a directory that can be used to store data. (Contributed by Tarek Ziadé; issue 6693.)

The site module now reports exceptions occurring when the sitecustomize module is imported, and will no longer catch and swallow the KeyboardInterrupt exception. (Fixed by Victor Stinner; issue 3137.)
The create_connection() function gained a source_address parameter, a (host, port) 2-tuple giving the source address that will be used for the connection. (Contributed by Eldon Ziegler; issue 3972.)

The recv_into() and recvfrom_into() methods will now write into objects that support the buffer API, most usefully the bytearray and memoryview objects. (Implemented by Antoine Pitrou; issue 8104.)
The SocketServer module’s TCPServer class now supports socket timeouts and disabling the Nagle algorithm. The disable_nagle_algorithm class attribute defaults to False; if overridden to be True, new request connections will have the TCP_NODELAY option set to prevent buffering many small sends into a single TCP packet. The timeout class attribute can hold a timeout in seconds that will be applied to the request socket; if no request is received within that time, handle_timeout() will be called and handle_request() will return. (Contributed by Kristján Valur Jónsson; issue 6192 and issue 6267.)
Updated module: the sqlite3 module has been updated to version 2.6.0 of the pysqlite package. Version 2.6.0 includes a number of bugfixes, and adds the ability to load SQLite extensions from shared libraries. Call the enable_load_extension(True) method to enable extensions, and then call load_extension() to load a particular shared library. (Updated by Gerhard Häring.)
The ssl module’s ssl.SSLSocket objects now support the buffer API, which fixed a test suite failure (fix by Antoine Pitrou; issue 7133) and automatically set OpenSSL’s SSL_MODE_AUTO_RETRY, which will prevent an error code being returned from recv() operations that trigger an SSL renegotiation (fix by Antoine Pitrou; issue 8222).

The ssl.wrap_socket() constructor function now takes a ciphers argument that’s a string listing the encryption algorithms to be allowed; the format of the string is described in the OpenSSL documentation. (Added by Antoine Pitrou; issue 8322.)

Another change makes the extension load all of OpenSSL’s ciphers and digest algorithms so that they’re all available. Some SSL certificates couldn’t be verified, reporting an “unknown algorithm” error. (Reported by Beda Kosata, and fixed by Antoine Pitrou; issue 8484.)

The version of OpenSSL being used is now available as the module attributes ssl.OPENSSL_VERSION (a string), ssl.OPENSSL_VERSION_INFO (a 5-tuple), and ssl.OPENSSL_VERSION_NUMBER (an integer). (Added by Antoine Pitrou; issue 8321.)
The struct module will no longer silently ignore overflow errors when a value is too large for a particular integer format code (one of bBhHiIlLqQ); it now always raises a struct.error exception. (Changed by Mark Dickinson; issue 1523.) The pack() function will also attempt to use __index__() to convert and pack non-integers before trying the __int__() method or reporting an error. (Changed by Mark Dickinson; issue 8300.)
New function: the subprocess module’s check_output() runs a command with a specified set of arguments and returns the command’s output as a string when the command runs without error, or raises a CalledProcessError exception otherwise.
```
>>> subprocess.check_output(['df', '-h', '.'])
'Filesystem     Size   Used  Avail Capacity  Mounted on\n
/dev/disk0s2    52G    49G   3.0G    94%    /\n'

>>> subprocess.check_output(['df', '-h', '/bogus'])
  ...
subprocess.CalledProcessError: Command '['df', '-h', '/bogus']' returned non-zero exit status 1
```
(Contributed by Gregory P. Smith.)

The subprocess module will now retry its internal system calls on receiving an EINTR signal. (Reported by several people; final patch by Gregory P. Smith in issue 1068268.)
New function: is_declared_global() in the symtable module returns true for variables that are explicitly declared to be global, false for ones that are implicitly global. (Contributed by Jeremy Hylton.)
The syslog module will now use the value of sys.argv[0] as the identifier instead of the previous default value of 'python'. (Changed by Sean Reifschneider; issue 8451.)
The sys.version_info value is now a named tuple, with attributes named major, minor, micro, releaselevel, and serial. (Contributed by Ross Light; issue 4285.)

sys.getwindowsversion() also returns a named tuple, with attributes named major, minor, build, platform, service_pack, service_pack_major, service_pack_minor, suite_mask, and product_type. (Contributed by Brian Curtin; issue 7766.)
The tarfile module’s default error handling has changed, to no longer suppress fatal errors. The default error level was previously 0, which meant that errors would only result in a message being written to the debug log, but because the debug log is not activated by default, these errors go unnoticed. The default error level is now 1, which raises an exception if there’s an error. (Changed by Lars Gustäbel; issue 7357.)

tarfile now supports filtering the TarInfo objects being added to a tar file. When you call add(), you may supply an optional filter argument that’s a callable. The filter callable will be passed the TarInfo for every file being added, and can modify and return it. If the callable returns None, the file will be excluded from the resulting archive. This is more powerful than the existing exclude argument, which has therefore been deprecated. (Added by Lars Gustäbel; issue 6856.) The TarFile class also now supports the context manager protocol. (Added by Lars Gustäbel; issue 7232.)
The wait() method of the threading.Event class now returns the internal flag on exit. This means the method will usually return true because wait() is supposed to block until the internal flag becomes true. The return value will only be false if a timeout was provided and the operation timed out. (Contributed by Tim Lesher; issue 1674032.)
The Unicode database provided by the unicodedata module is now used internally to determine which characters are numeric, whitespace, or represent line breaks. The database also includes information from the Unihan.txt data file (patch by Anders Chrigström and Amaury Forgeot d’Arc; issue 1571184) and has been updated to version 5.2.0 (updated by Florent Xicluna; issue 8024).
The urlparse module’s urlsplit() now handles unknown URL schemes in a fashion compliant with RFC 3986: if the URL is of the form "<something>://...", the text before the :// is treated as the scheme, even if it’s a made-up scheme that the module doesn’t know about. This change may break code that worked around the old behaviour. For example, Python 2.6.4 or 2.5 will return the following:
```
>>> import urlparse
>>> urlparse.urlsplit('invented://host/filename?query')
('invented', '', '//host/filename?query', '', '')
```
Python 2.7 (and Python 2.6.5) will return:
```
>>> import urlparse
>>> urlparse.urlsplit('invented://host/filename?query')
('invented', 'host', '/filename?query', '', '')
```
(Python 2.7 actually produces slightly different output, since it returns a named tuple instead of a standard tuple.)

The urlparse module also supports IPv6 literal addresses as defined by RFC 2732 (contributed by Senthil Kumaran; issue 2987).
```
>>> urlparse.urlparse('http://[1080::8:800:200C:417A]/foo')
ParseResult(scheme='http', netloc='[1080::8:800:200C:417A]',
            path='/foo', params='', query='', fragment='')
```
New class: the WeakSet class in the weakref module is a set that only holds weak references to its elements; elements will be removed once there are no references pointing to them. (Originally implemented in Python 3.x by Raymond Hettinger, and backported to 2.7 by Michael Foord.)
The ElementTree library, xml.etree, no longer escapes ampersands and angle brackets when outputting an XML processing instruction (which looks like <?xml-stylesheet href="#style1"?>) or comment (which looks like ). (Patch by Neil Muller; issue 2746.)
The XML-RPC client and server, provided by the xmlrpclib and SimpleXMLRPCServer modules, have improved performance by supporting HTTP/1.1 keep-alive and by optionally using gzip encoding to compress the XML being exchanged. The gzip compression is controlled by the encode_threshold attribute of SimpleXMLRPCRequestHandler, which contains a size in bytes; responses larger than this will be compressed. (Contributed by Kristján Valur Jónsson; issue 6267.)
The zipfile module’s ZipFile now supports the context management protocol, so you can write with zipfile.ZipFile(...) as f:. (Contributed by Brian Curtin; issue 5511.)

zipfile now also supports archiving empty directories and extracts them correctly. (Fixed by Kuba Wieczorek; issue 4710.) Reading files out of an archive is faster, and interleaving read() and readline() now works correctly. (Contributed by Nir Aides; issue 7610.)

The is_zipfile() function now accepts a file object, in addition to the path names accepted in earlier versions. (Contributed by Gabriel Genellina; issue 4756.)

The writestr() method now has an optional compress_type parameter that lets you override the default compression method specified in the ZipFile constructor. (Contributed by Ronald Oussoren; issue 6003.)

New module: importlib¶

Python 3.1 includes the importlib package, a re-implementation of the logic underlying Python’s import statement. importlib is useful for implementors of Python interpreters and to users who wish to write new importers that can participate in the import process. Python 2.7 doesn’t contain the complete importlib package, but instead has a tiny subset that contains a single function, import_module().

import_module(name, package=None) imports a module. name is a string containing the module or package’s name. It’s possible to do relative imports by providing a string that begins with a . character, such as ..utils.errors. For relative imports, the package argument must be provided and is the name of the package that will be used as the anchor for the relative import. import_module() both inserts the imported module into sys.modules and returns the module object.

Here are some examples:

>>> from importlib import import_module
>>> anydbm = import_module('anydbm')  # Standard absolute import
>>> anydbm
<module 'anydbm' from '/p/python/Lib/anydbm.py'>
>>> # Relative import
>>> file_util = import_module('..file_util', 'distutils.command')
>>> file_util
<module 'distutils.file_util' from '/python/Lib/distutils/file_util.pyc'>

importlib was implemented by Brett Cannon and introduced in Python 3.1.

New module: sysconfig¶

The sysconfig module has been pulled out of the Distutils package, becoming a new top-level module in its own right. sysconfig provides functions for getting information about Python’s build process: compiler switches, installation paths, the platform name, and whether Python is running from its source directory.

Some of the functions in the module are:

get_config_var() returns variables from Python’s Makefile and the pyconfig.h file.
get_config_vars() returns a dictionary containing all of the configuration variables.
getpath() returns the configured path for a particular type of module: the standard library, site-specific modules, platform-specific modules, etc.
is_python_build() returns true if you’re running a binary from a Python source tree, and false otherwise.

Consult the sysconfig documentation for more details and for a complete list of functions.

The Distutils package and sysconfig are now maintained by Tarek Ziadé, who has also started a Distutils2 package (source repository at http://hg.python.org/distutils2/) for developing a next-generation version of Distutils.

ttk: Themed Widgets for Tk¶

Tcl/Tk 8.5 includes a set of themed widgets that re-implement basic Tk widgets but have a more customizable appearance and can therefore more closely resemble the native platform’s widgets. This widget set was originally called Tile, but was renamed to Ttk (for “themed Tk”) on being added to Tcl/Tck release 8.5.

To learn more, read the ttk module documentation. You may also wish to read the Tcl/Tk manual page describing the Ttk theme engine, available at http://www.tcl.tk/man/tcl8.5/TkCmd/ttk_intro.htm. Some screenshots of the Python/Ttk code in use are at http://code.google.com/p/python-ttk/wiki/Screenshots.

The ttk module was written by Guilherme Polo and added in issue 2983. An alternate version called Tile.py, written by Martin Franklin and maintained by Kevin Walzer, was proposed for inclusion in issue 2618, but the authors argued that Guilherme Polo’s work was more comprehensive.

Updated module: unittest¶

The unittest module was greatly enhanced; many new features were added. Most of these features were implemented by Michael Foord, unless otherwise noted. The enhanced version of the module is downloadable separately for use with Python versions 2.4 to 2.6, packaged as the unittest2 package, from http://pypi.python.org/pypi/unittest2.

When used from the command line, the module can automatically discover tests. It’s not as fancy as py.test or nose, but provides a simple way to run tests kept within a set of package directories. For example, the following command will search the test/ subdirectory for any importable test files named test*.py:

python -m unittest discover -s test

Consult the unittest module documentation for more details. (Developed in issue 6001.)

The main() function supports some other new options:

-b or --buffer will buffer the standard output and standard error streams during each test. If the test passes, any resulting output will be discarded; on failure, the buffered output will be displayed.
-c or --catch will cause the control-C interrupt to be handled more gracefully. Instead of interrupting the test process immediately, the currently running test will be completed and then the partial results up to the interruption will be reported. If you’re impatient, a second press of control-C will cause an immediate interruption.

This control-C handler tries to avoid causing problems when the code being tested or the tests being run have defined a signal handler of their own, by noticing that a signal handler was already set and calling it. If this doesn’t work for you, there’s a removeHandler() decorator that can be used to mark tests that should have the control-C handling disabled.
-f or --failfast makes test execution stop immediately when a test fails instead of continuing to execute further tests. (Suggested by Cliff Dyer and implemented by Michael Foord; issue 8074.)

The progress messages now show ‘x’ for expected failures and ‘u’ for unexpected successes when run in verbose mode. (Contributed by Benjamin Peterson.)

Test cases can raise the SkipTest exception to skip a test (issue 1034053).

The error messages for assertEqual(), assertTrue(), and assertFalse() failures now provide more information. If you set the longMessage attribute of your TestCase classes to True, both the standard error message and any additional message you provide will be printed for failures. (Added by Michael Foord; issue 5663.)

The assertRaises() method now returns a context handler when called without providing a callable object to run. For example, you can write this:

with self.assertRaises(KeyError):
    {}['foo']

(Implemented by Antoine Pitrou; issue 4444.)

Module- and class-level setup and teardown fixtures are now supported. Modules can contain setUpModule() and tearDownModule() functions. Classes can have setUpClass() and tearDownClass() methods that must be defined as class methods (using @classmethod or equivalent). These functions and methods are invoked when the test runner switches to a test case in a different module or class.

The methods addCleanup() and doCleanups() were added. addCleanup() lets you add cleanup functions that will be called unconditionally (after setUp() if setUp() fails, otherwise after tearDown()). This allows for much simpler resource allocation and deallocation during tests (issue 5679).

A number of new methods were added that provide more specialized tests. Many of these methods were written by Google engineers for use in their test suites; Gregory P. Smith, Michael Foord, and GvR worked on merging them into Python’s version of unittest.

assertIsNone() and assertIsNotNone() take one expression and verify that the result is or is not None.
assertIs() and assertIsNot() take two values and check whether the two values evaluate to the same object or not. (Added by Michael Foord; issue 2578.)
assertIsInstance() and assertNotIsInstance() check whether the resulting object is an instance of a particular class, or of one of a tuple of classes. (Added by Georg Brandl; issue 7031.)
assertGreater(), assertGreaterEqual(), assertLess(), and assertLessEqual() compare two quantities.
assertMultiLineEqual() compares two strings, and if they’re not equal, displays a helpful comparison that highlights the differences in the two strings. This comparison is now used by default when Unicode strings are compared with assertEqual().
assertRegexpMatches() and assertNotRegexpMatches() checks whether the first argument is a string matching or not matching the regular expression provided as the second argument (issue 8038).
assertRaisesRegexp() checks whether a particular exception is raised, and then also checks that the string representation of the exception matches the provided regular expression.
assertIn() and assertNotIn() tests whether first is or is not in second.
assertItemsEqual() tests whether two provided sequences contain the same elements.
assertSetEqual() compares whether two sets are equal, and only reports the differences between the sets in case of error.
Similarly, assertListEqual() and assertTupleEqual() compare the specified types and explain any differences without necessarily printing their full values; these methods are now used by default when comparing lists and tuples using assertEqual(). More generally, assertSequenceEqual() compares two sequences and can optionally check whether both sequences are of a particular type.
assertDictEqual() compares two dictionaries and reports the differences; it’s now used by default when you compare two dictionaries using assertEqual(). assertDictContainsSubset() checks whether all of the key/value pairs in first are found in second.
assertAlmostEqual() and assertNotAlmostEqual() test whether first and second are approximately equal. This method can either round their difference to an optionally-specified number of places (the default is 7) and compare it to zero, or require the difference to be smaller than a supplied delta value.
loadTestsFromName() properly honors the suiteClass attribute of the TestLoader. (Fixed by Mark Roddy; issue 6866.)
A new hook lets you extend the assertEqual() method to handle new data types. The addTypeEqualityFunc() method takes a type object and a function. The function will be used when both of the objects being compared are of the specified type. This function should compare the two objects and raise an exception if they don’t match; it’s a good idea for the function to provide additional information about why the two objects aren’t matching, much as the new sequence comparison methods do.

unittest.main() now takes an optional exit argument. If False, main() doesn’t call sys.exit(), allowing main() to be used from the interactive interpreter. (Contributed by J. Pablo Fernández; issue 3379.)

TestResult has new startTestRun() and stopTestRun() methods that are called immediately before and after a test run. (Contributed by Robert Collins; issue 5728.)

With all these changes, the unittest.py was becoming awkwardly large, so the module was turned into a package and the code split into several files (by Benjamin Peterson). This doesn’t affect how the module is imported or used.

See also

http://www.voidspace.org.uk/python/articles/unittest2.shtml: Describes the new features, how to use them, and the rationale for various design decisions. (By Michael Foord.)

Updated module: ElementTree 1.3¶

The version of the ElementTree library included with Python was updated to version 1.3. Some of the new features are:

The various parsing functions now take a parser keyword argument giving an XMLParser instance that will be used. This makes it possible to override the file’s internal encoding:
```
p = ET.XMLParser(encoding='utf-8')
t = ET.XML("""<root/>""", parser=p)
```
Errors in parsing XML now raise a ParseError exception, whose instances have a position attribute containing a (line, column) tuple giving the location of the problem.
ElementTree’s code for converting trees to a string has been significantly reworked, making it roughly twice as fast in many cases. The ElementTree write() and Element write() methods now have a method parameter that can be “xml” (the default), “html”, or “text”. HTML mode will output empty elements as <empty></empty> instead of <empty/>, and text mode will skip over elements and only output the text chunks. If you set the tag attribute of an element to None but leave its children in place, the element will be omitted when the tree is written out, so you don’t need to do more extensive rearrangement to remove a single element.

Namespace handling has also been improved. All xmlns:<whatever> declarations are now output on the root element, not scattered throughout the resulting XML. You can set the default namespace for a tree by setting the default_namespace attribute and can register new prefixes with register_namespace(). In XML mode, you can use the true/false xml_declaration parameter to suppress the XML declaration.

New Element method: extend() appends the items from a sequence to the element’s children. Elements themselves behave like sequences, so it’s easy to move children from one element to another:

from xml.etree import ElementTree as ET

t = ET.XML("""<list>
  <item>1</item> <item>2</item>  <item>3</item>
</list>""")
new = ET.XML('<root/>')
new.extend(t)

# Outputs <root><item>1</item>...</root>
print ET.tostring(new)

New Element method: iter() yields the children of the element as a generator. It’s also possible to write for child in elem: to loop over an element’s children. The existing method getiterator() is now deprecated, as is getchildren() which constructs and returns a list of children.

New Element method: itertext() yields all chunks of text that are descendants of the element. For example:

t = ET.XML("""<list>
  <item>1</item> <item>2</item>  <item>3</item>
</list>""")

# Outputs ['\n  ', '1', ' ', '2', '  ', '3', '\n']
print list(t.itertext())

Deprecated: using an element as a Boolean (i.e., if elem:) would return true if the element had any children, or false if there were no children. This behaviour is confusing – None is false, but so is a childless element? – so it will now trigger a FutureWarning. In your code, you should be explicit: write len(elem) != 0 if you’re interested in the number of children, or elem is not None.

Fredrik Lundh develops ElementTree and produced the 1.3 version; you can read his article describing 1.3 at http://effbot.org/zone/elementtree-13-intro.htm. Florent Xicluna updated the version included with Python, after discussions on python-dev and in issue 6472.)

Build and C API Changes¶

Changes to Python’s build process and to the C API include:

The latest release of the GNU Debugger, GDB 7, can be scripted using Python. When you begin debugging an executable program P, GDB will look for a file named P-gdb.py and automatically read it. Dave Malcolm contributed a python-gdb.py that adds a number of commands useful when debugging Python itself. For example, py-up and py-down go up or down one Python stack frame, which usually corresponds to several C stack frames. py-print prints the value of a Python variable, and py-bt prints the Python stack trace. (Added as a result of issue 8032.)
If you use the .gdbinit file provided with Python, the “pyo” macro in the 2.7 version now works correctly when the thread being debugged doesn’t hold the GIL; the macro now acquires it before printing. (Contributed by Victor Stinner; issue 3632.)
Py_AddPendingCall() is now thread-safe, letting any worker thread submit notifications to the main Python thread. This is particularly useful for asynchronous IO operations. (Contributed by Kristján Valur Jónsson; issue 4293.)
New function: PyCode_NewEmpty() creates an empty code object; only the filename, function name, and first line number are required. This is useful for extension modules that are attempting to construct a more useful traceback stack. Previously such extensions needed to call PyCode_New(), which had many more arguments. (Added by Jeffrey Yasskin.)
New function: PyErr_NewExceptionWithDoc() creates a new exception class, just as the existing PyErr_NewException() does, but takes an extra char * argument containing the docstring for the new exception class. (Added by ‘lekma’ on the Python bug tracker; issue 7033.)
New function: PyFrame_GetLineNumber() takes a frame object and returns the line number that the frame is currently executing. Previously code would need to get the index of the bytecode instruction currently executing, and then look up the line number corresponding to that address. (Added by Jeffrey Yasskin.)
New functions: PyLong_AsLongAndOverflow() and PyLong_AsLongLongAndOverflow() approximates a Python long integer as a C long or long long. If the number is too large to fit into the output type, an overflow flag is set and returned to the caller. (Contributed by Case Van Horsen; issue 7528 and issue 7767.)
New function: stemming from the rewrite of string-to-float conversion, a new PyOS_string_to_double() function was added. The old PyOS_ascii_strtod() and PyOS_ascii_atof() functions are now deprecated.
New function: PySys_SetArgvEx() sets the value of sys.argv and can optionally update sys.path to include the directory containing the script named by sys.argv[0] depending on the value of an updatepath parameter.

This function was added to close a security hole for applications that embed Python. The old function, PySys_SetArgv(), would always update sys.path, and sometimes it would add the current directory. This meant that, if you

Python 3 文档(简体中文) 3.2.2 documentation

Python Documentation contents¶

What’s New in Python¶

What’s New In Python 3.2¶

PEP 384: Defining a Stable ABI¶

PEP 389: Argparse Command Line Parsing Module¶

PEP 391: Dictionary Based Configuration for Logging¶

PEP 3148: The concurrent.futures module¶

PEP 3147: PYC Repository Directories¶

PEP 3149: ABI Version Tagged .so Files¶

PEP 3333: Python Web Server Gateway Interface v1.0.1¶

Other Language Changes¶

New, Improved, and Deprecated Modules¶

email¶

elementtree¶

functools¶

itertools¶

collections¶

threading¶

datetime and time¶

math¶

abc¶

io¶

reprlib¶

logging¶

csv¶

contextlib¶

decimal and fractions¶

ftp¶

popen¶

select¶

gzip and zipfile¶

tarfile¶

hashlib¶

ast¶

os¶

shutil¶

sqlite3¶

html¶

socket¶

ssl¶

nntp¶

certificates¶

imaplib¶

http.client¶

unittest¶

random¶

poplib¶

asyncore¶

tempfile¶

inspect¶

pydoc¶

dis¶

dbm¶

ctypes¶

site¶

sysconfig¶

pdb¶

configparser¶

urllib.parse¶

mailbox¶

turtledemo¶

Multi-threading¶

Optimizations¶

Unicode¶

Codecs¶

Documentation¶

IDLE¶

Code Repository¶

Build and C API Changes¶

Porting to Python 3.2¶

What’s New In Python 3.1¶

PEP 372: Ordered Dictionaries¶

PEP 378: Format Specifier for Thousands Separator¶

Other Language Changes¶

New, Improved, and Deprecated Modules¶

Optimizations¶

IDLE¶

Build and C API Changes¶

Porting to Python 3.1¶

PEP 3148: The `concurrent.futures` module¶