What to Expect from Python 3.11?

Here are the most important updates on our favourite programming language.

Even Better Error Messages

When dealing with highly nested data structures (such like API responses) you might encounter a KeyError when trying to access undefined dictionary keys.

Before Python 3.11 this has been particularly painful. Consider this code snippet:

users = [
    {'id': 1, 'name': 'Bas', 'social': {'twitter': '@bascodes'}},
    {'id': 2, 'name': 'John Doe', 'social': {}},
]

def get_twitter(user):
    return user['social']['twitter']

get_twitter(users[1])

which will produce the following error message on Python 3.10:

Traceback (most recent call last):
  File "/Users/sebst/tmp311/py311.py", line 9, in <module>
    get_twitter(users[1])
  File "/Users/sebst/tmp311/py311.py", line 7, in get_twitter
    return user['social']['twitter']
KeyError: 'twitter'

In this small example, we immediately see that the error is because of the missing twitter account of user 2. The more nested these structures become, the more helpful is the new error message format in Python 3.11:

Traceback (most recent call last):
  File "/tmp/py311.py", line 9, in <module>
    get_twitter(users[1])
    ^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/py311.py", line 7, in get_twitter
    return user['social']['twitter']
           ~~~~~~~~~~~~~~^^^^^^^^^^^
KeyError: 'twitter'

The error message immediately tells us which part of the dictionary is missing. This will become even more helpful when we have two dictionary accesses in one line.

Typing: Adding the Self Type

PEP 673 introduces the Self type.

Here is an example from the PEP:

from typing import Self

class Shape:
    def set_scale(self, scale: float) -> Self:
        self.scale = scale
        return self


class Circle(Shape):
    def set_radius(self, radius: float) -> Self:
        self.radius = radius
        return self

Typing: Variadic Generics

PEP 646 discusses the introduction of Variadic Generics in the context of numerical libraries, such as NumPy. In that context, a variable is not only characterized by its type but also by its shape.

Particularly, the introduction of TypeVarTuple allows using an arbitrary-length type variable like so:

from typing import TypeVar, TypeVarTuple

DType = TypeVar('DType')
Shape = TypeVarTuple('Shape')

class Array(Generic[DType, *Shape]):

    def __abs__(self) -> Array[DType, *Shape]: ...

    def __add__(self, other: Array[DType, *Shape]) -> Array[DType, *Shape]: ...

TOML

The TOML format (maybe popular by using it in pyproject.toml files) can now be parsed with a tomllib, which will become part of Python 3.11’s standard library through PEP 680.

import tomllib
with open("pyproject.toml", "rb") as f:
    data = tomllib.load(f)

Exception Groups / except*

One important feature of Python 3.11 is discussed in PEP 654.

This PEP introduces ExceptionGroups. With them, we’re able to raise multiple exceptions simultaneously like so:

try:
    raise ExceptionGroup("Validation Errors", (
        ValueError("Input is not a valid user name."),
        TypeError("Input is not a valid date."),
        KeyError("Could not find associated project.")
    ))
except* (ValueError, TypeError) as exception_group:
    ...
except* KeyError as exception_group:
    ...

AsyncIO Task Groups

The idea of TaskGroups is to run nested tasks and continue running them even if one fails. Errors are raised using exception groups, so no error will pass silently.

In fact, the previously mentioned ExceptionGroups were needed to implement the TaskGroups feature.

For more information on TaskGroups, look at this tweet.

try:
    async with asyncio.TaskGroup() as tg:
        tg.create_task(coro1())
        tg.create_task(coro2())
except* ValueError as e:
    pass # Ignore all ValueErrors

Performance Improvements

The release notes of Python 3.11.0a6 claim that there is a performance improvement of approx. 19% in CPython 3.11 vs. CPython 3.10.

For a quick overview have a look at this tweet.