Type Checking Python

Learning that we can have our cake (using familiar Python) and eat it too (with type checking).

A couple of years ago, I wrote off Python in favor of Go as my go-to general-purpose programming language. This decision was largely informed by my experience living through the transition from JavaScript to TypeScript in frontend development; I came to depend on type checking.

Turns out that Python has, in recent years, has similarly adopted type checking. First there was a standard syntax.

PEP 3107 introduced syntax for function annotations, but the semantics were deliberately left undefined. There has now been enough 3rd party usage for static type analysis that the community would benefit from a standard vocabulary and baseline tools within the standard library.

This PEP introduces a provisional module to provide these standard definitions and tools, along with some conventions for situations where annotations are not available.

— PEP 484

Then there was a tool.

Mypy is a static type checker for Python.

Type checkers help ensure that you’re using variables and functions in your code correctly. With mypy, add type hints (PEP 484) to your Python programs, and mypy will warn you when you use those types incorrectly.

Python is a dynamic language, so usually you’ll only see errors in your code when you attempt to run it. Mypy is a static checker, so it finds bugs in your programs without even running them!

Mypy is designed with gradual typing in mind. This means you can add type hints to your code base slowly and that you can always fall back to dynamic typing when static typing is not convenient.

— python/mypy

Finally, there is a comprehensive tutorial, Python Type Checking (Guide), and evidence that it can be used at scale, Our journey to type checking 4 million lines of Python.

A Problem

Here we will walk through a concrete problem with Python without type checking.

Please note: The example code here is provided in the larkintuckerllc / typelearning Repository.

In the run function below:

• It creates a Resource object given a bytes and a tuple of MetadataItem objects as parameters
• Based on the usage, it appears that the Resource object’s read_metadata method returns a Dict[str, MetadataItem] object
• Finally, it passes the Dict[str, MetadataItem] to the has_name function and prints its return value (in this case True)

untyped/main.py

Say we instead pass the tuple (‘funky’,) to the read_metadata method, we might expect to get back an empty Dict[str, MetadataItem] and printing the return value of the has_name function of False

Looking at the source code for the untyped.resource package, we indeed confirm our assumption about the return value of the read_metadata method; a Dict[str, MetadataItem] object.

untyped/resource/__init__.py

Now let us change the behavior of the read_metadata method to instead return None instead of an empty Dict[str, MetadataItem] when the Resource object’s metadata does not have any of the items as keys.

untyped/resource/__init__.py

With this change, the application works as expected, printing True, when we pass the tuple (‘name’,) to the read_metadata method. Changing this to pass the tuple (‘funky’,), however, results in a runtime error.

$ python -m untyped
Traceback (most recent call last):
  File "/Users/jtucker/.pyenv/versions/3.10.0/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/jtucker/.pyenv/versions/3.10.0/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/jtucker/src/learning/typedlearning/untyped/__main__.py", line 4, in <module>
    run()
  File "/Users/jtucker/src/learning/typedlearning/untyped/main.py", line 18, in run
    print(has_name(metadata))
  File "/Users/jtucker/src/learning/typedlearning/untyped/main.py", line 7, in has_name
    return 'name' in metadata
TypeError: argument of type 'NoneType' is not iterable

The problem here is that it was not until we ran the application with an edge case did we discover our mistake (not updating the has_name method to handle the edge case).

Type Checking to the Rescue

On the other hand, let us say we wrote the same example using type checking.

typed/main.py

typed/resource/__init__.py

As before, we change the behavior of the read_metadata method to instead return None instead of an empty Dict[str, MetadataItem] when the Resource object’s metadata does not have any of the items as keys.

With this change, we can immediately see a problem with the has_name function in our editor (here with VSCode enabled with Python type checking).

Likewise we can see the same problem when using the mypy static type checker.

$ mypy  -p typed --strict
typed/main.py:21: error: Argument 1 to "has_name" has incompatible type "Optional[Dict[str, MetadataItem]]"; expected "Dict[str, MetadataItem]"
Found 1 error in 1 file (checked 4 source files)

Essentially, with type checking we have the opportunity to fix the error before it fails at runtime.

A Bonus

As a bonus, with the type checked code and a Python type checking enabled editor, we can hover over items and get details on the typing as we are coding.