OOP in Python, part 14: The Exception class hierarchy

MP 57: How exceptions are implemented in Python, and what they can show us about class hierarchies.

Note: This post is part of a series about OOP in Python. The previous post discussed how abstract base classes are used in the marshmallow serialization library. The next post looks at the class structure in the pathlib module.

---

In the last post we discussed one library in depth that used abstract base classes to build a class hierarchy. It’s really informative to see how class hierarchies are implemented in well-established real-world projects, so we’ll continue that trend with an examination of a class hierarchy from the Python standard library.

In this post we’ll look at how exceptions are implemented in Python. Exceptions are always interesting to look at in Python, because everyone has run into them. Even if you haven’t yet learned to write exception-handling code you’ve almost certainly written code that generated a traceback, and there’s an exception at the heart of every unintended traceback.

We’ll spend some time looking at how parts of Python are implemented in C, but you don’t need to know anything about C to understand the larger takeaways from this post.

The exception class hierarchy

One of the challenges of discussing Python class hierarchies in the standard library is that much of the standard library is implemented in C. It’s worth taking a peek at one of these hierarchies, though. For one thing, you can get a much clearer sense of how Python itself is written. But also, most relevant to our ongoing discussion about OOP, you can see that class hierarchies aren’t just a Python thing. They exist in all languages that support object-oriented programming. And even in languages like C that don’t directly support OOP, experienced programmers can still build out effective implementations of OOP concepts.

To understand how exceptions are implemented, let’s focus on a particular one: IndexError. Here’s a short Python program, bad_list.py, that generates an index error:

python_exceptions = [
    "SyntaxError",
    "NameError",
]

my_exception = python_exceptions[2]

In this example we’re asking for the third item in python_exceptions instead of the second item, which generates an error:

$ python bad_list.py 
...
IndexError: list index out of range

Let’s look at how IndexError is implemented, and then come back to this code.

How IndexError is implemented

The main source code for Python’s built-in exceptions is in a file called exceptions.c. This file is almost 4,000 lines long; searching for IndexError brings us to this snippet:

/*
 *    IndexError extends LookupError
 */
SimpleExtendsException(PyExc_LookupError, IndexError,
                       "Sequence index out of range.");

The first three lines here are a comment. IndexError inherits from, or extends, LookupError. SimpleExtendsException() is a C function that defines how one exception inherits from another.

It turns out IndexError is just a really thin wrapper around LookupError. All it adds to LookupError is a more specific error message, Sequence index out of range. We must be on the right track, because that’s the error message we saw in the output of bad_list.py.

Why would people make such a simple child class? Let’s think about usage for a moment. If you know you’re iterating over a list, you might want to catch IndexError and handle situations where they arise appropriately. But what if you’re writing a function that could receive either a list or a dictionary? In this situation you can iterate over the sequence, and catch a LookupError instead of the more specific IndexError. This will catch IndexError and KeyError exceptions, because both of those inherit from LookupError.

How LookupError is implemented

It turns out LookupError is another thin wrapper:

/*
 *    LookupError extends Exception
 */
SimpleExtendsException(PyExc_Exception, LookupError,
                       "Base class for lookup errors.");

LookupError in turn inherits from Exception.

Let’s keep going; Exception is yet another thin wrapper:

/*
 *    Exception extends BaseException
 */
SimpleExtendsException(PyExc_BaseException, Exception,
                       "Common base class for all non-exit exceptions.");

Exception inherits from BaseException, and here we’ve come to the end of child classes that act as thin wrappers.

How BaseException is implemented

The implementation of BaseException is hundreds of lines of C code. This code implements most of the exception handling behavior you’re probably familiar with if you’ve used try-except blocks to handle errors.

Here’s the first few lines of BaseException’s implementation, showing that there’s one more layer to the hierarchy:

/*
 *    BaseException
 */
static PyObject *
BaseException_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
    PyBaseExceptionObject *self;

    self = (PyBaseExceptionObject *)type->tp_alloc(type, 0);
    if (!self)
        return NULL;
    /* the dict is created on the fly in PyObject_GenericSetAttr */
    self->dict = NULL;
    self->notes = NULL;
    self->traceback = self->cause = self->context = NULL;
    self->suppress_context = 0;
    ...

BaseException actually consists of a number of functions that implement the behavior we associate with exceptions. This is the start of the BaseException_new() function, which defines how new exception objects are created. BaseException relies on PyObject, a C component that lies at the base of all Python class hierarchies.

You can see that the implementation in C looks much different than a Python class. Because C doesn’t support OOP directly, this code implements OOP concepts at a lower level. It’s a lot more work, but the benefit is a higher level of efficiency.

Even though the implementation looks a lot different in C, the concepts of OOP and inheritance still show through.

Connecting back to Python

If you’ve been poking around in exceptions.c, you might have noticed one more appearance of IndexError:

static struct static_exception static_exceptions[] = {
#define ITEM(NAME) {&_PyExc_##NAME, #NAME}
    ...
    ITEM(IndexError),  // base: LookupError(Exception)

This is the boundary between C and Python. When we work with an IndexError in Python, this is the code that connects back to that specific error’s implementation in C.

To wrap up this excursion into C, we’ve found that:

• IndexError inherits from LookupError;

• LookupError inherits from Exception;

• Exception inherits from BaseException;

• BaseException depends on the more primitive object.¹

Seeing this hierarchy in Python

Now that we know what the class hierarchy for IndexError looks like, we should be able to see it in a couple ways. Let’s go back to the original bad_list.py example, and catch the IndexError that was raised. Once we’ve caught the exception, we can examine it in several different ways.

We’ll first call help() on the IndexError object itself:

python_exceptions = [
    "SyntaxError",
    "NameError",
]

try:
    my_exception = python_exceptions[2]
except IndexError as e:
    help(e)
    raise e

We place the line that caused the error inside the try block. The except block catches IndexError, and assigns it to the variable e. It’s this variable that we can examine to learn more about IndexError objects.

We call help() on the IndexError object, and then re-raise the exception so the program exits with the same traceback.

Here’s the most relevant part of the help() output:

Help on IndexError object:

class IndexError(LookupError)
 |  Sequence index out of range.
 |  
 |  Method resolution order:
 |      IndexError
 |      LookupError
 |      Exception
 |      BaseException
 |      object

The help() output for an instance of a class includes the Method resolution order, or mro, of the class. This is a representation of the class hierarchy. If you call a method on the instance e, this is a list of the places Python will look for that method.

As an example, exception objects have an add_note() method, defined in BaseException:

 |  Methods inherited from BaseException:
 |  ...
 |  add_note(...)
 |      Exception.add_note(note) --
 |      add a note to the exception

This is at the heart of how inheritance works. If you make a call such as e.add_note("Oh no, index out of bounds!"), Python will first look in IndexError for the method add_note(). If it doesn’t find it there, it will look in the LookupError implementation. If it doesn’t find the requested method anywhere in the classes listed in the method resolution order, Python will raise an AttributeError because that method doesn’t exist in the hierarchy.

Calling isinstance()

We can use isinstance() to verify that the exception we caught is an instance of any class in the hierarchy. For example, here’s how we can prove the IndexError we caught is also an instance of LookupError:

python_exceptions = [
    ...

try:    
    my_exception = python_exceptions[2]
except IndexError as e:
    print(isinstance(e, LookupError))
    raise e

Here we’re asking Python whether e is an instance of LookupError.

It is:

True
Traceback (most recent call last):
    ...

You can replace LookupError with any class in the hierarchy: IndexError, LookupError, Exception, BaseException, or object. In all of these cases, the output will be the same. If you replace LookupError with any exception outside the hierarchy, such as SyntaxError or NameError, the output will be False.

Catching LookupError

This also means you can catch any exception in the hierarchy, and IndexError will be caught. Here’s what it looks like to catch LookupError instead of IndexError:

python_exceptions = [
    ...

try:
    my_exception = python_exceptions[2]
except LookupError as e:
    print(type(e))
    raise e

Here we’ve used LookupError in the except block, and we’re printing the type of the exception object that’s caught.

Here’s the output:

<class 'IndexError'>
Traceback (most recent call last):
    ...

We’ve still caught an IndexError, because that’s the exception object that Python raised. But an IndexError is a LookupError, so the except block still runs. This error-handling code will catch any exceptions that inherit from LookupError, and no other exceptions.

Examining the method resolution order explicitly

An instance by itself doesn’t have a method resolution order; that’s a property of a class. But if you have an instance you can get the class, and then you can get the method resolution order.

Here we’ll go back to catching the IndexError, and then examine its method resolution order:

python_exceptions = [
    ...

try:
    my_exception = python_exceptions[2]
except LookupError as e:
    print(type(e).mro())
    raise e

Calling type(e) returns the class that e is an instance of. Classes have a default mro() method, which returns the list of classes where Python will look for methods. That’s effectively a representation of the class hierarchy.

Here’s the output:

[<class 'IndexError'>, <class 'LookupError'>, <class 'Exception'>,
        <class 'BaseException'>, <class 'object'>]
Traceback (most recent call last):
    ...

This is the same set of classes we saw in the implementation chain for IndexError, and in the help() output for an IndexError object.

Conclusions

This was a deeper dive into class hierarchies in Python than I ever intended to do when starting this series. But exploring OOP has led us here, and I’ve learned a lot about Python and OOP by following these rabbit holes. I hope you have as well.

Exceptions are an interesting part of Python to focus on, because most people’s first experience of exceptions centers around seeing them appear when they inevitably make mistakes in their code. It’s easy to think exceptions are just a large group of things that can go wrong, with no real structure to them. But reality is far from that; there’s a whole lot of structure in the implementation of Python’s exceptions, and there’s a lot of purpose behind that structure as well.

The next time you see an exception, consider where it might live in the overall exception class hierarchy. And if you find yourself catching a number of different exceptions, ask yourself if you can move up one level in the hierarchy, and catch the exception that all those others inherit from. If you’re handling them all the same way, this approach might work. If you’re handling each one in a unique way, then you’re probably already working at the correct level in the hierarchy.

Resources

You can find the code files from this post in the mostly_python GitHub repository.

1. I didn’t show this connection as clearly as all the others. But Python’s base object class is at the bottom of all Python class hierarchies, and it has to do with the PyObject element we saw in the C implementation of BaseException_new().