What's Wrong with Python's super()?

This is Part 1: the problem.

Jump to Part 2: the solution.

Jump to Part 3: the implementation.

 

#Inheritance is one of the basic mechanisms of object-oriented programming.  Loosely speaking, it allows a subclass to extend the behavior of one or more superclasses by adding fields and overriding methods.  Often, the extended functionality of a method is quite similar to the version in the superclass being overridden; in such cases, it makes sense to utilize the original version rather than copying its code to the subclass.  However, the normal method-calling mechanism in most object-oriented languages (C++ being a notable exception) is dynamic binding, in which the implementation of a method chosen by the run-time system is based on the actual type of the receiver object, rather than its declared type (which may be a supertype of the actual type).  It is therefore necessary to provide a special mechanism to call the overridden implementation, and this is called super in many languages, including Java and #Python.

This is a relatively simple mechanism in languages (such as Java) that only support single inheritance, but things get more complex in languages (such as Python) with multiple inheritance.

As an example, consider a set of classes that represent elements of a document, such as headings, paragraphs, lists, and so on.  The top of the inheritance hierarchy could be a class called DocumentElement.  Classes such as Document, Heading, Paragraph, and List would inherit from DocumentElement.

Some document elements contain other elements; for example, a heading (such as a chapter or a section) contains the elements under that heading, and a list contains the list items.  In addition, we may want to endow some document elements with associated properties.  We can use two mixin classes to selectively make document elements containers or property owners; these could be classes such as:

class ContainerMixin:
    def __init__(self):
        self.contents = []

    def add_contents(self, *contents):
        self.contents.extend(contents)


class PropertyMixin:
    def __init__(self):
        self.properties = {}

    def add_properties(self, **properties):
        self.properties.update(properties)

If a document is supposed to have both contents and properties, the class would be declared like this:

class Document(DocumentElement,
               ContainerMixin,
               PropertyMixin):
    def __init__(self):
        super().__init__()

Since neither ContainerMixin nor PropertyMixin have any superclasses other than the default object, you would think that they don't need to call super().__init__(), right?  Wrong!  The super().__init__() call in Document calls the __init__ method of ContainerMixin, but the __init__ method of PropertyMixin will not be called, and its properties field will not be initialized.  This will happen if
the __init__ method of ContainerMixin calls super().__init__(); that call will invoke the __init__ method of PropertyMixin, even though PropertyMixin is not a superclass of ContainerMixin , or in any way related to it.  This is surprising to those used to other languages, such as Java, in which a super call always invokes a method of a superclass.

The Python super() function actually returns a proxy object, which contains the list of superclasses of the receiver object; this is sorted according to the Method Resolution Order, or MRO.  Method calls on this object will invoke the specified method implementations in the MRO classes, in order.  Because the actual object on which the original method is called can belong to any subclass of Document, even those defined later, and this subclass may inherit from arbitrary additional classes, it is impossible to predict what will be called when you write the code for any potential superclass.  This seems to imply that you must always call super().__init__() (or any other method), unless you know your class will never be subclassed.

Unfortunately, this won't work in general.  Suppose, for example, that the __init__ methods of the mixin classes take arguments:

class ContainerMixin:
    def __init__(self, *contents):
       self.contents = contents

class PropertyMixin:
    def __init__(self, **properties):
        self.properties = properties

We may decide to ignore these, since all are optional, but this may not be true in general.  What can we do if we want to accept the contents and properties arguments in the Document class and pass each to the appropriate mixin?

There is a workaround, but it doesn't work all cases.  The trick is to use the 2-argument form of super().  The first argument is one of the receiver object's superclasses, and the second is the object itself.  When using this form of super(), the implementation that will be invoked for the method call will be the one from the class that follows the given superclass on the MRO list.  The code will look like this:

class Document(DocumentElement,
               ContainerMixin,
               PropertyMixin):
    def __init__(self, *contents, **properties):
        super().__init__(*contents)
        super(ContainerMixin, self).__init__(**properties)

The first super() call, without arguments, will call the first __init__ method of the first superclass on the MRO (that is, one that follows the Document class itself) that provides such a method.  If you execute the command print(Document.__mro__), you will see the following output:

(<class '__main__.Document'>,
 <class '__main__.DocumentElement'>,
 <class '__main__.ContainerMixin'>,
 <class '__main__.PropertyMixin'>,
 <class 'object'>)

The first superclass is DocumentElement, which has no __init__ method.  The following one is ContainerMixin, which does, and therefore this is the implementation invoked, with the arguments from contents.

The second super call has ContainerMixin as its first argument.  The implementation invoked is therefore searched on the MRO starting with the following class, PropertyMixin, and this indeed accepts the properties argument.

This behavior means that the first super call could also have been written as:
        super(DocumentElement, self).__init__(*contents)

The reason that Python searches the MRO from the class after the one given as the first argument is to make it easy to invoke the default behavior by using the actual class of the object as the first argument.  Indeed, before Python 3, super required at least one argument, and so this was the common pattern.  The first super call in our example would have had to be written this way:

super(Document, self).__init__(*contents)

Note that this assumes that the actual type of the object is Document, but in fact it could be any subtype of this class, in which case the MRO will be different, with different classes following DocumentElement and ContainerMixin on it.  For example, if SomeMixin has an __init__ method that doesn't call super, the following class definition will not call PropertyMixin.__init__():

class MyDocument(Document, SomeMixin, PropertyMixin):
    pass

This definition has the effect of putting SomeMixin before PropertyMixin in the MRO of MyDocument,  so that the call super(ContainerMixin, self).__init__(**properties) in Document actually calls the __init__ method of SomeMixin, not that of PropertyMixin.

This may seem strange: why put PropertyMixin in the superclass list at all, given that Document already inherits from it?  But sometimes this is necessary in order to give precedence to methods of one class over another.

This behavior is unpredictable, and therefore dangerous, from the point of view of the developer who writes the super() call.  While the developers of a class must know the internals of all classes they inherit from, they have no way of knowing anything about subclasses of their class; such subclasses may be created at any future time by other people.

All the examples above used the __init__ method,  but, of course, the same issues apply to any method of the class.

Because of these issues, you will find a lot of opinions that state that classes must be designed together in order to make the correct use super() possible by subclasses.  Some will tell you that all superclasses that are designed to work together must have exactly the same signature for each method; others recommend using the most permissive signature (*args, **kwargs), and passing all arguments to all super methods.  Each of these has its own issues, and in any case severely limit the design if super() is to be supported.

The upshot of this is that the super mechanism in Python is flawed because it is unpredictable.  It is impossible in general to guarantee the behavior we want using this mechanism.  This stems from the absence of a good theoretical basis for this implementation of the super feature.

What is a sound theoretical basis?  I have mentioned Design by Contract in several previous posts, and it also explains how super calls need to be treated.  More on this in the next post!