The Principles of Deferring to a Superclass, or How to Do super() Right

In a previous post I showed how Python's implementation of super() is unpredictable and therefore impossible to use reliably.  What are the theoretical principles on which a correct implementation should be built?

I have written before on Design by Contract, a practical methodology for correct object-oriented design.  In a nutshell, it requires every class to have a class invariant, which specifies the acceptable states of objects of that class.  For example, a container class must ensure that its size as reported (or stored in a class field) is always non-negative.  The invariant must be maintained at all times, except when inside the implementation of a class method.  In the absence of concurrent executions of methods on the same object, it is sufficient to guarantee that the constructor establishes the invariant, and that every method maintains it.  In addition, each method has a precondition, which specifies the legal configurations (object state and argument values) in which it is possible to call the method, and a postcondition, which specifies the state in which the method must leave the object, as a function of its previous state and the argument values.  The precondition is the responsibility of the caller; for example, it is illegal to call pop() on a stack when its
empty() method returns true.  The postcondition, the responsibility of the method implementation; for example, after pushing an element to the stack, it must be the top element of the stack, and all other elements must stay the same.

From the point of view of the programmer who calls any operation on an object, the relevant contract is that of the static type of the object, since the dynamic type can't be known when writing the code; however, it must be a subtype of the static type.  The Design by Contract methodology follows Liskov's Substitution Principle, which implies that a subclass must obey the contracts of all its superclasses.  Therefore, relying on the static type is always correct.

However, in the Python implementation of super(), this isn't true.  As we saw in the previous post, a super call can invoke an implementation from a class that is not a superclass of the static type of the receiver object.  In the last example shown there, the call super(ContainerMixin, self).__init__(**properties) in the class Document could invoke a method of an unrelated class (SomeMixin).  Under such behaviors, it is impossible to know what is the contract of the called method, and therefore impossible to write correct code.  The only way to enable developers to write correct and reliable code is to restrict super calls to static superclasses; that is, superclasses of the class in which the super call appears, regardless of the dynamic type of the receiver object.

In addition, the super mechanism in Python (and most other object-oriented languages, including Java and C++) allows diagonal calls, in which one method can call a different method on the super object.  This is rarely done in practice, and should be disallowed in principle, since it bypasses the contract of the called method.  The implementation of a method can use implementations provided by superclasses for the same method, and will need to be mindful of their contracts; but if it needs a service from another method of the same object, it should call the implementation provided for that object, which is the only one that is responsible for the correct contract.  (Remember that the dynamic type of the object, which determines its contract, is not known when writing the code.)

Bertrand Meyer, the author of the Design by Contract methodology, embodied this methodology in the programming language #Eiffel.  In Eiffel, the super construct is called Precursor, and calls the implementation of the method in which it appears in the immediate (static) superclass, if there is exactly one such.  Otherwise, Precursor needs to be decorated with the name of one of the static superclasses, whose implementation is to be invoked.  This adheres to the principles above: it is based on static relationships between classes, and so the relevant contract is always known; and it prohibits diagonal calls.

This behavior can be implemented in Python, although a more efficient implementation will have to be done inside the compiler.  Before we show the implementation, let's look at a particularly difficult example.  This is taken from Meyer's book,
Object-Oriented Software Construction (2nd edition), and is based on an earlier example from Stroustrup's book The C++ Programming Language.

In this example, we have a Window class, describing a window or widget in a graphical user interface.  One of the important methods of this class is draw(), which draws the contents of the window on the screen.  Some windows have borders, and some have menus.  These are described by subclasses WindowWithBorder and WindowWithMenu, respectively.  The implementation of the draw() method in each of these subclasses first needs to draw the window, using the method in the parent Window class, and then add the border or menu.  The implementation of these classes could look like this, translated into Python syntax:

class Window:
    def draw(self):
        print('Window.draw()')


class WindowWithBorder(Window):
    def draw_border(self):
        print('WindowWithBorder.draw_border()')

    def draw(self):
        print('WindowWithBorder.draw()')
        precursor()
        self.draw_border()


class WindowWithMenu(Window):
    def draw_menu(self):
        print('WindowWithMenu.draw_menu()')

    def draw(self):
        print('WindowWithMenu.draw()')
        precursor()
        self.draw_menu()

Running the following test will print the output shown:

wb = WindowWithBorder()
wb.draw()
print('***')
wm = WindowWithMenu()
wm.draw()

Output:

WindowWithBorder.draw()
Window.draw()
WindowWithBorder.draw_border()
***
WindowWithMenu.draw()
Window.draw()
WindowWithMenu.draw_menu()

This demonstrates that the correct methods are called, in the correct order.

But what about windows that have both borders and menus?  These would be described by a class WindowWithBorderAndMenu, which should inherit from both WindowWithBorder and WindowWithMenu.  Unfortunately, its draw() method must not call the implementations in its immediate superclasses, since that will call Window.draw() twice; this is likely to undo the effects of the call from the previous superclass, and may well cause other issues.  Instead, the implementation of draw() in this class should call the Window implementation, followed by draw_border() and draw_menu().

class WindowWithBorderAndMenu(WindowWithBorder,
                              WindowWithMenu):
    def draw(self):
        print('WindowWithBorderAndMenu.draw()')
        precursor_of(Window)()
        self.draw_border()
        self.draw_menu()

The blue line shows how to call the implementation in a superclass, using a different function precursor_of, which takes the appropriate class object as a parameter and returns the method implementation from that class.

Eiffel prohibits calling a method implementation that is not in one of the immediate superclasses, based on the consideration that jumping over an implementation is likely to miss important parts of the implementation.  However, in this case this is justified, and it is possible to do this in Eiffel by inheriting yet again from Window, this time directly.  My implementation of precursor in Python does allow such jumps, although this can be changed.

The following test of WindowWithBorderAndMenu will give the output shown:

wbm = WindowWithBorderAndMenu()
wbm.draw()

Output:

WindowWithBorderAndMenu.draw()
Window.draw()
WindowWithBorder.draw_border()
WindowWithMenu.draw_menu()

How does this magic happen?  The short answer is "with some reflection."  Getting more technical, we use Python's inspect module to find the frame that called the precursor() function.  From that frame we can extract the name of the method, as well as the target of the call (that is, the receiver object), and the __bases__ field of its class gives the list of superclasses.  The precursor() function will only work if there is a single superclass, in which case it will call the implementation of the called method from that superclass (or whatever that superclass defers to).

The precursor_of() function is similar, but instead of searching for a unique superclass it accepts the superclass as an argument.  It then returns an internal function that will invoke the correct method implementation when it is called.  (This is why there is another set of parentheses in the call marked in blue in the code above; this is where you would provide any arguments to the super method.)

You can find the full implementation in my github repo yishaif/parseltongue, in the extensions/super directory; tests are in extensions/tests.

Of course, using reflection for every super call is not advisable.  This implementation is just an example of how this could work; an efficient implementation would have to be part of the Python compiler.

But what if you perversely want a static super implementation that can still perform diagonal calls?  I don't want to encourage you to do such a thing, but it can certainly be done.  Here is the windowing example using this style.  I am using the functions static_super() and  static_super_of()instead of super(), so as not to interfere with Python's builtin function.  The first is for the case in which the class has a single immediate superclass, and the second is for multiple inheritance, and takes a parameter that specified the class whose implementation is to be invoked.  The changes from the previous example are in blue.

class Window:
    def draw(self):
        print('Window.draw()')


class WindowWithBorder(Window):
    def draw_border(self):
        print('WindowWithBorder.draw_border()')

    def draw(self):
        print('WindowWithBorder.draw()')
        static_super().draw()
        self.draw_border()


class WindowWithMenu(Window):
    def draw_menu(self):
        print('WindowWithMenu.draw_menu()')

    def draw(self):
        print('WindowWithMenu.draw()')
        static_super().draw()
        self.draw_menu()


class WindowWithBorderAndMenu(WindowWithBorder, WindowWithMenu):
    def draw(self):
        print('WindowWithBorderAndMenu.draw()')
        static_super_of(Window).draw()
        self.draw_border()
        self.draw_menu()
Here are the tests, with the output:

wb = WindowWithBorder()
wb.draw()
print('***')
wm = WindowWithMenu()
wm.draw()
print('***')
wbm = WindowWithBorderAndMenu()
wbm.draw()

Output:

WindowWithBorder.draw()
Window.draw()
WindowWithBorder.draw_border()
***
WindowWithMenu.draw()
Window.draw()
WindowWithMenu.draw_menu()
***
WindowWithBorderAndMenu.draw()
Window.draw()
WindowWithBorder.draw_border()
WindowWithMenu.draw_menu()

As you can see, the results are exactly the same.  However, this form enables diagonal calls, since the static_super() object can invoke any method defined in the chosen superclass.

The code for static_super() can be found in the same github repo.  I will go into the details of the implementation in the next post.

What's Wrong with Python's super()?

This is Part 1: the problem.

Jump to Part 2: the solution.

Jump to Part 3: the implementation.

 

#Inheritance is one of the basic mechanisms of object-oriented programming.  Loosely speaking, it allows a subclass to extend the behavior of one or more superclasses by adding fields and overriding methods.  Often, the extended functionality of a method is quite similar to the version in the superclass being overridden; in such cases, it makes sense to utilize the original version rather than copying its code to the subclass.  However, the normal method-calling mechanism in most object-oriented languages (C++ being a notable exception) is dynamic binding, in which the implementation of a method chosen by the run-time system is based on the actual type of the receiver object, rather than its declared type (which may be a supertype of the actual type).  It is therefore necessary to provide a special mechanism to call the overridden implementation, and this is called super in many languages, including Java and #Python.

This is a relatively simple mechanism in languages (such as Java) that only support single inheritance, but things get more complex in languages (such as Python) with multiple inheritance.

As an example, consider a set of classes that represent elements of a document, such as headings, paragraphs, lists, and so on.  The top of the inheritance hierarchy could be a class called DocumentElement.  Classes such as Document, Heading, Paragraph, and List would inherit from DocumentElement.

Some document elements contain other elements; for example, a heading (such as a chapter or a section) contains the elements under that heading, and a list contains the list items.  In addition, we may want to endow some document elements with associated properties.  We can use two mixin classes to selectively make document elements containers or property owners; these could be classes such as:

class ContainerMixin:
    def __init__(self):
        self.contents = []

    def add_contents(self, *contents):
        self.contents.extend(contents)


class PropertyMixin:
    def __init__(self):
        self.properties = {}

    def add_properties(self, **properties):
        self.properties.update(properties)

If a document is supposed to have both contents and properties, the class would be declared like this:

class Document(DocumentElement,
               ContainerMixin,
               PropertyMixin):
    def __init__(self):
        super().__init__()

Since neither ContainerMixin nor PropertyMixin have any superclasses other than the default object, you would think that they don't need to call super().__init__(), right?  Wrong!  The super().__init__() call in Document calls the __init__ method of ContainerMixin, but the __init__ method of PropertyMixin will not be called, and its properties field will not be initialized.  This will happen if
the __init__ method of ContainerMixin calls super().__init__(); that call will invoke the __init__ method of PropertyMixin, even though PropertyMixin is not a superclass of ContainerMixin , or in any way related to it.  This is surprising to those used to other languages, such as Java, in which a super call always invokes a method of a superclass.

The Python super() function actually returns a proxy object, which contains the list of superclasses of the receiver object; this is sorted according to the Method Resolution Order, or MRO.  Method calls on this object will invoke the specified method implementations in the MRO classes, in order.  Because the actual object on which the original method is called can belong to any subclass of Document, even those defined later, and this subclass may inherit from arbitrary additional classes, it is impossible to predict what will be called when you write the code for any potential superclass.  This seems to imply that you must always call super().__init__() (or any other method), unless you know your class will never be subclassed.

Unfortunately, this won't work in general.  Suppose, for example, that the __init__ methods of the mixin classes take arguments:

class ContainerMixin:
    def __init__(self, *contents):
       self.contents = contents

class PropertyMixin:
    def __init__(self, **properties):
        self.properties = properties

We may decide to ignore these, since all are optional, but this may not be true in general.  What can we do if we want to accept the contents and properties arguments in the Document class and pass each to the appropriate mixin?

There is a workaround, but it doesn't work all cases.  The trick is to use the 2-argument form of super().  The first argument is one of the receiver object's superclasses, and the second is the object itself.  When using this form of super(), the implementation that will be invoked for the method call will be the one from the class that follows the given superclass on the MRO list.  The code will look like this:

class Document(DocumentElement,
               ContainerMixin,
               PropertyMixin):
    def __init__(self, *contents, **properties):
        super().__init__(*contents)
        super(ContainerMixin, self).__init__(**properties)

The first super() call, without arguments, will call the first __init__ method of the first superclass on the MRO (that is, one that follows the Document class itself) that provides such a method.  If you execute the command print(Document.__mro__), you will see the following output:

(<class '__main__.Document'>,
 <class '__main__.DocumentElement'>,
 <class '__main__.ContainerMixin'>,
 <class '__main__.PropertyMixin'>,
 <class 'object'>)

The first superclass is DocumentElement, which has no __init__ method.  The following one is ContainerMixin, which does, and therefore this is the implementation invoked, with the arguments from contents.

The second super call has ContainerMixin as its first argument.  The implementation invoked is therefore searched on the MRO starting with the following class, PropertyMixin, and this indeed accepts the properties argument.

This behavior means that the first super call could also have been written as:
        super(DocumentElement, self).__init__(*contents)

The reason that Python searches the MRO from the class after the one given as the first argument is to make it easy to invoke the default behavior by using the actual class of the object as the first argument.  Indeed, before Python 3, super required at least one argument, and so this was the common pattern.  The first super call in our example would have had to be written this way:

super(Document, self).__init__(*contents)

Note that this assumes that the actual type of the object is Document, but in fact it could be any subtype of this class, in which case the MRO will be different, with different classes following DocumentElement and ContainerMixin on it.  For example, if SomeMixin has an __init__ method that doesn't call super, the following class definition will not call PropertyMixin.__init__():

class MyDocument(Document, SomeMixin, PropertyMixin):
    pass

This definition has the effect of putting SomeMixin before PropertyMixin in the MRO of MyDocument,  so that the call super(ContainerMixin, self).__init__(**properties) in Document actually calls the __init__ method of SomeMixin, not that of PropertyMixin.

This may seem strange: why put PropertyMixin in the superclass list at all, given that Document already inherits from it?  But sometimes this is necessary in order to give precedence to methods of one class over another.

This behavior is unpredictable, and therefore dangerous, from the point of view of the developer who writes the super() call.  While the developers of a class must know the internals of all classes they inherit from, they have no way of knowing anything about subclasses of their class; such subclasses may be created at any future time by other people.

All the examples above used the __init__ method,  but, of course, the same issues apply to any method of the class.

Because of these issues, you will find a lot of opinions that state that classes must be designed together in order to make the correct use super() possible by subclasses.  Some will tell you that all superclasses that are designed to work together must have exactly the same signature for each method; others recommend using the most permissive signature (*args, **kwargs), and passing all arguments to all super methods.  Each of these has its own issues, and in any case severely limit the design if super() is to be supported.

The upshot of this is that the super mechanism in Python is flawed because it is unpredictable.  It is impossible in general to guarantee the behavior we want using this mechanism.  This stems from the absence of a good theoretical basis for this implementation of the super feature.

What is a sound theoretical basis?  I have mentioned Design by Contract in several previous posts, and it also explains how super calls need to be treated.  More on this in the next post!