The difference between __getattribute__ and __getattr__

The other day I was teaching Python meta-programming to a workmate. I think it’s a good way to learn about high order functions since meta-programming makes extensive use of closures, function builders, decorators… I was trying to make a concept probe about a very, very generic REST connector. Here is my first (and wrong) attempt:

class RESTConn(object):

  def __init__(self, entry_point):
    self.entry_point = entry_point

  def method_builder(self, method_name):
    verb, _, collection = method_name.split('_', 2)
    def do_verb(payload=None, **kwargs):
      uri = self.make_uri(collection)
      querystring = self.make_querystring(kwargs)
      print verb.upper(), self.combine(uri, querystring)
      if payload:
        print payload

    return do_verb

  def make_uri(self, collection):
    return '/'.join([self.entry_point, collection])

  def make_querystring(self, kwargs):
    return '&'.join(['='.join(pair) for pair in kwargs.iteritems()])

  def combine(self, uri, querystring):
    if querystring:
      return '&'.join([uri, querystring])

    return uri

  def __getattribute__(self, name):
    if not hasattr(self, name):
      method = self.method_builder(name)
      setattr(self, name, method)

    return super(RESTConn, self).__getattribute__(name)

Try this example by instantiating a new connector and trying to call something like:

c = RESTConn('unoyunodiez.com')
c.get_from_articles()

The program falls into an infinite recursion and do nothing before crashing. Why?

There are three problems here. First and most important is using __getattribute__(), second is using hasattr() and third is accessing self.method_builder().

The object’s method __getattribute__() is used to retrieve an attribute from an instance. It captures every attempt to access an instance attribute by using dot notation or getattr() built-in function. Unless it was overridden, the former expression is translated into object.__getattribute__(self, ‘get_from_article’). The default implementations looks into the instance’s namespace, then looks into the class namespace, then into each base’s namespace and so on. Finally, if not found, the default implementation calls the fallback __getattr__() method of the instance and it raises an AttributeError exception as default implementation.

This is not a problem by itself but if you pay attention enough you’ll notice we are trying to create the new method only if the object does not have the method yet. It is semantically the same as overriding __getattr__() because it is called only when the attribute was not found. So, even if we cannot explain the infinite recursion error yet, we can fix the class by replacing:

  def __getattribute__(self, name):
    if not hasattr(self, name):
      method = self.method_builder(name)
      setattr(self, name, method)

    return super(RESTConn, self).__getattribute__(name)

with:

  def __getattr__(self, name):
    method = self.method_builder(name)
    setattr(self, name, method)
    return getattr(self, name)

So, the difference between __getattribute__() and __getattr__() is that the first one is called unconditionally when an attribute is being retrieved from an instance while the second is called only when the attribute was not found.

But, what about the infinite recursion? Why the first example was failing?