thatmattbone.com

Most of my work on BetterXML was in the area of data binding. In both of our frameworks, we created mappings between classes and markup elements, and then bound each occurrence of a tag in a document to an instance of the mapped class. Our work there was implemented from scratch (i.e. on top of a SAX parser), but it turns out that some of our ideas are quite simple to implement in Python’s ElementTree. Let’s say we are parsing the xml version of someone’s twitter stream.  It looks a little something like:

<statuses type="array">
  <status>
    <created_at>Fri Dec 18 18:31:14 +0000 2009</created_at>
    <id>6804282413</id>
    <text>my frozen soup will never thaw </text>
    ...
  </status>
  ...
</statuses>

We may like to bind each instance of the <text> tag to an instance of a Tweet class:

class Tweet(_ElementInterface):
    def word_count(self):
        return len(self.text.split())

    def at_someone(self):
        for word in self.text.split(): #this is in the xml, but here we do it from scratch for fun
            if(word.startswith('@')):
                return True
        return False

The actual binding occurs by creating our own factory method, and supplying this to the ElementTree TreeBuilder and parser:

def bound_element_factory(tag, attrs):
    if(tag == 'text'):
        return Tweet(tag, attrs)
    elif(tag == 'statuses'):
        return Twitter(tag, attrs)
    else:
        return _ElementInterface(tag, attrs)

parser = XMLTreeBuilder(target=TreeBuilder(element_factory=bound_element_factory))
tree = ElementTree.parse('mytweets.xml', parser=parser)

So now as the document, is parsed, the bound_element_factory() method is called each time a tag is encountered. Instead of always instantiating an _ElementInterface class for each tag (what the default factory method does), this method instantiates specialized children classes when a “text” or “statuses” tag is encountered. These classes can obviously define or override methods willy-nilly.

While I think this is a pretty neat xml party trick, it is fair to question the utility of this approach in particular and data binding in general.   After all, ElementTree is pretty useful on its own. But data binding is appealing because it allows us to create the object and class hierarchy first, and worry about going to and from the markup representation later.

This version of data binding is a bit backwards, though. Because our objects extend the _ElementInterface object, we still must call the ElementTree’s insert() method build up a tree of objects.  Thus, when we write our classes and build up trees of objects, we’re aware that they’re going to or coming from xml.

However, all is not lost. This approach can be useful when going from existing markup (particularly if it is changing or poorly defined) to an object hierarchy.  Again, this is what ElementTree excels at, but here we have the opportunity to create some utility methods in the objects and hide the markup from the users of these objects.  So in this case, if twitter changes the markup from <text> to <tweet>, we need only change the factory and the call to find().  Users of the Tweet class will see no changes.

A complete working example can be found in my bitbucket (pardon the non-working thing in that repo, too). Also, a major hat tip to the flexibility of the ElementTree implementation. I high recommend reading that very pretty code, too.

One of the many things I like about Python is generators. They allow for the creation of iterators without the boilerplate imposed by Java or PHP. Furthermore, iterators can be thought of as streams, and many times we don’t really want to default to eager behavior (maybe eager behavior should be demanded ). For example, consider a class implementing a very simple binary tree:

class BinaryTree:
    def __init__(self, data, left=None, right=None):
        self.data = data
        self.left = left
        self.right = right

    def __unicode__(self):
        return '%s' % self.data

More of a struct, this simple class has slots (instance variables) for the left sub-node, the right sub-node, and the data. When we first encounter a structure like this in, say, a data structure course, our inclination is to traverse the thing. Remember, we can do this any number of ways: depth-first, breadth-first, pre-order, post-order (for the traversals in this article, I will only be concerned with node data, but all the algorithms can easily be modified to yield the nodes themselves). If we want to write a very simple, eager, depth-first (and also pre-order) traversal of a tree like this, we can do something as follows:

def recursive_dfs(tree):
    nodes = []
    if(tree != None):
        nodes.append(tree.data)
        nodes.extend(recursive_dfs(tree.left))
        nodes.extend(recursive_dfs(tree.right))
    return nodes

This is a great first step, but, as already mentioned, it is eager. By calling this function, we get a complete list of all the nodes in the tree whether we need them or not. With a few simple modifications, however, we can pull nodes out of a tree on demand in the same pre-order fashion by using Python generators. We simply start the traversal, yield the node data, yield all nodes in the left subtree, and then yield all nodes in the right subtree:

def basic_dfs(tree):
    if(tree!=None):
        yield tree.data
        for node_data in basic_dfs(tree.left):
            yield node_data
        for node_data in basic_dfs(tree.right):
            yield node_data

If we wanted a (not-quite)-post-order traversal, we would yield the nodes in the right subtree first. We could do this by simple rewriting the function above. However, we can take this a step further, and leave the the decision of what nodes to yield first to another function entirely (I will call this the ‘chooser’ function):

def left_then_right(tree):
    if(tree!=None):
        yield tree.left
        yield tree.right

def dfs(tree, chooser=left_then_right):
    if(tree!=None):
        yield tree.data
        for immediate_child in chooser(tree):
            for node_data in dfs(immediate_child, chooser):
                yield node_data

Thus dfs(sometree) will call the left_then_right() function by default, and perform a pre-order traversal. For our (not-quite)-post-order traversal, we define the right_then_left() function:

def right_then_left(tree):
    if(tree!=None):
        yield tree.right
        yield tree.left

And passing this function with dfs(sometree, right_then_left), we have our new traversal. To really see the benefit of our lazy traversals, though, we can go one step further, and implement a binary-search on top of our dfs() function as a chooser function. Instead of yielding the left subtree and the right subtree, the chooser function will yield the nodes in the left subtree if the value we’re searching for is less than the node data, the nodes in the right if the value is greater than the node data, or the node data itself if it is equal to the value:

def binary_search_chooser(value):
    def binary_search_chooser_inner(tree):
        if(tree!=None and tree.data!=None):
            if(value<=tree.data):
                yield tree.left
            else:
                yield tree.right

    return binary_search_chooser_inner

Don’t let the closure fool you, this is still simple stuff. A call to binary_search_chooer(5), returns a chooser function that will decide whether to go left or right down the tree based on a node’s value. So to search a BST for 5, we can just call bfs(tree, binary_search_chooser(5)). This will give us a list of nodes (the path to a leaf), the last of which will be 5 if that value is found.

For sure there are more efficient ways to do these kinds of traversal with pointer manipulation, etc, but this serves as fun exercise for fans of generators. The astute reader will also note that we’ve implemented the strategy pattern in a functional-programming type of way with our use of first class functions.