Monthly Archives: January 2010

palindromes on twitter – rettiwt no semordnilap

Lately I’ve been thinking about processing streams of data on the fly. Weirdly enough, I have a fascination with palindromes, so I wrote a very bad hack with Twisted to hookup to the streaming twitter feed and search for these things. Over the course of about a day, I found 259 distinct palindromes where palindromes are (very narrowly) defined as:

  1. Words without punctuation, numbers or accents (i.e. only a-z in regex).
  2. Longer than four characters.
  3. Consisting of three or more distinct characters.
  4. Not in a set of stop words I decided I was not interested in.

The astute reader will notice that these rules actual mean all valid palindromes will be 5 or more characters. Now for the results:

Twitter Palindromes

“Level” is the clear winner with 662 occurrences. The venerable palindrome “racecar” sped into the #30 spot with a mere 8 occurrences.

Unfortunately I did not keep track of how many tweets I scanned, but it appears a very small percentage of tweets contain this type of palindrome.  I would like to convert this to a full-on palindrome tracking twitter bot that will annoyingly tweet at you if you tweet something palindrome-ish, but I am slightly afraid someone would take that code and make streaming-twitter-pesterbot-v1™.  Hunting for palindromic sentences would be fun, too (a palindromic 144 character tweet would be pure genius).  Most importantly, I’d like to search for sarahpalindromes which I’ve defined as words that become palindromes with one transposition (sarahpalindrome detection algorithms are much appreciated…you could probably get a paper out of something like that).

Binding Classes to Tags in Python’s ElementTree

Most of my work on BetterXML was in the area of data binding. In both of our frameworks, we created mappings between classes and markup elements, and then bound each occurrence of a tag in a document to an instance of the mapped class. Our work there was implemented from scratch (i.e. on top of a SAX parser), but it turns out that some of our ideas are quite simple to implement in Python’s ElementTree. Let’s say we are parsing the xml version of someone’s twitter stream.  It looks a little something like:

<statuses type="array">
  <status>
    <created_at>Fri Dec 18 18:31:14 +0000 2009</created_at>
    <id>6804282413</id>
    <text>my frozen soup will never thaw </text>
    ...
  </status>
  ...
</statuses>

We may like to bind each instance of the <text> tag to an instance of a Tweet class:

class Tweet(_ElementInterface):
    def word_count(self):
        return len(self.text.split())
 
    def at_someone(self):
        for word in self.text.split(): #this is in the xml, but here we do it from scratch for fun
            if(word.startswith('@')):
                return True
        return False

The actual binding occurs by creating our own factory method, and supplying this to the ElementTree TreeBuilder and parser:

def bound_element_factory(tag, attrs):
    if(tag == 'text'):
        return Tweet(tag, attrs)
    elif(tag == 'statuses'):
        return Twitter(tag, attrs)
    else:
        return _ElementInterface(tag, attrs)
 
parser = XMLTreeBuilder(target=TreeBuilder(element_factory=bound_element_factory))
tree = ElementTree.parse('mytweets.xml', parser=parser)

So now as the document, is parsed, the bound_element_factory() method is called each time a tag is encountered. Instead of always instantiating an _ElementInterface class for each tag (what the default factory method does), this method instantiates specialized children classes when a “text” or “statuses” tag is encountered. These classes can obviously define or override methods willy-nilly.

While I think this is a pretty neat xml party trick, it is fair to question the utility of this approach in particular and data binding in general.   After all, ElementTree is pretty useful on its own. But data binding is appealing because it allows us to create the object and class hierarchy first, and worry about going to and from the markup representation later.

This version of data binding is a bit backwards, though. Because our objects extend the _ElementInterface object, we still must call the ElementTree’s insert() method build up a tree of objects.  Thus, when we write our classes and build up trees of objects, we’re aware that they’re going to or coming from xml.

However, all is not lost. This approach can be useful when going from existing markup (particularly if it is changing or poorly defined) to an object hierarchy.  Again, this is what ElementTree excels at, but here we have the opportunity to create some utility methods in the objects and hide the markup from the users of these objects.  So in this case, if twitter changes the markup from <text> to <tweet>, we need only change the factory and the call to find().  Users of the Tweet class will see no changes.

A complete working example can be found in my bitbucket (pardon the non-working thing in that repo, too). Also, a major hat tip to the flexibility of the ElementTree implementation. I high recommend reading that very pretty code, too.