A few weeks back I started looking at ElementPath package of the ElementTree module to explore a new syntax for finding things in xml documents, but this was a bust. The existing path syntax (a sort of limited and significantly less confusing subset of xpath) works quite nicely and I can’t see any reason to fiddle with it.  And while the implementation of ElementPath is simple and readable, I liked exploring it so much I thought I’d share some thoughts since it serves as a nice example of the power of generators and first class functions.

First consider this simple xml document as an example:

<root>
  <a>
    <b value="2"/>
  </a>
  <a>
    <b>
      <c value="101"/>
    </b>
  </a>
  <a value="will not be found"/>
</root>

The ElementPath expression for finding all “b” elements that are children of “a” elements is simply “a/b” (quite familiar if you’ve ever worked with xpath).  The code to build up the ElementTree and actually evaluate this expression is also fairly straightforward.  Assuming we’ve defined the string TEST_DOC to be the xml document in the code section above, to do this we simply build the ElementTree and call findall():

root = xml.etree.ElementTree.fromstring(TEST_DOC)
root.findall(path)

First it’s useful to think how we might write some code to find all “b” elements that are children of an “a” element without the benefit of a path language.  Writing this by hand, we simply walk the tree, finding all “a” elements and checking to see if they have a “b” element as a child. Here is one possible way to write this code:

def brute_find(root):
    def find_a(root):
        for node in root:
            if node.tag == "a":
                yield node
 
    def find_b(root):
        for node in root:
            if node.tag == "b":
                yield node
 
    for a_node in find_a(root):
        for b_node in find_b(a_node):
            yield b_node

In this particular implementation, we chain together two generators. The first yields all “a” elements, and the second yields all “b” elements that are children of those elements yielded by the first function (“a” elements). When evaluating the path expression “a/b” the actual ElementPath implementation ends up doing something quite similar. Let’s take a look at what happens step by step.

First the expression is tokenized using a regular expression.  The stream of tokens for “a/b” looks like:

[('', 'a'), ('/', ''), ('', 'b')]

The meat of the work in the path language happens when it uses the first first element of the first tuple in the stream of token-tuples to dispatch to the functions responsible for creating the code that will be used to find elements.

To build up code that will walk the tree like our hand-coded example above, the ElementPath implementation looks at the first item in the tuples of the stream and dispatches to helper functions. The first item in the first tuple of the stream of tokens above, the empty string, dispatches to the prepare_child() function, which looks this:

def prepare_child(next, token):
    tag = token[1]
    def select(context, result):
        for elem in result:
            for e in elem:
                if e.tag == tag:
                    yield e
    return select

Generally speaking, all tokens in the stream cause these helper functions to be dispatched (if you are looking at the ElementPath source code, each helper function starts with “prepare_”).  Each helper function returns a function that contains the code for walking the tree and finding nodes of interest (and optionally the helper function gobbles up more tokens in the stream using the next() iterator).  All the functions returned contain a common calling convention, taking a context and a result as parameters. I’ll ignore the context parameter except to say that it is used for path expressions that look backwards up the tree instead of just at attributes or child elements. The result parameter is the result of the previously filtered node (if any) or the root of the tree. In this particular instance, prepare_child() returns a function select() that ignores the context (it does no backtracking) and iterates over the result element, yielding children of the result element if their tag matches the second item in the provided token-tuple.

All of the functions returned by the helper functions are appended to a list, selector, and ultimately chained together:

result = [elem]
...
for select in selector:
    result = select(context, result)
return result

So, in our example path, “a/b”, the select() function looking for “a” elements is provided as input to the select() function looking for “b” elements.  Conceptually we can think of this as:

function_looking_for_b_elements(
   context={},
   result=function_looking for_a_elements())

Therefore when the first select function (looking for “b” elements) is evaluated, it forces the evaluation of the second function looking for “a” elements. Remember that these are all generators and inherently lazy, and this means that the entire process is lazy. If you’re only looking for one match, you’ll get the first one in the xml tree (by first I’m mean using a DFS) and no further evaluation is required.  The find(), findall(), and iterfind() methods on each ElementTree node let you find one, all, or as many nodes as you need respectively.

To summarize this process, the original path expression is tokenized.  Then functions are dispatched using these tokens.  The functions may consume more tokens in the stream, but ultimately return functions themselves.  The functions returned are chained together and the root of the tree is passed in.  All of the functions in the chain are generators, and asking for the first matched element will force evaluation all the way up the chain.

As a final thought, remember that all of this finding of elements happens against a tree that is completely loaded into memory. Obviously this isn’t always plausible.  Some subsets of this path language and others like xpath, can, however, be used in a streaming setting. Adding something like this to the standard library could be useful (especially if it was general enough to be applied to streams of JSON).

All of the code I wrote for this post can be found in this gist. It has been tested in Python 3.2 only.

Several years ago, nearing the end of my time at my first “real” job, an “agile consultancy” was brought in to evaluate the firms readiness to use agile techniques.  Things were bad at the time. Morale was low, many projects were perceived as failing, and nearly every meeting ended in frustration.  So when the agile consultants conducted their interviews and group sessions, these tended to devolve into an airing of grievances.

One of these sessions still sticks in my mind.  The consultants asked everyone in the group of about ten people to write one word on an index card describing what the firm could be doing better. We then posted the cards on the wall.  I knew exactly what to write, and within a few minutes my card, emblazoned with the world “TOOLS,” stood out in a field of other cards reading “process.”

At the time it was so painfully obvious to me that tools were the primary source of frustration, and I was disheartened to see my colleagues miss the mark so completely.  In my mind, process meant more meetings, more garbage, and more waste.  We had a tool problem! Most developers were using a shared development server which yielded combinations of code never represented in version control.  While we had individual development virtual machines,there was nothing close to a one click deploy, nothing to bootstrap a project out of version control.  Those of us actually running the projects on our development machines did it from scratch and administered our own boxes. Many of the developers were new to the LAMP stack, and didn’t have the skillset to accomplish these administrative tasks. Likewise, while some projects had functional and integration tests, there was little buy in and the continuous integration server sat dormant. So after the agile consultants asked me about my tool-related frustrations, the topic was gradually redirected to “process” and I zoned out. Process meant bullshit, and these guys were outsiders charging us an arm and a leg to feed us more of it.  And what did it matter? My foot was already halfway out the door, and I went home and sent off a few more resumes.

Unfortunately I was wrong.  Tools weren’t the issue.  We had plenty of tools, and while most were underutilized, this merely required an investment of time.  The problem was community.  Our community was dysfunctional, and fixing it would’ve subsumed all problems.

moving on

There are many ways to build quality software in a reasonable amount of time without wanting to scream, but no one solution is a fit for all situations.  In my current gig, I’m living in a bit of a utopia. I’ve been a lead on a project for an internal system replacing an older system (it sends email, but that’s not terribly important) for about nine months.  I have one extremely well-informed and expert internal client in the marketing department that is the key stakeholder in the project, and because this system is used by many other developers, my colleagues get to try out my APIs and offer feedback.  All told, we have the nicest piece of software I’ve ever written.  It’s internally successful, delivering results early and being improved incrementally, and a joy to work on.  We have tools: the system will bootstrap itself and start running locally with one call to a shell script, the unit tests run all the time, the build server builds the docs (we have docs!), and we’re able to contribute liberally to our shared internal library.  And we have just a bit of process, too: releases are often available in a staging environment, we deploy very rapidly,  feedback from the internal client is nearly instantaneous, progress is tracked in our ticket system, and we have zero standing meetings (instead preferring to do things with short, ad-hoc, high value interactions at the coffee machine).  But even in this same organization I’ve worked on projects that have left me profoundly unhappy, and obviously the approach for my current project won’t always work or scale to other systems.  So I’m left trying to identify the things that make development fun and make this project such a success.

This current project reminds me a lot of working in an academic setting.  In my pre-”real”-job days, the stakeholders were highly involved, highly informed, and wanted to be working on the same stuff that I wanted to be working on (because we were broke and doing it all for free).  But what’s going on now and what was going on then is that we’ve fostered a real sense of community.  Sure, we have some constituent elements that are required to build software in a professional setting (tools, process), and we’ve assembled these nuts and bolts in an orderly fashion. But much more importantly, we’ve fostered a sense of mutual respect.

My stakeholder in the marketing department functions as the closest thing to a project manager I have.  We plan things out and we repeatedly revise our estimates and expectations.  Together he knows that context switching between feature requests slows me down, but I’ve also come to realize that when he asks for something quickly, he’s already thought through the ramifications, and the overhead of the context switch is worthwhile for the business.  The key element here is that our relationship is not adversarial.  Quite the opposite.  I respect his insight into his area of expertise, and he respects mine.  I’ve seen so many project manager/developer relationships go south because both parties perceive the other as the adversary.  The PM thinks the developer is dragging his or her feet, and the developer is irrationally convinced the that fucking over developers was in the PM’s job description. This makes work suck. This makes software suck. Warring factions can’t collaborate to build something great; warring factions can only destroy.

I’m not sure how mutual levels of respect can be earned or fostered.  Of all things, the possibility of this might be the most innate thing in the organization’s human constituents (i.e. we have to presuppose that we’ve hired good people on all sides if this is ever to happen).  But, assuming that respect and trust can be built between the nerds and the stake holders over time, there are still some other factors that have to exist to make a happy project.  One important thing is continued learning on all sides.  Everyone has to be willing to learn new things.  Sometimes this means that business types have to learn new vocabulary.  The might have to know how a cookie works or what it means for the web to be stateless.  And the same holds for developers.  We may have to hold our nose at the imprecise language used in marketing meetings and learn some acronyms and business strategies.  If an unwillingness to learn exists, the project will fail.

Transparency is also key.  Developers need to know how things work.  They need to know the background story around the business decisions that went into a project.  Likewise, the stakeholders need to see that progress is being made.  If we’re building a web app, then the latest stable(ish) release should be available for the business types to tinker with.  The transparency helps both sides.  Sometimes the analytical nerd brain can raise legitimate business questions.  Other times, the stakeholders can stem a failure or misread specification much earlier if they’re interacting with the project on a continual basis.

Finally, fault tolerance and resilience is key to a pleasant project.  Both sides, programmers and stakeholders, have to understand that the other will occasionally make mistakes, misread a spec, or be unclear.  Again, this can’t be adversarial.   For a happy project, we programmers must realize there is no suit and tie cabal maliciously typing away unclear requirements documents.  No, just like we forget to add files to version control and fat finger Unix commands, things get misinterpreted and misread.  So everyone should be aware that mistakes will happen.  Some ideas might have to be delayed, and some 500 pages might be raised.  But, if we respond to these quickly, refuse to panic, and fix both types of problems, the project can will continue to be fun and happy.  And it will succeed.

sum(it_up)

For me, something like a Project Euler problem is the most satisfying computing exercise.  It’s well defined, there is a solution, and I know when I’ve found it.  But this is a mirage.  Very few computing problems are solved with such simple and clear algorithms. Everything is fuzzy and we rarely go it alone.  But if we respect each other along the way, keep and eye on how things are going, and respond to problems as they come we can get closer and closer to a clear solution and enjoy the journey along the way.

Now we all know exec() is cruise-control for awesome. Despising readability, sanity, and performance, I sprinkle my code liberally with this MSG and joyfully await the ensuing migraine. But sometimes I ask myself, is it enough?

Correct Answers Considered Harmful

A wonderfully dynamic language like python lets us take exec() where no string literal has gone before by allowing us to pass in global and local environments that are used for variable lookup. The high-priests of python would have us use this functionality to constrain the side-effects of gratuitious exec()ing, but we can do so much better. A simple subclass of dict() that overrides __getitem__ and only returns the item we’re looking for 50% of the time gives us some ammo:

class AreYouFeelingLuckyDict(dict):
    #this is idealized. see complete source at end of post.
    def __getitem__(self, item):
        if random.choice([True, False]):
            return dict.__getitem__(self, item)
        else:
            raise KeyError(item)

With this dictionary implementation, we can move on to more interesting exec()s:

exec("""a = 1; print a""",
     AreYouFeelingLuckyDict(), #globals
     AreYouFeelingLuckyDict()) #locals

Because python is using instances of AreYouFeelingLuckyDict for global and local variable lookups inside our string of code, half the time the code will work properly. The other half of the time a NameError will be raised. Introducing another variable brings the odds of success to 1 in 4:

exec("""a = 1; b=2; print(a); print(b)""",
     AreYouFeelingLuckyDict(),
     AreYouFeelingLuckyDict())

Of course we can program a bit more defensively and ensure that we’ll get the output we’re looking for. If we encounter a NameError along the way, we simply try, try, and try again:

    exec("""
a=1
b=2
while True:
  try:
    print(a)
    print(b)
    break
  except NameError:
    pass
""", AreYouFeelingLuckyDict(), AreYouFeelingLuckyDict())

Keep in mind that “1″ will most likely be printed out several times as its lookup occurs first (and most persistently) and the exception handling is coarse-grained. The obvious solution is to pipe the output through tail -n 2.

But Wait, There’s More

Becaue of a debilitating gambling problem, bogosort is my favorite sorting algorithm. Bogosort is the equivalent of throwing an array of items onto the floor and seeing if they’ve come up in the proper order. Given a list of N unique items, the odds of getting the answer on the first go round of bogosort are 1 in N!. Running bogosort in production on your company’s machines is like playing roullette (with an enormous wheel and your career). Feel that rush.

But bogosort gets even more awesomer with our exec() trick:

    myglobals = AreYouFeelingLuckyDict(random=random)
    mylocals = AreYouFeelingLuckyDict()
    exec("""
def sorted(is_sorted):
    prev = is_sorted[0]
    for x in is_sorted:
        if prev > x:
            return False
        prev = x
    return True
 
mylist = [4, 1, 9, 5]
runs = 1
while True:
    try:
        is_sorted = []
        while True:
            mutable_mylist = list(mylist)
 
            for i in range(len(mutable_mylist)):
                index = random.randint(0, len(mutable_mylist)-1)
                is_sorted.append(mutable_mylist.pop(index))
 
            if sorted(is_sorted):
                break
            else:
                runs+=1
                is_sorted=[]
        break
    except NameError:
        runs += 1
        pass
""", myglobals, mylocals)

Because I’m a generous man, I’ve modified the AreYouFeelingLuckyDict to always return a result when the item being looked up is “runs”, “NameError”, “True”, “random”, “len”, “list”, “range”, or “sorted.” It’s also worth noting that as soon as we drop into is_sorted() we’re back to a plain-jane, well-behaved locals() with boringly reliable dict-like properties.

With this bogosort modification (err, improvement) and considering a list of length 4, I figure the odds of getting an answer the first time out is 1 in 3,145,824. Now we’re talking! Of course, the code above keeps going until it sorts our list. Running it a few times, I’ve seen the answer produced in as little as 700,000 tries. A paragon of efficiency.

Wink

I hope you’ve enjoyed this. I’ve been playing around with this post for a while and have had a great time. The source to this party can be found in this gist:
https://gist.github.com/945666

Also, please check out Be careful with exec and eval in Python which inspired this post.

Sure, we all know that “default parameter values are evaluated when the function definition is executed” in python. But what may not be clear is a fun way to introduce time-dependent bugs. See, this:

def my_busted_record_keeper(last_update=datetime.date.today()):
    ...
    print("last update was %s" % last_update)

Is totally broken.

If the code runs for more than a day, we aren’t going to get the results we expect since last_update will have been evaluated at definition time which is now yesterday. Instead we should do something like:

def my_busted_record_keeper(last_update=None):
    if last_update is None:
        last_update = datetime.date.today()
    ...
    print("last update was %s" % last_update)

Usually this isn’t a problem because my code crashes at least once a day , but I know I’ve made this mistake before and there’s a non-zero chance something like this running in production. Quite embarrassing.

On Monday I was discussing the y combinator at Rossi’s.  The next morning I wrote an example in scheme to send to my friends, but they don’t speak scheme.  Sadly, this is tricky to write in python because of the lack of multi-line lambdas, but it’s damned easy in javascript:

(function(x, thunk) {
 if(x==0) {
   return 0;
 } else {
   return x + thunk(x-1, thunk);
 }
})(5, function(x, thunk) {
        if(x==0) {
          return 0;
        } else {
          return x + thunk(x-1, thunk);
        }
});

See, there’s something to love about every language. And the beauty of this example is that you can probably type it write into the javascript console of you browser. REPL included.

In the spirit of ChiPy meetings, I’m declaring 2011 to be the best year yet. I can only assume the best-ness of each passing year will increase monotonically until the year of my death, at which point, all bets are off. Here are some reasons why 2010 was pretty great, too:

• I rode my bike to Saint Louis (without being eaten by possums).
• I went to PyCon.
• My friend Brian and I went to Las Vegas and did not lose large sums of money.
• Erin and Dan took my old apartment and are tending to the compost pile with great fervor.
• Camping with friends occurred more frequently.

And here are some reasons why 2011 will also be great:

• I have no plans to land in the ER, break a bone, receive an IV, or be eaten by possums.
• I will go to PyCon.
• Camping with friends will occur more frequently.
• I will read more books.
• Rob Pike will release Plan10 for download in the iTunes App Store.
• Bjarne Stroustrup will announce that this joke has gone on long enough.
• Cable news channels will not mention the woes of a single teenage starlet, instead focusing on well-researched and thoughtful journalism that is neither sensationalist nor pandering to one side of the aisle or another.
• Robots will stop attempting to submit spam comments to my blog.
• We will meet back here and discuss all this next year.

This is kind of silly, but I like the nonlocal keyword in Python 3 (well, ok, I like let-bindings in Lisps much much better, but this will do) and how you can get a closure and then poke the insides of the closure to see what it’s up to.  I’m sure this isn’t recommended, but it’s a fun trick:

def counter():
    a = 0
    def inc():
        nonlocal a
        a+=1
        return a
    return inc
 
c = counter()
print(c())
print(c())
print(c())
 
print(c.__closure__[0].cell_contents)

I’d guess the locations of cells in the closure tuple are determined at compile time (i.e. they won’t change between runs), but I haven’t investigated this.

About the time the nosql movement started to get legs, I started to get interested in relational databases.  They’re awesome beasts, and horizontal scalability issues aside, they handle a lot of the very hard problems in the web programming space (i.e. durability and concurrency).  But when data gets large and you can’t find the 2 million quarters in your sofa for that oracle license (or the 2 million braincells for that oh-so-perfect sharding scheme),  the nosql databases start to show their appeal.

But maybe there’s another way.  This article is mostly focussed on relaxing the durability constraints of postgres but also mentions a possible mixing of relational and non-relational systems.  What a cool idea!  Use the somewhat batch-oriented, distributed, and non-relational system as a large backing store, and populate a traditional relational database (almost) on demand by transforming and moving (ETLing) the data there.  With the right transforms, we get all the ad-hoc queries of our beloved star schema data warehouses without the expense of a big-honkin’ database server.  Certainly these ‘transforms’ are the tricky part (and slicing the data out of the non-relational store along a particular dimension (probably time) will almost always be necessary), but the gains seems pretty great.  Maybe we’ll call this approach a “pattern” someday.

Isn’t big data exciting?

• close your files, flush your files, fsync() your files, sync() if you have to, just don’t be stupid.
• don’t use globals for mutable state. especially when you’re in a multi-threaded environment. or you’re interested in correct answers.
• there’s a good chance the ORM is writing shitty sql.
• coffee != sleep
• more tests!

These tiny things make Python fun and useful:

import datetime
thirty_days = datetime.timedelta(days=30)
thirty_days_ago = datetime.date.today() - thirty_days