Several years ago, nearing the end of my time at my first “real” job, an “agile consultancy” was brought in to evaluate the firms readiness to use agile techniques.  Things were bad at the time. Morale was low, many projects were perceived as failing, and nearly every meeting ended in frustration.  So when the agile consultants conducted their interviews and group sessions, these tended to devolve into an airing of grievances.

One of these sessions still sticks in my mind.  The consultants asked everyone in the group of about ten people to write one word on an index card describing what the firm could be doing better. We then posted the cards on the wall.  I knew exactly what to write, and within a few minutes my card, emblazoned with the world “TOOLS,” stood out in a field of other cards reading “process.”

At the time it was so painfully obvious to me that tools were the primary source of frustration, and I was disheartened to see my colleagues miss the mark so completely.  In my mind, process meant more meetings, more garbage, and more waste.  We had a tool problem! Most developers were using a shared development server which yielded combinations of code never represented in version control.  While we had individual development virtual machines,there was nothing close to a one click deploy, nothing to bootstrap a project out of version control.  Those of us actually running the projects on our development machines did it from scratch and administered our own boxes. Many of the developers were new to the LAMP stack, and didn’t have the skillset to accomplish these administrative tasks. Likewise, while some projects had functional and integration tests, there was little buy in and the continuous integration server sat dormant. So after the agile consultants asked me about my tool-related frustrations, the topic was gradually redirected to “process” and I zoned out. Process meant bullshit, and these guys were outsiders charging us an arm and a leg to feed us more of it.  And what did it matter? My foot was already halfway out the door, and I went home and sent off a few more resumes.

Unfortunately I was wrong.  Tools weren’t the issue.  We had plenty of tools, and while most were underutilized, this merely required an investment of time.  The problem was community.  Our community was dysfunctional, and fixing it would’ve subsumed all problems.

moving on

There are many ways to build quality software in a reasonable amount of time without wanting to scream, but no one solution is a fit for all situations.  In my current gig, I’m living in a bit of a utopia. I’ve been a lead on a project for an internal system replacing an older system (it sends email, but that’s not terribly important) for about nine months.  I have one extremely well-informed and expert internal client in the marketing department that is the key stakeholder in the project, and because this system is used by many other developers, my colleagues get to try out my APIs and offer feedback.  All told, we have the nicest piece of software I’ve ever written.  It’s internally successful, delivering results early and being improved incrementally, and a joy to work on.  We have tools: the system will bootstrap itself and start running locally with one call to a shell script, the unit tests run all the time, the build server builds the docs (we have docs!), and we’re able to contribute liberally to our shared internal library.  And we have just a bit of process, too: releases are often available in a staging environment, we deploy very rapidly,  feedback from the internal client is nearly instantaneous, progress is tracked in our ticket system, and we have zero standing meetings (instead preferring to do things with short, ad-hoc, high value interactions at the coffee machine).  But even in this same organization I’ve worked on projects that have left me profoundly unhappy, and obviously the approach for my current project won’t always work or scale to other systems.  So I’m left trying to identify the things that make development fun and make this project such a success.

This current project reminds me a lot of working in an academic setting.  In my pre-”real”-job days, the stakeholders were highly involved, highly informed, and wanted to be working on the same stuff that I wanted to be working on (because we were broke and doing it all for free).  But what’s going on now and what was going on then is that we’ve fostered a real sense of community.  Sure, we have some constituent elements that are required to build software in a professional setting (tools, process), and we’ve assembled these nuts and bolts in an orderly fashion. But much more importantly, we’ve fostered a sense of mutual respect.

My stakeholder in the marketing department functions as the closest thing to a project manager I have.  We plan things out and we repeatedly revise our estimates and expectations.  Together he knows that context switching between feature requests slows me down, but I’ve also come to realize that when he asks for something quickly, he’s already thought through the ramifications, and the overhead of the context switch is worthwhile for the business.  The key element here is that our relationship is not adversarial.  Quite the opposite.  I respect his insight into his area of expertise, and he respects mine.  I’ve seen so many project manager/developer relationships go south because both parties perceive the other as the adversary.  The PM thinks the developer is dragging his or her feet, and the developer is irrationally convinced the that fucking over developers was in the PM’s job description. This makes work suck. This makes software suck. Warring factions can’t collaborate to build something great; warring factions can only destroy.

I’m not sure how mutual levels of respect can be earned or fostered.  Of all things, the possibility of this might be the most innate thing in the organization’s human constituents (i.e. we have to presuppose that we’ve hired good people on all sides if this is ever to happen).  But, assuming that respect and trust can be built between the nerds and the stake holders over time, there are still some other factors that have to exist to make a happy project.  One important thing is continued learning on all sides.  Everyone has to be willing to learn new things.  Sometimes this means that business types have to learn new vocabulary.  The might have to know how a cookie works or what it means for the web to be stateless.  And the same holds for developers.  We may have to hold our nose at the imprecise language used in marketing meetings and learn some acronyms and business strategies.  If an unwillingness to learn exists, the project will fail.

Transparency is also key.  Developers need to know how things work.  They need to know the background story around the business decisions that went into a project.  Likewise, the stakeholders need to see that progress is being made.  If we’re building a web app, then the latest stable(ish) release should be available for the business types to tinker with.  The transparency helps both sides.  Sometimes the analytical nerd brain can raise legitimate business questions.  Other times, the stakeholders can stem a failure or misread specification much earlier if they’re interacting with the project on a continual basis.

Finally, fault tolerance and resilience is key to a pleasant project.  Both sides, programmers and stakeholders, have to understand that the other will occasionally make mistakes, misread a spec, or be unclear.  Again, this can’t be adversarial.   For a happy project, we programmers must realize there is no suit and tie cabal maliciously typing away unclear requirements documents.  No, just like we forget to add files to version control and fat finger Unix commands, things get misinterpreted and misread.  So everyone should be aware that mistakes will happen.  Some ideas might have to be delayed, and some 500 pages might be raised.  But, if we respond to these quickly, refuse to panic, and fix both types of problems, the project can will continue to be fun and happy.  And it will succeed.

sum(it_up)

For me, something like a Project Euler problem is the most satisfying computing exercise.  It’s well defined, there is a solution, and I know when I’ve found it.  But this is a mirage.  Very few computing problems are solved with such simple and clear algorithms. Everything is fuzzy and we rarely go it alone.  But if we respect each other along the way, keep and eye on how things are going, and respond to problems as they come we can get closer and closer to a clear solution and enjoy the journey along the way.

A few weeks back, I boasted in a bar that anyone could write a simple s-expression parser from scratch (no flex, no yacc, no bison, no antler, etc).  Well, I could not boast without doing it myself, so I finally did.  My s-expression lexer/parser weighs in at about 170 lines without tests (450 lines with them), and parsers integers and symbols.  I even created a little repl which takes this:

(+ 1 2)

And converts it into this:

[Symbol(+), Number(1), Number(2)]

A pretty-printed python list consisting of objects representing symbols and numbers.

I think the design is fairly readable if a bit unusual. There are classes for each token (instances of which are used for fully and partially formed tokens) and we “add” these tokens together to form a larger token and ultimately invoke a next_token_exception allowing the lexer to yield the largest, greediest, most fully-formed token possible.

Here’s an example of the lexer in action:

>>> symbol_token("h")
<parse_sexp.symbol_token object at 0x7fb9e2d253d0>
>>> _ + symbol_token("i")
<parse_sexp.symbol_token object at 0x7fb9e2d25710>
>>> formed = _
>>> formed + " "
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "parse_sexp.py", line 33, in __add__
    raise next_token_exception()
parse_sexp.next_token_exception
>>> formed.value
'hi'

After lexing, we’re left with a stream of fully-formed tokens; it’s a simple matter to filter out the whitespace tokens and start yielding instances of Symbol() and Number() classes with proper, pythonic types for the data.

Although I used regexes (quite unnecessarily) and generators, this little thing could probably be ported to C. It’d be the fastest, most useless parser/lexer around

Happy Hacking!

• close your files, flush your files, fsync() your files, sync() if you have to, just don’t be stupid.
• don’t use globals for mutable state. especially when you’re in a multi-threaded environment. or you’re interested in correct answers.
• there’s a good chance the ORM is writing shitty sql.
• coffee != sleep
• more tests!

I created this image using matplotlib and the data from my CPS Racial Data Warehouse project.  Circle size represents school population.  The axes are latitude and longitude (as a reference, the loop is the empty rectangle just below the left corner of the legend).  Also I should note the Native American population was excluded from this image because of a technical issue.

Full size image here, more on this project soon (in the meantime, the code is here).

I’ve been thinking a lot about software quality and testing. This is what I’ve come up with:

When I was teaching data structures, one of the things that commonly came up was the whole issue of “pass by value” versus “pass by reference.” This was a very difficult thing to teach;  somehow I’ve internalized the common C/Java style memory model, and these things are fairly second nature.  While lecturing, I was definitely guilty of resorting to mind-numbing aphorisms like “integers are passed by value while objects are passed by reference”.  These rules of thumb just don’t work (and make it incredibly difficult to get your head around the idea that object _references_ are passed by value). This (fabulous) article over at MSDN has a really nice description of the stack as an implementation detail.  Perhaps the key to solving this problem is really to just make the first course in computer science programs a course in programing language implementation.  Then you can move on to harder things like control flow and datastructures (I’m kind of kidding here).

But in reality, the vagaries of “value types” versus heap-allocated objects are too pointy for a data structures course.  Their guts are easily exposed in something like C while being appropriately hidden in an everything-is-an-object language like Python.  The Java/C# mix of these memory allocation strategies is confusing for the beginner.  Expose the details or don’t.  Exceptions to the rules breed bugs and confusion.

I spent a significant portion of my undergraduate career fooling around with the outdated seismology lab in the Loyola Physics department. Though I did eventually connect the seismometers to a computer and was able to record activity, they usually are not running as there is currently no student interested in checking in daily. However, we do turn them on every now and then for a lab that the physics students do. So, we hooked everything up yesterday, and BAM, the biggest earthquake in many years hit Chicago this morning. We captured the quake, and I am positively thrilled. Two traces are below. The times are way off (clock drift), but you can clearly see the quake on the 14:00 line. Also, apparently the after shock was centered in “Bone Gap, IL”! Ha! Amazing.

We’ve created an ETL Yahoo pipe that aggregates the content of members’ blogs tagged with ‘etl.luc.edu’. What I’m wondering is how quickly the pipe gets updated, and where things get cached along the line…

After some discussions with Dr. George recently, I was interested in exploring the relationship between xml and s-expressions. This has certainly been done before, but I thought I would try my hand at it. In the end result, I’m able to convert an s-expression such as:

(a :a1 "test" :a2 "test2" "mycdata" (b :a3 "c"))

to xml:

<a a1="test" a2="test2">mycdata<b a3="c"></b></a>

I do so with a somewhat strange but more explicit intermediate format:

(:element-name "a" :attributes (("a1" . "test") ("a2" . "test2")) :cdata nil                   :children (:element-name "b" :attributes (("a3" . "c"))                                :cdata "My CDATA" :children nil))

This shows pretty clearly how a single namespace xml document can be represented with an s-expression. What I’m wondering now is if I can go from the intermediate representation to something like YAML or JSON. George and I have been calling these ‘tree languages’; I wonder if some sort of overarching equivalence for these ‘tree languages’ can be shown?

I don’t think this has much practical use, but the code is here.

(I’m guessing that a lot of what I say here applies to all distributed version control systems, but since I’ve only used darcs, that is what I’ll focus on.)

One of the neatest things about darcs is the fact that every checked out copy is itself a repository. This is perfect for university courses! Students can check out the professor’s code, make their own changes to the code (i.e. if the code is the starting point for some assignment…quite common), and commit their changes to their own repository without ‘disconnecting’ it from the professor’s version. All the student needs to do is pull changes from the professors server periodically and deal with any conflicts that may arise. The student can even push their copy to a university server for backup. It’s perfect!

If only Google code had darcs support, then I could force my 271 students to do this. As it stands, I’m not so sure a data structures course is the ideal place to introduce version control, but I may end up doing this anyway.