Most of the Erlang-in-Lisp issues discussed here and on the mailing list this summer have been rolled into the user manual in some form. Get it here in pdf format or here in HTML.

First off, I have added some support for timeouts in receive blocks (in erlang this is done with the ‘after’ block). For now this is only supported on the unix-process implementation, but should be easy to implement. While I know that the timeouts are ‘soft’ I do wonder how they are implemented in Erlang. For example, does the timeout usually just measure the amount of time spent blocking, or does it include the amount of time spent searching messages already in the process’ mailbox (which could be lengthy if the process has accumulated large amounts of messages for whatever reason).

There is also some preliminary support for process linking. More specifically, if one process is linked to another, it behaves like an Erlang system process and is notified via a message if the other linked process experiences an error.

Also there has been some seriuos refactoring. Much like the Closette implementation discussed in The Art of the Metaobject protocol, the implementation of erlang-in-lisp is separated into a thin macro-layer and a heavier layer based on generic functions (thus most behavior can be overridden if necessary). The thin macro layer is where something like muproc compatibility would be implemented. It is essentially the public facing side of erlang-in-lisp. The generic function layer is for those wishing to extend the functionality of erlang-in-lisp by, for example, introducing a new concurrency strategy.

On Monday the erlang-in-lisp operation is moving to Bloomington, Indiana. I’ll even be able to write code in my new backyard (or on top of my in-apartment washer and dryer!).

What has been accomplished?
EiL has an implementation of the actors model as illustrated in Erlang. New processes can be spawned, and with the send() and receive() primitives these processes can communicate via message passing. While the spawning of processes is currently limited to a single machine (i.e. not distributed), distribution is possible if processes are spawned by hand on remote machines. Erlang style process registration (mapping a particular process to a name) is supported and is in fact written in EiL using the send() and receive() primitives.

Currently EiL contains two concurrency strategies: unix processes and threads. The decision about which concurrency mechanism to use must be made when EiL is started, and thus the two concurrency strategies cannot be mixed at the moment (some changes to the API should make this possible, and this should be a goal). However, programs written for EiL are unaware of the underlying concurrency mechanism. For example, to run the ping-pong example with unix-process based concurrency we simply:

ERLANG-IN-LISP> (toplevel :process-class ‘unix-process)
ERLANG-IN-LISP> (ping-pong 3)
……
ERLANG-IN-LISP> (toplevel-exit)

To use thread-based concurrency we need only change the keyword to the toplevel function:

ERLANG-IN-LISP> (toplevel :process-class ‘thread-process)

Thus the goal of pluggable concurrency as described in the original proposal has been accomplished. To a certain extent, so has the goal of pluggable messaging. However, messaging is tightly coupled to the concurrency mechanism (i.e. processes serialize messages and communicate via sockets while threads simply exchange references to the messages in memory). Truly pluggable messaging is best left to each concurrency strategy (as opposed to introducing another layer of abstraction). Thus unix-processes may choose to communicate via secure or unsecure sockets while these options would be unavailable to threads.

What remains to be accomplished?
Much remains to be accomplished. Most important is linked processes. I believe this can be accomplished by relying on the Common Lisp condition system. Processes will maintain a link set as in Erlang, and conditions will be handled and forwarded via the messaging primitives to interested processes. While I have sketched out a plan to this end, no code has yet been written.

True distribution including the remote spawning of processes remains as well. The main sticking point is how to serialize closures and ship them off to a remote system. While we can easily send a function name and the required arguments to a remote process and hope that the remote process knows about that function, this is not ideal (and, in fact spawn() only takes a thunk at the moment, so we’d need to figure out how to serialize an anonymous function to capture these arguments…that or have two separate APIs like spawn() and remote-spawn()). We may also wish to generate bindings for the erl_interface so that EiL nodes and code written in EiL can play nicely in an Erlang cluster.

Finally, our ultimate goal of compiling Erlang to EiL remains. To this end, we should focus on compiling Core Erlang only. It may be helpful to develop ‘core EiL’ and then build a Lispier macro-based front end for user consumption.

Some people in the weblocks community have been thinking about some of the things I’ve been pondering. No real answers. I need to play with the continuations in arnesi and cl-cont, but if we can’t move them between address spaces (either distributed or in different unix processes) it’s all for naught.

On Sunday and Monday I patched our version of receive. It now matches the Erlang semantics of blocking until a suitable match is found. This works in the unix-process and thread based concurrency models (it required changes to the API, but shows how we should grow the pluggable concurrency API in an organic fashion (and it shows that it actually works)). Receive timeouts are not implemented, though, so that remains.

My biggest worry at the moment is the half-assed serialization strategy I have for shipping off PIDs and how this isn’t working for spawning remote processes. Sure I can ship off a function name, arguments, and invoke on a remote server no problem…but this doesn’t really match the Lisp semantics (or common sense semantics). I really need some closure serialization ala Termite.

I just committed the last touches on some pretty major changes in the erlang-in-lisp epmd branch. I was distracted from my linking/registering work on Tuesday by the fragility of the original design. I had an idea to use CLOS and hoped this would help achieve our goals of pluggable concurrency and pluggable networking as set out in our proposal. I think it worked out; the two branches (main and epmd) currently have the same functionality. However, the epmd branch has an additional layer of abstraction. The concrete implementations of send/spawn/receive can be switched an runtime, and if the appropriate methods are implemented communication should be possible between implementations (currently there is only one implementation).

Next steps are to finally publish my todo list, test distribution (the way things are setup now, it should be working already), and return to my work on registering/linking/spawning remote processes. The CLOS experiment did suck up some considerable time, but I’d still like to finish remote spawning and registration by monday.

One final note, this is really the first time I’ve used CLOS extensively. My initial attraction to Lisp and other languages with an appreciation for functional style came out of a frustration with the tedium of Java. I guess I closely associated OOP with Java and was never very interested in persuing an OO style in these languages. Working with CLOS and generic methods, however, really is OO done right and appropriately melds FP and OOP. For example, specializing on two variables in method feels very functional (actually it makes me feel like I’m using Haskell…is that a good thing?), but we can still have side-effects within the objects themselves. I’m rambling; I like CLOS.

One of the things I’ve always been unclear on is how a server binds to particular address. This became apparent today in my erlang-in-lisp work with iolib and erlang itself. I had been messing around with distributing the ping-pong example last week and things were not going as smoothly as I’d initially hoped. It was all coming down to dns, the hosts file, and binding the server to the wrong address. Today I tried the example in erlang, and my mistake became clear when I read up on the behavior of erlang with regards to ‘short names’ and ‘long names’ (it seems erlang is fond of adopting somewhat unusual terminology).

It turns out that some of my thinking last week was unacceptable; I’d been expecting self() to return information that could only be retrieved after a connection to a particular process. This is impossible given the semantics of self(). Furthermore, I’d forgotten about the node() BIF in erlang. By matching the behavior of the sname and name command line parameters in erlang, all has become clear. The user specifies the hostname (or sometimes the IP address) to which the erlang (or erlang-in-lisp) process should be bound. While this may seem a bit tedious at first blush, it really does (along with the cookie system) prevent unsuspecting users from casually opening up erlang processes to the outside world. This will be (has been) duplicated in eil (though there is still a runtime typing problem keeping things from working properly at the moment…hopefully trivial).

Tomorrow I’m planning to swing by Loyola to pick up my ‘alumni card’. I hope it allows me into the library (it should) to check out “Unix Network Programming” as I think this reference will be useful in the coming days. I’m planning to tackle distribution and linking processes simultaneously this week. I’m also hoping to be diligent in writing unit tests as I’ve found myself retyping the same test code at the repl over the last several days.

I’ve also been wondering about an erlang grammar. Does one exist and could be use it in something like a lisp packrat parser? Off to bed now…I hope this post does not sound like gibberish in the morning

The ping-pong example is working with one caveat. Matches against patterns with just one atom must still be wrapped in a list. This is annoying, but we can fix it.

This first step happened a bit more slowly than I’d like, but I can see where we’re going now. We’re on our way.

One of the things we may want to think about for erlang-in-lisp is a top level. In erlang, one can send and receive messages right at the REPL because it is a process with a mailbox. For us, we have to do some additional bookkeeping (thus the standard REPL is not an erlang-in-lisp process). This should be easy to implement and handy to some.

I have a very simple example (sort of) working in erlang-in-lisp/examples.lisp. I did a quick hack to integrate fare-match into send/receive. The matching really isn’t working because the message is being mangled before it is sent over the socket. More tomorrow…