thatmattbone.com

First off, I have added some support for timeouts in receive blocks (in erlang this is done with the ‘after’ block). For now this is only supported on the unix-process implementation, but should be easy to implement. While I know that the timeouts are ‘soft’ I do wonder how they are implemented in Erlang. For example, does the timeout usually just measure the amount of time spent blocking, or does it include the amount of time spent searching messages already in the process’ mailbox (which could be lengthy if the process has accumulated large amounts of messages for whatever reason).

There is also some preliminary support for process linking. More specifically, if one process is linked to another, it behaves like an Erlang system process and is notified via a message if the other linked process experiences an error.

Also there has been some seriuos refactoring. Much like the Closette implementation discussed in The Art of the Metaobject protocol, the implementation of erlang-in-lisp is separated into a thin macro-layer and a heavier layer based on generic functions (thus most behavior can be overridden if necessary). The thin macro layer is where something like muproc compatibility would be implemented. It is essentially the public facing side of erlang-in-lisp. The generic function layer is for those wishing to extend the functionality of erlang-in-lisp by, for example, introducing a new concurrency strategy.

On Monday the erlang-in-lisp operation is moving to Bloomington, Indiana. I’ll even be able to write code in my new backyard (or on top of my in-apartment washer and dryer!).

What has been accomplished?
EiL has an implementation of the actors model as illustrated in Erlang. New processes can be spawned, and with the send() and receive() primitives these processes can communicate via message passing. While the spawning of processes is currently limited to a single machine (i.e. not distributed), distribution is possible if processes are spawned by hand on remote machines. Erlang style process registration (mapping a particular process to a name) is supported and is in fact written in EiL using the send() and receive() primitives.

Currently EiL contains two concurrency strategies: unix processes and threads. The decision about which concurrency mechanism to use must be made when EiL is started, and thus the two concurrency strategies cannot be mixed at the moment (some changes to the API should make this possible, and this should be a goal). However, programs written for EiL are unaware of the underlying concurrency mechanism. For example, to run the ping-pong example with unix-process based concurrency we simply:

ERLANG-IN-LISP> (toplevel :process-class ‘unix-process)
ERLANG-IN-LISP> (ping-pong 3)
……
ERLANG-IN-LISP> (toplevel-exit)

To use thread-based concurrency we need only change the keyword to the toplevel function:

ERLANG-IN-LISP> (toplevel :process-class ‘thread-process)

Thus the goal of pluggable concurrency as described in the original proposal has been accomplished. To a certain extent, so has the goal of pluggable messaging. However, messaging is tightly coupled to the concurrency mechanism (i.e. processes serialize messages and communicate via sockets while threads simply exchange references to the messages in memory). Truly pluggable messaging is best left to each concurrency strategy (as opposed to introducing another layer of abstraction). Thus unix-processes may choose to communicate via secure or unsecure sockets while these options would be unavailable to threads.

What remains to be accomplished?
Much remains to be accomplished. Most important is linked processes. I believe this can be accomplished by relying on the Common Lisp condition system. Processes will maintain a link set as in Erlang, and conditions will be handled and forwarded via the messaging primitives to interested processes. While I have sketched out a plan to this end, no code has yet been written.

True distribution including the remote spawning of processes remains as well. The main sticking point is how to serialize closures and ship them off to a remote system. While we can easily send a function name and the required arguments to a remote process and hope that the remote process knows about that function, this is not ideal (and, in fact spawn() only takes a thunk at the moment, so we’d need to figure out how to serialize an anonymous function to capture these arguments…that or have two separate APIs like spawn() and remote-spawn()). We may also wish to generate bindings for the erl_interface so that EiL nodes and code written in EiL can play nicely in an Erlang cluster.

Finally, our ultimate goal of compiling Erlang to EiL remains. To this end, we should focus on compiling Core Erlang only. It may be helpful to develop ‘core EiL’ and then build a Lispier macro-based front end for user consumption.

Some people in the weblocks community have been thinking about some of the things I’ve been pondering. No real answers. I need to play with the continuations in arnesi and cl-cont, but if we can’t move them between address spaces (either distributed or in different unix processes) it’s all for naught.

On Sunday and Monday I patched our version of receive. It now matches the Erlang semantics of blocking until a suitable match is found. This works in the unix-process and thread based concurrency models (it required changes to the API, but shows how we should grow the pluggable concurrency API in an organic fashion (and it shows that it actually works)). Receive timeouts are not implemented, though, so that remains.

My biggest worry at the moment is the half-assed serialization strategy I have for shipping off PIDs and how this isn’t working for spawning remote processes. Sure I can ship off a function name, arguments, and invoke on a remote server no problem…but this doesn’t really match the Lisp semantics (or common sense semantics). I really need some closure serialization ala Termite.