thatmattbone.com

I just committed the last touches on some pretty major changes in the erlang-in-lisp epmd branch. I was distracted from my linking/registering work on Tuesday by the fragility of the original design. I had an idea to use CLOS and hoped this would help achieve our goals of pluggable concurrency and pluggable networking as set out in our proposal. I think it worked out; the two branches (main and epmd) currently have the same functionality. However, the epmd branch has an additional layer of abstraction. The concrete implementations of send/spawn/receive can be switched an runtime, and if the appropriate methods are implemented communication should be possible between implementations (currently there is only one implementation).

Next steps are to finally publish my todo list, test distribution (the way things are setup now, it should be working already), and return to my work on registering/linking/spawning remote processes. The CLOS experiment did suck up some considerable time, but I’d still like to finish remote spawning and registration by monday.

One final note, this is really the first time I’ve used CLOS extensively. My initial attraction to Lisp and other languages with an appreciation for functional style came out of a frustration with the tedium of Java. I guess I closely associated OOP with Java and was never very interested in persuing an OO style in these languages. Working with CLOS and generic methods, however, really is OO done right and appropriately melds FP and OOP. For example, specializing on two variables in method feels very functional (actually it makes me feel like I’m using Haskell…is that a good thing?), but we can still have side-effects within the objects themselves. I’m rambling; I like CLOS.

One of the things I’ve always been unclear on is how a server binds to particular address. This became apparent today in my erlang-in-lisp work with iolib and erlang itself. I had been messing around with distributing the ping-pong example last week and things were not going as smoothly as I’d initially hoped. It was all coming down to dns, the hosts file, and binding the server to the wrong address. Today I tried the example in erlang, and my mistake became clear when I read up on the behavior of erlang with regards to ‘short names’ and ‘long names’ (it seems erlang is fond of adopting somewhat unusual terminology).

It turns out that some of my thinking last week was unacceptable; I’d been expecting self() to return information that could only be retrieved after a connection to a particular process. This is impossible given the semantics of self(). Furthermore, I’d forgotten about the node() BIF in erlang. By matching the behavior of the sname and name command line parameters in erlang, all has become clear. The user specifies the hostname (or sometimes the IP address) to which the erlang (or erlang-in-lisp) process should be bound. While this may seem a bit tedious at first blush, it really does (along with the cookie system) prevent unsuspecting users from casually opening up erlang processes to the outside world. This will be (has been) duplicated in eil (though there is still a runtime typing problem keeping things from working properly at the moment…hopefully trivial).

Tomorrow I’m planning to swing by Loyola to pick up my ‘alumni card’. I hope it allows me into the library (it should) to check out “Unix Network Programming” as I think this reference will be useful in the coming days. I’m planning to tackle distribution and linking processes simultaneously this week. I’m also hoping to be diligent in writing unit tests as I’ve found myself retyping the same test code at the repl over the last several days.

I’ve also been wondering about an erlang grammar. Does one exist and could be use it in something like a lisp packrat parser? Off to bed now…I hope this post does not sound like gibberish in the morning

The ping-pong example is working with one caveat. Matches against patterns with just one atom must still be wrapped in a list. This is annoying, but we can fix it.

This first step happened a bit more slowly than I’d like, but I can see where we’re going now. We’re on our way.

One of the things we may want to think about for erlang-in-lisp is a top level. In erlang, one can send and receive messages right at the REPL because it is a process with a mailbox. For us, we have to do some additional bookkeeping (thus the standard REPL is not an erlang-in-lisp process). This should be easy to implement and handy to some.

I have a very simple example (sort of) working in erlang-in-lisp/examples.lisp. I did a quick hack to integrate fare-match into send/receive. The matching really isn’t working because the message is being mangled before it is sent over the socket. More tomorrow…

One of the interesting things about processes in Unix is that they are tree structured. This is easily seen with the pstree command; init is the root of all processes. Several things, can, however go wrong with this tree. Defunct or zombie processes show up when a child process dies and the parent is unaware. Likewise, orphan processes occur when the parent dies and the child is still running. Despite the names, neither is a tragedy. Zombie processes are removed from the process table when the parent dies, and orphans become the children of init. Thus, the strict tree structure is maintained.

Compare this to a multithreaded system. Threads are not forked from other threads. They have no explicit parent other than the enclosing process or address space. If we treat threads like objects in a garbage collected OO environment (i.e. we need to have a reference to a thread in order for it to survive), we quickly see they are not tree structured like Unix processes. One thread can create five new threads, pass the references to other threads, and disappear. There is no parent/child relationship, and the newly created threads needn’t care about the status of the thread in which they were originally created. The representation of these relationships is not a tree, but rather a graph that evolves over time.

What does this have to do with erlang-in-lisp? The light-weight processes in Erlang are much closer to the graph-nature of multithreaded systems, and with our current approach, we will have to map this graph onto a the Unix process tree.

The simplest approach is to have each process wait for its children to complete before exiting or returning. This should probably be our first approach. However, in larger or long running systems, this could become a burden, with many short-lived processes hanging around and taking up memory (though not cycles) while their longer-lived children run to completion.

As I suspected, make-socket-pair worked just fine. I discovered this yesterday when I set the input and output buffer sizes to 1. Then, I had the sinking sensation that while I’d already tried flushing the buffer, I had been flushing the wrong side.

I think one of the side items that should come out of erlang-in-lisp is a simple iolib tutorial.

Well the first week of SoC has come and gone. I’m disappointed with my progress. I had hoped to have some simple ping/pong examples working by the end of the first week, but I’ve yet to resolve some issues with iolib and message passing. Realistically, I think I can fix these problems and have the examples working today or tomorrow.

My problems stem from local domain sockets in iolib. I can express what I’d like to do in C with socketpair/fork/read/write (and get it working), and the iolib API should, of course, make this much simpler. But for whatever reason, the streams created by make-socket-pair, don’t seem to be connected; when I write to the one side, I cannot read from the other. Late last night I started to suspect this was some buffering issue, and fired up the slime inspector on these objects. I didn’t learn much. I also looked closely at the iolib code, and everything seems to be in order. The problem is on my end. My next step is to try using them as binary streams (I’ve been treating them as character streams) ala the tcp server in philip-jose. Either that or raise hell in #lisp.

I view these hiccups as noise rather than signal, and I still think we’re on track for an interesting summer. I have a much more fundamental question about our approach in the next posting.