Copyright © 2003, 2004 Gene Michael Stover. All rights reserved. Permission to copy, store, & view this document unmodified & in its entirety is granted.
I implemented a full TCPMUX server as a single-threaded program in Lisp. This document describes why I did it, what I did, how I did it, & the results.
The TCP Port Service Multiplexer (TCPMUX) is a network service described in RFC 1078 ([Lot88]).
A simplified description of TCPMUX is in Figure 2.1.
When the client requests a service by supplying its name, it can request all sorts of things. The TCPMUX protocol description is not concerned with what services are provided. It's only concerned with how the client connects to a true service through the TCPMUX server.
RFC 1078 doesn't specify any true services except that a HELP service sends the names of all the services the TCPMUX server supports, & RFC 1078 recommends that the host computer makes many of the standard services available through their names in /etc/services.
Service implementors could give their services any name that would be unique. Presumably, service implementors would have chosen a naming system like Sun recommends for classes in Java: the implementor's domain name reversed, followed by package names & finally the class's name. See Chapter 10 for a more detailed discussion.
You can see a TCPMUX server in action without installing any new software because you can use telnet as a simple client. Table 3.1 outlines the steps to see TCPMUX work using telnet as a client.
In the first step from Table 3.1, run ``telnet cybertiggyr.com 41397''.3.1 Notice that the port number is 41397, not the standard TCPMUX port 1. I am running my TCPMUX server on 41397 because I would need to run it from root for it to use port 1, & I don't at this time want to run it as root.
After you run the telnet command, wait for telnet to indicate that you are connected. You'll probably see a message like ``Escape character is ]''. You will not see a ``login'' prompt.
Type the name of a service supported by the TCPMUX server. A good choice is ``help'' because that should be supported by all TCPMUX servers, & it prints the names of the other services the TCPMUX server supports. Be careful when you type the service name; TCPMUX does not support backspaces. You must type it correctly the first time. You might type it correctly into another window, then copy-&-paste it into the terminal window where you are connecting to TCPMUX. Case is not significant, so you could type ``help'', ``HELP'', ``HeLp'', or any other permutation of capitals & lower case letters.
The TCPMUX server will first send a one-line reply. If the first character in the line is a plus (+), the service name was recognized & accepted. If the first character in the status line is a minus character (-), it means the service name was in error.
If you requested the ``help'' service & it was accepted (status message began with ``+''), you'll then see a list of all the services the TCPMUX server supports.
After sending all of its results, my TCPMUX server waits five seconds before closing the connection. I had to do this so that telnet clients would print all of the results. Without the delay, I found that telnet clients usually printed their ``Connection closed by foreign host'' message before it had printed the last line of the results. Or maybe telnet printed its message on top of the last line. I'm not sure. Anyway, a five-second delay before closing the socket solved the problem. So be patient for five seconds while you wait for my TCPMUX server to close the connection with your telnet client.
So why did I implement TCPMUX? And single-threaded? And in Lisp of all things?
The question about Lisp is easiest to answer. Though I program in all sorts of languages for a living, my favorite language is Lisp. When I write programs for myself or for the sake of programming, I usually write them in Lisp. This TCPMUX project was for myself & for the sake of programming. So my first requirement was ``Write it in Lisp because I want to see it run in Lisp''.
I think threads are over-used. They are too often used at the expense of simplicity & maintainability. In fact, threads are often used at the expense of correctness.
A single thread can achieve most of the performance you can achieve with multiple threads unless you have multiple CPUs and you know what you are doing with multiple-threads. Multiple CPUs are fairly common these days, but most humans can't do a very good job with multiple threads.
I've written single-threaded servers that support multiple connections & have good performance, but until now, they've all been owned by my employers. I wanted to implement such a server that was free software4.1 as proof that single-threaded network servers work just fine. So my second requirement was ``Use a single thread because I want to demonstrate that it can be done with a single thread''.
Why TCPMUX? As my first experiment in network programming with Lisp4.2, I wanted something that was ...
TCPMUX is all of those things. TCPMUX is an excellent service to implement if you want your main work to deal with the network code in the server, not with the service provided by the server. Also, I've always had a fondness for TCPMUX & the vision that must have inspired it (Chapter 10), but I have never seen an implementation. So my third requirement was ``The server I will implement will be TCPMUX''.
Common Lisp does not include functions for networking, so a network server must use some non-standard Lisp functions. Nevertheless, I wished to keep the number of such functions to a minimum because ...
So my fourth requirement was ``Minimize the number of non-standard Lisp functions I use''.
My final requirement, or maybe it was my original & most important requirement, was that I wanted to see it run.
The current release is 04. It was released on 19 January 2004. The source code archive is available for download in two formats.
In addition, you may browse the source code online at http://lisp-p.org/tcpmux/tcpmux-04/.
As of 19 January 2004, the requirements & instructions in this section have not been tested. They may be incomplete or otherwise erroneous.
Here is a list of the software that you must install before installing TCPMUX. My TCPMUX distribution contains client & server programs. The server is written in Lisp & will require a Lisp system & other libraries, but the client programs are written in plain C & probably do not require software that is not already on most unix-like systems.6.1
I wrote TCPMUX for clisp, but I kept the non-standard functions to a minimum.
If you are not a Lisp programmer, you probably must have clisp installed. I used release 2.31, which is the current release as I write this in December 2003.
If you are a Lisp programmer & want to port it to another Common Lisp, you will need to implement four non-standard functions that I used from clisp. Those functions are listed in Table 6.1.
These special functions from clisp are documented in The Implementation Notes for Gnu Clisp ([aDSS03]).
If your Common Lisp has its own networked functions, it's probable you can write wrappers for them & give your wrappers the names & semantics of clisp's functions. Otherwise, you might have to resort to your Lisp's foreign function interface.
cpio -i''.
tar xf -''.
![\begin{table}{\tt\small
\begin{itemize}
\item (SOCKET:SOCKET-SERVER \&OPTIONAL [...
...-stream-or-list \&OPTIONAL [seconds [microseconds]])
\end{itemize}}\end{table}](img3.png)
--help''
to get a list.
If everything worked, your TCPMUX server is bin/tcpmuxd. If the tcpmux-04 directory is your current directory, you can start the server by running ./bin/tcpmuxd. The server & the client programs read the port number from ./share/tcpmux.port. You can stop the server by running ./bin/shutdown.
There are some client programs; all are named bin/clnt-*. All of the programs are designed to be run with tcpmux-04 as the current directory. They all try to connect to TCPMUX on localhost by default, but some of them allow you to specify an alternate server with a ``-s hostname'' command line option. Only some of the client programs allow that option; I have been inconsistent (& delinquent) in implementing it.
The other chapters I wrote about the architecture were really bad. Perplexes me because I don't normally have trouble writing. Sure, you might not what I normally write, but I do & I don't feel like it was work writing it. But the description of TCPMUX's architecture is different. So this chapter is a second attempt at it. If I can make this chapter work, I'll probably remove the others, & I'll of course remove this paragraph.
The foundation of my TCPMUX application server is the event loop. Its primary responsibility is to allow other functions the opportunity to handle I/O events on sockets. The event loop's secondary responsibility is to allow other functions to schedule yet more functions to be evaluated at specific times or at specific periods.
The entry point to the event loop is the function RUN in the package CYBERTIGGYR-TCPMUX-SERVER in the file server.lisp. It has one argument, which is a done-function. The RUN function loops, calling the done-function with FUNCALL until the done-function returns true.
The event loop needs to know what sockets it's monitoring. Instead of monitoring the sockets directly, it monitors objects which obey a connection protocol, which is a set of generic functions that classes may implement. Those generic functions are:
Returns the socket that obj reads or writes.
Returns true if & only if obj wants to read from its socket.
Returns true if & only if obj wants to write to its socket.
Tells obj that an input event is pending on its socket. If obj's socket is a server socket, it should accept the new socket, create a new & appropriate connect object around it, & insert that new object into the event loop's list of sockets. Otherwise, obj's socket is a regular, non-server socket, so obj should read one octet from it. Obj should not attempt to read more than one octet because the operation might block.
Tells obj to write one octet over its socket.
Notifies obj that its socket was closed from the other side or that the connection was broken. In other words, notifies obj that its socket has closed for some reason other than a request from obj to close it.
All of these generic functions are declared in server.lisp. To find the start of them, search that file for ``defgeneric socket''. When I talk about a connection or a connection object, I mean any object which obeys this Connection Protocol, regardless of the object's actual class.
The event loop needs to keep a list of the connection objects its monitoring. The list isn't public, but there are two functions for manipulating it. Both of the functions are in the package CYBERTIGGYR-TCPMUX-SERVER.
Inserts the connection into the list of connections the event loop is monitoring.
Removes the connection from the list of connections the event loop is monitoring. It is not an error if the connection is not in the list.
Neither function deals well with a connection that has been added to the list multiple times, so don't insert a connection multiple times.
These two functions are in the server.lisp file.
The event loop needs to make one more component visible. That component is the event queue. It's an instance of CyberTiggyr Flez.
Notice that the event loop doesn't actually know anything about TCPMUX specifically. It could be used for any sort of network service, not just for TCPMUX. The flip-side of this is that we need to create connection objects that know about TCPMUX.
We need two types of connection objects to implement TCPMUX. The first & simpler type of connection is the socket server which accepts new sockets & creates the other type of TCPMUX connection. The second & more complex type of TCPMUX connection object is what I'll call the TCPMUX session connection.
The TCPMUX server connection just needs to create new sockets with the accept function, then create a TCPMUX session connection on the new socket. There will most likely be exactly one TCPMUX server connection object in a TCPMUX network server process.
The TCPMUX server connection type is defined in the file tcpmux.lisp. It begins with ``defclass service'' in that file. It's pretty simple. It always wants to ``read'' from its socket, & when it reads in its EREAD method, it simply creates a new TCPMUX session connection on the new socket & inserts the session connection into the event loop's list of connections. You can create a TCPMUX server connection object on a particular TCP port with the MAKE-SERVICE function.
So the TCPMUX server connection object is pretty simple. I suspect that nearly any server connection for any other service will be equally simple. In other words, the TCPMUX server connection object is representative of server connection objects for other types of services, if we were to use this TCPMUX network server program for services other than TCPMUX.
The TCPMUX session connection object is where it gets interesting. I suspect it is not representative of the session connection objects for other types of services.
The TCPMUX session connection object reads the name of a service that the client process (on the other end of the socket) wants. I'll call this the true service. If the TCPMUX session connection object recognizes the true service name, it creates a connection object for that type of service. The new, true connection object will read & write the same socket, which is unusual. The TCPMUX session connection object will then insert the new, true connection into the event loop's list of connections, & it'll remove itself. That's what's unusual about the TCPMUX session connection object.
Notice that though the TCPMUX session connection object is unusual & makes careful use of the event loop's list of connections, it does not rely on a privileged interface to the event loop. It uses the same interface to the event loop that any other service can use, but it makes use of a solid understanding of that interface.
The TCPMUX session connection type is defined in the file tcpmux.lisp, but I called the class TRANSIENT instead of session connection. Search the file for ``defclass transient'' to find the beginning of TCPMUX session connection's definition.
If connection objects are event-driven tasks, then the purpose of the TCPMUX session connection object is for one task to create a new task that reads & writes a particular socket. See Unix's dup/fork/exec Dance .
How does the TCPMUX session connection know how to create a true connection from the true service's name? It keeps a dictionary that maps true service names to functions. Each of the functions has one argument which is the socket for the connection it will create & return.
If you make a service, you can add it to TCPMUX's dictionary of names-to-functions with the REGISTER-SERVICE function.
Register a service for TCPMUX to know about. The name is a string. The maker is a function of one argument (a socket) which creates a new connection object for that socket & returns the new connection object. The case of the name string is irrelevant; it will be normalized before it is stored in the dictionary.
Return a list of the names of the services that have been registered.
Before I wrote a bunch of services, I implemented some classes that would do some of the work that many services would need. The first of those classes is BASIC-CONNECTION.
Class BASIC-CONNECTION is in server.lisp. Search for ``defclass basic-connection'' to find where it begins.
BASIC-CONNECTION has one attribute, the socket it uses. The accessor for that socket is called SOCKET.
I created two generic functions for BASIC-CONNECTION. They are NREAD & NWRITE:
A BASIC-CONNECTION's EREAD method calls NREAD on the BASIC-CONNECTION. Obj is the BASIC-CONNECTION. Ch is the octet that was just read from the BASIC-CONNECTION's socket. The BASIC-CONNECTION should handle the octet immediately. It might want to append it to a buffer, or write it to a file, or handle a command that has been buffered if the octet ends a command. The BASIC-CONNECTION should do whatever is appropriate; it all depends on the protocol the BASIC-CONNECTION is using to communicate with the process on the other and of the socket.
BASIC-CONNECTION's EWRITE method calls NWRITE on itself to get the next octet it should write over the socket. NWRITE should return an octet to write if there is something to write, the symbol :CLOSE if there is nothing more to write & the socket should be closed, or NIL if there is nothing to write now but there might be something to write later. (Rather than returning NIL from NWRITE repeatedly, a connection object should indicate that it is no interested in writing. Then NWRITE will not be called at all.)
BASIC-CONNECTION's EREAD method reads the next octet from the socket & sends an NREAD message to itself with the new octet. Subclasses will not need to re-implement the EREAD method.
BASIC-CONNECTION's EWRITE method sends a NWRITE message to itself to determine what to write. If there is an octet to write, it writes it. If NWRITE returned :CLOSE, the BASIC-CONNECTION calls ``(remove-connection self :close t)'' so that the event framework will remove the BASIC-CONNECTION from the list of connections & will then close the socket. If NWRITE returned NIL, the BASIC-CONNECTION's EWRITE method does nothing. Subclasses will probably not need to re-implement EWRITE.
Subclasses of BASIC-CONNECTION will want to implement their own WANT-READ-P & WANT-WRITE-P methods.
Subclasses might want to implement their own DID-CLOSE methods.
You can create a BASIC-CONNECTION with the MAKE-BASIC-CONNECTION function. You can initialize or re-initialize an existing BASIC-CONNECTION with the INIT-BASIC-CONNECTION function.
Built on top of BASIC-CONNECTION is class ZENDPOINT.7.1Zendpoint provides even more services to its subclasses than BASIC-CONNECTION does.
The main service ZENDPOINT provides subclasses is allowing instances to schedule multiple octets to be sent. ZENDPOINT's EWRITE & NWRITE methods will send all the queued data & send DONE-WRITING to the object when everything is sent. In its DONE-WRITING method, the object may generate & queue more data if it wants.
Subclasses of ZENDPOINT should not re-implement WANT-WRITE-P, NWRITE, EWRITE, or ENQUEUE-WRITE.
Subclasses will probably need to re-implement DONE-WRITING.
Subclasses are on their own with respect to their NREAD methods.
Class ZENDPOINT is in server.lisp. Search for ``defclass zendpoint''.
The next type of service that might be useful to discuss is the Help service.
RFC 1078: TCP Port Service Multiplexer (TCPMUX) defines a help service. It prints the names of all the services that TCPMUX knows. I thought it would be a good type of connection to implement after the TCPMUX connection types.
My TCPMUX HELP service is in help.lisp. Notice that it is a subclass of ZENDPOINT . HELP has to implement only one method, DONE-WRITING, though it is critical that CYBERTIGGYR-TCPMUX-HELP:INIT send DONE-WRITING to the HELP object. See that HELP's DONE-WRITING queues the next item in its list of things to write, until there are no items remaining in the list. Then it calls (REMOVE-CONNECTION SELF :CLOSE T) so the event framework will forget about the HELP instance & will close the socket when it's safe to do so.
According to RFC 1078: TCP Port Service Multiplexer (TCPMUX) , a TCPMUX might want to offer some of the usual network services. Many of the usual network services comprise many of the lines in the file /etc/services. For example, TCPMUX might offer SMTP as ``smtp''.
One way to add all those services would be to implement connection objects to implement each of those services, but that would require a lot of programming. It would be easier to reuse the code from the existing services. One way to do that is to write a ``proxy'' service.
A proxy service can connect a client to a real service. Once TCPMUX creates the proxy connection object to talk to the client, that proxy connection creates yet another proxy connection object to talk to the network service that the client requested. The two proxy connection objects share two queues. One queue is for writing octets in one direction, form client to real service. The other queue is for writing from the real service to the client. Each of the proxy connection objects reads octets from a socket & inserts them into one of the queues, & it removes octets from the other queue & writes them to the socket.
The proxy connection objects are instances of class PROXY-ENDPOINT in proxy.lisp, but the more interesting part of the proxy service is the PROXY-MAKER function in the same file.
The PROXY-MAKER function has one argument which is the TCP port number for the true service. It returns a function which creates a proxy connection object, but where the connection-creating functions for the other services create just one connection object, this new function creates two. It creates two PROXY-ENDPOINTs which share two queues. The function returns one of the end points, but it silently inserts the other into the event loop's list of connections. Notice that this does not use a priviledged interface to the event loop's data; it's a careful use of the interface that any other service may use.
The proxy connection objects are pretty simple. Each has a queue that it reads. If its queue is not empty, then it returns true from its WANT-WRITE-P method. If its peer's queue is not full, it returns true from its WANT-READ-P method.
The subtleties are the DID-CLOSE method & the DONE-WRITING method. In its DID-CLOSE method, a PROXY-ENDPOINT makes its peer endpoint forget about it, thereby making its queue permanently full, then it uses REMOVE-CONNECTION to make the event loop forget about it. In its DONE-WRITING method, if its peer proxy connection object has closed, a proxy connection object uses REMOVE-CONNECTION on itself.
Also see Compare Lisp to unix for some more thoughts about these proxy connections.
I wrote a service which sends a lot of ASCII characters to the client before it closes the connection. The full name I gave it is ``cybertiggyr.tcpmux.sometext''.
You could connect to the ``sometext'' service with something as simple as telnet, but I also wrote a client program in C for it. The client program prints the rate at which it received characters. The rate includes the time to connect & the time that the event loop delays before closing the socket. So it's sort of a worst-case rate.
The cybertiggyr.tcpmux.sometext service is in sometext.lisp. The client for it is clnt-sometext.sh, but that is just a tiny Bourne shell program around clnt-generic.c.
Yet another simple service I implemented is ``cybertiggyr.tcpmux.time''. It is not the same as the ``time'' service in /etc/services & which my TCPMUX server offers through its proxy endpoint class.
My time service prints the time in ISO format, like this: ``2004-FEB-11T23:45:29''. That's about fourteen & one half minutes before the midnight which begins 12 February 2004.
There is no special reason for this time service. I wrote it as a simple service I could implement for my TCPMUX server.
The cybertiggyr.tcpmux.time service is implemented in time.lisp. I had to call the class XTIME because TIME is in use by package COMMON-LISP.
I also wrote a custom client for the time service, though it's hardly necessary to use that client. The client is clnt-time.sh, but that is a brief shell program that wraps clnt-generic.c.
I mostly use the conventional, barely object-oriented languages at work: C++, Java, & Perl.7.2 Since becoming a programmer who chooses Lisp or unix shell with some C, I've acquired a disdain for object orientation. I think it's because Lisp gives you so many more options than objects & because most of what you need to do can be done by piping together a few other programs, some of which might need to be written as small C programs.
Nevertheless, something told me I needed to use an object oriented style for the protocol by which the services communicate with the TCPMUX event loop, so I did it with some simple CLOS.
In retrospect, I don't like it.
It tries to make the protocol easy to understand by making it explicit & declaring it within the language, but somehow that obscures the protocol. It would have been better to spend the same time documenting a protocol that exists in the human mind than one that exists in the programming language.
Another problem with having the protocol exist in the programming language is that it creates a dependency between the protocol & the services which implement it. Sure, the dependency is only on symbols, but it's there.
The thing I dislike the most is the class hierarchy. It separates the functionality of an object, obscures it by defining it in multiple places. When one class provides features for the convenience of its subclasses, it broadens or restricts the protocol that already exists, & that's more complexity, more ways in which the functionality of the ultimate class is obscured.
If I wrote it again, here's what I think I would do differently.
I would separate the socket from the service object. The event framework would associate them, but the service object would not know its socket. Instead of telling the service object when to read or write its socket, the event framework would tell it what was read or would ask it what to write. This would mean that the event loop would need to know whether a socket was a server socket or a data socket.
I would not use generics to define the protocol for the event loop to communicate with the services. Instead, the function which created a service would return an assoc list of the functions to call on different events. The key symbols in the assoc list would be from the keyword packages, so there would be no dependency whatsoever, even on symbols, between the event loop's package (cybertiggyr-tcpmux-server) & the services.
True, generics remove the need for functions to return assoc lists of other functions, but in this case, the protocol is simple enough that I don't need generics. Generic functions create symbol dependencies, which I neither want nor need.
Once the event loop obtains the assoc list of functions for a new service object, it extracts the functions & inserts them into some private type of object that allows it to look up the function quickly. Then the event loop can call the function with FUNCALL . I suspect that an indirect function call like that is faster than a method look-up. (On the other hand, one of my mantras is ``Run time isn't as valuable as you think it is'', so what do I care?)
Class hierarchies obscure functionality. Instead of creating a class hierarchy of services, I would use functional composition. (That's the functions that return assoc lists of functions which comprise the service objects.) I would create helper functions for use by the functions that create the service objects. To see the definition of a type of service object, you could look at the ``maker'' function for it. You would know what functions went into a service object because you could see what functions were used to create it. The functionality would be separated, just like the functionality in a class hierarchy, but it would be explicit.
Here's an example of how I might implement the sometext service if I did a rewrite.
The ``maker'' function for sometext might look like this:
;;; inside a SOMETEXT package
(defconstant *max* 1000007)
(defun make-sometext ()
(let ((count 0))
(list
(cons :notify-read
#'(lambda (octet)
"Ignore input."
(declare (ignore octet))))
(cons :notify-write
#'(lambda ()
(if (< (incf count) *max*)
(random-character)
:close)))
(cons :notify-closed
#'(lambda ()
"Ignore it. Assume the framework
automagically forgets this
service object."
nil))
(cons :want-read-p
#'(lambda ()
"Sometext never wants to read."
nil))
(cons :want-write-p
#'(lambda ()
"Sometext always wants to write."
t)))))
The sometext service is really simple. More complex services might need to buffer incoming data until some type of packet terminator, record terminator, line terminator, or command terminator is received; or a service might want to generate data for output in chunks, queue it, & forget about it until all of the chunk was sent. Helper functions could provide these service. The helper functions would not perform the services; they would create & return functions which did the services. I realize that this is a form of inheritance like classes offer, but it would be explicit, thereby more easily maintained.
I'd further simplify the protocol between the event loop & the service objects by making it purely one-way. The event loop would call functions on the service objects; it would never work the other way. Service objects could communicate with the event loop by their return values. Then I would not need the insert-connection & remove-connection functions.
I would retain the Flez event queue in the event loop.
I used some functions that are not part of the Common Lisp standard. Because my main Lisp is clisp, I used the socket functions from its library. To port TCPMUX to another Lisp, a programmer will need to implement these functions for the target Lisp. I presume they could be implemented on top of that target Lisp's own non-standard socket functions.
All of these functions are in a package called ``SOCKET''.
socket-accept socket-server &key
:element-type :external-format :buffered :timeout 
new-socket
Creates & returns a new, regular (a.k.a. client, non-server) socket. The properties are determined by the keyword arguments.
The :ELEMENT-TYPE keyword argument determines whether the new socket reads & writes characters or bytes. The value for :ELEMENT-TYPE must be CHARACTER, ``(UNSIGNED BYTE 8)'', or nil. If it's nil, it's the same as CHARACTER.
The :EXTERNAL-FORMAT keyword argument is complex. See the description in the clisp implementation notes.
The :BUFFERED argument, if nil, means the new stream will be unbuffered, & that is appropriate for this TCPMUX server. For all I know, buffered streams would be incorrect, though I haven't tested it yet.
The :TIMEOUT keyword argument provides the maximum number of seconds to wait for a connection. Since my TCPMUX server doesn't call socket-accept until a new connection is pending, this argument is unused.
Whatever. I mean, None.
None that I know.
Whatever.
None.
socket-connect port &optional host
&key :element-type :external-format :buffered :timeout
stream
Creates & returns a new, connected regular socket. The new socket will be connected to the indicated port on the indicated computer. The properties are determined by the keyword arguments.
The :ELEMENT-TYPE keyword argument determines whether the new socket reads & writes characters or bytes. The value for :ELEMENT-TYPE must be CHARACTER, ``(UNSIGNED BYTE 8)'', or nil. If it's nil, it's the same as CHARACTER.
The :EXTERNAL-FORMAT keyword argument is complex. See the description in the clisp implementation notes.
The :BUFFERED argument, if nil, means the new stream will be unbuffered, & that is appropriate for this TCPMUX server. For all I know, buffered streams would be incorrect, though I haven't tested it yet.
The :TIMEOUT keyword argument provides the maximum number of seconds to wait for a connection. Since my TCPMUX server doesn't call socket-accept until a new connection is pending, this argument is unused.
Whatever. I mean, None.
None that I know.
Whatever.
None.
socket-server &optional port-or-socket
stream
Creates & returns a server socket on the indicated port. When a connection is pending on the serve socket, you may accept it. To close the server socket, use server-close.
Whatever. I mean, None.
None that I know.
Whatever.
None.
socket-server-close socket-server
Closes the server socket. The server socket must have been created by socket-server.
Whatever. I mean, None.
None that I know.
Whatever.
None.
socket-status lst-sock &optional seconds microseconds
lst-op
This is an interface to the unix select system call.
Whatever. I mean, None.
None that I know.
Whatever.
None.
One of the reasons for writing my TCPMUX in Lisp was to see the performance of a single-threaded, multi-connection server that used Common Lisp's READ-BYTE & WRITE-BYTE for its input & output.
To measure the performance of this TCPMUX server, I used the com.cybertiggyr.tcpmux.sometext service because it spews about one million characters to the client. I used my clnt-sometext program for the client because it prints timing information.
The TCPMUX server ran on a computer called Palsy. It's about a 450 MHz Intel Pentium something-or-other with 128 (or is it 256?) megabytes of memory running Open BSD 2.7.9.1 The TCPMUX client ran on a computer called Plague. It's a Sony Vaio laptop of some sort with about an 800 MHz Intel Pentium whatever with 256 megabytes of memory. Palsy has a 100 megabite-per-second Ethernet card, & my network hub works at 100 megabites per second, but Plague's PCMCIA Ether card is old & runs at 10 megabits per second, so the maximum throughput is about 1 megabyte per second.
I wanted to see how performance degrades as the number of simultaneous connections increases, so I made many ``test runs'', each with a different number of simultaneous connections. The ``1'' run launched a single clnt-sometext client & waited for it to finish. The ``2'' run launched two clnt-sometext processes in parallel & waited for them to finish. I did runs for many numbers of simultaneous connections up to 64, which launched 64 clnt-sometext processes in parallel & waited for all of them to finish.
So that I could compare TCPMUX's performance to some real quantity, I also created a file containing the same type of data that the sometext service sends & copied that file from Palsy to Plague using nfs, ftp, & rcp. (For ``nfs'', I put the file on a drive on Palsy which is NFS-mounted onto Plague, then I copied the file to Plague's local drive with the cp command. For ftp, I transfered the file in binary mode.) For these three control groups9.2, I ran only one connection, each. I did not attempt multiple simultaneous connections.
The program which runs all of the performance tests is speed.sh.
So much for what I did. Let's look at the data.
Table 9.1 compares the performances of TCPMUX with various numbers of simultaneous connections to the performances of nfs, ftp, & rcp. Each row is for a unique combination of ``method'' & ``connections'', where method is nfs, ftp, rcp, or tcpmux, & connections is the number of simultaneous connections. The ``characters'' column is the number of characters that were transfered for a single connection, & some people might wish that I had called it the ``bytes'' column, but I wish I had called it the ``octets'' column. The ``time'' column is the number of seconds required for the average connection in a run. The ``abs. rate'' column is the data rate in characters per second for the average connection in a run. The ``rel. rate'' column is the data rate for the average connection in a run, relative to the data rate for the average connection in the fastest method/connections row (which was nfs/1).
For example, the row for tcpmux/64 shows a time of 2,650.380 seconds. The mean transfer time for the test run for 64 simultaneous connections was 2,650.380 seconds. Some of the connections in the run might have required less time, & some might have required more, but the sum of their times, divided by the number of connections (64), was 2,650.380 seconds. Both of the data rate columns were computed using this mean time, so the data rates reflect the mean for the 64-connection test run, too.
You can see that my TCPMUX server is not very speedy.
Remember that speed is relative: If you use telnet to
connect to my server & request the
``com.cybertiggyr.tcpmux.sometext'' service, you'll probably
see text scroll faster than you can read it. Nevertheless,
for one connection, its data rate is about
those of nfs, ftp, or rcp.
But wait. Remember that my TCPMUX server waits five seconds before closing a connection. I had to do that for telnet clients.9.3 I'm willing to accept that this is a bug in my implementation or in clisp's socket library or in a telnet which aggressively closes sockets early, so I'm going to do a controversial thing. I'm going to add five seconds to the times of nfs, ftp, & rcp. Now take a peek at Table 9.2. It is like the previous table except that it simulates the five-second delay by increasing the transfer times of nfs, ftp, & rcp. I used the same data for this new table; I just added five seconds to the first three rows, then re-computed the data rates.
In the new table, TCPMUX does considerably better, though nfs, ftp, & rcp are still one hundred times faster.
I'm convinced that the low performance occurs because Common Lisp's input & output functions are for single bytes. If I used a foreign function interface so that I could use unix's read & write functions or use buffering , it could be a hundred times faster, says my gut, which would make it almost as fast as nfs, ftp, & rcp if you allow for the five-second delay as I have in Table 9.2. Unfortunately, the only way to confirm or refute this belief is to write such a program, & I don't feel like doing that now. It would make a good follow-up article, though.
Another thing to notice from the tables is the rate at which
performance degrades as the number of connections
increases. The per-connection rate at 64 connections is
about
that if the per-connection rate
for one connection.9.4 That's better than I expected; I expected it to
be
.
The remaining tables show the individual connection times for the test runs. For each number of simultaneous connections I desired (1, 2, 4, 8, 16, 32, & 64), I made a test run, & each test run is in one table. Except for the final three rows in each table, there is one row in a table for each client process in the test run. Notice that the client processes in a test run executed simultaneously, so the duration of a test run was not the sum of the times of the client processes in it. The duration of a test run was probably slightly longer than the longest client process in the test run.
The rows labeled ``mean'' in those tables became the rows labeled ``tcpmux'' in Table 9.1 & Table 9.1.
|
|
|
|
The short answer is that TCPMUX isn't significant because it was never popular. The longer answer is that TCPMUX is interesting in many ways.
It's interesting to compare TCPMUX with web services.
First, TCPMUX is not a web server. If you compare TCPMUX to a first-general, plain, vanilla web server (which just handed out files over HTTP), you see a lot of differences. A web server with CGI doesn't have much in common with TCPMUX, either.
Now that web services are part of web servers, there is a resemblance to TCPMUX. Web services multiplex different services over HTTP, which commonly runs on port 80. Each service has a name encoded in an URL. The web server handles the HTTP connection & might provide a platform on which the web service runs, but it isn't concerned with the internals of the web server. This is the same relationship a TCPMUX server has to the services it supports.
If TCPMUX had become popular before the World Wide Web was created, it might have become more common for software developers to create their own services. They might not have advertised those services, but they could have named them easier than the reality, which was that they needed to reserve a port. TCPMUX servers might have developed into platforms for pluggable services, & that's pretty much what web servers are now.
The web would have developed anyway, but HTTP might have been the name of a service accessed through TCPMUX rather than on port 80.
When in reality we started adding services to web servers, in a world where TCPMUX had become popular first, we probably would have added those services through TCPMUX instead.
HTTP dos not provide interactive communication during a single session. In a single session, the client connects to the web server & sends its entire query. Then the server sends its entire reply.
This non-interactive session protocol has presented some hurdles for the web. Do you remember ``server push'' from the late 1990s?
TCPMUX does not imply this limitation. Once the TCPMUX server hands the socket to the true server, the entire session is under the control of the client & the true server. It can be full duplex, half duplex, test, binary, or whatever. What different types of services would have developed, or been easier to develop, in a world where TCPMUX had become popular before the web?
I'm not saying TCPMUX is better than web services. I just think it's interesting to compare what we have with what we would have had. TCPMUX could provide a more general equivalent to web services & without at least one of the limitations that web services have inherited from HTTP.
Why didn't TCPMUX become popular? I suspect it was about ten years ahead of its time.
TCPMUX is a reasonable programming project for those learning to write network servers. That is because TCPMUX describes all the features of a modern network server except a service itself. I mean that a TCPMUX server should support multiple simultaneous connections, but the work it does is really simple, almost non-existant, so you can concentrate on the networking parts of the program.
It's possible to implement a single-threaded network server that supports multiple simultaneous connections in Common Lisp plus a handful of non-standard functions for managing sockets.
Performance is tolerable but not great. I believe it would be improved if instead of the Common Lisp READ-BYTE & WRITE-BYTE functions, I used a foreign function interface facility to access the unix read & write functions.
I am not pleased with my application of CLOS to the problem (Section 7.9). If I were to write a new version , I would use functional composition instead.
Notice that the TCPMUX session connection object's purpose in life is a high-level Lisp equivalent of the dup/fork/exec idiom in unix.
Notice that in unix, an easier way to implement the proxy service would have been to juggle the sockets with dup, then do the fork & exec dance to hand it to a child process which would have handled the entire session between the client & the true service. It might have been possible to do this in Lisp, but not with the self-imposed requirement to minimize the number of non-standard functions I use. Besides, I wanted to see how the proxy technique performed in Lisp.
It is possible that clisp uses unix's read & write functions under the covers. I don't know for certain that it does or doesn't because I haven't checked the implementation notes, & I don't care to do it. It occurred to me that one way a Lisp could make use of buffered streams is to use the operating system's more efficient I/O functions.
I tested it by changing the socket-accept function call in tcpmux.lisp to create buffered sockets. It didn't work because clisp's SOCKET-STATUS function never indicates that a buffered socket is ready for writing. It is possible that the problem was in my code; I did not do a lot of research to see what was wrong when I tried buffered sockets.
This document is available in multi-file HTML format at http://lisp-p.org/tcpmux/.
This document is available in DVI format at http://lisp-p.org/tcpmux/tcpmux.dvi.
This document is available in PostScript format at http://lisp-p.org/tcpmux/tcpmux.ps.
I have no plans to make this document available in Pointless Document FormatB.1 (PDF), ever.
