Generating HTML with Lisp & Templates

created Tuesday, 27 January 2004
updated Wednesday, 15 July 2004

In Generating HTML with Lisp / a tutorial for new programmers, I described a way to produce HTML documents from data structures expressed in Lisp. In this article, I discuss another way of making HTML documents. It uses Lisp, but instead of expressing the content in Lisp data structures which convert to HTML, it reads a file of HTML that also contains Lisp code to execute to fill-in parts of the HTML file. These files are called ``templates''.

HTML templates are also used in Perl & probably some other languages. A PHP page is inherently a template, too.

2 To Do

This article is still under construction. The biggest problem is that the source code in the figures is out of date. You might check back here after 15 February 2004.

Make sure all links to Lisp CGI programs are relative to CyberTiggyr.COM, not to the base URL. This is necessary because the server for lisp-p.org does not have clisp.

Consider installing a better Lisp on lisp-p.org. Might want to update Lisp/CGI Programming Experiments. Might convert that article into a tutorial that shows how to use many Lisps for CGI, though it would not discuss techniques such as generating HTML from Lisp expressions ([Sto03a]) or HTML templates ([Sto04a]).

3 Basic Idea

We'll use a macro to read the template file & produce Lisp code for it. When you execute that Lisp code, it sends the HTML directly to the browser, & it evaluates the embedded Lisp code.

Instead of a macro, we could use a function that produced Lisp code, but to execute the Lisp code, we'd need to use the EVAL function. The EVAL function executes Lisp code at the top level, so local variables, such as those within LET , are not in effect. That's a bit of a bummer, so I chose a macro.

What to call the macro? I think CALL-TEMPLATE is most appropriate. Actually, the name CALL-TEMPLATE is not all that great, but it leaves LOAD-TEMPLATE for a function which produces Lisp code in memory from the template file.

4 Function LOAD-TEMPLATE

The top-level function is LOAD-TEMPLATE. It reads an HTML template from a file & then evaluates it. An optional argument lets you identify the stream to which the template's output is sent.

Assuming a SLURP-FILE function returns the entire contents of a file as a string, & a CALL-TEMPLATE function evaluates an HTML template (as a string), then LOAD-TEMPLATE could be implemented as shown in Figure 1.

**Figure 1:** The LOAD-TEMPLATE function
$\begin{figure}\begin{verbatim}(defun load-template (pn &optional (strm t)) ''... ...of the file.'' (call-template (slurp-file pn) strm))\end{verbatim} \end{figure}$

5 Function SLURP-FILE

The simplest implementation of this function is probably to read characters from the file & push them onto a vector which will later be the string containing the entire contents of the file. This implementation is in Figure 2, but it is probably a slow implemtation of the SLURP-FILE function ([Sto03b]). It's kind of a cheat because it delegates most of the work to a SLURP-STREAM-SIMPLE function, which is in Figure 3.

**Figure 2:** A simple but probably slow implementation of function SLURP-FILE
$\begin{figure}\begin{verbatim}(defun slurp-file-simple (pn) ''Return a string... ...ith-open-file (strm pn) (slurp-stream-simple strm)))\end{verbatim} \end{figure}$

**Figure 3:** A simple but probably slow implementation of function SLURP-STREAM-SIMPLE
$\begin{figure}\begin{verbatim}(defun slurp-stream-simple (strm) ''Return a st... ...) ((eq ch strm) str) (vector-push-extend ch str))))\end{verbatim} \end{figure}$

I wrote separate functions for SLURP-FILE-SIMPLE & SLURP-STREAM-SIMPLE because it seems like I might sometimes have a stream to read, not always a file.

I expect that it would be faster to read the file in big chunks of characters, then concatenate the chunks into a single string. Common Lisp does not offer a ``read an entire buffer'' function like unix's read function. The closest we can get is Common Lisp's READ-LINE function. Let's use that to create a list of lines. Once we have all the lines, we'll concatenate them into a string.

When creating a list that will be concatenated later, it's generally faster to treat the list as a stack while we build it, then reverse it before concatenating it.¹

A twist is that READ-LINE removes the end-of-line character. We want to preserve those characters. I also think it's easier if our replacement for READ-LINE returns one value: the string in most cases & the stream on end-of-file.

The new SLURP-FILE-FASTER function is in Figure 4. The SLURP-STREAM-FASTER function is in Figure 5. Some functions & a global object that they use are in Figure 6.

**Figure 4:** The SLURP-FILE-FASTER function
$\begin{figure}\begin{verbatim}(defun slurp-file-faster (pn) ''This turned out... ...ith-open-file (strm pn) (slurp-stream-faster strm)))\end{verbatim} \end{figure}$

**Figure 5:** The SLURP-STREAM-FASTER function
$\begin{figure}\begin{verbatim}(defun slurp-stream-faster (strm) ''Slurp the c... ...the ;; concatenation. (strcatlst (nreverse lst)))))\end{verbatim} \end{figure}$

**Figure 6:** Some objects used by SLURP-STREAM-FASTER
$\begin{figure}\begin{verbatim}(defconstant newline-string (format nil ''~%''... ...str) (setq str (replace str (first x) :start1 i)))))\end{verbatim} \end{figure}$

The SLURP-STREAM-FASTER function is more complex but hopefully faster than SLURP-STREAM-SIMPLE. I ran some performance comparisons & learned that SLURP-FILE-FASTER is pretentiously named. Where SLURP-FILE-SIMPLE required 3.42 seconds, SLURP-FILE-FASTER required 6.98 seconds. Some of the functions I used to run the performance comparison are in Figure 7.

**Figure 7:** Some of the functions I used to compare the two slurp-file functions
$\begin{figure}\begin{verbatim}(defun make-tester (fn) ...$

So much for performance tests. The final SLURP-FILE function is in Figure 8. It is effectively the same as the SLURP-FILE-SIMPLE function.

**Figure 8:** The final SLURP-FILE function
$\begin{figure}\begin{verbatim}(defun slurp-stream (strm) (let ((str (make-arr... ...PN.'' (with-open-file (strm pn) (slurp-file strm)))\end{verbatim} \end{figure}$

I guess another implementation could push characters onto a list, then reverse the list & COERCE the list to a string. The vector-appending SLURP-FILE function I have is good enough, though. So much for SLURP-FILE.

6 Function CONVERT-TEMPLATE

Fix this. The code in the figures here isn't up-to-date with the code in lht.lisp.

The CONVERT-TEMPLATE function has one argument, which is the HTML template as a string. It must return an expression in Lisp which produces the HTML for the template.

For example, if we have the HTML template from Figure 9, we want CONVERT-TEMPLATE to produce something like the Lisp expression in Figure 10.

**Figure 9:** An example HTML template
$\begin{figure}\begin{verbatim}<html> <head> <title>Titular Title</title> </hea... ...li> <lisp> ) </lisp> </ol><p>Whew!</p></body></html>\end{verbatim} \end{figure}$

The Lisp expressions are surrounded by lisp tags, as if they were HTML. They aren't HTML; I just thought that was a clever way to identify the Lisp parts of the template. Notice that you can mix Lisp with HTML. See how I've done that with the DOLIST ; part of it's body is Lisp, & part is HTML. Personally, I think that's ugly, but I wanted to show that it can be done.

**Figure 10:** One possible Lisp expression for the HTML template in Figure 9
$\begin{figure}\begin{verbatim}(progn (princ ''<html> <head> <title>Titular Ti... ...') ) (princ '' </ol><p>Whew!</p></body></html> ''))\end{verbatim} \end{figure}$

To create the Lisp expression in Figure 10, I placed ``(progn (princ "'' at the beginning of the template & ``"))'' after the template. Then I replaced every ``<lisp>'' with ``")'' & every ``</lisp>'' with ``(princ "''. I didn't reformat the code; that's in the hopes of making my changes more aparent. The resuling code is formatted terribly if you want to see what it does, so Figure 11 has the same code formatted more readably. Because it contains so many string constants with embedded newlines, it's still fairly difficult to read.

**Figure 11:** Indented Lisp code that was produced for the example HTML template
$\begin{figure}\begin{verbatim}(progn (princ ''<html> <head> <title>Titular Ti... ...') ) (princ '' </ol><p>Whew!</p></body></html> ''))\end{verbatim} \end{figure}$

The description of the changes I made by hand to convert the example HTML template to an example Lisp expression tell what the CONVERT-TEMPLATE function must do. Assuming we have a function STRING-REPLACE-ALL which replaces all occurences of an old substring with a new substring in a big string, then CONVERT-TEMPLATE may be implemented as in Figure 12.

**Figure 12:** The CONVERT-TEMPLATE function
$\begin{figure}\begin{verbatim}(defun convert-template (template) ''Evaluates ... ...rinc \uml '' template)) ''\uml strm) strm)''))))))\end{verbatim} \end{figure}$

It's ugly as sin because of those string constants which themselves contain fragments of Lisp expressions, but it's a really simple function. Now we need a STRING-REPLACE-ALL function.

Lisp doesn't have many string functions. I tried to find a string library which did what I need here. I found a couple but had troubles installing them. So let's write STRING-REPLACE-ALL here & now. It could be useful in other libraries.

If I'm not worried about efficiency, the STRING-REPLACE-ALL function could search for the old string & replace it with the new string. It could continue the process until it cannot find any occurrences of the old string. Such an implementation is in Figure 13.

**Figure 13:** The STRING-REPLACE-ALL function
$\begin{figure}\begin{verbatim}(defun string-replace-all (old new big) ''Repla... ...eq big 0 i) new (subseq big (+ i (length old)))))))\end{verbatim} \end{figure}$

If it turned out that I needed a faster version, I guess we could give SEARCH the index after the last occurence we found because it wouldn't need to search earlier occurrences, but I don't think it'll be necessary.

First, rename the function to STRING-REPLACE-ALL-SIMPLE. Though that change is trivial, Figure 14 shows the new function.

**Figure 14:** The STRING-REPLACE-ALL-SIMPLE function
$\begin{figure}\begin{verbatim}(defun string-replace-all-simple (old new big) ... ...eq big 0 i) new (subseq big (+ i (length old)))))))\end{verbatim} \end{figure}$

My idea for an optimization is to alter the second SEARCH callso that it won't examine the part of the big string that it has already examined. The index where it should start searching is the location it last found, plus the length of the new string. While we're at it, let's take the length of the new string once at the beginning of the loop & store it into a temporary variable.

The new, hypothetically optimized function is STRING-REPLACE-ALL-FASTER, & it's in Figure 15.

**Figure 15:** The STRING-REPLACE-ALL-FASTER function
$\begin{figure}\begin{verbatim}(defun string-replace-all-faster (old new big) ... ...eq big 0 i) new (subseq big (+ i (length old)))))))\end{verbatim} \end{figure}$

Those functions & others are in the file lht.lisp. The expressions I used for the performance comparison & their results are in Figure 16.

**Figure 16:** Comparison of the two implementations of STRING-REPLACE-ALL
$\begin{figure}\begin{verbatim}> (setq big (make-string 10000 :initial-element ... ...MPLE 1.82E+1 2 STRING-REPLACE-ALL-FASTER 1.69E+1 NIL\end{verbatim} \end{figure}$

The ``simple'' function required 18.2 seconds to complete the test, while the ``faster'' function required 16.9 seconds. So the faster function is about 7 percent faster. So I'll use it for STRING-REPLACE-ALL.

7 Bringing the Library Together

All the source code for this tiny library is in the file lht.lisp. There are test programs in the file test.lisp.

8 Example CGI Programs

8.1 No-Lisp Example

For a first example, let's use an HTML template that does not embed any Lisp at all. The source code for the program is in ex1.lisp. The template file it uses is ex1.tmpl. You can see the program run at ex1.cgi.

8.2 A Little Lisp Example

This second example generates some of its output with Lisp code. It doesn't use any arguments from get or post. The source code for the program is in ex2.lisp. The template file it uses is ex2.tmpl. You can see the program run at ex2.cgi.

Generating HTML with Lisp & Templates

Contents

1 Introduction