As I discussed in the previous chapter, Common Lisp provides an abstraction, the pathname, that's supposed to insulate you from the details of how different operating systems and file systems name files. Pathnames provide a useful API for manipulating names as names, but when it comes to the functions that actually interact with the file system, things get a bit hairy.
The root of the problem, as I mentioned, is that the pathname abstraction was designed to represent filenames on a much wider variety of file systems than are commonly used now. Unfortunately, by making pathnames abstract enough to account for a wide variety of file systems, Common Lisp's designers left implementers with a fair number of choices to make about how exactly to map the pathname abstraction onto any particular file system. Consequently, different implementers, each implementing the pathname abstraction for the same file system, just by making different choices at a few key junctions, could end up with conforming implementations that nonetheless provide different behavior for several of the main pathname-related functions.
However, one way or another, all implementations provide the same basic functionality, so it's not too hard to write a library that provides a consistent interface for the most common operations across different implementations. That's your task for this chapter. In addition to giving you several useful functions that you'll use in future chapters, writing this library will give you a chance to learn how to write code that deals with differences between implementations.
The basic operations the library will support will be getting a list of files in a directory and determining whether a file or directory with a given name exists. You'll also write a function for recursively walking a directory hierarchy, calling a given function for each pathname in the tree.
In theory, these directory listing and file existence operations are
already provided by the standard functions
PROBE-FILE. However, as you'll see, there are enough different
ways to implement these functions--all within the bounds of valid
interpretations of the language standard--that you'll want to write
new functions that provide a consistent behavior across
Before you can implement this API in a library that will run correctly on multiple Common Lisp implementations, I need to show you the mechanism for writing implementation-specific code.
While most of the code you write can be "portable" in the sense that it will run the same on any conforming Common Lisp implementation, you may occasionally need to rely on implementation-specific functionality or to write slightly different bits of code for different implementations. To allow you to do so without totally destroying the portability of your code, Common Lisp provides a mechanism, called read-time conditionalization, that allows you to conditionally include code based on various features such as what implementation it's being run in.
The mechanism consists of a variable
*FEATURES* and two extra
bits of syntax understood by the Lisp reader.
*FEATURES* is a
list of symbols; each symbol represents a "feature" that's present in
the implementation or on the underlying platform. These symbols are
then used in feature expressions that evaluate to true or false
depending on whether the symbols in the expression are present in
*FEATURES*. The simplest feature expression is a single symbol;
the expression is true if the symbol is in
*FEATURES* and false
if it isn't. Other feature expressions are boolean expressions built
OR operators. For instance, if
you wanted to conditionalize some code to be included only if the
bar were present, you could write the
(and foo bar).
The reader uses feature expressions in conjunction with two bits of
#-. When the reader sees either of these
bits of syntax, it first reads a feature expression and then
evaluates it as I just described. When a feature expression following
#+ is true, the reader reads the next expression normally.
Otherwise it skips the next expression, treating it as whitespace.
#- works the same way except it reads the form if the feature
expression is false and skips it if it's true.
The initial value of
*FEATURES* is implementation dependent, and
what functionality is implied by the presence of any given symbol is
likewise defined by the implementation. However, all implementations
include at least one symbol that indicates what implementation it is.
For instance, Allegro Common Lisp includes the symbol
:allegro, CLISP includes
:clisp, SBCL includes
:sbcl, and CMUCL includes
:cmu. To avoid dependencies
on packages that may or may not exist in different implementations,
the symbols in
*FEATURES* are usually keywords, and the reader
*PACKAGE* to the
KEYWORD package while reading
feature expressions. Thus, a name with no package qualification will
be read as a keyword symbol. So, you could write a function that
behaves slightly differently in each of the implementations just
mentioned like this:
(defun foo () #+allegro (do-one-thing) #+sbcl (do-another-thing) #+clisp (something-else) #+cmu (yet-another-version) #-(or allegro sbcl clisp cmu) (error "Not implemented"))
In Allegro that code will be read as if it had been written like this:
(defun foo () (do-one-thing))
while in SBCL the reader will read this:
(defun foo () (do-another-thing))
while in an implementation other than one of the ones specifically conditionalized, it will read this:
(defun foo () (error "Not implemented"))
Because the conditionalization happens in the reader, the compiler doesn't even see expressions that are skipped.1 This means you pay no runtime cost for having different versions for different implementations. Also, when the reader skips conditionalized expressions, it doesn't bother interning symbols, so the skipped expressions can safely contain symbols from packages that may not exist in other implementations.
You can implement the function for listing a single directory,
list-directory, as a thin wrapper around the standard function
DIRECTORY takes a special kind of pathname,
called a wild pathname, that has one or more components
containing the special value
:wild and returns a list of
pathnames representing files in the file system that match the wild
pathname.2 The matching algorithm--like most
things having to do with the interaction between Lisp and a
particular file system--isn't defined by the language standard, but
most implementations on Unix and Windows follow the same basic
DIRECTORY function has two problems that you need to address
list-directory. The main one is that certain aspects of
its behavior differ fairly significantly between different Common
Lisp implementations, even on the same operating system. The other is
DIRECTORY provides a powerful interface for listing
files, to use it properly requires understanding some rather subtle
points about the pathname abstraction. Between these subtleties and
the idiosyncrasies of different implementations, actually writing
portable code that uses
DIRECTORY to do something as simple as
listing all the files and subdirectories in a single directory can be
a frustrating experience. You can deal with those subtleties and
idiosyncrasies once and for all, by writing
and forget them thereafter.
One subtlety I discussed in Chapter 14 is the two ways to represent the name of a directory as a pathname: directory form and file form.
DIRECTORY to return a list of files in
/home/peter/, you need to pass it a wild pathname whose
directory component is the directory you want to list and whose name
and type components are
:wild. Thus, to get a listing of the
/home/peter/, it might seem you could write this:
(directory (make-pathname :name :wild :type :wild :defaults home-dir))
home-dir is a pathname representing
This would work if
home-dir were in directory form. But if it
were in file form--for example, if it had been created by parsing the
"/home/peter"--then that same expression would list
all the files in
/home since the name component
would be replaced with
To avoid having to worry about explicitly converting between
representations, you can define
list-directory to accept a
nonwild pathname in either form, which it will then convert to the
appropriate wild pathname.
To help with this, you should define a few helper functions. One,
component-present-p, will test whether a given component of a
pathname is "present," meaning neither
NIL nor the special value
tests whether a pathname is already in directory form, and the third,
pathname-as-directory, converts any pathname to a directory
(defun component-present-p (value) (and value (not (eql value :unspecific)))) (defun directory-pathname-p (p) (and (not (component-present-p (pathname-name p))) (not (component-present-p (pathname-type p))) p)) (defun pathname-as-directory (name) (let ((pathname (pathname name))) (when (wild-pathname-p pathname) (error "Can't reliably convert wild pathnames.")) (if (not (directory-pathname-p name)) (make-pathname :directory (append (or (pathname-directory pathname) (list :relative)) (list (file-namestring pathname))) :name nil :type nil :defaults pathname) pathname)))
Now it seems you could generate a wild pathname to pass to
DIRECTORY by calling
MAKE-PATHNAME with a directory form
name returned by
pathname-as-directory. Unfortunately, it's
not quite that simple, thanks to a quirk in CLISP's implementation of
DIRECTORY. In CLISP,
DIRECTORY won't return files with no
extension unless the type component of the wildcard is
:wild. So you can define a function,
directory-wildcard, that takes a pathname in either directory
or file form and returns a proper wildcard for the given
implementation using read-time conditionalization to make a pathname
:wild type component in all implementations except for
NIL in CLISP.
(defun directory-wildcard (dirname) (make-pathname :name :wild :type #-clisp :wild #+clisp nil :defaults (pathname-as-directory dirname)))
Note how each read-time conditional operates at the level of a single
#-clisp, the expression
either read or skipped; likewise, after
is read or skipped.
Now you can take a first crack at the
(defun list-directory (dirname) (when (wild-pathname-p dirname) (error "Can only list concrete directory names.")) (directory (directory-wildcard dirname)))
As it stands, this function would work in SBCL, CMUCL, and LispWorks.
Unfortunately, a couple more implementation differences remain to be
smoothed over. One is that not all implementations will return
subdirectories of the given directory. Allegro, SBCL, CMUCL, and
LispWorks do. OpenMCL doesn't by default but will if you pass
DIRECTORY a true value via the implementation-specific keyword
subdirectories only when it's passed a wildcard pathname with
:wild as the last element of the directory component and
NIL name and type components. In this case, it returns only
subdirectories, so you'll need to call
DIRECTORY twice with
different wildcards and combine the results.
Once you get all the implementations returning directories, you'll
discover they can also differ in whether they return the names of
directories in directory or file form. You want
to always return directory names in directory form so you can
differentiate subdirectories from regular files based on just the
name. Except for Allegro, all the implementations this library will
support do that. Allegro, on the other hand, requires you to pass
DIRECTORY the implementation-specific keyword argument
NIL to get it to return
directories in file form.
Once you know how to make each implementation do what you want,
list-directory is simply a matter of
combining the different versions using read-time conditionals.
(defun list-directory (dirname) (when (wild-pathname-p dirname) (error "Can only list concrete directory names.")) (let ((wildcard (directory-wildcard dirname))) #+(or sbcl cmu lispworks) (directory wildcard) #+openmcl (directory wildcard :directories t) #+allegro (directory wildcard :directories-are-files nil) #+clisp (nconc (directory wildcard) (directory (clisp-subdirectories-wildcard wildcard))) #-(or sbcl cmu lispworks openmcl allegro clisp) (error "list-directory not implemented")))
clisp-subdirectories-wildcard isn't actually
specific to CLISP, but since it isn't needed by any other
implementation, you can guard its definition with a read-time
conditional. In this case, since the expression following the
#+ is the whole
DEFUN, the whole function definition will
be included or not, depending on whether
clisp is present in
#+clisp (defun clisp-subdirectories-wildcard (wildcard) (make-pathname :directory (append (pathname-directory wildcard) (list :wild)) :name nil :type nil :defaults wildcard))
PROBE-FILE, you can define a function called
file-exists-p. It should accept a pathname and return an
equivalent pathname if the file exists and
NIL if it doesn't. It
should be able to accept the name of a directory in either directory
or file form but should always return a directory form pathname if
the file exists and is a directory. This will allow you to use
file-exists-p, along with
directory-pathname-p, to test
whether an arbitrary name is the name of a file or directory.
file-exists-p is quite similar to the standard
PROBE-FILE; indeed, in several implementations--SBCL,
LispWorks, and OpenMCL--
PROBE-FILE already gives you the
behavior you want for
file-exists-p. But not all
PROBE-FILE behave quite the same.
Allegro and CMUCL's
PROBE-FILE functions are close to what you
need--they will accept the name of a directory in either form but,
instead of returning a directory form name, simply return the name in
the same form as the argument it was passed. Luckily, if passed the
name of a nondirectory in directory form, they return
with those implementations you can get the behavior you want by first
passing the name to
PROBE-FILE in directory form--if the file
exists and is a directory, it will return the directory form name. If
that call returns
NIL, then you try again with a file form name.
CLISP, on the other hand, once again has its own way of doing things.
PROBE-FILE immediately signals an error if passed a name in
directory form, regardless of whether a file or directory exists with
that name. It also signals an error if passed a name in file form
that's actually the name of a directory. For testing whether a
directory exists, CLISP provides its own function:
probe-directory (in the
ext package). This is almost
the mirror image of
PROBE-FILE: it signals an error if passed a
name in file form or if passed a name in directory form that happens
to name a file. The only difference is it returns
T rather than
a pathname when the named directory exists.
But even in CLISP you can implement the desired semantics by wrapping
the calls to
(defun file-exists-p (pathname) #+(or sbcl lispworks openmcl) (probe-file pathname) #+(or allegro cmu) (or (probe-file (pathname-as-directory pathname)) (probe-file pathname)) #+clisp (or (ignore-errors (probe-file (pathname-as-file pathname))) (ignore-errors (let ((directory-form (pathname-as-directory pathname))) (when (ext:probe-directory directory-form) directory-form)))) #-(or sbcl cmu lispworks openmcl allegro clisp) (error "file-exists-p not implemented"))
pathname-as-file that you need for the CLISP
file-exists-p is the inverse of the
pathname-as-directory, returning a pathname
that's the file form equivalent of its argument. This function,
despite being needed here only by CLISP, is generally useful, so
define it for all implementations and make it part of the library.
(defun pathname-as-file (name) (let ((pathname (pathname name))) (when (wild-pathname-p pathname) (error "Can't reliably convert wild pathnames.")) (if (directory-pathname-p name) (let* ((directory (pathname-directory pathname)) (name-and-type (pathname (first (last directory))))) (make-pathname :directory (butlast directory) :name (pathname-name name-and-type) :type (pathname-type name-and-type) :defaults pathname)) pathname)))
Finally, to round out this library, you can implement a function
walk-directory. Unlike the functions defined
previously, this function doesn't need to do much of anything to
smooth over implementation differences; it just needs to use the
functions you've already defined. However, it's quite handy, and
you'll use it several times in subsequent chapters. It will take the
name of a directory and a function and call the function on the
pathnames of all the files under the directory, recursively. It will
also take two keyword arguments:
:directories is true, it will call the
function on the pathnames of directories as well as regular files.
:test argument, if provided, specifies another function
that's invoked on each pathname before the main function is; the main
function will be called only if the test function returns true.
(defun walk-directory (dirname fn &key directories (test (constantly t))) (labels ((walk (name) (cond ((directory-pathname-p name) (when (and directories (funcall test name)) (funcall fn name)) (dolist (x (list-directory name)) (walk x))) ((funcall test name) (funcall fn name))))) (walk (pathname-as-directory dirname))))
Now you have a useful library of functions for dealing with
pathnames. As I mentioned, these functions will come in handy in
later chapters, particularly Chapters 23 and 27, where you'll use
walk-directory to crawl through directory trees containing
spam messages and MP3 files. But before we get to that, though, I
need to talk about object orientation, the topic of the next two
annoying consequence of the way read-time conditionalization works is
that there's no easy way to write a fall-through case. For example,
if you add support for another implementation to
foo by adding
another expression guarded with
#+, you need to remember to
also add the same feature to the
or feature expression after
#- or the
ERROR form will be evaluated after your new
2Another special value,
appear as part of the directory component of a wild pathname, but you
won't need it in this chapter.
3Implementations are allowed to return
:unspecific instead of
NIL as the value of pathname
components in certain situations such as when the component isn't
used by that implementation.
4This is slightly broken in the sense that if
PROBE-FILE signals an error for some other reason, this code
will interpret it incorrectly. Unfortunately, the CLISP documentation
doesn't specify what errors might be signaled by
probe-directory, and experimentation seems to show that they
simple-file-errors in most erroneous situations.