2.1 WHAT MAKES A GOOD WEB SERVER

2.2 SECURING THE SITE

Whether you have just built a brand new machine or you are recycling an existing

server, it’s time for a security review. It may seem early in the process to be worrying

about this, but in my experience, starting with a secure system is better than trying to

lock down a finished installation. The former builds good habits early, while the latter

is prone to loose threads. The particulars are OS-specific, but for most Unix-like systems the procedure is

roughly the same:

1 Go to the password file (/etc/passwd or /etc/shadow) and disable shell access for all accounts that aren’t necessary for a functioning server. You can take that to mean everything but root, although some people prefer to have one non-privileged account with normal access.

2 If you are running inetd, open /etc/inetd.conf (or whatever the configuration file is on your system) and start commenting out services. Which services do you need? Possibly none, in which case you can just shut down inetd all together: chances are however, that you’ll use ftp for maintaining your site, and you’ll need telnet to log in and do the rest of the configuration. Consider replacing both of these with ssh; it provides scp to perform secure, password-encrypted file transfers as well as a secure shell that doesn’t expose passwords to plain text network traffic. In chapter 11 we’ll discuss rsync and other configuration management tools that will ease your site-management tasks.

3 Moving on to tougher tasks, find out what scripts and configuration files your system uses to start other services. Some have a master script (rc.boot), some have a program that executes a series of scripts (often located in /etc/rc.d). On my system, /etc/rc.d has the boot-time scripts: rc.sysinit runs first, then rc executes a series of scripts from one of the subdirectories, then rc.local executes. Examine the scripts to find out what services they start, and which of those services respond to commands from the outside world.

4 Disable services that you don’t need for a web server. Some things you should not need are: nfs or other networked file systems; network printer services; SMB or other Windows connectivity services; Yellow Pages (yp) or NIS services; and any remote management utilities you can live without on your system. If you aren’t expecting to receive email on this site, you can shut down sendmail, imap, pop, and other such tools. You will probably find inetd’s startup somewhere along the line, and you can shut it down also if you aren’t using its services.

5 Create a nonprivileged account that the web server will use to own its files, with a name such as www, web, or apache. If anyone other than the system administrator will be putting files on the server, let him use that account; otherwise disable shell access.

6 Change the root password, preferably using mkpasswd or a similar tool to generate a random string of 10–15 letters and numbers. Do the same for any remaining accounts with shell access. I keep such passwords in a small notebook that I can lock in a drawer or cabinet.

Now you are ready to reboot your system and verify that all is well and your system is secure. While booting you may notice other services that should be disabled, so go back to the research step and find out how to remove them. You might also investigate tools such as nessus (http://www.nessus.org) that will help check your security.

Some systems don’t need to be locked down this tightly. In particular, an intranet server can be considered secure if it has its own root password and there are no shell accounts that have privileges to change configurations. Since the server is inside a protected network, you can take advantage of nfs and other LAN-level services to make your workgroup’s life easier.

If your server has an open connection to the Internet, all these steps are required and should be taken before you proceed with any other installations. From this point on you’ll be working with the machine either by logging in directly or via a remote session (preferably protected by ssh). There should be no other way to get to the server’s files.

Systems exposed to the Internet get attacked. It is a sad fact of life that crackers are always finding new ways to find and exploit vulnerable systems. But protecting your system isn’t as difficult as the media sometimes portrays: remove vulnerabilities by disabling services that your system doesn’t need, and tightly secure the rest with passwords that can’t be guessed (or generated from a dictionary) and use configurations that make sense.

From here on, I’ll assume that the system is secure. As configuration issues or new potential vulnerabilities come up, I’ll highlight the steps needed to keep things that way.

2.3 THE CASE FOR APACHE

Now it is time to apply the Open Source value considerations and buyer’s guide principles to a real-world choice: what web server should you use for your site?

In the rest of the book I’ll present alternatives where possible, but in this case there is only one strong choice: the Apache web server. There are other Open Source choices (including thttpd, which we’ll discuss in section 2.9), but most are either experimental or specialized; Apache is hardened by years of use at millions of web sites. Of the commercial choices available, many are actually made from Apache itself; the code is offered under a BSD-like license that doesn’t constrain commercial use. In fact, the developers encourage other companies to use it as a reference implementation for the HTTP protocol.

30 Million

18 Million

2.4 APACHE CONFIGURATION

Like most Open Source programs, Apache gets its knowledge of the system from text files.¹ Get out your favorite text editor and examine the contents of the conf directory in Apache’s home; it contains a handful of files which are all readable by humans. Lines beginning with ‘#’ (or any text after a ‘#’ that’s not in a quoted string) are comments; the rest is information for the server.

The main configuration file for Apache is conf/httpd.conf. The other files in the directory are auxiliaries (such as magic and mime.types) or vestigial configuration files that are left over from the way NCSA did things (access.conf and srm.conf). The .default files provide the original configuration info shipped with Apache, and can be used as a reference if you want to go back to the way things started.

Note that in discussing configuration here, we mean run-time configuration. Apache also has compile-time configuration, which uses a separate configuration file (src/Configuration in the source distribution) as well as other sources that say how Apache should be put together. One of Apache’s strengths is the flexibility each site’s management has in deciding what to include in its server. We’ll discuss these compile-time options in each section that requires them.

2.5 PRODUCTION SERVER

When actively building web sites, a development group often needs a set of servers in different roles: a “toy” server where the developers can experiment with things, a production server where the finished product runs, and perhaps something in between.

If you are working alone and your server isn’t exposed to the great wide world, the minimum configuration is fine for getting started and learning what to do next. If, however, you are setting up a group of servers, you should secure your production server immediately. While it’s unlikely that some cracker is lurking in the shadows waiting to pounce on your hapless web site, securing the server from the start will establish good habits all around by making developers learn how things need to work in the final installation.

Apache’s default configuration has reasonable security practices in place, but not as good as you might like. Read through the Security Tips page in the Apache online documentation, set the protections on your files and directories as shown there, then consider how permissive you want to be with your users.

After deciding on your site’s policies, you’ll want to look for the following directives in httpd.conf and change them accordingly:

AllowOverride—Permits or disables the use of local configuration files (usually called .htaccess) in directories, so examine carefully each occurrence in httpd.conf. In general, a production server should disable this at the top directory, and then enable it again if needed in specific directories or locations:

<Directory />

AllowOverride None
</Directory>

This directive turns off the use of local configuration files in all directories. If you need to enable it again for a given directory—say, the document directory of a trusted user—you can use AllowOverride again to enable some features:

<Directory /home/sysmgr/public_html>

AllowOverride FileInfo AuthConfig Limit </Directory>

This allows sysmgr to set some directives in a .htaccess file in /home/sysmgr/ public_html. Not all directives will be honored; look up AllowOverride’s documentation to see exactly what we’ve permitted.

Note that allowing local configuration files slows down the server, as it has to parse the local file for each request to a URL that permits them. This is not how you want to run your primary applications.

Options—Permits or disables a grab bag of server features, including the execution of CGI scripts and browsing of directories. For a secure server, the top directory should turn off everything:

<Directory />

Options None </Directory>

Options None is as tight as it gets, but you might consider allowing FollowSym-Links or SymLinksIfOwnerMatch. These two permit the use of symbolic links on Linux and Unix, which is convenient as long as users don’t have general write permissions to important directories. SymLinksIfOwnerMatch allows links only if the link and the file it points to are owned by the same user. Using just FollowSym-Links is the best option performance-wise, since it doesn’t require Apache to do extra look-ups along file paths. See chapter 12 for more information.

Again, the Options directive can be used in specific <Directory> or <Location> sections to open up permissions as needed.

What if we had this section and the previous one setting AllowOverride? They would both apply; Apache neatly merges sections that apply to the same target in the order they occur. The same applies to <Location>, <Files>, and so on. Sane administrators will want to do that merging manually though, so that it is more obvious what directives apply to which sections.

UserDir—This directive controls the mapping of username URLs; that is, http:// www.example.site/~user. The default is to enable user directories for all users and map them to public_html, meaning that http://www.example.site/~bob/resume.html gets mapped to /home/bob/public_html/resume.html (assuming /home/ bob is Bob’s home directory). The argument can either be disabled or enabled (with a list of users following either) or a path can be mapped onto the user’s home directory.

If your site doesn’t have general user accounts, you can turn this feature off: setting UserDir disabled will turn off the mapping functions. Better yet, if you build Apache from source files you can leave out mod_userdir entirely. See the helpful files on compile-time configuration in the source distribution. If you want to let a few users do this but disable it for the rest, then use:

UserDir public_html UserDir disabled UserDir enabled bob carol ted alice

The more permissive variation is to disable user directories for sensitive accounts and leave it open otherwise:

UserDir disabled root www nobody ftp

If you allow user directories, then you should also have a <Directory> section that specifies AllowOverride and Options to control what users can do. The default httpd.conf contains such a section (possibly commented out), so modify it accordingly:

<Directory /home/*/public_html> AllowOverride None

Options Indexes SymLinksIfOwnerMatch </Directory>

This section disables the use of local configuration files and allows browsing of directories (that’s what Indexes does) and symbolic links that are owned properly.

Mapping ~user URLs to the user’s public_html directory is a typical scheme, but UserDir can also be set to map those requests onto an entirely separate directory tree with a subdirectory for each user. For example, this arrangement directs the requests to appropriate directories under /www/users:

UserDir /www/users

Whether using subdirectories of the user’s home or a separate directory tree, each user chooses what files to expose on the web site by moving those files to the appropriate directory. Directory ownership and write permissions should be set accordingly. UserDir should never map requests directly to a user’s home directory since that could make all subdirectories visible, thus removing the active choice. Another explanation given for this arrangement is that by not exposing the user’s home directory, you also don’t expose the various hidden files (those beginning with ‘.’ on Linux and Unix) that contain sensitive information such as passwords for mail servers, which are stored by some mail clients. It’s not a good idea to use such things on a server that has an open Internet connection.

A third possibility is to direct ~user URLs to another server entirely:

UserDir http://another.example.site/home_pages

This version causes UserDir to redirect requests for http://www.example.site/~user to http://another.example.site/home_pages/user.

ScriptAlias—Specifies a directory that contains executable scripts. This is one way to get the Apache server to execute a script for a URL, and is very secure as long as the administrator controls the scripts. A typical setting is:

ScriptAlias /cgi-bin/ /usr/local/cgi/

The first argument matches the beginning of a URL, and the second specifies the directory that contains the script for the match. For example, http://www.example.site/cgi-bin/hello_world.cgi would map to /usr/local/cgi/hello_world.cgi. Note that directories containing executable scripts should not be viewable by browsers. Set DocumentRoot appropriately and don’t mingle the two.

That’s one way to handle executable scripts. The others are via the ExecCGI option in the Options directive and by setting special handlers for given files. We’ll cover special handlers in later chapters. Think twice about using ExecCGI, especially in user directories. In conjunction with AddHandler, this option can lead to Apache running arbitrary code. Even if your users are trustworthy, you have to trust that they are taking proper precautions with their directories, passwords, and so forth.

If you have opened up directory permissions, then also use a <Directory> section to lock down each script directory:

<Directory "/usr/local/cgi">

AllowOverride None

Options None

</Directory>

The ScriptAlias tells Apache that the directory contains executable scripts in spite of Options None, so everything is set and ready for testing.

After making all these changes, restart Apache and test the server again. You can even skip to the next chapter and grab a few CGI sample scripts to make sure that works. Your production server is now secure and running!

2.6 DEVELOPMENT SERVER

While it is possible to build a site using a single server, it’s often handy to have a separate place for working on new programs and content (a “toy” server where security is loosened and developers can try out their code without a lot of bother). As compared to the production environment, a development server can allow open access to configuration files, documents, and scripts so that programmers can drop in new works quickly. Of course, this assumes that the server is not on an open Internet connec-tion—it should be in a protected network or otherwise configured to refuse requests from outside its LAN.

Here are a few possible configuration scenarios:

2.7 USING APACHECTL

Having chosen a configuration and set things up properly, test your server again. This is a good idea after any change to a configuration file, even if the changes look trivial. If your configuration allows CGI then take an example from the next chapter and try that out too.

The apachectl program includes helpful code for verifying configuration files. This command checks that things are all proper:

/usr/local/apache/bin/apachectl configtest

There are three options for restarting the server, in order of severity: graceful, immediate, and hard.

A graceful restart allows Apache to complete work in progress; servers with open connections won’t restart until they finish their transfers or time out waiting for clients. A server won’t close an open log file until the server shuts down. Your users will appreciate having a chance to finish their business, but you might find it troubling if you are looking for an immediate change.

Trigger this restart using apachectl or kill:

/usr/local/apache/bin/apachectl graceful

or

kill -USR1 httpd

If you use apachectl, it will also run a configtest for you automatically, so you can be assured the server will start up again properly.

An immediate restart tells Apache to close log files and open connections and then read the configuration files again:

/usr/local/apache/bin/apachectl restart

or

kill -HUP httpd

The apachectl program will confirm the configuration files with configtest before stopping the server. If the server isn’t running in the first place, it will just start it for you without an error.

A hard restart is necessary if you have changed the Apache program itself or made system changes that the server won’t normally track:

/usr/local/apache/bin/apachectl stop /usr/local/apache/bin/apachectl start

or

kill -9 httpd /usr/local/apache/bin/apachectl start

Shutting the server down with apachectl is preferable, but not always possible if things are going wrong.

2.8 SERVING DOCUMENTS

Your site now has a functional web server that can present static documents and run CGI scripts. Most sites have some static content, and it’s easy to manage once you learn how Apache translates a URL into a file path.

We’ve discussed URL mapping previously in the sections on UserDir. When Apache receives a request for a particular URL, it translates it into a file path and then figures out how to serve up that file. The rules for the latter can be quite complicated. Later chapters will explore them in examples that use handlers to take over this process for certain files or directories. The mapping rules themselves are generally simple:

1 If Apache is configured to handle multiple sites using <VirtualHost> directives, it looks at the site specification to see which set of rules to use. The site is the part of the URL between http:// and the next /. If there aren’t any virtual hosts, this part of the URL is ignored and the general rules are applied.

2 The section of the URL after the site specification is the path, composed of words separated by /s; it can end in a file name such as document.html, a trailing path component or just a /. Apache evaluates UserDir, Script-Alias, <Location>, and other directives to see if they match this path.

3 If the beginning of the path matches a ScriptAlias directive, the rest of the path is mapped onto the given directory and the resulting file is executed as a CGI script. Similarly Alias directives, Redirects, and other rewriting rules are applied.

4 If Apache is built with mod_userdir and UserDir isn’t disabled, it checks to see if the path begins with a ~. If so, the first path component is considered to be a user name and the UserDir rules discussed previously are applied to map the rest of the URL as a file path onto the user’s document directory.

5 If the rules in number four didn’t satisfy the most, the path is considered as a file path relative to the directory given by the DocumentRoot directive, usually the htdocs subdirectory of Apache’s home. Any <Directory> directives that match some or all of that file path are applied to the URL.

6 If the path ends in a file name and that file exists, directives are checked and the file is served if permissions allow. The browser receives the file and displays it according to its rules and idiosyncrasies.

7 If the path ends in a trailing /, Apache checks for the existence of a default file (usually index.html) and serves that if it exists. Otherwise, if directory browsing is allowed (Options Indexes), Apache creates a document on the fly that represents a directory listing. Depending on the other options you allow and the number of icons supplied for file types, this directory listing can look like a desktop file browser or an FTP site listing.

8 If Apache hasn’t figured out any other way of handling the path, it sends back an error document. These too are configurable; some sites have nice, apologetic error documents that offer an email address for sending complaints, while others reroute the user to a site navigation page. My favorite variations are those that use haiku:

You step into the stream, but the water has moved on. Document not found.

There is a bit more to it than that; when Apache decides what file to send to the browser, it also tries to figure out what type of file it is so that it can send along appropriate headers to the browser. These types are managed in the mime.types file in Apache’s configuration directory. If you are serving up unusual content, you’ll need to add types to this file so that browsers know what to do with the documents you send.

Suppose a browser sends a request for http://www.example.site/hello_web.html to our server. The path consists of just a file name, so there isn’t much in the way of analysis to do; Apache looks up hello_web.html in the DocumentRoot directory and sends it back to the browser.

That file contains an IMG tag specifying a relative URL for images/hi.jpg. Assuming the browser is displaying images, it sends a request for that URL back to Apache. The URL has a path, images, and a file name, hi.jpg. Apache looks for directives that apply to the path, and finding none, maps it onto the document root as a simple directory path. It sends the file back with appropriate headers and the image is displayed.

That’s static document handling in a nutshell. As I mentioned, there are plenty of ways to make even this process more complicated. Apache has a rich set of directives for rewriting URLs and managing changes of directories, file names, and so on that are inevitable over the lifetime of a site.

If that’s all you need, Apache will serve you and your documents well. You might consider an alternative, however, one specially built for speed at just this task.

2.9 THTTPD

thttpd is a small and versatile web server. Its flexibility begins with its name: the ‘t’ stands for tiny, turbo or throttling, take your pick. The author (Jef Poskanzer) offers the software from its web page (http://www.acme.com/software/thttpd/thttpd.html). While thttpd’s design goals are much the same as those for Apache—a secure, stable web server that handles static documents and other tasks—thttpd is built for speed, while Apache is meant to be a general platform for many other pieces of software.

thttpd’s feature list shows its focus: it serves static documents with a minimal implementation of the HTTP 1.1 protocol, offers a few utilities such as CGI processing, and a unique throttle utility which lets a webmaster allocate bandwidth to different sections of a site. It handles virtual hosts (see chapter 11), an absolute necessity for modern web sites. Other features such as redirection and server-side include (SSI) are given over to external programs, keeping thttpd small.

One reason for its lean size is it runs as a single process, unlike Apache with its separate children. The server listens on its socket for incoming requests and handles each in turn, but it doesn’t bog down on any particular client. thttpd uses nonblocking I/O via select to fill each waiting socket and move on to the next, so it can feed documents to a large number of browsers concurrently. That single process is smaller than just one normal Apache child (of which a typical configuration has at least several hanging around waiting for work).

Thus by keeping a tight focus and using a minimum of system resources, thttpd provides a very high performance level for servers that run in small spaces. In a race with Apache, thttpd can serve documents faster per second than the larger server can, but the author points out that the bandwidth limitations of most sites limit the performance of either server more than memory or other system resources. Don’t expect thttpd to improve on a site that has a network bottleneck.

In chapter 12 we’ll discuss how thttpd can be used as a front-end server to handle static documents while a more heavyweight Apache serves dynamic content. Don’t consider it just for a secondary role, however; it’s a fine server for many workloads.

It is also easy to extend thttpd for more than static documents. PHP users can use the popular mod_php module directly to run their applications, and Freshmeat lists a thttpd variant, pthttpd, which has an embedded Perl interpreter for speeding up the kind of code I’ll be talking about in the rest of the book. The web page for thttpd lists add-ons for SSL (see chapter 6) and other frequently needed options.

If you need only to publish static documents, Apache or thttpd will work fine for you. Chances are good you want more from your web server, however, and dynamic content is where the action is, so let’s go on to the tools that make the exciting stuff happen.