Skip to main content.

Web Based Programming Tutorials

Homepage | Forum - Join the forum to discuss anything related to programming! | Programming Resources

Web Programming Unleashed

Chapter 24 -- CGI Scripting with the UNIX Shell

Chapter 24

CGI Scripting with the UNIX Shell

by Dave Taylor


So far you've spent a lot of your time learning about complex and sophisticated languages for working with the Internet. All of them are powerful approaches to developing Common Gateway Interface (CGI) solutions for the World Wide Web. If you've been around the UNIX system for a while, though, you already know the majority of what it takes to make a powerful programming environment from your day-to-day command-line usage: the UNIX shell. This chapter explores the capabilities of the shell and looks at how it can be the ideal solution for a variety of simple to sophisticated Web back-end processes.

How CGI Scripts Work, a Redux

First, you can admit it: The Web, and all that it has spawned, is ugly and pretty much a huge hack. It's somewhat miraculous that it all works as well as it does and that it appears so seamless, when underneath, it is weird and remarkably inconsistent. Not just the markup language itself, mind you, but the actual protocol and even the way that the server identifies the type of information being sent to the client is the result of this combination of software and luck.

It all works similar to the depiction in Figure 24.1.

Figure 24.1 : The basic client/server communication.

The simplest case is when the client program (the Web browser) asks for a specific file using the file's address, or Uniform Resource Locator (URL). The server looks for the file, and if it finds it, sends back the contents of the specified file. If not, an error occurs and you get an ugly one-liner such as 404 File not found.

However, instead of the server sending back the contents of a file, the request can instead be "run this program on the server and send the output." Better yet, what if the program itself could output information in HTML so that the results are attractive and consistent? That's what interactivity is all about on the Web.

To have a program, rather than a file, be the target of a URL, you need to have that program live on the server and output HTML code. At its simplest, a shell script like the following does the trick:

#!/bin/sh -f

echo "Content-type: text/html"

echo ""

echo "<HTML><BODY>"

echo "<H1>Hello World</H1>"

echo "</BODY></HTML>"

exit 0

Web servers typically execute programs only in certain directories on the server, most commonly cgi-bin (Common Gateway Interface Binaries). Drop your programs into that directory on your server if that's how your server is configured, or you have another possibility. All scripts must have the .cgi suffix, even if they're shell scripts, but they can live anywhere in the file system. Check with the person who built the server-or, for your own system, look in the config files-to find out how the system is installed

After you've figured out the necessary naming scheme for your server, you're ready to roll with simple programs; change the ownership of the script using the UNIX chmod command to ensure that the script is executable, and figure out the full URL including your host name (such as Any output that your program emits is sent to the browser (the client program) on the other end of the wire.

Local Time and Who's Logged In

You can write a simple output-only CGI script that shows what time it is and who is logged on to the server itself by using the UNIX date and who commands, all wrapped up with standard HTML commands (see Listing 24.1).

Listing 24.1. Providing the time and a list of users to the client.

#!/bin/sh -f

echo "Content-type: text/html"

echo ""

echo -n "<h1>Welcome to "


echo -n " It's "


echo "</h1>"

echo "Here's Who is Logged In Right Now:"



echo "</PRE></BLOCKQUOTE>"

echo "</BODY></HTML>"

exit 0

The output looks like the following:

Welcome to It's Fri Jul 26 14:06:41 CUT 1996

Here's Who is Logged In Right Now:

       sol           pts/2       Jul 06 00:48           (

       taylor      pts/5       Jul 26 13:56          (

This script isn't too gorgeous, but it's quite utilitarian and serves as a basis for some interesting status scripts and such.

In addition to the standard HTML formatting instructions output from your script, the first two lines of output are critical to the Web browser's understanding of what to do with the information in the output: the two lines must be "Content-type: text/html" followed by a blank line. If you omit this, you'll see "document has no data" error messages when you try to access the scripts from a browser

The Environment Wrapped Up in the CGI Transmission

The request for a URL from the Web browser comes wrapped in a package that contains various snippets of useful information. When that information gets to the CGI program, it becomes part of the calling environment. Listing 24.2 shows a simple script that lets you see what you've got.

Listing 24.2. Obtaining browser information.

#!/bin/sh -f

echo "Content-type: text/html"

echo ""

echo "<PRE>"

env || printenv         # SVR4 or BSD - skip errors

echo "</PRE>"

exit 0

The output from the server when I connect via the Web shows what kind of client I'm using, from what system I'm connected, and lots of other interesting information, if you look closely.



HTTP_ACCEPT=*/*, image/gif, image/x-xbitmap, image/jpeg

HTTP_USER_AGENT=Mozilla/2.1N (Macintosh; I; PPC)





SCRIPT_NAME= /test.cgi




Notice that there's no remote login name or e-mail address in this collection of information. The best you can get automatically from a connection (today) is the user's remote host name (variable REMOTE_HOST) and the type of browser that the client is running (variable HTTP_USER_AGENT). You can extract the specific browser with a couple of simple UNIX tools:

x="`echo $HTTP_USER_AGENT | sed 's:/: :g'`"

browser="`echo $x | cut -d\   -f1`"

The sed invocation replaces the slash that separates the browser from its version ID, so it has arguments separated by spaces. Then, cut gives you the first field (-f1) in the argument as variable browser.

There are a variety of different browsers on the Net today, but a small number represent a large percentage of the user population. Here are some of the most common user agent identification strings:

HTTP_USER_AGENT=Lynx/2-4-2  libwww/2.14

HTTP_USER_AGENT=Microsoft Internet Explorer/4.40.308 (Windows 95)

HTTP_USER_AGENT=Mozilla/0.96 Beta (Windows)

HTTP_USER_AGENT=Mozilla/1.1N (Macintosh; I; PPC)

HTTP_USER_AGENT=Mozilla/1.22 (Windows; I; 32bit)

HTTP_USER_AGENT=NCSA Mosaic for the X Window System/2.4 libwww/2.12 modified



HTTP_USER_AGENT=Spyglass Mosaic/1.0  libwww/2.15_Spyglass

Mozilla is the internal name for Netscape's Navigator and has crept out in various releases of the browser

Being able to automatically identify the browser software within your script offers a terrific capability: browser-sensitive pages.

Pages Based on Browser Software

Suppose you want to have two versions of the same Web page, one for people with graphic capabilities and another for those using the utilitarian Lynx text-only Web browser. Listing 24.3 shows how you can do it.

Listing 24.3. Producing browser-dependent Web pages.

#!/bin/sh -f

echo "Content-type: text/html"

echo ""

x="`echo $HTTP_USER_AGENT | sed 's:/: :g'`"

browser="`echo $x | cut -d\   -f1`"

if [ $browser = "Lynx" ] ; then

  cat $text_only_home_page


  echo $graphical_home_page


exit 0

One drawback to this approach is that you must maintain two parallel HTML pages with the same basic information. Web page development tools that can help with this problem are just starting to show up on the market; you could also use m4 or a similar UNIX macro processor to allow #ifdef-style conditionals within a single version of the file that would automatically split into separate browser-specific documents.

Checking the Host Domain

One common use for the Web is to provide information that's available for the public and simultaneously provide other information only for people in the same institution or type of business. There's a very straightforward way to accomplish this by using the REMOTE_HOST environment variable. Its value is example, To extract the relevant information, use a pair of UNIX commands:

TOPMOST="`echo $REMOTE_HOST | rev | cut -d. -f1 | rev`"

DOMAIN="`echo $REMOTE_HOST | rev | cut -d. -f1,2 | rev`"

Note the double use of the obscure UNIX rev command, which reverses characters in the line. By reversing the remote domain name (to moc.evitiutni.4www), you can get the first field only (-f1 to cut) as the topmost domain and the first two fields as the regional domain. Then, flip it back with another call to rev, and you're ready to use the domain name.

How could you use this? The following shows an example:

echo "<h2>Our Favorite Links</h2>"

if [ $TOPMOST = "com" ] ; then

  cat commercial_links

elif [ $TOPMOST = "edu" ] ; then

  cat educational_links


  cat other_links


The rage in Web site promotion is advertising, and this tool gives you the capability to tailor a portion of your page-perhaps to your advertisers-based on their domain. For example, you can list different sets of advertisements or sponsors for students and commercial users (which includes subscribers America Online, the Microsoft Network, and all the dial-up services, too).

How Fast Is Your Connection?

When a browser connects to a server, its host name is sent as the environment variable REMOTE_HOST. Knowing that, you can write a shell script to run on a server and indicate the speed of the connection between the visitor and the server system. Listing 24.4 shows how that would look.

Listing 24.4. Checking the client/server connection.

#!/bin/sh -f

echo "Content-type: text/html"

echo ""

echo "<HTML>"

echo "<H1>Ping info to host $REMOTE_HOST:</h1>"


ping -c 10 $REMOTE_HOST

echo "</HTML>"

exit 0

If you connect to this script via the Web, you might see the following output:

Ping info to host

       PING ( 56 data bytes

       64 bytes from icmp_seq=0 ttl=47 time=351 ms

       64 bytes from icmp_seq=1 ttl=47 time=286 ms

       64 bytes from icmp_seq=2 ttl=47 time=310 ms

       64 bytes from icmp_seq=3 ttl=47 time=293 ms

       64 bytes from icmp_seq=4 ttl=47 time=291 ms

       64 bytes from icmp_seq=5 ttl=47 time=293 ms

       64 bytes from icmp_seq=6 ttl=47 time=302 ms

       64 bytes from icmp_seq=7 ttl=47 time=284 ms

       64 bytes from icmp_seq=8 ttl=47 time=289 ms

       64 bytes from icmp_seq=9 ttl=47 time=297 ms

       --- ping statistics ---

       10 packets transmitted, 10 packets received, 0% packet loss

       round-trip min/avg/max = 284/299/351 ms

Because ping computes an average round-trip time (see the last line of the preceding output), you could even add some smarts to your script that would let you take different actions based on the speed of the connection between the server and the browser system.

The first step is to extract the average round-trip packet speed:

ping -c 10 $REMOTE_HOST > /tmp/pingme.$$

average="`tail -1 /tmp/pingme.$$ | awk -F/ '{ print $4 }'`"

Now average has the average ping speed, in milliseconds, which you can then use as a gauge to deliver different information based on connection speed. The following is a very simple example of different graphics that you might include on your page based on whether the client has a fast or slow connection (fast connections get larger images in this case). Listing 24.5 shows how you could write a script to exploit this idea.

Listing 24.5. Delivering graphics based on connection speed.

if [ $average -lt 100 ] ; then

  echo "<img src=hi-rez.jpg>"

elif [ $average -lt 200 ] ; then

  echo "<img src=low-rez.jpg>"


  echo "<img src=black+white.gif>"


Users who connect over a slow line see the black-and-white GIF format, but users with a reasonable speed connection see a color graphic. However, if your users connect to the same site over a really fast line, they are surprised to see a beautiful, high-resolution JPEG image.

Remote ping

What if you want to specify what machine you want to ping? It turns out that there's another environment variable that can be sent from the browser to the server called QUERY_STRING. If you simply specify the URL of a particular CGI program or script and add a ?, anything following that question mark-up to the first space-is sent as the value of the QUERY_STRING variable.

Listing 24.6 shows how you could easily modify the ping script to check the connection between the server and any arbitrary system on the Internet.

Listing 24.6. Using ping for a specific machine.

#!/bin/sh -f

echo "Content-type: text/html"

echo ""

echo "<HTML>"

if [ "$QUERY_STRING" = "" ] ; then

  echo "<h1>no query string? No host to check</h1>"


  echo "<H1>Ping info to host $QUERY_STRING:</h1>"

  echo "<blockquote><pre>"

  ping -c 10 $QUERY_STRING

  echo "</pre></blockquote>"


echo "</HTML>"

exit 0

Suppose the script in Listing 24.6 is called query.cgi and that it lives on The URL for that particular script is then Now you can actually send it some arguments-in this case, the host name of a machine to check-by appending the desired system name to the URL itself. For example, within Netscape Navigator, you can click the Open URL button within your browser and type

The following output is shown as the contents of that page:

Ping info to host

       PING ( 56 data bytes

       64 bytes from icmp_seq=0 ttl=244 time=39 ms

       64 bytes from icmp_seq=1 ttl=244 time=36 ms

       64 bytes from icmp_seq=2 ttl=244 time=41 ms

       64 bytes from icmp_seq=3 ttl=244 time=42 ms

       64 bytes from icmp_seq=4 ttl=244 time=29 ms

       64 bytes from icmp_seq=5 ttl=244 time=32 ms

       64 bytes from icmp_seq=6 ttl=244 time=31 ms

       64 bytes from icmp_seq=7 ttl=244 time=30 ms

       64 bytes from icmp_seq=8 ttl=244 time=32 ms

       64 bytes from icmp_seq=9 ttl=244 time=31 ms

       --- ping statistics ---

       10 packets transmitted, 10 packets received, 0% packet loss

       round-trip min/avg/max = 29/34/42 ms

As you can see, Pipeline in New York has a much, much faster TCP packet turn-around speed with the server than the test workstation used in the previous example (34 milliseconds compared to 299 milliseconds); that's almost 10 times faster!

A Form Front End

Being able to add arguments to a URL as a way to pass information is useful, but it's somewhat limited. After all, what you really want to have on your Web pages are boxes and checklists-places where users can specify information and then press a Do It or Submit button and have that information quickly relayed to the waiting CGI script.

You include such elements with the <FORM> HTML tag set within your documents, as shown in Listing 24.7.

Listing 24.7. Creating a form to reach your script.


<h1>How fast is your connection?</h1>




Look for?

<input type=string name=ping>


<input type=submit value="ping this host">



Most of this script is typically straightforward Web markup until you get to the <FORM> section. The FORM tag has two attributes: the mechanism by which the information should be transmitted to the server and the URL of the CGI program that should receive the information. In this case, you can see that the ACTION specifies that the remote script is referenced as

and that the METHOD is Get.

The two basic METHODs for sending information from a browser to a CGI script are Get and Post. Get is the easiest to work with because the information is all tucked neatly into the QUERY_STRING environment variable, but it has some serious size limitations. Instead, complex Web forms invariably use Post, which sends all the information as standard input to the CGI script, allowing an arbitrary amount of data to flow from the browser to the CGI script.

Every input field from an HTML form is sent as a name=value pair with multiple pairs separated by an ampersand. In the case of this particular form, the <input type=string> HTML tag produces a small box within which the user can type the name of a system to ping. That information is actually sent to the CGI script as ping=whatever-they-typed (that's what the name=ping does in that <INPUT> tag). There's a variety of different input type fields, including those shown in Table 24.1.

Table 24.1. Form input types.

Input typeMeaning
stringOne line of text requested from the browser
passwordOne line of text-not echoed as typed
radioOne of a set of radio buttons
checkboxA yes/no checkbox
submitThe submit or do it button
resetThe reset or restore default values button

Many good references are available both online and in books at your local bookstore, so I won't belabor the point here.

Now you have a simple Web page that prompts for a host to ping and presents an action button right below the prompt that users can click to have their information sent to the script and acted upon. This is shown in Figure 24.2.

Figure 24.2 : A simple form-based Web page inviting input.

How does this variation of the CGI script look? It's surprisingly similar to the last version you saw; indeed, all you need to add is the capability to extract the host name from its name=value form, as shown in Listing 24.8.

Listing 24.8. ping script that accepts input.

#!/bin/sh -f

# modified to accept name=value pairs...

echo "Content-type: text/html"

echo ""

echo "<HTML>"

if [ "$QUERY_STRING" = "" ] ; then

  echo "<h1><I>no query string? No host to check</i></h1>"


  host="`echo $QUERY_STRING | awk -F= '{print $2}'`"

  echo "<H1>Ping Info to Host $host:</h1>"

  echo "<blockquote><PRE>"

  ping -c 10 $host

  echo "</pre></blockquote>"


echo "</HTML>"

exit 0

You use the awk program to split the information received at the =, which works fine for a single value. However, if you move to a multiple-variable script, a more sophisticated technique is required.

With this HTML page and the script shown in Listing 24.8, users can now pop over to the Web site and ping any host on the network with wild and merry abandon. Entering and pressing the Submit button even produces results identical to those shown earlier.

Another Example: finger

You can apply the same technique you used for ping to another simple UNIX command-one that would be useful to access from within the Web environment: the finger command. This is an interesting command because its behavior is dependent on the type of information that you give it; use it to finger a name, and it searches for everyone with that information in the password file, showing you all the results. Give it a remote host name instead-in the form it tells you who is logged in to that machine. Use a fully qualified e-mail it tells you, the requestor, about that person if it can connect to the machine.

Listing 24.9 shows an HTML file that's a quick and simple finger front end. You'll see that it's remarkably similar to the ping page.

Listing 24.9. Form front end for finger queries.


<h1>Finger:</h1><h2>find out about users or computers</h2>


<form method=get action="


Look for?

<input type=string name=finger>


<blockquote><font size=+1>

<I>Try an email address for a specific user, or just the '@hostname'

format to see who is using a particular computer on the net</i>


<CENTER><input type=submit value="look up this user or host">




This time, you've also included some helpful information for the user in the form of a brief italicized comment below the input box, as you can see in Figure 24.3.

Figure 24.3 : Simple Web front end for the finger command.

The script at the other end looks like Listing 24.10.

Listing 24.10. The finger CGI script.

#!/bin/sh -f

# Finger user or user@host or @host

echo "Content-type: text/html"

echo ""

echo "<HTML>"

if [ "$QUERY_STRING" = "" ] ; then

  echo "<h1><I>no user or host to check?</i></h1>"


  value="`echo $QUERY_STRING | awk -F= '{print $2}'`"

  echo "<H1>Finger information for $value:</h1>"

  echo "<blockquote><PRE>"

  finger $value

  echo "</pre></blockquote>"


echo "</HTML>"

exit 0

In the spirit of good coding, you again include some error checking-this time, the CGI script can produce an error message (as proper HTML, of course) if you hit the script without going through the HTML page. (If you leave the box blank, it sends finger=, so there's still some data.) If the client does give the script something, it shows you, the client, the results of the finger command run on the server with the information you've specified.

Take a quick look at some of the possible output formats. Note in these examples that the $value sent also appears as part of the output. As an interface rule, this is a great bit of positive user feedback, allowing users to verify that what they sent was processed accurately.

First, what happens if the client doesn't specify anything?

Finger information for :

     Login       Name              TTY   Idle    When    Site Info

     taylor   Dave Taylor           p0     2 Thu 12:57

What happens if you specify to see who might be logged in there?

Finger information for


       Login       Name              TTY Idle    When    Where

       zanna    Zanna Knight       KA 1:46 Thu 09:00  Zanna's Mac:8.23

       toni     Toni Veglia           co  13: Tue 14:59

       diane    Diane DeMartini   p6    8 Thu 08:44

       ellie    Ellie Young           p7 1:22 Thu 09:04  boss:0.0

       toni     Toni Veglia           pb      Thu 09:12

       carolyn  Carolyn Carr       pe 1:26 Thu 09:20  bigx:0.0

       ah       Alain Henon           q0   12 Thu 09:29

       zanna    Zanna Knight        q3   59 Thu 09:48

       eileen   Eileen Curtis         q5   36 Thu 10:04

The Usenix Assocation has a lot of people and a lot of things going on. You can also pick someone and submit that person's name to find out more about him or her:

Finger information for


       Login name: zanna                       In real life: Zanna Knight

       Directory: /staff/zanna                 Shell: /bin/csh

       On since Dec  7 09:00:41 on KA0.0 from Zanna's Mac:8.23

       1 hour 45 minutes Idle Time

       No unread mail

       No Plan.

       Login name: zanna                       In real life: Zanna Knight

       Directory: /staff/zanna                 Shell: /bin/csh

       On since Dec  7 09:48:06 on ttyq3 from

       58 minutes Idle Time

Here, Zanna is actually logged in on two different lines, which is why you see two entries for her in this output. Notice that the top entry indicates that she's actually connected from a Macintosh, too. Ah, the things you can glean when you poke around on the networkÉ

Another Script Trick: Redirection

One nuance of CGI response that's quite cool is if your program emits a line "Location: url" with some valid URL for a page anywhere on the network, the connection to your server is instantly replaced with a connection to the specified other server. Suppose you're at and you have a CGI script in the cgi-bin directory. The script proves to be a tiny script that looks like this:

#!/bin/sh -f

echo "Content-type: text/html"

echo "Location:"

echo ""

exit 0

You connect to the URL and what's returned is actually The net result of invoking this CGI script is that you'll be looking at the home page of TNT Media at

The Location: line is output in addition to the Content-Type: header. Notice that the blank line still must appear after these lines

What can you do with this? The most obvious answer is a random forwarding service, a so-called URL roulette, where you connect to this URL, and it randomly picks another URL from a file and sends you there instead-somewhat like the following pseudo code:

randomurl = `pick-random-line-from  urllist`

print "Location: " $randomurl

More interestingly, you can combine what I've been discussing to make a page that takes people to the home page of their browsers, as shown in Listing 24.11.

Listing 24.11. Jumping to the right home page.

#!/bin/sh -f

echo "Content-type: text/html"

x="`echo $HTTP_USER_AGENT | sed 's:/: :g'`"

browser="`echo $x | cut -d\   -f1`"

case $browser in

 Cello )        loc=;     ;;

 Spyglass )     loc=;           ;;

 Lynx )         loc=; ;;

 NCSA )       loc=;              ;;

 Mozilla )      loc=;                           ;;

 NetCruiser )   loc=;                         ;;

 Microsoft )

          loc=;  ;;

       *   )    loc=;                              ;;


echo Location: $loc

echo ""

exit 0

If you don't know what to do with their browsers, you send 'em to Yahoo!


This chapter shows only the tip of the proverbial iceberg for shell-based CGI scripting. In particular, any time you either display pages based on processing, without user input, or process only a single variable of information, a shell script is probably the fastest and easiest solution available within UNIX.

Other scripting languages are available on UNIX-notably TK, TCL, and Python-and they have very specific capabilities that make them useful for UNIX programming but not for CGI work.