Lecture 2: 14 Jan, 2009

Khaled Harras (with a nod to Mark Stehlik and Greg Kesden for this lecture)

Administrative

Lab 1 released.

The Shell and Shell Scripting

Most of us are pretty familiar with the "UNIX shell". We use it, whether sh, csh, bash, tcsh, zsh, or other variants, to start and stop processes, control the terminal, and to otherwise interact with the system.

Many of you have heard of, or made use of, "shell scripting" — the process of providing instructions to the shell in a simple, interpreted programming language. In many ways the language of the shell is very powerful — it has functions, conditionals, loops, for example. In other ways, it is weak &mdash it is completely untyped (everything is a string).

But the real power of shell scripting doesn't come from the language but from the diverse library that it can call upon — any program. Shell programming remains popular because it provides a quick and easy way to integrate command-line tools and filters to solve often complex problems.

While we'll demonstrate lots of programs in this lecture, I don't expect you to write down or remember detail of every command. UNIX has a wonderful documentation system of man pages that can be used when you need to know how a particular command works. The first 3 sections (of 8!) will be of most use to you in this course. Section 1 covers UNIX commands, Section 2, System calls, and Section 3, C library functions. The UNIX mantra is RTFM (try man man and, of course, man woman)!

Simple Scripts

The simplest scripts of all are nothing more than lists of commands. Consider the script below, which we will enter in a file called, first:
    #!/bin/sh

    # my first shell script

    date    # Wed Jan 16 09:01:19 AST 2008
    pwd     # /afs/qatar.cmu.edu/usr16/msakr/public/123-dist/scripts
    users   # kharras
  
To run this script, which we've placed in the file, first, we just execute the file (we need to use ./ to indicate that we're running the program out of the current directory):
    ./first
  
What happens? The shell tells us "Permission denied" — it's not an executable file. To make it so, we need to do the following:
    chmod a+x first
  
Now we can run our script (and you can RTFM on chmod).

Note that anything after a # is a comment and is ignored by the shell. We have one entire line commented as well parts of lines where we show sample output for each of the commands. But what about that first line? When the two symbols, #!, (read "hash-bang") are the first two characters in a script, the operating system is to execute the script using the interpreter that is specified after the #!.

Thus, this line tells the shell where the script was run to invoke /bin/sh to run the script. This is necessary because different users might be using different shells: sh, csh, bash, tcsh, zsh, tcsh, etc. And these shells have slightly different languages and build-in features. In order to ensure consistent operation, we want to make sure that the same shell is used to run the script each time.

Aside: the various shells are more the same than different. As a result, on many systems, there is actually one shell program capable of behaving with different personalities. On these systems, the personality is often selected by soft linking different names to the same shell binary. Then, the shell looks at argv[0] to observe how it was invoked, sets some flags to enable/disable behaviors, and goes from there.

The bulk of this simple script is a list of commands. These commands are executed, in turn, and the output is displayed to the console (the terminal). The commands are found by searching the standard search path, PATH. PATH is a : delimited list of directories which is to be searched for executable programs. Here's my current search path (yours probably looks a little different):
    /afs/qatar.cmu.edu/usr4/mjs/bin:/usr/local/bin:/usr/edu/bin:/bin:/usr/bin:/usr/contributed/bin:.
  
The command which, used as in which ls, will tell you which version of a command is being executed. This is useful if different versions might be in your search path. In general, the search path is traversed from left to right.

Aside: Notice that ".", the current working directory, is the last directory listed. This should almost certainly be the case. Placing it at the very beginning of the path is dangerous (why?)!
Also realize that if "." is at the end of your search path and you name a script with the same name as a UNIX command, e.g., test, and don't specify ./test, you will get execute the UNIX command, not your script since the UNIX command will be found first along the search path!

Variables

PATH discussed above is one example of a variable. It is what is known as an environment variable. It reflects one aspect of the shell environment — where to look for executables. Changing it changes the environment in which the shell executes programs. Environment variables are special in that they are defined before the shell begins (at login). There are also shell variables, like history, which apply only to the current instance of the shell.

Environment variables, like most other variables, can be redefined simply by assigning them a new value. I appended . to my default path using the following set command:
    set path=($path .)
  
And, they are evaluated (i.e., their value is examined), using the $ operator, as below:
    echo $PATH
  
To create new variables in a shell script, you simply assign them a value:
    echo $prof
    prof="Khaled"
    echo $prof
  
All shell script variables are untyped (well, they really are strings) — how they are interpreted often depends on what program is using them or what operator is manipulating or examining them.

Positionals, e.g. Command Line Arguments

Several special variables exist to help manage command-line arguments to a script:

$# - represents the total number of arguments (much like argv)

$0 - represents the name of the script, as invoked

$1, $2, $3, .., $8, $9 - The first 9 command line arguments

$* - all command line arguments, starting with $1
"$@" - all command line arguments, where each argument is individually quoted (useful if some arguments include spaces).

Unlike other variables, positions can't be assigned values using the = operator. Instead, they can only be changed in a very limited way.
The set command sets these values. Consider the following example:
    set a b c
    # $1 is now a
    # $2 is now b
    # $3 is now c
  
If there are more than 9 command-line arguments, there is a bit of a problem — there are only 9 positionals: $1, $2, ..., $9. $0 is special and is the shell script's name. To address this problem, the shift command can be used. It shifts all of the arguments to the left, throwing away $1. What would otherwise have been $10 becomes $9 — and addressable. We'll talk more about shift after we've talked about while loops.

Quotes, Quotes, and More Quotes

Shell scripting has three different styles of quoting — each with a different meaning:

unquoted strings are normally interpreted

"double-quoted strings are basically literals — but $variables are evaluated"

'single-quoted strings are absolutely literally interpreted'

`commands in back-quotes like this are executed, their output is then inserted as if it were assigned to a variable and then that variable was evaluated`

I think "quotes" and 'quotes' are pretty straight-forward — and will be constantly reinforced. Here's an example using `back-quotes` (don't mind the use of cut — we'll get to that (or RTFM)):
    day=`date | cut -d " " -f1`
    printf "Today is %s.\n" $day
  

expr

The expr program can be used to manipulate variables, normally interpreted as strings, as integers. Consider the following "adder" script:
    sum=`expr $1 + $2`

    printf "%s + %s = %s\n" $1 $2 $sum
  

A Few Other Special Variables

$? - the exit status of the last program to exit

$$ - the shell's pid

Predicates

The convention among UNIX programmers is that programs should return a 0 upon success. Typically a non-0 value indicates that the program couldn't do what was requested. Some (but not all) programmers return a negative number upon an error, such as file not found, and a positive number upon some other terminal condition, such as the user choosing to abort the request.

As a result, the shell's notion of true and false is a bit backward from what most of us C programmers might expect: to the shell, 0 is considered to be true and non-0 is considered to be false!

We can use the test command to evaluate an expression. The following example will print 0 if mjs is the user and 1 otherwise. It illustrates not only the test but also the use of the status variable. status is automatically set to the exit value of the most recently exited program. Recall that the notation $var, such as $test, evaluates the variable.
    test "$LOGNAME" = mjs
    echo $?
  
Shell scripting languages are typeless. By default everything is interpreted as a string. So, when using variables, we need to specify how we want them to be interpreted. Thus, the operators we use vary with how we want the data interpreted.

Operators for strings, ints, and files

string x = y,
comparison: equal x != y,
comparison: not equal x,
not null/not 0 length -n x,
is null

ints x -eq y,
equal x -ge y,
greater than or equal x -le y,
less than or equal x -gt y,
strictly greater x -lt y,
strictly lesser x -ne y,
not equal

file -f x,
is a regular file -d x,
is a directory -r x,
is readable by this script -w x,
is writeable by this script -x x,
is executable by this script

logical x -a y,
logical and, like && in C (0 is true, though!) x -o y,
logical or, like || in C (0 is true, though!)

Operators for strings, ints, and files
string	x = y, comparison: equal	x != y, comparison: not equal	x, not null/not 0 length	-n x, is null
ints	x -eq y, equal	x -ge y, greater than or equal	x -le y, less than or equal	x -gt y, strictly greater	x -lt y, strictly lesser	x -ne y, not equal
file	-f x, is a regular file	-d x, is a directory	-r x, is readable by this script	-w x, is writeable by this script	-x x, is executable by this script
logical	x -a y, logical and, like && in C (0 is true, though!)	x -o y, logical or, like \|\| in C (0 is true, though!)

[ Making the Common Case Convenient ]

We've looked at expressions evaluated as below:
    test -f somefile.txt
  
Although this form is the canonical technique for evaluating an expression, the shorthand, as shown below, is universally supported and much more reasonable to read:
    [ -f somefile.txt ]
  
You can think of the [ ] operator as a form of the test command. But, one very important note — there must be a space to the inside of each of the brackets. This is easy to forget or mistype. But, it is quite critical.

Conditional Execution

Like most programming languages, shell script supports an if statement, with or without an else. The general form is below:
    if command
    then
        command
        command
        ...
        command
    else
        command
        command
        ...
        command
    fi
  
    if command
    then
        command
        command
        ...
        command
    fi
  
The command used as the predicate can be any program or expression. The results are evaluated with a 0 return being true and a non-0 return being false.

If ever there is the need for an empty if-block, the null command, a :, can be used in place of a command to keep the syntax legal.

The following is a nice, quick example of an if-else:
    if [ "$LOGNAME" = "kharras" ]
    then
      printf "%s is logged in; all is well" $LOGNAME
    else
      printf "Someone else is logged in.  Beware!"
    fi
  

The elif construct

Shell scripting also has another construct that is very helpful in reducing deep nesting. It is unfamiliar to those of us who come from languages like C. It is the elif, the "else if". This probably made its way into shell scripting because it drastically reduces the nesting that would otherwise result from the many special cases that real-world situations present — without functions to hide complexity (shell does have functions, but not parameters — and they are more frequently used by csh shell scripters than traditionalists).
    if command
      command
      command
      ...
      command
    then
      command
      command
      ...
      command
    elif command
    then
      command
      command
      ...
      command
    elif command
    then
      command
      command
      ...
      command
    fi
  