Bristle Software Unix Tips

This page is offered as a service of Bristle Software, Inc.  New tips are sent to an associated mailing list when they are posted here.  Please send comments, corrections, any tips you'd like to contribute, or requests to be added to the mailing list, to tips@bristle.com.

Table of Contents:

  1. General
    1. Commands
      1. The help system
      2. find - Find and operate on files
    2. Command Line
      1. Command Line Redirection
        1. Redirection Summary
        2. Output Redirection
        3. Input Redirection
        4. Here Document
        5. Tab-stripped Here Document
        6. Here Document Without Expansion
        7. Piping
        8. Filters
        9. Command Substitution
        10. tee
        11. set noclobber
        12. Standard Input, Standard Output, and Standard Error
        13. Redirecting Standard Output and Error Together
        14. Redirecting Standard Error in csh
        15. Reading and Writing the Same File
        16. Redirecting Other File Descriptors
        17. DOS and Windows Redirection Gotchas
    3. Shell Scripts
      1. coming soon...
  2. Unix Shells
    1. csh and tcsh
      1. csh and tcsh Command Line
        1. set cdpath - Navigating directories
        2. set filec - Filename completion
        3. set prompt - Setting the Prompt
        4. alias - Define a new command
        5. alias dirr - Search a directory tree for a file
        6. alias confirm - Apply a command to multiple files
      2. csh and tcsh Shell Scripts
        1. coming soon...
      3. csh and tcsh Environment
        1. Startup Files
    2. sh
      1. sh Command Line
        1. coming soon...
      2. sh Shell Scripts
        1. coming soon...
      3. sh Environment
        1. Startup Files
    3. bash
      1. bash Command Line
        1. coming soon...
      2. bash Shell Scripts
        1. coming soon...
      3. bash Environment
        1. Startup Files
    4. ksh
      1. ksh Command Line
        1. set CDPATH - Navigating directories
      2. ksh Shell Scripts
        1. coming soon...
      3. ksh Environment
        1. Startup Files
  3. Unix Flavors
    1. BSD
      1. BSD Command Line
        1. coming soon...
      2. BSD Shell Scripts
        1. coming soon...
    2. System V
      1. System V Command Line
        1. coming soon...
      2. System V Shell Scripts
        1. coming soon...
    3. Solaris
      1. Solaris Command Line
        1. coming soon...
      2. Solaris Shell Scripts
        1. coming soon...
    4. Linux
      1. Linux Command Line
        1. coming soon...
      2. Linux Shell Scripts
        1. coming soon...
  4. Unix Utilities
    1. perl Scripts
      1. See also
    2. awk Scripts
      1. coming soon...
    3. sed Scripts
      1. coming soon...
    4. yacc Grammars
      1. coming soon...
    5. lex Grammars
      1. coming soon...
    6. make files
      1. coming soon...
    7. rcs
      1. coming soon...
    8. sccs
      1. coming soon...

Details of Tips:

  1. General

    1. Commands

      1. The help system

        Last Updated: 11/4/2001
        Applies to: All shells, All Unix flavors

        How to get started if you don't know the Unix commands?  Here are some starting points:

        Command  What it does
        apropos Searches the one-line summaries of each command in the "man pages" (Unix manual pages) for the specified string (not necessarily as a full word).  

        For example:
                apropos file
        shows the summaries that contain the string "file" -- a good way to find lots of commands that operate on files.

        You can also combine this with the "grep" (search) command to do things like:
                apropos file | grep -i open | grep -i dialog
        which shows the summaries that contain all the strings "file", "open", and "dialog" -- a good way to find support for popping up a dialog box to allow the user to choose a file to open.

        Note:  If your system has no "apropos" command, use "man -k" instead.

        whatis Shows the one-line summary of the specified command.

        Useful to see quickly whether a certain command does the kind of thing you want.

        man Shows the entire man page, one screen at a time, for the specified command.  

        Useful to see all the details about a command, once you know which command to ask for.

        whereis Looks in a bunch of standard places for the source, binary, and documentation files for the specified command.

        Useful to find a standard command that isn't yet on your PATH.  You can then run it by typing its full path (for example:  /usr/sbin/useradd), or by adding its directory to your PATH.

        which Looks in all the places on your PATH for the specified command, showing the first match.

        Useful to find out exactly which file would execute, if you were to issue a command.

        which -a Looks in all the places on your PATH for the specified command, showing all matches.

        Useful to find out which command file is hiding which other command files by the same name.

        where (tcsh only)
        type -a (ksh, bash only)
        whence -p (ksh only)
        command -v (ksh only)
        Like "which -a", but also takes into account "aliases" (see alias - Define a new command) and commands that are built in to the Unix shell (not stored as separate files). 
        find Searches the specified directory tree for the specified file.
        locate GNU utility to search a database maintained by the system administrators for the specified file.   

        Note:  Faster than find, but relies on the database being current.

        Now that you know the names of these commands, use the man command to find out more about each one.

        Note:  The Unix "man pages" document more than just commands.  They also document APIs that can be called from a program, "daemon" processes that run in the background, the format of configuration files, etc.  Each one-line summary includes a number in parentheses, with the following meaning:

            (1)    Interactive commands
            (2)    System calls
            (3)    Subroutines
            (4)    Special files
            (5)    File formats and conventions
            (6)    Games
            (7)    Macro packages and language conventions
            (8)    Maintenance commands and procedures.    

        Thanks to Tom Dickey and Thor Collard for their contributions to this tip!

        --Fred

      2. find - Find and operate on files

        Last Updated: 5/26/2000
        Applies to: All shells, All Unix flavors

        The find command is used to find and operate on files.  It searches an entire directory tree starting at the specified directory for a file with a specified name, size, date, etc.  For each file it finds, it can display the name, execute a user-specified command, etc.  It supports a huge number of options, but is a little tricky to use.  If you are a DOS, OS/2, Windows, or VMS user, looking for the recursive options on commands like DIR, and noticing that ls -R doesn't do what you expect, this is probably what you want (and a whole lot more).  See man find for details.

        --Fred

    2. Command Line

      1. Command Line Redirection

        1. Redirection Summary

          Last Updated: 2/24/2002
          Applies to: All shells, All Unix flavors

          All Unix shells offer a wide variety of techniques to redirect the input and output streams from a command to a file or to another command.

          The following forms are valid in all shells:

          Syntax (one line) What it does
          command > filename Redirects the output of command to write to filename, creating filename if it doesn't exist, and overwriting its current contents if it does exist.
          command >> filename Redirects the output of command to write to filename, creating filename if it doesn't exist, and appending to the end of its current contents if it does exist.
          command < filename Redirects the input of command to read from filename.
          command << word Redirects the input of command to read from the current shell script, until the first line that starts with word.  Known as a "here document".  Only useful in shell scripts.  Many people use "EOF" as the word to indicate the EOF (end of file) of the here document.
          command1 | command2 "Pipes" the output of command1 into command2 by redirecting the output of command1 and the input of command2.
          command1 `command2` Redirects the output of command2 into the command line (not into standard input) of command1.  The output of command2 is used as command line options and arguments to command1.  
          Note:  You must use the backtick character (`), not the apostrophe (').  On most keyboards, it is just left of the number 1 key, with tilde (~) as its shift character.
          command | tee filename Creates 2 copies of the standard output of command, redirecting one copy to filename but not affecting the other copy.  Very useful when you want to watch the output of a long-running process on the screen, but also capture it all to a log file.
          set noclobber Sets the "noclobber" shell variable which causes > to fail if filename already exists, and >> to fail if filename does not already exist.

          The following forms are valid in csh-based shells (csh and tcsh):

          Syntax (one line) What it does
          command >! filename Same as without the exclamation point (!), but suppresses the "noclobber" check.
          command >>! filename
          command >& filename Same as without the ampersand (&), but redirects standard error, as well as standard output.
          command >>& filename
          command >&! filename
          command >>&! filename
          command1 |& command2
          (command > filename1) >& filename2   Redirects standard output of command to filename1, and standard error to filename2.

          The following forms are valid in sh-based shells (sh, bash, and ksh):

          Syntax (one line)  What it does
          command >| filename Same as without the pipe character (|), but suppresses the "noclobber" check.
          command >>| filename
          command 0< filename Same as command < filename.  A way of explicitly referring to standard input (file descriptor 0).
          command 1> filename Same as command > filename.  A way of explicitly referring to standard output (file descriptor 1).
          command 2> filename Redirects standard error (file descriptor 2) of command to filename without affecting standard output.
          command 1> filename1 2> filename2 Redirects standard output (file descriptor 1) of command to filename1, and standard error (file descriptor 2) to filename2.
          command > filename 2>&1 Same as without the 2>&1, but redirects standard error, as well as standard output.
          This is the same 2> as above (redirect standard error), but the special syntax &1 redirects it into file descriptor 1 (standard output) instead of into a named file.
          Note:  The location of the syntax 2>&1 is important.  It must occur, as shown here, after the redirection of the merged stream to filename, but before the pipe character (|).
          command >> filename 2>&1
          command1 2>&1 | command2
          command &> filename Same as command > filename 2>&1
          [Note:  bash only; not sh and ksh]
          command >& filename
          command <<- word Same as command << word, but strips all leading tab characters from each line of the "here document", allowing you to indent it within the shell script.
          command <> filename Same as command < filename, but opens filename for reading and writing.  Only useful if command writes to standard input.
          command 0<> filename Same as command <> filename.  A way of explicitly referring to standard input (file descriptor 0).
          command 1<> filename Same as command 1> filename, but opens filename for reading and writing.  Only useful if command reads from standard output.
          command 2<> filename Same as command 2> filename, but opens filename for reading and writing.  Only useful if command reads from standard error.
          command n< filename Redirects file descriptor n of command to read from filename.  This is the general form of the 0< syntax shown above.  It is only useful for programs that read from file descriptors other than 0.
          command n> filename Redirects file descriptor n of command to write to filename.  This is the general form of the 1> and 2> syntax shown above.  It is only useful for programs that write to file descriptors other than 1 and 2.
          command n>> filename All the forms shown above (>, >>, >|, >>|, &>, <>, etc.) can explicitly use 0, 1, 2, or any other file descriptor n.  The number for the file descriptor is always placed immediately before the first < or > character.
          command n>| filename
          command n<> filename
          etc...

          For more information, see the tips below, or type:

                  man csh
                  man tcsh
                  man sh
                  man bash
                  man ksh

          --Fred

        2. Output Redirection

          Last Updated: 2/24/2002
          Applies to: All shells, All Unix flavors

          The most commonly used form of command line redirection is redirecting the output of a command to a file, as:

                  ls > out.txt

          which sends the output of the ls command (a list of filenames) to the file out.txt instead of to the screen.

          A variation on this is:

                  ls >> out.txt

          which appends the output to out.txt, instead of overwriting the contents of out.txt.

          These forms are supported by all Unix shells.  They are also supported by DOS and Windows (but there, you use dir instead of ls to list the names of files) .

          --Fred

           

        3. Input Redirection

          Last Updated: 2/24/2002
          Applies to: All shells, All Unix flavors

          You can also redirect the input of a command to read from a file, as:

                  sort < in.txt

          which causes the sort command to read from the file in.txt, instead of from the keyboard.

          This form is supported by all Unix shells, and also by DOS and Windows.

          --Fred

           

        4. Here Document

          Last Updated: 2/24/2002
          Applies to: All shells, All Unix flavors

          A variation on input redirection is:

                  sort <<EOF
                  line of text
                  another line of text
                  third line of text
                  EOF

          which is only useful in a shell script.  This form, called a "here document", causes the sort command to read from the shell script that contains it ("read from here"), instead of from the keyboard, until it finds a line that consists entirely of the word that occurs after the << ("EOF" in this case).

          This form is supported by all Unix shells, but not by DOS or Windows.

          --Fred

           

        5. Tab-stripped Here Document

          Last Updated: 2/24/2002
          Applies to: sh-based shells, All Unix flavors

          Your shell scripts will be more readable if you indent the lines of the here document to make them stand out from the rest of the script.  However, you don't want the indentation to be treated as part of the input stream.  You can do this, as:

                  sort <<-EOF
                      line of text
                      another line of text
                      third line of text
                  EOF

          by adding a minus sign (-) immediately after the <<, which causes the leading tab characters to be ignored.

          This form is supported by sh-based Unix shells (sh, bash, ksh), but not by csh-based shells (csh, tcsh), and not by DOS or Windows.

          --Fred

           

        6. Here Document Without Expansion

          Last Updated: 3/16/2002
          Applies to: All shells, All Unix flavors

          Ordinarily lines of a "here document" are subject to various forms of expansion (command substitution, parameter expansion, arithmetic expansion, etc) before they are passed to the command.  For example, the script:

                  sort <<EOF
                  line of text
                  another line of text
                  my username is $LOGNAME
                  EOF

          produces:

                  another line of text
                  line of text
                  my username is fred

          To suppress such expansion, quote the word that follows the << as:

                  sort <<"EOF"
                  line of text
                  another line of text
                  my username is $LOGNAME
                  EOF

          which produces:

                  another line of text
                  line of text
                  my username is $LOGNAME

          You can use double quotes as shown above ("EOF"), or single quotes ('EOF'), or backslash quoting (\EOF).  Also, you can quote the entire word, or just part of it (E"OF", E'OF', E\OF).

          This form is supported by sh-based Unix shells (sh, bash, ksh).  

          It is also supported by csh-based shells (csh, tcsh), but in that case, you must include exactly the same quotes in the closing word as you did in the opening word.  For example:

                  sort <<"EOF"
                  line of text
                  another line of text
                  my username is $LOGNAME
                  "EOF"

          It is not supported by DOS or Windows.

          Thanks to Tom Dickey for reminding me of this form!

          --Fred

           

        7. Piping

          Last Updated: 2/24/2002
          Applies to: All shells, All Unix flavors

          You can combine input redirection and output redirection, directing the output of one command into the input of another command, as:

                  ls | sort

          which directs the output of ls into the sort command, so that the list of filenames produced by ls is sorted.  Since the output of sort is not redirected, the sorted list is displayed on the screen.

          You can also string multiple pipes together, as:

                  ls | grep abcd | sort | more

          which pipes the output of ls into the grep command to discard all lines not containing the string "abcd", pipes the output of grep into the sort command to sort it, and pipes the output of the sort command into the more command to cause it to be displayed one screenful at a time.

          You can also combine piping with other forms of redirection.  For example:

                  ls | grep abcd | sort > out.txt

          which does the same as the previous example, except it sends the output of the sort command into the file out.txt, instead of paging it to the screen via the more command.

          These forms are supported by all Unix shells, and also by DOS and Windows.  However, in DOS and Windows, you must use the find command instead of grep, the dir command instead of ls, and must put the string in quotes as:

                  dir | find "abcd" | sort | more
                  dir | find "abcd" | sort > out.txt

          --Fred

           

        8. Filters

          Last Updated: 11/12/2004
          Applies to: All shells, All Unix flavors

          To take advantage of easy file redirection and piping, many Unix commands are written as "filters".  That is, they don't require command line parameters to tell them what file to read and what file to write.  Instead, they read from standard input and write to standard output.  You can redirect to the desired files, or pipe to other commands.  Some of the most useful Unix filters are:

          Filter What it does
          more Waits for you to ask for more.  It reads standard input, writes a screenful of it to standard output, then waits for keystrokes to tell it how much more of standard input to write to standard output before waiting again (return = one more line, space = one more screen, b = back one screen, h = help, etc.) .  This prevents long output streams from flowing past faster than you can read them.
          sort Sorts the input lines alphabetically, sorting upper case letters before lower case or treating them the same (option -f), optionally ignoring leading blanks (option -b).  With option -n, it sorts  numerically (2 before 10, but 20 after 10) instead of alphabetically (2 and 20 both after 10).
          grep Searches the input lines for a specified string or regular expression, filtering out lines that don't (or do -- see -v option) match.  Named for the "vi" command "g/re/p" where g means search globally in the file, "re" is any regular expression, and p means print the matching lines.
          uniq Discards consecutive duplicate lines
          tr Translates characters from one set to another.
          sed Stream editor.  Applies a specified sequence of editing commands to each line.
          awk More powerful editor.  Named for its authors Aho, Weinberger, and Kernighan.
          cat Catenates (concatenates) multiple files into one.
          tee "Tee" pipe fitting.  Copies standard input to 2 places:  standard output and the specified file. 
          head Shows only the first few lines.
          tail Shows only the last few lines.

          And there are many, many more.  Almost all Unix commands read from standard input, write to standard output, or both.  However, for convenience when you're not thinking of them as filters, many of these commands also allow their input and output files to be specified directly as command line parameters.  That's why you can display of the contents of a file via commands, like:

                  cat file.txt
                  more file.txt

          rather than having to use longer forms, like:

                  cat < file.txt
                  more < file.txt
                  cat < filename | more

          Use the man command to learn more about these Unix commands.

          DOS and Windows offer only 3 standard filters:

          Filter What it does
          more Same as Unix more, but far less flexible.  Until Win2000, any keystroke was treated as space (one more screen).
          sort Sorts the input lines alphabetically.
          find Same as Unix grep, but far less flexible.  No regular expression support, and far fewer options.

          Some of the newer (NT-based) versions of Windows offer a help command that can tell you more about these commands.

          In both Unix and Windows, you can write additional filters of your own.

          Thanks to Tom Stluka for reminding me of the sort options!

          --Fred

           

        9. Command Substitution

          Last Updated: 2/24/2002
          Applies to: All shells, All Unix flavors

          Sometimes you want to redirect the output of a command into the command line of another command, not into its standard input.  You can do this as:

                  cc `ls *.c`

          Which causes the list of filenames produced by the ls *.c command to be used as command line parameters to the cc command (the C compiler).  This compiles all the C source files in the current directory.

          Note:  You must use the backtick character (`), not the apostrophe ('). On most keyboards, it is just left of the number 1 key, with tilde (~) as its shift character.  

          This form is supported by all Unix shells, but not by DOS or Windows.  However, in NT-based versions of Windows, there is a similar construct, using the FOR /F command.  For details, see:

                  Windows.htm#parsing_command_output

          --Fred

           

        10. tee

          Last Updated: 4/5/2002
          Applies to: All shells, All Unix flavors

          You can split the output of a command into two identical copies with the tee filter, as:

                  make | tee make.log

          This sends one copy of the standard output of the make command to the file make.log and another copy to standard output (the screen).  For commands like make, which may take a long time to run, but may generate output continuously, this is a useful way to monitor the progress (by watching it on the screen) while still creating a log file (in case you look away at a critical moment, or need a permanent record).

          The tee filter is standard in all flavors of Unix, but not in DOS or Windows.  

          Good news and bad news...  The good news is that tee is a very simple program that you can write yourself.  Write a simple loop that reads from standard input, and writes each line or character to both standard output and a named file.  A simple C version looks like:

                  while (c = getchar())
                  {
                      putchar(c);
                      fputc( c, fp);
                  }

          The bad news is that, on some versions of Windows, it still won't do what you expect.  In Unix and in NT-based versions of Windows (WinNT, Win2000, WinXP), each command in a pipeline runs as a separate process.  They all run concurrently.  Therefore, the tee program is running at the same time as the make program, so it can write output to the log file and to the screen as that output is generated by make.  This is what you want because it allows you to monitor the progress of the long-running make command.  However, in DOS and non-NT-based version of Windows (Win 3.1, Win95, Win98, WinME), the commands in a pipeline run sequentially, not concurrently.  Therefore, this pipeline runs the make command first, sending the output to a temporary holding area.  Then it runs the tee command, copying the output to the screen and to the log file.  You see no output at all until the make command is completed, so this is useless as a way to monitor the progress of the make command.

          Thanks to Andy Glick for pointing out that the NT-based versions of Windows *do* run pipelines concurrently.

          --Fred

           

        11. set noclobber

          Last Updated: 2/24/2002
          Applies to: All shells, All Unix flavors

          To protect yourself from overwriting a critical file by accidentally redirecting output to it, you may want to use the command:

                  set noclobber

          This command adds error checking to file redirections.  When the noclobber variable is set, the form:

                  command > filename

          fails if the specified file (to be created) already exists, and the form:

                  command >> filename

          fails if the specified file (to be appended to) does not already exist.

          I recommend setting this variable in your Unix startup file.  You can override it explicitly, in csh-based shells with the forms:

                  command >! filename
                  command >>! filename

          and in sh-based shells with the forms:

                  command >| filename
                  command >>| filename

          This variable is supported by all Unix shells, but not by DOS or Windows.

          --Fred

           

        12. Standard Input, Standard Output, and Standard Error

          Last Updated: 3/3/2002
          Applies to: All shells, All Unix flavors

          So far, we've covered standard input and standard output.  Unix defines another output stream called "standard error".  By convention, programs write their regular output to standard output, but their error messages to standard error.  You may never have noticed because both streams default to the screen.  Thus, if the current directory contains one or more files that have names starting with the letter f, but no file named crapola, the command:

                  ls f* crapola

          writes the names of the files starting with f to standard output, and an error message about crapola not existing to standard error.  The output looks like:

                  ls: crapola: No such file or directory
                  f1 f2 f3

          and the difference between standard output and standard error doesn't matter much.  

          However, once you start redirecting standard output to a file or pipe, you may notice that error messages don't get redirected.  For example:

                  ls f* crapola > out.txt

          sends the names of the files starting with f to the file out.txt, but the error message about crapola still comes to the screen.

          In sh-based shells, you can explicitly refer to the 3 files streams by using the associated "file descriptor" numbers 0 (standard input), 1 (standard output), and 2 (standard error).  Thus:

                  sort 0< in.txt
                  ls 1> out.txt
                  ls 2> out.txt

          The first two are more explicit forms of the same thing without the 0 or 1:

                  sort < in.txt
                  ls > out.txt

          but the third form allows you to explicitly redirect standard error.  Thus, the command:

                  ls f* crapola 1> out.txt 2> err.txt

          sends the names of the files starting with f to the file out.txt, and the error message about crapola to the file err.txt.

          Windows (and I think, DOS) also support the concept of standard error, but programs written for DOS and Windows are pretty lax about distinguishing between them.  You'll find many programs that write errors to standard output, or non-errors to standard error.  For programs written correctly, you can use the same syntax as the sh-based shells.  

          I tried to re-create the example above for DOS/Windows using dir instead of the Unix ls to show the names of files, as:

                  dir f* crapola 1> out.txt 2> err.txt

          However, dir has an unfortunate "feature".  Instead of reporting an error when any of the specified files is missing, it reports an error only if all of the specified files are missing.  So the fact that there were filenames starting with f prevented the non-existence of crapola from being reported at all.  No error messages -- therefore not a good example to use here.  

          Then I tried re-creating the example using the type command (show the contents of the files), as:

                  type f* crapola 1> out.txt 2> err.txt

          This works better, but shows that the type command is also written in an "interesting" way.  It sends the contents of the files to standard output, but the names of the files to standard error, along with the error messages.  In any case, you can play with it, and get the idea.

          --Fred

           

        13. Redirecting Standard Output and Error Together

          Last Updated: 4/5/2002
          Applies to: All shells, All Unix flavors

          Sometimes, you want to redirect standard output and standard error to the same file, so that you can see the error messages intermixed with the regular output to give some context to the error messages.  You can do this, in csh-based shells (csh and tcsh), by adding an ampersand (&) to the regular redirections and pipes, as:

                  ls f* crapola >& out.txt
                  ls f* crapola >>& out.txt
                  ls f* crapola |& more

          and in sh-based shells (sh, bash, ksh) as:

                  ls f* crapola > out.txt 2>&1
                  ls f* crapola >> out.txt 2>&1
                  ls f* crapola 2>&1 | more

          The special syntax 2>&1 says "redirect file descriptor 2 (standard error) to the same place as file descriptor 1 (standard output)".  Note that the 2>&1 occurs after the redirections (in the first and second lines), but before the pipe (in the 3rd line).

          NT-based versions of Windows use the same syntax as the sh-based shells:

                  type f* crapola > out.txt 2>&1
                  type f* crapola >> out.txt 2>&1
                  type f* crapola 2>&1 | more

          Non-NT-based versions of Windows don't support this feature.

          The bash shell supports the syntax of the sh shell from which it is derived, but also supports one combination from the csh shell:

                  ls f* crapola >& out.txt

          and a slight variation formed by reversing the special characters:

                  ls f* crapola &> out.txt

          but not the other csh shell formats:  >>& and |&  and not their reverses:   &>> and &|.

          --Fred

           

        14. Redirecting Standard Error in csh

          Last Updated: 3/3/2002
          Applies to: csh-based shells, All Unix flavors

          The syntax for separately redirecting standard output and standard error:

                  ls f* crapola 1> out.txt 2> err.txt

          applies to sh-based shells (sh, bash, ksh) only.  However, you can achieve the same effect in csh-based shells (csh, tcsh) as:

                  (ls f* crapola > out.txt 2) >& err.txt

          This uses the general purpose grouping operators (parentheses) supported by all Unix shells (and DOS and Windows).  The ls f* crapola command is executed with its standard output redirected to out.txt.  Since its standard error is not redirected, it goes to the normal place.  However, the next part of the grouped command redirects standard error and whatever is left of the standard output (nothing, since it was already redirected) to err.txt.  The next effect is that standard output and standard error go to separate files.

          --Fred

           

        15. Reading and Writing the Same File

          Last Updated: 3/3/2002
          Applies to: sh-based shells, All Unix flavors

          The sh-based shells also support the following formats, which have no equivalent in csh-based shells or in DOS or Windows.

          Syntax (one line)  What it does
          command <> filename Same as command < filename, but opens filename for reading and writing.  Only useful if command writes to standard input.
          command 0<> filename Same as command <> filename.  A way of explicitly referring to standard input (file descriptor 0).
          command 1<> filename Same as command 1> filename, but opens filename for reading and writing.  Only useful if command reads from standard output.
          command 2<> filename Same as command 2> filename, but opens filename for reading and writing.  Only useful if command reads from standard error.

          --Fred

           

        16. Redirecting Other File Descriptors

          Last Updated: 3/3/2002
          Applies to: sh-based shells, All Unix flavors

          The sh-based shells also support the following formats, which have no equivalent in csh-based shells or in DOS or Windows.

          Syntax (one line)  What it does
          command n< filename Redirects file descriptor n of command to read from filename.  This is the general form of the 0< syntax shown above.  It is only useful for programs that read from file descriptors other than 0.
          command n> filename Redirects file descriptor n of command to write to filename.  This is the general form of the 1> and 2> syntax shown above.  It is only useful for programs that write to file descriptors other than 1 and 2.
          command n>> filename All the forms shown above (>, >>, >|, >>|, &>, <>, etc.) can explicitly use 0, 1, 2, or any other file descriptor n.  The number for the file descriptor is always placed immediately before the first < or > character.
          command n>| filename
          command n<> filename
          etc...

          --Fred

           

        17. DOS and Windows Redirection Gotchas

          Last Updated: 4/5/2002
          Applies to: DOS and Windows, not Unix

          There are a couple of other quirks to watch out for in the DOS and Windows implementation of command line redirection, especially older versions:

          1. Redirection and pipe characters are interpreted even inside comments.  I tripped across this one accidentally when I created a block comment header in a BAT file, and set it off with a "box", as:
                REM +----------------------------+
                REM | This BAT file does...      |
                REM +----------------------------+
            It tried to execute a command called "This".  Also, someone recently showed me a trick:
                REM > out.txt
            as a way to create an empty file by redirecting a comment (nothing) to it.  It didn't work on my Windows 2000 machine, but he had used it for years on older versions of DOS/Windows.
          2. You can't redirect the output of a BAT file from the command line that called the BAT file.  Instead, you have to redirect each line of the BAT file separately, from within the BAT file.

          These have both been fixed in the latest versions of Windows.

          Also, as described in tee, even in the latest of the non-NT-based versions, pipe segments run sequentially, not concurrently.

          Finally, beware that NT-based versions of Windows have 2 different command line interpreters:

          1. CMD.EXE - The newer NT-based command line interpreter with more advanced features.
          2. COMMAND.COM - The older pre-NT version, still provided for compatibility with older versions of DOS and Windows.

          and CMD.EXE has a set of "command extensions" that can be enabled or disabled.  For best results, use CMD.EXE with command extensions enabled.

          --Fred

           

  2. Unix Shells

    There are several different Unix shells (sh, csh, tcsh, ksh, bash, etc.)   Each is a command line interpreter that can be run under any Unix flavor (BSD, System V, Solaris, HP/UX, Linux, BSDI, SCO Unix, etc.).  They have similar but not identical syntax, predefined commands (ls, cd, set, etc.), environment variable settings, etc.  They were each written by a different set of authors, but were intended to be compatible with all Unix flavors.  For the most part, all shells are included with the distribution of each Unix flavor. 

    Any Unix program (cc, make, touch, more, etc.) can be launched from any of the shells.  Once running, it is under the control of the Unix flavor, not the shell, so a Unix program should behave the same with all shells, but not necessarily the same with all flavors. 

    The first shell was sh, the Bourne Shell, named for it's author.   Later, csh, the C Shell, was written to be an improvement on sh, using a syntax more like that of the C programming language in its scripts.  More recently, a couple of sh deriviatives have appeared:  ksh, the Korn Shell, and bash, the Bourne Again Shell.  These attempt to be strict supersets of sh, adding features, but maintaining 100% compatibility with sh.  Also, a csh derivative has appeared:  tcsh, the T C Shell, which attempts to be a strict superset of csh, adding command line editing and filename completion features.

    All shells are still considered current, and are widely used; none has successfully replaced another.  It is common to find different users using different shells at the same time on any Unix system.  In fact, it is common to find the same user using different shells at the same time.  In the 1980's I typically had multiple windows open on my Unix workstation, running csh in most of them, but sh in one.  Also, even though I typically used csh interactively, I wrote most shell scripts in sh.  Since the first line of each shell script identifies the shell that it is written in, you can run a script written in any shell language from any other interactive shell, or even call a script in one language from a script in another language.  These days, I tend to use tcsh interactively, and write scripts in perl.

    1. csh and tcsh

      1. csh and tcsh Command Line

        1. set cdpath - Navigating directories

          Last Updated: 10/4/1999
          Applies to: csh, tcsh
          See also:  CDPATH in ksh

          The shell variable cdpath specifies the directories in which the cd and pushd commands look for subdirectories.  For example, in my .cshrc file, I set:

          	set cdpath = (.. ~ ../.. ../../.. \
                              ~/ste/dat ~/ste/layout ~/ste/adt ~/ste)

          Then, when I type:

          	cd src

          it looks for the following (in order):

          	./src    (always the first place checked)
          	../src
          	~/src
          	../../src
          	../../../src
          	~/ste/dat/src
          	~/ste/layout/src
          	~/ste/adt/src
          	~/ste/src

          This makes it much less tedious to navigate a complex directory tree.

          --Fred

        2. set filec - Filename completion

          Last Updated: 5/12/2000
          Applies to: csh, tcsh

          Setting the shell variable filec enables filename completion in csh.   For example, in my .cshrc file, I put:

          	set filec

          Then, when I type a unique prefix of a filename and hit ESC, it completes the filename.  If the prefix I typed is not unique to a single file, it does nothing.  Hitting Ctrl-D after a non-unique prefix shows all possible matches.  Examples:

          	cat abc<ESC>		# Local file starting with abc
          	/usr/ucb/m<Ctrl-D>	# All files in /usr/ucb/ starting with m
          	/usr/ucb/mer<ESC>	# /usr/ucb/merge
          	ls ~f<ESC>		# Home directory of user starting with f

          For more details, see man csh.

          A similar capability exists in tcsh using TAB (not ESC) and operating on commands as well as filenames.  It is enabled by default, so you don't need to set the filec variable. 

          If you type a prefix of a command and hit TAB, tcsh completes the command if the prefix is unique for commands on the PATH.  Hitting Ctrl-D shows all matches.  Setting the autolist shell variable causes TAB to show all matches, making Ctrl-D unnecessary.

          Similarly for filenames, if you type a command, then a space, then a unique prefix of a filename, then TAB, it completes the filename.  Ctrl-D shows all matches.  Setting autolist makes Ctrl-D unnecessary.

          For more details, see man tcsh.

          For both shells, the fignore variable can be used to specify extensions of files that should be skipped when looking for unique matches to the prefixes.

          This makes it much less tedious to type long detailed filenames.

          --Fred

        3. set prompt - Setting the Prompt

          Last Updated: 9/14/2001
          Applies to: csh, tcsh

          Setting the shell variable prompt changes the prompt string on the command line.  You can arrange for the prompt string to always reflect the current working directory or any other piece of dynamic information by aliasing the commands that change that dynamic information to have the side effect of also updating the prompt variable.  As an extreme example, here is the code from my .cshrc file to maintain the prompt:

          	# ---------------------------------------------------------------------------
          	# These commands maintain the prompt with the following features:
          	# - History number is displayed in the prompt [Now commented out --Fred]
          	# - Time at which this directory was entered is displayed.
          	# - Your username (or the one you are currently su'd to) is displayed.
          	# - Machine name is displayed.
          	# - Full name of current directory is displayed.
          	#   - Your own home directory when in the prompt is always displayed as 
          	#	 "~" (not /home/fred, for example).
          	#   - Other people's main directories are displayed like "~zebrom00".
          	#   - When you switch to a directory via a soft link, the link path is
          	#	 displayed instead of the real path.  Use pwd to see the real path.
          	# - Number of nested shells is indicated (in tcsh, not csh).
          	# - Number of directories stacked by pushd/popd, if any, is displayed.
          	# They are needed only in interactive mode.
          	# Note:  Moving these to non-interactive section would cause C shell to 
          	#        lose ability to distinguish between interactive and non-intractive
          	#        shells because the prompt would always be defined.
          	# ---------------------------------------------------------------------------
          	if ($?prompt) then
          	   alias newdir1 'set noglob;set d=(`dirs`);unset noglob'
          	                                          # Noglob needed to prevent ~ from being 
          	                                          # expanded
          	   alias newdir2 'set n="";@ m = $#d - 1;if ($m > 0) set n="$m"; unset m'
          	   alias newdir3 'set t=`date`;set t=`echo "$t[4]x" | sed "s/:..x//"`'
          	   alias newdir4 'set c=`whoami`@`hostname -s`'
          	   alias newdir5 'set s=""'
          	   if ($?shlvl) then
          	      alias newdir5 'set s="";if ($shlvl > 0) set s="$shlvl/"'
          	   endif
          	#   alias newdir6 'set prompt="\\! $t $c [$s$n] $d[1] % ";unset c t d n s'
          	   alias newdir6 'set prompt="$t $c [$s$n] $d[1] % ";unset c t d n s'
          	   alias newdir 'newdir1;newdir2;newdir3;newdir4;newdir5;newdir6'
          	                                          # Set prompt
          	   alias cd 'set old=$cwd;chdir \!*;newdir'
          	                                          # Remember old dir, change dir and 
          	                                          # update prompt
          	   alias back 'set back=$old;cd $back;unset back'
          	                                          # Toggle back to previous dir and 
          	                                          # update prompt
          	   alias pushd 'pushd \!*;newdir'         # Push dir and update prompt
          	   alias popd 'popd \!*;newdir'           # Pop dir and update prompt
          	   alias up 'cd ..'                       # Move up one dir, updating prompt
          	   alias dn 'cd ./\!*'                    # Move down to subdir, updating prompt
          	   alias ac 'cd ../\!*'                   # Move across to sibling dir, updating
          	                                          # prompt
          	   alias pd pushd                         # Push dir and update prompt
          	   alias pop popd                         # Pop dir and update prompt
          	   alias md  mkdir
          	   alias rd  rmdir
          	   alias mcd 'md \!* && cd \!*'   # Create and move to directory
          	endif

          Note:  Beware of the -s option of hostname.  It doesn't exist on all Unix systems, in which case the -s is interpreted as the first argument to hostname, causing it to try to change the name of the computer.  An error message is reported unless you are a superuser.   Solaris (and other System V derivatives?) has no -s.  Apollo Domain/IX and BSDI (and other BSD derivatives?) do have a -s.  Linux also has a -s.  Others?

          --Fred

        4. alias - Define a new command

          Last Updated: 5/26/2000
          Applies to: csh, tcsh

          You can redefine an existing command with the alias command and restore its original meaning with the unalias command.  For example, I add the following lines to my .cshrc file:

          	alias cp cp -i		# Prompt before overwriting
          	alias mv mv -i		# Prompt before overwriting
          	alias rm rm -i		# Prompt before removing each file
          	alias diff diff -l -s	# Long format, list files that are same

          Commands redefined in this way are available only at the interactive command line, not in shell scripts, so they don't break existing shell scripts by changing the behavior of the commands used in the scripts. 

          To bypass the alias, simply precede the command with a backslash:

          	rm *		# Prompts before removing each file
          	\rm *		# Doesn't prompt

          You can also use alias and unalias to define new commands of your own.  For example, I add the following lines to my .cshrc file:

          	alias pd pushd
          	alias po popd
          	alias dir ls -Fla
          	alias del rm
          	alias search grep
          	alias type cat
          	alias up 'cd ..'	# Move up one dir
          	alias dn 'cd ./\!*'	# Move down to subdir
          	alias ac 'cd ../\!*'	# Move across to sibling dir

          Notice that you use exclamation point (a.k.a. "bang"), which is the history mechanism, to insert the command line parameters used on the alias into the definition of the alias (escaped with a backslash to prevent the insertion from happening while we are still defining the alias).  Here, we use "!*" to insert all command line parameters.  See "History substitutions" in man csh for details.

          Again, such commands are not available in shell scripts.  To define commands for use in a script, you must write an additional short script for each command.  For example, create a file named dir, containing the lines:

          	#!/bin/csh -f
          	ls $*

          put it in a directory on the PATH, make it executable via the chmod command, and use the rehash command to update the command cache.

          Notice that you use dollar sign to expand command line parameters used on a shell script into the body of the shell script.  Here we use "$*" to insert all command line parameters.  See "Variable substitution" in man csh for details.

          --Fred

        5. alias dirr - Search a directory tree for a file

          Original Version: 5/26/2000
          Last Updated: 9/18/2008
          Applies to: csh, tcsh

          Here's an alias to search the current directory and all subdirectories for a file with a specified name:

          	alias dirr 'find . -name \!* -print | sort'

          Wildcards can be used, but then the parameter must be enclosed in double quotes.  Examples:

          	cd /starting/point/of/search
          	dirr file.txt
          	dirr "*.txt"

          Note: This is different from ls -R because it does not follow symbolic links, and because it searches the entire current directory tree for files that match the pattern, rather than searching the subdirectories that match the pattern for all files.

          --Fred

        6. alias confirm - Apply a command to multiple files

          Last Updated: 6/16/2000
          Applies to: csh, tcsh

          Here's an alias to execute a specified command against all specified files in the current directory and all subdirectories:

          	alias confirm '(set echo;find . -name \!:2 -ok \!:1 "{}" ";")'

          It applies the command specified as the first parameter to the files specified as the second parameter (wildcards allowed) in the current directory tree, prompting for confirmation at each file. You must enclose the filename in double quotes if you use wildcards.  It displays the find command that it generates before executing it, which is useful if you are trying to learn how to use find.   Examples:

          	cd /starting/point/of/search
          	confirm rm tempfile
          	confirm rm "*.tmp"
          	confirm cc "*.c"

          If you want the same capability without being prompted at each file, use -exec instead of -ok, as:

          	alias noconfirm '(set echo;find . -name \!:2 -exec \!:1 "{}" ";")'

          --Fred

      2. csh and tcsh Shell Scripts

        1. coming soon...

      3. csh and tcsh Environment

        1. Startup Files

          Last Updated: 1/13/2001
          Applies to: csh, tcsh

          An instance of csh begins by executing commands from the system file /etc/csh.cshrc and, if a login shell, /etc/csh.login. It then executes commands from the user's file ~/.cshrc, and, if a login shell, from ~/.login.   Then, it executes commands in the specified shell script file or from the interactive command line.  When a login shell terminates normally, it executes commands from the files ~/.logout and /etc/csh.logout.   See the csh man page for more details.

          The sequence for tcsh is similar.  It executes the same system files.  It also executes the same user files, except that if ~/.tcshrc exists, it executes that file instead of ~/.cshrc.  However, you can call ~/.cshrc from ~/tcshrc (via the source command) to run both.  Login shells execute the additional files:  ~/.history and ~/.cshdirs.   The system can be configured to run the system and user .login files before, instead of after, the .cshrc files.  See the tcsh man page for more details.

          --Fred

    2. sh

      1. sh Command Line

        1. coming soon...

      2. sh Shell Scripts

        1. coming soon...

      3. sh Environment

        1. Startup Files

          Last Updated: 1/13/2001
          Applies to: sh

          A login instance of sh begins by executing commands from the system file /etc/profile, and then the user's file ~/.profile.  It then follows the sequence followed by non-login instances, executing commands in the file named by the value of the ENV environment variable, if any.  There is no equivalent of the .logout files of csh.  See the sh man page for more details.

          --Fred

    3. bash

      1. bash Command Line

        1. coming soon...

      2. bash Shell Scripts

        1. coming soon...

      3. bash Environment

        1. Startup Files

          Last Updated: 1/13/2001
          Applies to: bash

          A login instance of bash begins by executing commands from the system file /etc/profile.   Then, it looks for ~/.bash_profile, ~/.bash_login, and ~/.profile, in that order, and reads and executes commands from the first one that exists and is readable.  It then follows the sequence followed by non-login instances, executing commands in the file ~/.bashrc.   Then, it executes commands in the specified shell script file or from the interactive command line.  When a login shell terminates normally, it executes commands from the file ~/.bash_logout.

          You can cause a shell script file to execute at the end of each shell (not only login shells), by adding the following line to your ~/.bashrc file:

              trap '. $HOME/.sh_logout; exit' 0

          Almost all of this behavior can be altered via options on the command line used to invoke bash.  There are options to tell bash to skip various files, to run different files, to run in posix mode, etc.  Also, bash follows the startup sequence of sh instead, if invoked by the verb "sh" (via a Unix link or by renaming the bash executable file to "sh").  Finally, it behaves differently if invoked as a remote shell via rshd.

          See the bash man page for more details.

          --Fred

    4. ksh

      1. ksh Command Line

        1. set CDPATH - Navigating directories

          Last Updated: 10/5/1999
          Applies to: ksh
          See also:  cdpath in csh and tcsh

          The parameter CDPATH specifies the directories in which the cd command looks for subdirectories.  For example, in my .profile file, I set:

          	export CDPATH=".:..:~:../..:../../..:~/ste/dat:~/ste/layout:~/ste/adt:~/ste"

          Then, when I type:

          	cd src

          it looks for the following (in order):

          	./src
          	../src
          	~/src
          	../../src
          	../../../src
          	~/ste/dat/src
          	~/ste/layout/src
          	~/ste/adt/src
          	~/ste/src

          This makes it much less tedious to navigate a complex directory tree.

          Note:  If you set CDPATH, and don't include a "." entry or a null entry (2 consecutive colons, or a leading or trailing colon), the cd command does not search the current directory.

          Thanks to David "Thor" Collard for this tip.

          --Fred

      2. ksh Shell Scripts

        1. coming soon...

      3. ksh Environment

        1. Startup Files

          Last Updated: 1/13/2001
          Applies to: ksh

          A login instance of ksh begins by executing commands from the system file /etc/profile, and then the user's file ~/.profile.  It then follows the sequence followed by non-login instances, executing commands in the file named by the value of the ENV environment variable, defaulting to the file ~/.kshrc

          There is no equivalent of the .logout files of csh, but you can mimic the effect, by adding the following line to your ~/.profile file:

              trap '. $HOME/.sh_logout; exit' 0

          See the ksh man page for more details.

          --Fred

  3. Unix Flavors

    There are several different Unix flavors (BSD, System V, Solaris, HP/UX, Linux, BSDI, SCO Unix, etc.)  Each is a different version of the operating system, written by a different set of authors, but intended to be compatible. You can run any of the Unix shells (sh, csh, tcsh, ksh, bash, etc.) under any Unix flavor.  The flavors ship with similar, but not identical, sets of programs (cc, make, touch, more, etc.)  Any Unix program can be launched from any of the shells.  Once running, it is under the control of the Unix flavor, not the shell, so a Unix program should behave the same with all shells, but not necessarily the same with all flavors.

    The name Unix is a reaction to the Multics system of the time that was considered too complex.  System V (System Five) was written by AT&T, presumably as a follow on to Systems I, II, III, and IV.  BSD (Berkeley Software (Systems?) Distribution) originates from UCB (University of California at Berkeley).  For many years, these were the only two flavors.  The source code for Unix was freely distributed, but support was not always provided.  Eventually, companies started producing, selling, and offering support for their own versions.  Solaris is a System V derivative from Sun MicroSystems.  HP/UX is a BSD (I think) deriviative from Hewlett Packard.  SCO Unix (derived from ?) and BSDI (obviously a BSD deriviative) are ports to the IBM PC (Intel x86).  And everyone's heard of GNU/Linux, the wildly popular free version, with the Linux kernel written by Linus Torvald, and the GNU utilities provided by the Free Software Foundation.

    For more info, see:  http://www.unix-systems.org/what_is_unix/flavors_of_unix.html

  4. Unix Utilities

    1. perl Scripts

      1. See also

        Bristle Software Perl Tips

©Copyright 1998-2008, Bristle Software, Inc.  All rights reserved