Shell Scripting


Need for scripting

Note:

  • .sh is typically used as extension for shell scripts
  • Material presented here is for GNU bash, version 4.3.11(1)-release


Hello script

#!/bin/bash

# Print greeting message
echo "Hello $USER"
# Print day of week
echo "Today is $(date -u +%A)"

# use single quotes for literal strings
echo 'Have a nice day'

The first line has two parts

  • /bin/bash is path of bash
    • type bash to get path
  • #! called as shebang), directs the program loader to use the interpreter path provided

Comments

  • Comments start with #
  • Comments can be placed at end of line of code as well
    • echo 'Hello' # end of code comment
  • Multiline comments

Single quotes vs Double quotes

  • Single quotes preserves the literal value of each character within the quotes
  • Double quotes preserves the literal value of all characters within the quotes, with the exception of '$', '`', '\', and, when history expansion is enabled, '!'
  • Difference between single and double quotes

echo builtin command

  • help -d echo Write arguments to the standard output
  • By default, echo adds a newline and doesn't interpret backslash
  • -n do not append a newline
  • -e enable interpretation of the following backslash escapes
  • -E explicitly suppress interpretation of backslash escapes
  • echo Q&A on unix stackexchange
$ chmod +x hello_script.sh
$ ./hello_world.sh 
Hello learnbyexample
Today is Wednesday
Have a nice day


Sourcing script

$ help -d source
source - Execute commands from a file in the current shell.
  • If script should be executed in current shell environment instead of sub-shell, use the . or source command
    • For example, after editing ~/.bashrc one can use source ~/.bashrc for changes to be immeditely effective
$ # contents of prev_cmd.sh
prev=$(fc -ln -2 | sed 's/^[ \t]*//;q')
echo "$prev"
  • For example, to access history of current interactive shell from within script
$ printf 'hi there\n'
hi there
$ bash prev_cmd.sh 

$ printf 'hi there\n'
hi there
$ source prev_cmd.sh
printf 'hi there\n'


Command Line Arguments

#!/bin/bash

# Print line count of files given as command line argument
echo "No of lines in '$1' is $(wc -l < "$1")"
echo "No of lines in '$2' is $(wc -l < "$2")"
  • Command line arguments are saved in positional variables starting with $1 $2 $3 etc
  • If a particular argument requires multiple word string, enclose them in quotes or use appropriate escape sequences
  • $0 contains the name of the script itself - useful to code different behavior based on name of script used
  • $@ array of all the command line arguments passed to script
  • $# Number of command line arguments passed to script
  • Use double quotes around variables when passing its value to another command
  • bash special parameters reference
$ ./command_line_arguments.sh hello_script.sh test\ file.txt 
No of lines in 'hello_script.sh' is 9
No of lines in 'test file.txt' is 5


Variables and Comparisons

  • dir_path=/home/guest space has special meaning in bash, cannot be used around = in variables
  • greeting='hello world' use single quotes for literal strings
  • user_greeting="hello $USER" use double quotes for substitutions
  • echo $user_greeting use $ when variable's value is needed
  • no_of_lines=$(wc -l < "$filename") use double quotes around variables when passing its value to another command
  • num=534 numbers can also be declared
  • (( num = 534 )) but using (( )) for numbers makes life much easier
  • (( num1 > num2 )) number comparisons are also more readable within (( ))
  • [[ -e story.txt ]] test if the file/directory exists
  • [[ $str1 == $str2 ]] for string comparisons

Further Reading


Accepting User Input interactively

#!/bin/bash

# Get user input
echo 'Hi there! This script returns the sum of two numbers'
read -p 'Enter two numbers separated by spaces: ' number1 number2

echo -e "\n$number1 + $number2 = $((number1 + number2))"
echo 'Thank you for using the script, Have a nice day :)'
  • help -d read Read a line from the standard input and split it into fields
  • -a array assign the words read to sequential indices of the array variable ARRAY, starting at zero
  • -p prompt output the string PROMPT without a trailing newline before attempting to read
  • -s do not echo input coming from a terminal
  • More examples with read and getting input from stdin
$ ./user_input.sh 
Hi there! This script returns the sum of two numbers
Enter two numbers separated by spaces: 7 42

7 + 42 = 49
Thank you for using the script, Have a nice day :)


if then else

#!/bin/bash

if (( $# != 2 ))
then
    echo "Error!! Please provide two file names"
    # simple convention for exit values is '0' for success and '1' for error
    exit 1
else
    # Use ; to combine multiple commands in same line
    # -f option checks if file exists, ! negates the value
    # white-space around [[ and ]] is necessary
    if [[ ! -f $1 ]] ; then
        echo "Error!! '$1' is not a valid filename" ; exit 1
    else
        echo "No of lines in '$1' is $(wc -l < "$1")"
    fi

    # Conditional Execution
    [[ ! -f $2 ]] && echo "Error!! '$2' is not a valid filename" && exit 1
    echo "No of lines in '$2' is $(wc -l < "$2")"
fi
  • When handling user provided arguments, it is always advisable to check the sanity of arguments. A simple check can reduce hours of frustrating debug when things go wrong
  • The code inside if [[ ! -f $1 ]] ; then block is only intended for demonstration, we could as well have used error handling of wc command if file doesn't exist
  • Default exit value is 0 , so need not be explicitly written for successful script completion
  • Use elif if you need to test more conditions after if
  • The operator && is used to execute a command only when the preceding one successfully finishes
  • To redirect error message to stderr, use echo "Error!! Please provide two file names" 1>&2 and so on
  • Control Operators && and ||
  • More examples for if conditional block
$ ./if_then_else.sh 
Error!! Please provide two file names
$ echo $?
1

$ ./if_then_else.sh hello_script.sh 
Error!! Please provide two file names
$ echo $?
1

$ ./if_then_else.sh hello_script.sh xyz.tzt
No of lines in 'hello_script.sh' is 9
Error!! 'xyz.tzt' is not a valid filename
$ echo $?
1

$ ./if_then_else.sh hello_script.sh 'test file.txt' 
No of lines in 'hello_script.sh' is 9
No of lines in 'test file.txt' is 5
$ echo $?
0

Combining if with exit status of command executed

Sometimes one needs to know if intended command operation was successful or not and then take action depending on outcome. Exit status of 0 is considered as successful condition when used with if statement. When avaiable, use appropriate options to suppress stdout/stderr of command being used, otherwise redirection might be needed to avoid cluttering output on terminal

$ grep 'echo' hello_script.sh 
echo "Hello $USER"
echo "Today is $(date -u +%A)"
echo 'Have a nice day'

$ # do not write anything to standard output
$ grep -q 'echo' hello_script.sh 
$ echo $?
0

$ grep -q 'echo' xyz.txt
grep: xyz.txt: No such file or directory
$ echo $?
2
$ # Suppress error messages about nonexistent or unreadable files
$ grep -qs 'echo' xyz.txt
$ echo $?
2

Example

#!/bin/bash

if grep -q 'echo' hello_script.sh ; then
    # do something
    echo "string found"
else
    # do something else
    echo "string not found"
fi


for loop

#!/bin/bash

# Ensure atleast one argument is provided
(( $# == 0 )) && echo "Error!! Please provide atleast one file name" && exit 1

file_count=0
total_lines=0

# every iteration, variable file gets next positional argument
for file in "$@"
do
    # Let wc show its error message if file doesn't exist
    # terminate the script if wc command exit status is not 0
    no_of_lines=$(wc -l < "$file") || exit 1
    echo "No of lines in '$file' is $no_of_lines"
    ((file_count++))
    ((total_lines = total_lines + no_of_lines))
done

echo -e "\nTotal Number of files = $file_count"
echo "Total Number of lines = $total_lines"
  • This form of for loop is useful if we need only element of an array, without having to iterate over length of an array and using an index for each iteration to get array elements
  • In this example we use the control operator || to stop the script if wc fails i.e 'exit status' other than 0
$ ./for_loop.sh 
Error!! Please provide atleast one file name
$ echo $?
1

$ ./for_loop.sh hello_script.sh if_then_else.sh command_line_arguments.sh
No of lines in 'hello_script.sh' is 9
No of lines in 'if_then_else.sh' is 21
No of lines in 'command_line_arguments.sh' is 5

Total Number of files = 3
Total Number of lines = 35
$ echo $?
0

$ ./for_loop.sh hello_script.sh xyz.tzt
No of lines in 'hello_script.sh' is 9
./for_loop.sh: line 14: xyz.tzt: No such file or directory
$ echo $?
1

Index based for loop

#!/bin/bash

# Print 0 to 4
for ((i = 0; i < 5; i++))
do
    echo $i
done

Iterating over used defined array

$ files=('report.log' 'pass_list.txt')
$ for f in "${files[@]}"; do echo "$f"; done
report.log
pass_list.txt

Files specified by glob pattern

A common mistake is to use output of ls command which is error prone and needless. Instead, the arguments can be directly used.

$ ls
pass_list.txt  power.log  report.txt

$ for f in power.log *.txt; do echo "$f"; done
power.log
pass_list.txt
report.txt


while loop

#!/bin/bash

# Print 5 to 1
(( i = 5 ))
while (( i != 0 ))
do
    echo $i
    ((i--))
done
  • Use while when you need to execute commands according to a specified condition
$ ./while_loop.sh 
5
4
3
2
1


Reading a file

Reading line by line

#!/bin/bash

while IFS= read -r line; do
    # do something with each line
    echo "$line"
done < 'files.txt'
  • IFS is used to specify field separator which is by default whitespace. IFS= will clear the default value and prevent stripping of leading and trailing whitespace of lines
  • The -r option for read will prevent interpreting \ escapes
  • Last line from input won't be read if not properly terminated by newline character
$ cat files.txt 
hello_script.sh
if_then_else.sh
$ ./while_read_file.sh 
hello_script.sh
if_then_else.sh

Reading line as different fields

  • By default, whitespace is delimiter
  • Specify a different one by setting IFS
$ cat read_file_field.sh
#!/bin/bash

while IFS=: read -r genre name; do
    echo -e "$genre\t:: $name"
done < 'books.txt'
$ cat books.txt 
fantasy:Harry Potter
sci-fi:The Martian
mystery:Sherlock Holmes

$ ./read_file_field.sh 
fantasy :: Harry Potter
sci-fi  :: The Martian
mystery :: Sherlock Holmes

Reading 'n' characters at a time

$ while read -n1 char; do echo "Character read is: $char"; done <<< "\word"
Character read is: w
Character read is: o
Character read is: r
Character read is: d
Character read is: 

$ # if ending newline character is not desirable
$ while read -n1 char; do echo "Character read is: $char"; done < <(echo -n "hi")
Character read is: h
Character read is: i

$ while read -r -n2 chars; do echo "Characters read: $chars"; done <<< "\word"
Characters read: \w
Characters read: or
Characters read: d


Debugging

  • -x Print commands and their arguments as they are executed
  • -v verbose option, print shell input lines as they are read
  • set -xv use this command to enable debugging from within script itself
$ bash -x hello_script.sh 
+ echo 'Hello learnbyexample'
Hello learnbyexample
++ date -u +%A
+ echo 'Today is Friday'
Today is Friday
+ echo 'Have a nice day'
Have a nice day

$ bash -xv hello_script.sh 
#!/bin/bash

# Print greeting message
echo "Hello $USER"
+ echo 'Hello learnbyexample'
Hello learnbyexample
# Print day of week
echo "Today is $(date -u +%A)"
date -u +%A
++ date -u +%A
+ echo 'Today is Friday'
Today is Friday

# use single quotes for literal strings
echo 'Have a nice day'
+ echo 'Have a nice day'
Have a nice day


Real world use case

With so much copy-paste of commands and their output involved in creating these chapters, mistakes do happen. So a script to check correctness comes in handy. Consider the below markdown file

## <a name="some-heading"></a>Some heading

Some explanation

```bash
$ seq 3
1
2
3

$ printf 'hi there!\n'
hi there!
```

## <a name="another-heading"></a>Another heading

More explanations

```bash
$ help -d readarray
readarray - Read lines from a file into an array variable.

$ a=5
$ printf "$a\n"
5
```
  • The whole file is read into an array so that index of next line to be read can be controlled dynamically
  • Once a command is identified to be tested
    • the expected output is collected into a variable. Multiple lines are concatenated. Some commands do not have stdout to compare against
    • accordingly the index of next iteration is corrected
  • Note that this is a sample script to demonstrate use of shell script. It is not fool-proof, doesn't have proactive check for possible errors, etc
  • Be sure eval is being used for known commands like is the case here
  • See Parameter Expansion for examples and explanations on string processing constructs
#!/bin/bash

cb_start=0
readarray -t lines < 'sample.md'

for ((i = 0; i < ${#lines[@]}; i++)); do
    # mark start/end of command block
    # Line starting with $ to be verified only between ```bash and ``` block end
    [[ ${lines[$i]:0:7} == '```bash' ]] && ((cb_start=1)) && continue
    [[ ${lines[$i]:0:3} == '```' ]] && ((cb_start=0)) && continue

    if [[ $cb_start == 1 && ${lines[$i]:0:2} == '$ ' ]]; then
        cmd="${lines[$i]:2}"

        # collect command output lines until line starting with $ or ``` block end
        cmp_str=''
        j=1
        while [[ ${lines[$i+$j]:0:2} != '$ ' && ${lines[$i+$j]:0:3} != '```' ]]; do
            cmp_str+="${lines[$i+$j]}"
            ((j++))
        done
        ((i+=j-1))

        cmd_op=$(eval "$cmd")
        if [[ "${cmd_op//$'\n'}" == "${cmp_str//$'\n'}" ]]; then
            echo "Pass: $cmd"
        else
            echo "Fail: $cmd"
        fi
    fi
done
  • Note how sourcing the script is helpful to take into consideration commands dependent on previous commands
$ ./verify_cmds.sh 
Pass: seq 3
Pass: printf 'hi there!\n'
Pass: help -d readarray
Pass: a=5
Fail: printf "$a\n"

$ source verify_cmds.sh 
Pass: seq 3
Pass: printf 'hi there!\n'
Pass: help -d readarray
Pass: a=5
Pass: printf "$a\n"


Resource lists

The material in this chapter is only a basic introduction

Shell Scripting

Specific topics

Handy tools, tips and reference

results matching ""

    No results matching ""