Shell


What is Shell?

Quoting from wikipedia

A Unix shell is a command-line interpreter or shell that provides a traditional Unix-like command line user interface. Users direct the operation of the computer by entering commands as text for a command line interpreter to execute, or by creating text scripts of one or more such commands. Users typically interact with a Unix shell using a terminal emulator, however, direct operation via serial hardware connections, or networking session, are common for server systems. All Unix shells provide filename wildcarding, piping, here documents, command substitution, variables and control structures for condition-testing and iteration

  • Interprets user commands
    • from terminal, from a file or as a shell script
    • expand wildcards, command/variable substitution
  • Command history, command completion and command editing
  • Managing processes
  • Shell variables to customize the environment
  • Difference between shell, tty and console


Like any indispensible software, Shell has undergone transformation from the days of basic sh shell that was used in 1970s Unix. While bash is default shell in most distros and most commonly used, powerful and feature rich shells are still being developed and released

  • sh bourne shell (light weight Linux distros might come with sh shell only)
  • bash bourne again shell
  • csh C shell
  • tcsh tenex C shell
  • ksh Korn shell
  • zsh Z shell (bourne shell with improvements, including features from bash, tcsh, ksh)
  • cat /etc/shells displays list of login shells available in the current Linux distro
  • echo $SHELL path of current user's login shell

Further Reading


Wildcards

It is easy to specify complete filenames as command arguments when they are few in number. But suppose, one has to delete hundreds of log files, spread across different sub-directories? Wildcards, or also known as globbing patterns help in such cases, provided the filenames have a commonality to exploit. We have already seen regular expressions used in commands like grep and sed. Shell wildcards are similar but has fundamental and syntactical differences

  • * match any character, 0 or more times
    • as a special case, * won't match the starting . of hidden files and has to be explicity specified
  • ? match any character exactly 1 time
  • [aeiou] match any vowel character
  • [!aeiou] exclude vowel characters, i.e match a consonant
  • [!0-9] match any character except digits
  • [a-z] match any lower case alphabets
  • [0-9a-fA-F] match any hexademical character
  • {word1,word2} match either of the specified words
    • words can themselves be made of wildcards

Examples

  • ls txt* list all files starting with txt
  • ls *txt* list all files containing txt anywhere in its name
  • ls *txt list all files ending with txt in the current directory
  • ls -d .* list only hidden files and directories
  • rm *.??? remove any file ending with . character followed by exactly three characters
  • ls bkp/201[0-5] list files in bkp directory matching 2010/2011/2012/2013/2014/2015
  • echo *txt for dry runs, use echo command to see how the wildcard expands

Brace Expansion

  • ls *{txt,log} list all files ending with txt or log in the current directory
  • cp ~/projects/adders/verilog/{half_,full_}adder.v . copy half_adder.v and full_adder.v to current directory
  • mv story.txt{,.bkp} rename story.txt as story.txt.bkp
  • cp story.txt{,.bkp} to create bkp file as well retain original
  • mv story.txt{.bkp,} rename story.txt.bkp as story.txt
  • mv story{,_old}.txt rename story.txt as story_old.txt
  • touch file{1..4}.txt same as touch file1.txt file2.txt file3.txt file4.txt
  • touch file_{x..z}.txt same as touch file_x.txt file_y.txt file_z.txt
  • rm file{1..4}.txt same as rm file1.txt file2.txt file3.txt file4.txt
  • echo story.txt{,.bkp} displays the expanded version 'story.txt story.txt.bkp' , useful to dry run before executing actual command

Extended globs

From info bash, where pattern-list is a list of one or more patterns separated by a |

  • ?(pattern-list) Matches zero or one occurrence of the given patterns
  • *(pattern-list) Matches zero or more occurrences of the given patterns
  • +(pattern-list) Matches one or more occurrences of the given patterns
  • @(pattern-list) Matches one of the given patterns
  • !(pattern-list) Matches anything except one of the given patterns

To check if extglob is enabled or to enable/disable:

$ shopt extglob 
extglob            on

$ # unset extglob
$ shopt -u extglob 
$ shopt extglob 
extglob            off

$ # set extglob
$ shopt -s extglob 
$ shopt extglob 
extglob            on

Examples

$ ls
123.txt  main.c  math.h  power.log

$ echo +([0-9]).txt
123.txt

$ ls @(*.c|*.h)
main.c  math.h

$ ls !(*.txt)
main.c  math.h  power.log
$ ls !(*.c|*.h)
123.txt  power.log

Recursively search current directory and its sub-folders

Set globstar and prefix pattern with **/ to search recursively

$ find -name '*.txt'
./song_list.txt
./bar/f1.txt
./bar/baz/f2.txt

$ shopt -s globstar
$ ls **/*.txt
bar/baz/f2.txt  bar/f1.txt  song_list.txt

Further Reading


Redirection

By default all results of a command are displayed on the terminal, which is the default destination for standard output. But often, one might want to save or discard them or send as input to another command. Similarly, inputs to a command can be given from files or from another command. Errors are special outputs generated on a wrong usage of command or command name

  • < or 0< is stdin filehandle
  • > or 1> is stdout filehandle
  • 2> is stderr filehandle

Redirecting output of a command to a file

  • grep -i 'error' report/*.log > error.log create new file, overwrite if file already exists
  • grep -i 'fail' test_results_20mar2015.log >> all_fail_tests.log creates new file if file doesn’t exist, otherwise append the result to existing file
  • ./script.sh > /dev/null redirect output to a special file /dev/null that just discards everything written to it, whatever may be the size
  • explicitly override the setting of noclobber with the >| redirection operator

Redirecting output of a command to another command

  • ls -q | wc -l the 'pipe' operator redirects stdout of ls command to wc command as stdin
  • du -sh * | sort -h calculate size of files/folders, display size in human-readable format which is then sorted
  • ./script.sh | tee output.log the tee command displays standard output on terminal as well as writes to file

Combining output of several commands

  • (head -5 ~/.vimrc ; tail -5 ~/.vimrc) > vimrc_snippet.txt multiple commands can be grouped in () and redirected as if single command output
    • commands grouped in () gets executed in a subshell environment
  • { head -5 ~/.vimrc ; tail -5 ~/.vimrc ; } > vimrc_snippet.txt gets executed in current shell context
  • Command grouping

Command substitution

  • sed -i "s|^|$(basename $PWD)/|" dir_list.txt add current directory path and forward-slash character at the start of every line
    • Note the use of double quotes to perform command substitution
  • file_count=$(ls -q | wc -l) save command output to a variable
  • Command Substitution

Process Substitution

Redirecting error

  • xyz 2> cmderror.log assuming a non-existent command xyz, it would give an error and gets redirected to specified file
  • ./script.sh 2> /dev/null discard error messages

Combining stdout and stderr

Assume that the file 'report.log' exists containing the text 'test' and non-existing file 'xyz.txt'

Bash version 4+:

  • grep 'test' report.log xyz.txt &> cmb_out.txt redirect both stdout and stderr to a file
  • grep 'test' report.log xyz.txt &>> cmb_out.txt append both stdout and stderr to a file
  • ls report.log xyz.txt |& grep '[st]' redirect both stdout and stderr as stdin

Earlier versions:

  • grep 'test' report.log xyz.txt > cmb_out.txt 2>&1 redirect both stdout and stderr to a file
  • grep 'test' report.log xyz.txt 2> cmb_out.txt 1>&2 redirect both stdout and stderr to a file
  • grep 'test' report.log xyz.txt >> cmb_out.txt 2>&1 append both stdout and stderr to a file
  • ls report.log xyz.txt 2>&1 | grep '[st]' redirect both stdout and stderr as stdin

Redirecting input

  • tr a-z A-Z < test_list.txt convert lowercase to uppercase, tr command only reads from stdin and doesn't have the ability to read from a file directly
  • wc -l < report.log useful to avoid filename in wc output
  • < report.log grep 'test' useful to easily modify previous command for different command options, search patterns, etc
  • grep 'test' report.log | diff - test_list.txt output of grep as one of the input file for diff command
  • difference between << , <<< and < <

Using xargs to redirect output of command as input to another command

  • grep -rlZ 'pattern' | xargs -0 sed -i 's/pattern/replace/' search and replace only those files matching the required pattern (Note: search pattern could be different for grep and sed as per requirement)
    • the -Z option would print filename separated by ASCII NUL character which is in turn understood by xargs via the -0 option. This ensures the command won't break on filenames containing characters like spaces, newlines, etc
  • When to use xargs

Further Reading


Process Control

  • Process is any running program
    • Program is a set of instructions written to perform a task
  • Daemon to simply put, are background processes
  • Job in Shell parlance is a process that is not a daemon, i.e an interactive program with user control

ps

report a snapshot of the current processes

kill

send a signal to a process

top

display Linux processes

  • Press M (uppercase) to sort the processes by memory usage
  • Press q to quit the command
  • Press W (uppercase) to write your favorite view of top command to ~/.toprc file and quit immediately, so that next time you use top command, it will display in the format you like
  • htop is better/prettier alternative to top
    • install instructions here
  • top Q&A on unix stackexchange

free

Display amount of free and used memory in the system

  • free -h shows amount of free and used memory in human readable format

pgrep

look up or signal processes based on name and other attributes

  • pgrep -l foobar search for process names containing foobar, displays PID and full process name
  • pgrep -x gvim search for processes exactly named gvim
  • pgrep -c chrom total number of processes matching chrom
  • pgrep -nl chrom most recently started process matching chrom


Further Reading


Running jobs in background

Often commands and scripts can take more than few minutes to complete, but user might still need to continue using the shell. Opening a new shell might not serve the purpose if local shell variable settings are needed too. Shell provides the & operator to push the commad (or script) execution to background and return the command prompt to the user. However, the standard outputs and errors would still get displayed on the terminal unless appropriately redirected

  • tkdiff result_v1.log result_v2.log & tkdiff, if installed, shows differences between two files in a GUI. If & is not used, the program would hog the command prompt

Pushing current job to background

What if you forgot to add & and using kill on the process might corrupt lot of things?

  • Ctrl+z suspends the current running job
  • bg push the recently suspended job to background
  • Continue using shell
  • fg bring the recently pushed background job to foreground
  • jobs built-in command - Display status of jobs
  • nohup command - run a command immune to hangups, with output to a non-tty
  • job control

results matching ""

    No results matching ""