&& The Bourne Again Shell -------------------------: quote: "bash scripts are dyslexic poetry" (anon) The Bourne Again Shell, or 'Bash' shell is one of the ways that a user may interact with the Linux Operating System. The Bash shell is extremely flexible and powerful since it has a full range of programming constructs, such as loops and tests as well as the ability to join commands in command chains using the pipe "|" symbol. The name 'Bourne Again' comes from the fact that the 'bash' shell was a rewrite of an earlier unix shell called the 'Bourne Shell'. GETTING HELP * view help for the built-in bash commands >> help * view help for the 'pwd' print working directory command >> help pwd * show all software about optical character recognition or speech >> apropos speech ocr | less [= image/eg-apropos-ocr.png] Optical character recognition means software which can 'recognise' written characters (letters) which form part of an image and can turn those characters into text. * display a short help message for all programs in the system >> whatis -r '.*' | less * show some installed programs having something to do with 'java' >> whatis -r java | less >> apropos -r java | less Apropos is better than 'whatis' because it actually searches the man pages for mentions of the given term, where as 'whatis' just looks at the short description of the program * show the version number of the bash shell which you are using >> echo $BASH_VERSION * Hold down the escape key for a few seconds to see all commands >> ESC WEB LINKS @@ http://tldp.org/LDP/abs/html/ a tutorial @@ www.commandlinefu.com lots of recipes @@ www.shelldorado.com more bash tips @@ http://www.bash-hackers.org/wiki/doku.php stuff about the bash shell @@ http://bashcurescancer.com/10-linux-commands-youve-never-used.html about little used but useful commands @@ www.stackoverflow.com This is a forum about any programming language, but it generally contains intelligent answers to bash questions. @@ http://www.faqs.org/faqs/unix-faq/shell/bash/ a bash faq TRAPS AND GOTCHAS This section details some unpleasant surprises awaiting the unwary bash novice. * be careful about white space in bash statements >> d=2 NOT d = 2 >> if [ $a -le 5 ] NOT if [ $a -le 5] * there must be a space after a { bracket >> for j in 1 2 3 4; {echo "j=$j"; } ##(NO!! doesnt work) * No!!! you need spaces after "$a" and before "$b", for some reason >> a='ss'; [ "$a"=="$b" ] && echo yes * scripts with MS Windows newlines \r\n will not run >> #!/bin/bash\r\n ##(doesnt work, use the 'dos2unix' tool) * the so called "back-ticks" (`) need to be escaped in shell scripts >> \`\` ##(the ` character is similar to the '$(...\)' construct) * echo doesnt show newlines. It removes the newlines from string variables >> s="first "$'\n'"2nd "$'\n'"3rd"; echo $s This prints "first 2nd 3rd" all on one line, BUT the variable "s" actually spans 3 lines. use 'echo "$s"' to actually show the real contents of the variable 's' with newline characters * cant use bash -v to get the version, use the following instead >> echo $BASH_VERSION * brace expansion cant have spaces * you cant background a job on the same line with another command >> ls &; clear ##(No!! this doesnt work) >> ls &; ##(No!! also doesnt work) * just omit the ';' semi-colon, but how does this work? >> ls & clear JARGON This section attempts to explain some of the technical terms which are often used when describing the bash shell. Many of these terms have a long (more than 30 years) tradition of use in Unix-type operating systems. @@ terminal An rs-232 serial device consisting of a keyboard and screen which communicates with a computer, or else software which emulates this. @@ console @@ shell This is a way of interacting with the operating system by typing a series of very cryptic and precise commands. Most users have an instinctive dislike and dread of this type of activity. @@ built-in command a command which is part of bash (does not need any external program to execute). You can view some help for these commands by typing 'help command' in a console. @@ variable A variable is an entity which may contain a value such as '2' or "The tree". The variable can change its value as often as is required (this is why its called a 'variable') @@ untyped and typed variables variables can contain various values and those values can be classified into 'types' such as text (also called 'strings') whole numbers (also called 'integers') or decimal numbers (also called 'floating point numbers') Bash normally uses 'untyped' variables because it does not really care what type of value the variable holds (whether the variable is text, an number or something else). The bash shell attempts to 'guess' what type of value a variable holds, and then tries to do something 'sensible' with that value. @@ variable interpolation This term refers to the process of inserting or substituting string (text) variables which are contained within other text variables. So if we have s=4; t="1 2 3 $s" then the value of the variable 't' will be '1 2 3 4' because the variable 's' was 'interpolated' within the variable 't'. This is a very handy feature of many modern scripting languages such as Perl, Ruby, Python etc @@ string In the world of programming a string refers to a chunk of text. This is because 'strings' were first thought of as a string of characters. @@ BSD Berkeley Standard Distribution of Unix. This is, more or less the version of Unix used with MacOSX and many of the standard programs are slightly different to their Gnu-Linux equivalents. For this reason, when writing bash scripts that need to run on both Mac OSX and Linux, the programmer must be sure to only use posix switches @@ FreeBSD A free version of the BSD Unix operating system. This is not a popular as Linux, but has the reputation for being more secure and better for older machines. FreeBSD may be capable of running on old 486 machines @@ Posix After the development of BSD Unix, there began to appear different versions of Unix which had small but significant differences. The Posix standard was an attempt to 'reunify' different Unix versions by specifying standards that all Unix type operating systems (including Linux) should implement. By conforming to posix standards, one is able to write bash shell scripts which should run correctly on a wide variety of different Unix-type systems, such as Mac OSX Linux and FreeBSD @@ globbing The bash shell expands certain special characters (such as '*') into a list of file names in the current folder. @@ brace expansion expands an expression containing braces '{}' into a series of numbers or files names. @@ script A script is a set of instructions contained within a text file in a format which is readable and understandable by a person (after some training) which is also readable and executable by a computer program in order to carry out some particular task. @@ backtics The 'backtic' (`) along with the more modern $() construct is used to insert the output of a command within another command or piece of text. It is an extemely powerful technique. The more modern $() may be nested but the older `` may not @@ option, switch These 2 terms which are equivalent are used to alter the behavior of the application or script. Options or switches are usually written with a hyphen before a letter (for example '-a' or '-b') or with 2 hyphen before a word or words (eg: '--list' or '--version') @@ short option The short option to a command is one hyphen followed by one letter (eg: '-a' or '-b') @@ long option an option to a command written with words rather than a single letter (eg --debug) @@ parameter Parameters are values which are passed to scripts and functions in order to provide information to the script or function. For example if a script is executed with './test.sh green white' then the first parameter has the value 'green' and the 2nd parameter has the value 'white' The parameters can be used within the script by accessing the variables $1, $2, ... etc. @@ bash function a function is a way of coding a task that is opten repeated. A function is slightly more flexible than an alias, and slightly less flexible than a script. @@ alias This is the simplest way to create new commands for tasks which you often carry out using the bash shell. Aliases cannot have parameters. @@ shebang line The 'shebang' line is the line at the top of a unix script which begins with '#!' and then a path. This line tells the unix or linux operating system what program to use to execute the script @@ executable, application, program All these terms are more or less equivalent and mean a file or files containing instructions which a computer can carry out in order to perform a particular task (such as surf the internet or write a document) @@ binary file @@ text file A text file represents a series of characters in some human writing system. Each character in the writing system (such as the chinese ideograph for 'rain' or the roman alphabet letter 'z' or the greek 'alpha' character) is given a unique number and is then encoded into a binary code which may be stored on a computer system. @@ character @@ text encoding This is the way that each character in a text file is transformed into a unique number (in binary code). Ascii was an old text encoding which was used for encoding the roman alphabet. Utf8 is a more modern encoding which is capable of encoding any character in any known human writing system. @@ Unicode Unicode is a very useful standard developed over a number of years by a committee to provide a way to categorise and enumerate all the characters in all the writing systems which are currently known in the world. This standard makes the job of writing text documents which can be used and read over the Internet in different countries, much easier. @@ locale Different human cultures use different writing systems to write documents, display numbers in different ways and use different languages. For example, in Colombia the number 2.200,33 means '2 thousand, 2 hundred and 1/3 (approximately) but in an Anglo-Saxon country, the number would be written 2,200.33. Linux attempts to cater for these different cultures using 'locales'. @@ interface The interface is the way that a human being interacts with a computer program. This is usually by typing text into small boxes, clicking 'buttons', choosing elements from lists and so on. An interface could also be based apon voice recognition and sound. In old unix programs the interface consists purely of text messages. @@ curses curses is a type of user interface which runs within a terminal. It allows the user to use the arrow keys to move around the screen. It is essentially a more 'primitive' version of graphical windows. @@ OTHER SHELLS While bash is considered the most popular and widely used shell for linux and other unixes, other shells exist. == different shells .. .. csh - a shell with a c like syntax .. ksh - the korn shell .. dash - a 'sh' implementation .. sh - an old shell .. fish - a simple non-compatible shell .. * find out the name of the running shell or script >> echo $0 PROCESSES AND JOBS Any unix-style operating system (and most modern operating systems) can do more than one thing at once. Each of these tasks is called a 'job' or a 'process'. There is a subtle difference but we wont worry about it right now. * run the command apropos shell > shell-commands as a background job >> apropos shell > shell-commands & * use export to make a variable available to subprocesses >> export name=value * pause a currently executing job >> [control] z * show what jobs the shell is executing >> jobs * stop the second job in the job list >> kill %2 ##(a job number is different from a process number) * stop the process with id '1423' >> kill -9 1423 * bring a job which is executing in the background to the foreground >> fg ##(if there is more than 1 background job, add a job-number) * put into the background a job running in the foreground >> bg * show all executing processes using 'unix' options >> ps -e * show the process id and initiating command of all processes >> ps -eo pid,comm * run a command which will not be interrupted when the user logs out. >> nohup command * run a command with a priority less than normal >> nice command * display a list box of running processes >> echo "$(ps -eo pid,comm | zenity --list --column= --height 800)" * display a list- box of running processes and kill the process chosen ---------------------------------------------------------------------- p="$(ps -eo pid,comm | zenity --list --column= --height=800)"; kill -9 $(echo $p | sed 's/ .*$//'); ,,, TEXT FILES * the right way to read a text file ----------------------------------- cat test.txt | while read l; do echo "$l" done ,,, * create a new file 'list' and type text into it >> cat > list - ##(press [control] + c to stop this) * read 3 characters from an input stream >> echo abcdef > junk.txt; read -n 3 a > for i in $(cat example.txt); do echo $i; done >> for i in $(> while read -r a ;do echo $a; done < file.txt * loop through the characters of a text file >> while read -n 1 a ;do echo $a; done < file.txt * insert "yes" at the 4th character in a text file >> exec 3<> file.txt ; read -n 4 <&3 ; echo -n yes >&3 ; exec 3>&- ; * a very simple text translation script ------------------------------------- exec 3< file.txt while read a <&3 do echo -ne "--\n$a\n>>"; read b; echo $b >> file.tr.txt done; exec 3<&- ,,, * a very simple text translation script with line numbers ------------------------------------- exec 3< test.txt n=1 while read a <&3 do echo -ne "$n--\n$a\n>>"; read b; echo $b >> eg.tr.txt ((n++)) done; exec 3<&- ,,, * create a small amount of random text for filling up documents >> cat /usr/share/dict/words | shuf | head -100 | tr '\n' ' ' >> shuf -n 100 /usr/share/dict/words | tr '\n' ' ' ##(same, better) * substitute the word 'blah' in 'f.txt' with lots of random text >> sed "s/blah/$(shuf -n 100 /usr/share/dict/words | tr '\n' ' ')/" f.txt * the same, but really do the substitution in the file & make a backup >> sed -i.bak "s/blah/$(shuf -n 100 /usr/share/dict/words | tr '\n' ' ')/" f.txt * append contents of the file 'file.txt' to itself >> cat file.txt | tee >> file.txt MERGING TEXT FILES .... * print lines between 2 markers >> awk '/abc/{flag=1;next}/mno/{flag=0}flag' file * replace text between 2 markers sed or awk ----- SED: View Raw Code sed -i -e "/^##START\$/,/^##STOP\$/c ##START\n${REPLACEMENT}\n##STOP" test AWK: View Raw Code ,,, awk "/##START/{p=1;print\"###START\n${REPLACEMENT}\n###STOP\";next}/##STOP/{p=0;next}!p" test.tmp > test To insert before the ###marker### line : // for each of second_file.txt : // if matches regexp ###marker###, outputs first_file.txt. // **without any condition :** print awk '/###marker###/ { system ( "cat first_file.txt" ) } \ { print; } \' second_file.txt To insert after the ###marker###line : // for each of second_file.txt : // **without any condition :** print // if matches regexp ###marker###, outputs first_file.txt. awk '{ print; } \ /###marker###/ { system ( "cat first_file.txt" ) } \' second_file.txt To replace the ###marker### line : // for each of second_file.txt : // if matches regexp ###marker###, outputs first_file.txt. // **else**, print awk '/###marker###/ { system ( "cat first_file.txt" ) } \ !/###marker###/ { print; }' second_file.txt If you want to do in-place replacement, use a temp file for being sure the pipe doesn't start before awk has read the entire file; add : > second_file.txt.new mv second_file.txt{.new,} // (like "mv second_file.txt.new second_file.txt", but shorter to type !) EDITING TEXT FILES .... The $EDITOR variable determines the default text editor. * Select and Edit a File in the Current Directory >> PS3="Enter a number: "; select f in *.txt;do $EDITOR $f; break; done But the command above breaks if any of the file names contain spaces You may be able to use the IFS variable to solve this. * choose a file to edit, even if the name contains a space >> IFS=$'\n'; PS3="Choose:"; select f in $(ls);do vim $f; break; done SEARCHING TEXT * search for comment lines in a file >> grep '^#' > grep -ril text * * make searches within the 'less' pager case-insensitive >> alias less='less -i' * A function to search through files for the given words -------------------------------------------------------- function has { help=" - searches for files containing a given text" if [ -z "$1" ]; then echo -e "usage: $FUNCNAME searchterm \n $help" return 3 fi find . | xargs grep -l "$*" | less } ,,, * view all the files containing text 'screen saver' >> has screen saver REMOTE DEVELOPMENT * using a remote shell >> putty (from MS Windows), ssh, telnet STREAMS The idea of a 'stream' is very important in unix and bash, since most work is carried out in this paradigm. Streams can be redirected and piped etc. * Append stdout and stderr to a file, and print stderr to the screen >> somecommand 2>&1 >> logfile | tee -a logfile * add some text to the begging and end of standard input >> echo hello | (echo yes; cat -; echo no) UNIX THREADS Unix provides several simple mechanisms for allowing 'threads' (also called processes, or running programs) to communicate between each other. This is probably the most powerful single feature of the unix operating system. see also: message queues, signals REDIRECTION .... * redirect stdout and stderr at the same time >> $> ##(only the bash shell) >> command 2>&1 ##(older style) * redirect both standard output error messages to the file "output" >> command > output.txt 2>&1 * Pipe stdout and stderr, etc., to separate commands >> some_command >>(/bin/cmd_for_stdout) 2>>(/bin/cmd_for_stderr) * redirect error messages to "stdout" and output to the file "save.txt" >> fond "dd" 2>&1 >save.txt * swap the stdout and stderr stream for the 'boggle' command >> boggle 3>&1 1>&2 2>&3 * accept input from a file >> grep text > diff -u <(ls -c1 dir1) <(ls -c1 dir2) * save the output of "sort" in "sorted" and also pipe to "uniq" >> sort list.txt | tee sorted.txt | uniq -c | less * send a list of users to the printer and to a file "users.txt" >> awk -F: '{ print $1}' /etc/passwd | sort | tee users.txt | lp TEE .... * show the date and also append it to 2 files >> date | tee -a file1 file2 * Record output of any command using 'tee' at backend >> ssh user@server | tee logfilename PIPES .... * cat large file to clipboard with speed-o-meter >> pv large.xml | xclip * write text or append to a file >> cat <<.>> somefilename COMPLEX AND NAMED PIPES .... 00- Named pipes are like files, they need to be explicitly deleted when they are no longer needed. - A named pipe can remove the need to save data to a temporary file - mknod and mkfifo are equivalent? * a simple pipeline and its equivalent named pipe version --------------------------------------------------------- # the anonymous pipe version echo "test" | wc # the named pipe version which is equivalent mknod apipe p; wc apipe; echo "test" > apipe ,,, >> sudo mkfifo mypipe; wc -l < mypipe >&1 ; ls > mypipe; rm mypipe; * create a named pipe which compresses things sent to it, send something * to it to get compressed and then delete it. --------------------- mkfifo mypipe gzip -9 -c < mypipe > out.gz cat file > mypipe rm mypipe ,,, * create a pipe which compresses a file and loads the data into mysql --------------------------------------------------------------------- mkfifo /tmp/namedPipe chmod 0666 /tmp/namedPipe gzip -d file.gz > /tmp/namedPipe LOAD DATA INFILE '/tmp/namedPipe' INTO TABLE tableName; ,,, * make some named pipes and redirect to them >> mkfifo pipe1 pipe2; ls | tee pipe1 | less ##(delete pipe1 later) * pipe the output of 2 commands to another command >> (cd /user/home && ls) | less * Who needs pipes? >> B <<< $(A) MULTIPLE COMMANDS * only execute "ls" if the change directory "cd" command was successful >> cd /user/home && ls USING FILE HANDLES * open a for reading and give it the 'file handle' 3 >> exec 3< file.txt * open a file for reading and writing, write the text "yes", then close it >> exec 3<> file.txt ; echo yes >&3 ; exec 3>&- ; LOOPS -----------------------------------------: Bash has both 'while' and 'for' loops but sometimes it is possible to use 'xargs' or 'parallel' instead, which may be simpler and neater. * an interpreting loop which breaks when the user enters 'q' >> while true; do read a; echo $a; [ "$a" = "q" ] && break; done * skip a broken piece of a loop but not exit the loop entirely >> ctrl + \ * display files with 4 files on each line >> ls | xargs -L4 * use xargs interactively prompting at each command >> ls | xargs -p -I{} cp "{}" "{}.bak" * execute multiple commands for every 'wav' file (play/sleep) >> locate '*.wav' | shuf | xargs -I{} echo "play {};sleep 2" | bash INTERPRETING LOOPS .... Interpreting loops can just receive an input from the user and do some action. The following script allows a user to create a transcription from an audio file (containing people speaking). * A long loop doing audio stuff ------------------------ #while read -sn 1 -p "?" com ; do f=jenny.txt audio=jenny.wav at=0 dur=0 echo "An audio transcription script..." echo "audio file '$audio', transcription file '$f'" while read -sn 1 com ; do echo $com if [ "$com" == "p" ]; then at=$[at + dur] start=$(date +'%s') echo "play from $at seconds" play -q $audio trim $at & pid=$! # echo "pid=$pid" fi if [ "$com" == "a" ]; then start=$(date +'%s') echo "play from ${at}s for $dur s" play -q $audio trim $at $dur & fi if [ "$com" == "s" ]; then #pkill play kill -9 $pid 2>/dev/null end=$(date +'%s') dur=$[end - start] echo "duration $dur" fi if [ "$com" == "b" ]; then read -p "Start time back (seconds):" new at=$[at - new] dur=0 fi if [ "$com" == "f" ]; then read -p "Start time forward (seconds):" new at=$[at + new] dur=0 fi if [ "$com" == "t" ]; then read -p "Enter new start time (seconds):" new at=$new dur=0 fi if [ "$com" == "m" ]; then read -p "Enter new start time (minutes):" new at=$[new * 60] dur=0 fi if [ "$com" == "w" ]; then echo "seconds: $at" >> $f echo " " >> $f # start vim at last line in insert mode with easy exit (X/Z) vim + -c 'nmap X :wq' -c 'nmap Z :wq' -c start $f fi if [ "$com" == "i" ]; then echo "transcription file: $f" echo "audio file: $audio" echo "current section duration (in seconds): $dur" echo "current play start point (in seconds): $at" fi if [ "$com" == "h" ]; then echo "p play" echo "a play same section again" echo "s stop playing" echo "b set start time back n seconds" echo "f set start time forward n seconds" echo "t enter a new start point in audio (in seconds)" echo "m enter a new start point in audio (in minutes)" echo "w transcribe" echo "i show info" echo "h show help" echo "x exit" fi if [ "$com" == "x" ]; then pkill play echo "goodbye!" exit fi done ,,,, FOR LOOPS .... * show the bash help for the 'for' loop >> help for | less * a simple loop >> for j in 1 2 3 4; do echo "j=$j"; done * a for loop with variable and computed limit values >> a=4; for n in $(seq $a $((a+4))); do echo $n; done * an arithmetic style for loop >> for ((i=0; i<$MAX; i++)) do echo $i; done * in recent versions of bash we can use curly braces too. >> for j in 1 2 3 4;{ echo "j=$j"; } * print the numbers 1 to 20 >> for j in $(seq 1 20); do echo "j=$j"; done >> for j in {1..20}; do echo "j=$j"; done ##(bash 3.0 and after) * a for loop with leading zero in bash version 3 >> for i in {0..1}{0..9}; do echo $i; done * for loop with leading zero in bash 3 >> seq -s " " -w 3 20 * print the numbers 1 to 20 with a step of 2 >> for j in $(seq 1 2 20); do echo "j=$j"; done * loop through the files in a directory >> for file in dir/*; do echo $file; done * loop through the files in the current folder and subdirectories >> for file in $(ls *); do echo $file; done * find all file with names ending in '.html' or '.php' containing the word 'big' >> find / \(-name \*.html -o -name \*.php\) | xargs grep -i "big" ##(this uses the tool 'xargs' instead of a 'for' loop) * a for loop with filling 0 format, with seq >> for i in `seq -f %03g 5 50 111`; do echo $i ; done * using 'jot' in a for loop >> for f in `jot - 0 50 5` ; do ping -c 1 -m 50 10.0.2.$f ; done * nested brace expansion in a for loop -------------------------------------- for host in {frontend{1..5},backend{1..3}}.mycompany.com do ssh $host "echo -n $host; uptime" done ,,, * loop over input line by line, not word by word -------------------------------------------- IFS=$'\n' for f in $(ls -ls); do echo "loop=$f" done ,,, * another way to read input line by line, not word by word ----------------------------------------------------- ls -thor | while read f; do echo "loop=$f" done ,,, WHILE LOOP .... * show the bash help for 'while' >> help while * loop while the counter is less than 5 >> c=0; while [ $c -lt 5 ]; do echo c is $c; let c=c+1; done * a loop which reads a line of text from the user infinitely >> while read s; do echo =$s; done * a loop through files and delete each if user enters 'd' >> for f in *; do echo $f; read x; [ "$x" == "d" ] && rm $f; done INFINITE LOOPS .... ----------------------------------------------------------- * make an infinite loop with while (stop it with ) >> while true ; do echo yes; done >> while test 1 ; do echo yes; done ##(the same) >> while [ 1 ] ; do echo yes; done ##(the same) * loop until the user types 'q' using ">" as a prompt >> while [ "$r" != "q" ] ;do read -p'>' r; done; unset r BREAK AND CONTINUE .... Break and continue are used within loops to skip certain iterations, for example to stop processing once a certain item it found within a list. * use break to break out of the loop >> for j in $(seq 1 20); do break; done ##(does nothing) * continue goes immediately to the next iteration >> for j in $(seq 1 20); do continue; done ##(does nothing) IF STATEMENT ----------------------------------------------- * show the bash help for the 'if' command >> help if | less * a basic if statement >> if [ ${f} == "xx" ]; then echo $f; fi * a short-hand way of writing an 'if' statement >> [ this -eq this ] && echo 'true' ##(prints 'true') * a short hand way of writing an if else statement >> [ this -eq that ] && echo 'true' || echo 'false' * print a message and exit if no 'c' files are found in the folder >> ls *.c 2>/dev/null || { echo "No C files found"; exit; } IF ELSE .... * construct an if else alternative ---------------------------------- if [ "1" == "" ]; then echo 'true' else echo 'false' fi ,,, * if with 'elif' (else if) statement >> if [ "${f} == "xx" ]; then echo $f; elif [ "${f} == "yy" ]; then echo $f; fi CASE STATEMENT * an example of a case statement -------------------------------- echo "choose"; read $o case "$o" in 1) station="mms://server:port/path" ;; 2) station="mms://otherserver:port/otherpath" ;; [3-9]) echo "wrong number" [[:lower:]]) echo "lower case" [qQ]) exit ;; *) echo "invalid" esac ,,, TESTS ----------------------------------------------------------- In the bash shell tests return either 0 (true) or 1 (false) * test if one number is less than another >> c=1; if [ $c -lt 5 ]; then echo $c; fi * check if the 'zenity' program is installed >> if ! type zenity >/dev/null 2>&1; then echo no; fi * test if the variable is a directory >> test -d "$HOME"; echo $? ##(prints either 0 or 1) * combine two tests with the shell "and" operator ------------------------------------------------- if [ -n "$var"] && [ -e "$var"]; then echo "\$var is not null and a file named $var exists!" fi ,,, * combine two test with the "and" -a operator (a less common way) >> [ $c -lt 4 -a $c -gt 1 ] * combine two test with the "or" operator >> [ $c -lt 4 -o $c -gt 1 ] NEGATING TESTS .... The negation operator is the exclamation mark "!" * A space between "!" and the next argument is necessary. * A space between '[' and '!' is also necessary * negate a test with a "!" symbol at the front of the test >> [ ! -e ~/plans.txt ] && echo "no plans" This prints "no plans" if the file "plans.txt" does not exist * one can also use 'test' instead of the '[]' brackets >> test ! -e ~/plans.txt && echo "no plans" * echo 'does not exist' if 'plans.txt' doesnt exist --------------------------------------------------- if test ! -e ~/plans.txt then echo "no plans" fi ,,, * test if a string variable is not empty >> i=" x"; test ! -z $i && echo "not empty" ##(prints "not empty") FILE AND DIRECTORY NAMES ==moving around folders .. cd - - go to the previous folder .. cd - go to the users home folder .. ls /usr/; cd !$ - jump to the last folder mentioned .. * list only subdirectories in the current >> ls -d */ >> find -type d -maxdepth 1 ##(similar) * print file information in a readable format >> ls -thor ##(this is more compact than 'ls -la') * Find all dot files and directories >> echo .* * a list of all files on one line separated by commas >> echo * | tr ' ' ',' * an alias to go back to the previous directory >> alias b='cd -' * strip out the path from a file name >> f=$(pwd); f=$(basename $f); echo $f * display the fifth file in the current folder >> a=($(ls)); echo ${a[5]} This prints the current folder without the path. * a bash shell alternative to 'basename' >> echo ${file##*/} * rename files replacing one text with another ---------------------------------------------- for f in $(find . -name '*replaceme.jpg') ; do mv $f ${f/replaceme/withme}; done ,,, * rename files ending in '.JPG' with '.jpg' even if file has a space in it >> locate '*.JPG' | xargs -I{} rename 's/\.JPG$/.jpg/' "{}" The '-n' option to "rename" is handy because it allows you to see what would happen without actually doing it. * rename files replacing the extension ".html" with ".php" >> for f in *.html; do mv $f ${f/.html/.php}; done * display only directory names in a listing >> ls -l | grep '^d' * jump to the folder used in the last command >> ls /usr/share/doc/; cd !$ * just the directory name of the last path entered >> !$:h * just the file name of the last path entered >> !$:t FINDING FILES The 2 command line tools are 'find' and 'locate'. Locate is fast and simple, but cant find very recent files. Find is slow and complicated but can do everything. * find files with a '.mp3' extension >> locate -r '.mp3$' FILE TESTS ----------------------------------------------------------- == Some file tests .. -d , is the file a directory .. -f , is the file a normal file (not a directory) .. -e , does the file or folder exist .. -r , is the file readable .. -w , is the file writeable .. -s , is the file non-empty .. -S , is the file a socket .. -z , is the file or variable empty .. * compare file ages >> -nt newer than, -ot older than * test if a file exists >> if [ -f file.ext ]; then echo ${file.ext}; done SCRIPTS A script is a set of bash commands placed into a text file and made executable. By convention bash script files often have a filename extension of '.sh' although this is not necessary. If your script is very small consider making it a function instead and place the function in the '.bashrc' configuration file == the steps to run a bash script .. find out where you bash is with 'which bash' .. put #!/bin/bash at the top of the script (see previous step) .. write your bash script .. make your script executable with 'chmod +x script' .. * make a script (text file) executable >> chmod +x list.sh * make a script (text file) executable for all users >> chmod a+x list.sh * a simple example script which lists a file ------------------------- #!/bin/bash [ -z $1 ] && echo usage: $0 filename && exit cat $1 ,,, EXECUTING BASH SCRIPTS .... Bash scripts are often given a file name extension of '.sh' such as 'smallscript.sh' but this is not necessary. * place a reference to the bash executable at the top of the script >> #!/bin/bash This is called the 'shebang' line - from hash-bang * execute a bash script called 'showpeople' in the current folder >> ./showpeople * execute a script which is in the 'tree' sub-folder >> tree/showpeople The methods above are the most usually way to execute to a script. * another way to execute a script >> eval $(cat script) * execute a (non-executable) bash script >> source script ##(??) * find out where the 'bash' executable is on the computer >> which bash * execute the bash commands contained in the file 'script' >> cat script | bash Using the technique above, the file 'script' does not have to be executable. * run the bash script 'script.sh' in debugging mode >> bash -x ./script.sh * run a bash script in debug mode, show output and save it >> bash -x script.sh 2> log * execute a shell script in the background and dont interrupt it >> nohup /bin/sh myscript.sh 1>&2 &>/dev/null 1>&2 &>/dev/null& * Read commands from string and assign any arguments to positional params >> bash -c 'set w x y z; IFS=":-;"; echo "$*"' SCRIPT PARAMETERS .... ----------------------------------------------------------- # This section deals with parameters which are passed from the # command line to a Bash script. * get the absolute path to your bash-script >> scriptpath=$(cd $(dirname $0);pwd) * display a message and exit if no parameters were set >> [ -z "$1" ] && echo "no parameters were given" && exit 1 * exit if exactly 3 parameters were not given --------------------------------------------- if [ $# -ne 3 ]; then echo "usage $0 " exit 1 fi ,,, * the string $@ contains an array of all the parameters >> a="$@" * all the parameters as a string not an array >> s="$*" * get the 2nd and rest of the parameters >> echo ${*:2} >> echo ${@:2} * get the 2nd and 3rd parameters -------------------------------- echo ${@:2:3} if the script is invoked with ./script one 2 three 4 then this will print "2 three" ,,, * do something for each parameter given >> files="$@"; for f in files; do echo $f; done * show the number of script parameters >> echo $# * test if a particular parameter exists >> if test -z "$1" ;then exit; fi ##(exits if no first parameter) >> if [ -z "$1" ] ;then exit; fi ##(the same) * assign a value to a variable depending if a parameter exists or not >> [ -z "$1" ] && x=”$1” || x="/bin/date" (works ???) * if 'foo' has no value, assign it the value 'one' >> foo=""; echo ${foo:=one} ##(prints 'one') * maybe the quickest way to get the current program name minus the path >> programname="${0##*/}" PUTTING A SCRIPT IN THE PATH .... * to execute a script without the './' set the path to a user 'bin' folder >> mkdir ~/bin; [in ~/.bashrc add] export PATH=$PATH:$HOME/bin * execute the 'showpeople' bash script which is located 'on the path' >> showpeople Another technique to make a bash script executable by everybody is to put a symbolic link to it in the '/usr/local/bin' folder. * put a symbolic link in the /usr/local/bin >> ln -s /usr/local/bin/go ~/scripts/super.sh (wrong ??) * search for a file in the system path >> type SCRIPT CONFIGURATION .... * check the bash version number in a script ------------------------------------------- if [[ $BASH_VERSION > "2.05a" ]]; then ... fi ,,, * check if the script has been run as root ------------------------------------------ if [ $(id -u) != 0 ]; then echo "This script must be run as root" 1>&2 exit 1 fi ,,, USER INTERACTION ----------------------------------------------------------- == tools .. read - allow the user to input some text .. select - presents the user with a list to select from .. whiptail - displays 'dialog boxes' in a shell window .. dialog - like whiptail .. osd_cat - display text on the desktop (not is the console) .. zenity - display nice windows from shell scripts (uses gtk) .. kdialog - like zenity but for the kde linux desktop .. wish - run a tk window script .. complete - automatically completes commands .. perl tk - display windows from within a perl script .. gtk - a library for writing window applications .. .. Traditionally Unix and Linux was the domain of command line hackers who enjoyed hours of typing cryptic commands into a black screen with green text in a strange language devoid of vowels and meaning. But the fact remains that the average human being has a gut-instinct aversion to being forced to using the command line to do things. It is possible to use various Linux tools to present the user with a slightly more enjoyable interface. USER INPUT .... The easiest way to interact with the user is just to ask the user to enter some information at a shell prompt. This can be down with the 'read' command. * display a prompt and get a response from the user >> echo enter name; read s; echo you typed $s; * get more than one value from the user >> read d m; echo day $d month $m * get a single key press from the user >> echo press key; key=$(dd bs=1 count=1 2> /dev/null); echo $key; see http://tldp.org/LDP/abs/html/commandsub.html for more information (the terminal mode needs to be changed first) * press any key to continue >> read -sn 1 -p "press any key to continue..."; echo * function that outputs dots every second until command completes >> sleeper(){ while `ps -p $1 &>/dev/null`; do echo -n "${2:-.}"; sleep * display todays date on the desktop >> date | osd_cat * display a webpage as text and update the view it every 10 seconds >> watch --interval=10 lynx -dump http://dslrouter/stats.html * a loop through files and delete each if user enters 'd' >> for f in *; do echo $f; read x; [ "$x" == "d" ] && rm $f; done DIALOG AND WHIPTAIL .... Whiptail and dialog are basically one step up from 'select', and one step down from 'yad' and 'zenity'. Dialog and whiptail are basically equivalent, but are both quircky and less useful than they should be. Whiptail installed by default on most debian systems * a whiptail menu with a return value >> r=$(whiptail --title "files" --menu "choose" 20 60 15 a app b ball c cat 3>&1 1>&2 2>&3); echo $r * with brace expansion >> r=$(whiptail --title files --menu "choose" 20 60 15 {A..z} 3>&1 1>&2 2>&3); echo $r * a whiptail file listing >> r=$(whiptail --title files --menu "choose" 20 60 13 $(ls | sed "s/$/ ./" | tr '\n' ' ';) 3>&1 1>&2 2>&3); echo $r * Show a 'curses' based menu selector >> whiptail --checklist "Simple checkbox menu" 11 35 5 tag item status repeat tags 1 * a trick to get a return value from whiptail >> foobar=$(whiptail --inputbox "Enter some text" 10 30 3>&1 1>&2 2>&3) * return values from dialog are more easily got >> result=$(dialog --output-fd 1 --inputbox "Enter some text" 10 30) PRESENT A CHOICE TO THE USER .... The 'select' tool is a simple way to present the user with a choice and perform and action based on that choice. * choose a file in the current folder to edit >> PS3="Enter a number: "; select f in *; do vim $f; break; done [= image/eg-select-edit.png] * choose an image for a list and open it >> IFS=$'\n'; select f in $(locate -i '/home/*.jpg'); { xdg-open $f; break; } Notice that select can use curly braces (like 'for') is modern versions of the bash shell. * set the 'IFS' variable if file names may contain spaces >> IFS=$'\n'; PS3="Choose dir:"; select f in $(find . -type d);{ cd $f;break;} * a function to list folders named with some text and jump to selected >> ju() { PS3="dir:"; select f in $(find ~ -type d -iname "*$1*"); do cd $f; break; done; }; ju doc; * choose from 2 phrases to print >> select f in "1 ball" "2 balls"; do echo $f; break; done * choose a big file to delete >> select f in $(find . -size +2M); { rm $f; echo $f; break; } * present the user with a choice of options -------------------------------------------- if [ "1" == "" ]; then echo " [1] Radio 1 [2] Radio 2 Enter your choice: " read choice else choice="1" fi case "$choice" in 1) station="mms://server:port/path" ;; 2) station="mms://otherserver:port/otherpath" ;; *) echo "Wrong choice!" exit esac ,,, * use select to jump to a line in a text file containing the text 'determ' -------------------------------------------------------------------------- IFS=$'\n'; PS3="Choose:"; n=bash-book.txt; select f in $(grep -n determ $n) do l=$(echo "$f" | cut -d' ' -f2) vim +$l $n; break; done ,,,, Notice the judicious use of the 'IFS' variable above. IFS determines how input will be split into elements. By default it will split all input on all whitespace (that is, into words) but here we want it to split the input into lines so we have to set IFS to only the new-line character. The problem is that if the output contains more lines than the screen, the user will never see the beginning of the output. * use select to jump to a line in a text file containing the text 'determ' -------------------------------------------------------------------------- IFS=$'\n'; PS3=""; n=test.txt; select f in $(grep -n determ $n); do l=$(echo "$f" | cut -d' ' -f2); vim +$l $n; break; done ,,,, * search for a word in 'words' then jump to the one selected in a window >> f=/usr/share/dict/words; vim +/$(grep -i jump $f | zenity --list --column= --height=800)/ $f * try to kill a process chosen by the user >> IFS=$'\n'; PS3="which?:"; select f in $(ps -ef|tr -s ' ');{ echo $f;break;} YAD .... Yad is a replacement for zenity, and is much better. ZENITY .... zenity is a way of easily displaying windows with user interface elements which allow the user to carry out actions or answer questions etc. Or use tcl/tk * view the man page for zenity >> man zenity * show a calender in a window 200 by 800 pixels >> zenity --calendar --title 'select a file' --height=800 --width=200 * show a list box 800 pixel high and 800 pixels wide >> ls -thor | zenity --list --column= --height=800 --width=800 FILE SELECTION ........ * allow the user to select a file and echo the name of the chosen file >> zenity --file-selection --title 'select a file' * select a file in the /home/bob/os/ folder >> zenity --file-selection --filename="/home/bob/os/" * a function which searches for files and then opens the selected one ------------------------------------------------------------------- function op { [ -z "$1" ] && echo "usage: $FUNCNAME " && return 1; xdg-open "$(find ~ -iname '*$1*' | zenity --list --column= --width=700 --height=700)" } ,,, * display a list of all pngs on the computer and display selected one >> locate '*.png' | zenity --list --column=Images | xargs -I{} xdg-open "{}" The 'xargs -I{} xdg-open "{}"' is important for the case where a file name contains a space. * locate all jpgs in user home folders >> xdg-open "$(locate -i '/home/*.jpg' | zenity --list --column=)" The recipe above also handles filenames with spaces in them, owing to the judicious use of double quotes >> xdg-open "$(locate -i '/home/*.jpg' | zenity --list --column=)" TEXT ENTRY ........ * display a dialog which allows the user to enter some text >> zenity --title="New File" --entry * display a window titled 'Name' with a text box and text prompt >> zenity --title "Name" --entry --text "Enter your name" LIST BOXES ........ * make a window with a list-box with the files from current folder >> ls | zenity --list --column="test" --title='choose a file' [= image/eg-zenity-list.png] * display a list of folders in the current folder with no window title >> ls -d */ | zenity --list --column="test" * display a list of files in a window with no title or column-title >> ls | zenity --list --column= * show a list of files and then display the file-name which was chosen >> echo "you chose $(ls | zenity --list --column=)" * jump to any folder in the current folder >> cd $(ls -d */ | zenity --list --column= --title="change folder") * jump to any folder in the users home directory (could be slow) >> cd $(find ~ -type d | zenity --list --column= ) * an alias to display lists with the zenity tool >> alias zenls='zenity --list --column=' We can then use the alias above in a command pipe-line such as >> ls | zenls * make an editable text box with a file listing saving changes to 'j.txt' >> ls | zenity --text-info --editable > j.txt MULTICOLUMN LIST BOXES .... * create a list with 2 columns, one all 'false' and the other files >> zenity --list --column=buy --column=item $(echo * | sed 's/ / FALSE /g;s/^/FALSE /') ls | tr '\n' ' ' * create a list with 2 columns, the first check-boxes and the other files >> zenity --checklist --list --column=buy --column=item $(echo * | sed 's/ / FALSE /g;s/^/FALSE /') * a zenity mulicolumn example checkbox example >> zenity --list --checklist --column=Buy --column=Item TRUE "Red Apples" TRUE Oranges FALSE 'Green Pears' FALSE Toothpaste == some zenity options .. --text "hello" - display hello at the top of the window .. --column "file" - display 'file' at the top of the column .. ZENITY NOTIFICATIONS ... * show an icon in the notification area >> zenity --notification --window-icon=up.png --text "System update necessary!" KDIALOG TK AND TCL .... It should be possible to use the 'wish' interpreter and the tk language to interact with the user in a 'graphical' fashion from a bash script, but this is a large topic. @@ http://pages.cpsc.ucalgary.ca/~saul/personal/archives/Tcl-Tk_stuff/tcl_examples/ some tk examples * a simple tk label example --------------------------- label .l1 -text "ground" -foreground blue label .l2 -text "big" -font {-family times -size 24} grid .l1 -row 0 grid .l2 -row 1 ,,, * buttons --------- #!/usr/bin/wish button .my_button -text "Hello World" -command exit pack .my_button ,,, * a button which exits when clicked >> echo 'button .b -text "Hello World" -command exit; pack .b' | wish [=0.2 image/eg-tk-button-exit.png] * a tk list-box with text items and file names >> (echo -n 'listbox .lb -selectmode multiple -height 14; .lb insert 0 sample stuff colors red yellow green '; echo -n * ; echo '; pack .lb') | wish [=0.2 image/eg-tk-listbox-files.png] * another way to execute some tk or tcl commands >> cat script | wish PERL TK .... Using perl with 'tk' is another way to interact with the user graphically. * install perl/tk with apt on a debian linux type system >> apt-get install perl-tk * another way to try to install the perl 'Tk' module >> perl -MCPAN -e 'install "Tk"' ##(didnt work!!) * view the very good help for the perl tk module >> man perl-tk * run the demonstration program showing all perl tk possibilities >> widget [= image/eg-perltk-widget.png] * a perl-tk one liner to create a button >> echo "use Tk; new MainWindow->Button(-text=>'click!')->pack; MainLoop;" | perl * tk example ------------ #!/usr/bin/perl -w use Tk; use strict; my $main = new MainWindow; my $button = $main->Button(); $button -> configure(-text => 'Press me!'); $button -> pack; MainLoop; ,,, * display a directory tree. --------------------------- use Tk; use Tk::DirTree; my $top = MainWindow->new; my $dl = $top->Scrolled('DirTree')->pack(-expand => 1, -fill => 'both'); MainLoop; ,,, PROGRAM OUTPUT ----------------------------------------------------------- * suppress program output messages >> prog &> /dev/null * suppress error messages >> prog 2> /dev/null * print the return value of a program >> prog; echo $? * return a value to the shell >> exit 31 ##(exits the script and returns 31) RANDOMNESS * play a random 'wav' sound file, but it fails if names have spaces >> a=($(ls *.wav)); play "${a[$RANDOM%${#a}]}" * play a random wav file in the current folder >> f=$(ls *.wav | sed -n "$[$RANDOM%$(ls | wc -l)]p" ); play "$f" * play a random wav, but this gives an error if no wavs available >> play "$(ls *.wav | shuf | head -1 )" * only print a message sometimes (randomly) >> (($RANDOM%6)) || echo 'hello world!' * display a random file in the current folder >> a=($(ls)); echo ${a[$RANDOM%${#a}]} * randomly shuffle lines of a file (but needs gnu sort, not bsd) >> sort --random-sort * another random file lines sort >> cat test.txt | while read f ; do printf "$RANDOM\t%s\n" "$f"; done | sort -n | cut -f2- * shuffle lines in a file on a system without 'shuf' >> cat /usr/share/dict/words | perl -MList::Util -e 'print List::Util::shuffle <>' | less >> perl -MList::Util=shuffle -e'print shuffle<>' SPECIAL VARIABLES ----------------------------------------------------------- * The script name is contained in $0 >> echo usage: $0 filename * $? contains 0 or 1 the result of the last program (or test) * the process id of the script >> $$ SPECIAL CHARACTERS * add 2 newlines to a string variable >> s="tree"$'\n'$'\n'; echo "$s" ##(try "echo $s" to see an echo "gotcha") BASH EXPANSION COMMAND COMPLETION .... * execute the last command >> !! * execute the last command entered with root privileges >> sudo !! * enter the last command typed on the command line >> alt + . * inserts the last used arguments >> . * Inserts the results of an auto-completion in the command line >> ESC * CONFIGURING COMMAND COMPLETION .... * some useful settings in ~/.inputrc: ------------------------------------- set completion-ignore-case on set show-all-if-ambiguous on set show-all-if-unmodified on ,,, * Enable ** to expand files recursively (>=bash-4.0) >> shopt -s globstar * Enable automatic typo correction for directory names >> shopt -s cdspell * Bash autocomplete case insensitive search >> shopt -s nocaseglob * bash glob dot-files (such as '.bashrc') >> shopt -s dotglob LISTS IN BASH Bash has several methods of generating lists, including "globbing", and "brace expansion". GLOBBING .... * rename a file using special brace expansion >> cp /long/path/file.{txt,doc} * assign a list of folder names to a variable >> d=$(echo */); echo $d Be careful here: "d=*/; echo $d" appears to have the desired effect but in reality, the variable 'd' does not contain a list, just the characters '*/' * get a list of folder names and replace all '/' with ':' >> d=$(echo ~/sf/htdocs/books/*/); echo ${d//\//:} * get a list of folder names but only print the last element of the path >> d=$(echo ~/sf/htdocs/*/); d="basename "${d// /;basename }; eval $d The recipe above is fairly nifty. It used globbing, string substitution and 'eval' to execute the bash commands, contained in the variable "d". In these capabilities it gets close to the power of a functional or "list-based" language such as lisp. * get a list of folder names and double up each folder name >> d=$(echo ~/sf/htdocs/*/); d="basename "${d// /;basename }; d=$(eval $d); echo $d | sed 's/\w\+/& &/g' This prints some like 'docs docs html html img img ...' etc ARITHMETIC BRACE LISTS .... * print the numbers from 1 to seven >> echo {1..7} ##(modern bash versions) * enumerate with the numbers padded with zeroes >> echo {001..5} * a for loop with leading zero (bash version 3) >> for i in {0..1}{0..9}; do echo $i; done * use a brace arithmetic series with a 'step' value (bash version 4) >> echo {1..20..2} BRACE EXPANSION Brace expansion is another way of generating lists of strings in the bash shell. In some ways this is reminiscent of list based programming languages, such as 'lisp' or 'scheme'. "Globbing" is another technique for generating lists of file names. * use a numeric sequence 'glob' (expansion) to create speech >> echo {1..3}" o'clock" ROCK | espeak * print an alphabetic expansion >> echo {a..z}"x " ##(prints 'ax bx cx ... zx') * multiply by nothing, zero string, create a file list >> echo *.html{"",.gz} * use an explicit list for with brace expansion >> echo n:{pin,nail,ball} This will print 'n:pin n:nail n:ball' * an html style brace expansion >> echo "
  • "{a..z}"
    • " * dynamic globbing can be done like this >> e=4; eval "echo {1..$e}"; echo {1..4} * print a comma delimited file list >> a=$(echo *); echo ${a//" "/, } * print a colon delimited list of folders in the current folder >> a=$(echo */); echo ${a//" "/:} * do a 'dynamic' brace expansion on a list of file names >> echo "echo "{$(echo * | tr ' ' ',')}"xx" | bash * another way >> a=$(echo *); echo 'echo file:{'${a//" "/,}} | bash * make the listing one per line with the -e switch >> a=$(echo *); echo 'echo -e "\nfile:{'${a//" "/,}} | bash * make an html directory listing out of the current folder >> echo "echo \"{$(echo * | tr ' ' ',')}"\" | bash The technique used above could equally be done using a 'for' loop such as 'for f in *; do ... ; done' but I find this brace expansion concept interesting. * make 2 backups of a file using brace expansion >> cp list{,bak,bak1} * using nested string brace expansion (bash version >= 3.0) >> echo {frontend{1..5},backend{1..3}}.server.net * list all the directories in the current folder >> echo */ >> ls -d */ >> ls -F | grep '/$' * list all hidden files (starting with '.') in the current folder >> echo .* EXECUTING BASH COMMANDS * find out where the 'wget' program is >> type -a wget >> which wget ##(similar) * execute bash commands from standard input >> echo "cd /; ls" | bash * execute commands in a string >> bash -c 'ls' * remove lots of files, but first check which ones will be removed >> find dir -name \*~ | xargs echo rm >> find dir -name \*~ | xargs echo rm | bash * use eval to execute the command 'ls' >> eval 'l''s' COMMAND SUBSTITUTION ----------------------------------------------------------- * delete a list of files named in a text file >> rm `cat file` >> rm $(cat file) ##(this is the same) * execute a command which is contained in a variable >> cmd=ls; $cmd PROCESS SUBSTITUTION * compare the packages installed on 2 servers >> diff <(ssh server1 'rpm -qa | sort') <(ssh server2 'rpm -qa | sort') VARIABLE ASSIGNMENT ----------------------------------------------------------- * use export to make a variable available to subprocesses >> export name=value * set a variable to the contents of a file >> v=$(> v=$(cat file); ##(the same) VARIABLE TYPE Variables in bash are normally 'untyped' but can be declared as a certain 'type' using the 'declare' keyword * check if a variable is not a number >> v='blah';if [ -z $(echo $v | grep [0-9]) ]; then echo "NON NUMERIC"; fi * check if a variable is not a number >> v='1';[ -z $(echo $v | grep [0-9]) ] && echo "not a number" || echo 'number' * declare variable as integer >> declare -i aa ; aa=3*8 ; echo $aa >> declare -i aa ; aa=3*8 ;aa='tree'; echo $aa If a string is assigned to the variable then the variable will be set to '0'. ARITHMETIC EVEN AND ODD .... * check if a number is even >> [ $((a%2)) -eq 0 ] && echo "even" * check if no of files in dir is even or odd >> n=$(ls | wc -l); [ $((n%2)) -ne 0 ] && echo "odd" * define a quick calculator function '?' >> ? () { echo "$*" | bc -l; } The function above can be called with '? 4*5' for example, which will print '20'. * add two numbers using bash operators >> echo $((4+5)) >> echo $[4+5] * perform addition with the 'bc' tool >> echo 1+1|bc * c style variable incrementation, >> a=8; (( a++ )); echo $a; ##(prints 9) * integer arithmetic and assignment >> a=3; b=$(( a + 5 )); echo $b; ##(prints 8) * add up all the numbers from 1 to 1000 >> echo {1..1000} | sed 's/ \+/+/g' | bc >> seq 1 1000 | (sed 's/^/x+=/'; echo x) | bc ##(another way) * sum all the numbers in the file 'list.txt' with each number on a line >> (sed 's/^/x+=/' list.txt ; echo x) | bc * display the product of 512 and 7 >> expr 512 \* 7 * creating a list of numbers for calculation >> echo $( du -sm /var/log/* | cut -f1 ) | sed 's/ /+/g' * a short counter >> yes '' | cat -n FLOATING POINT ARITHMETIC .... Unfortunately many linuxes dont come with 'bc' installed by default, which means you may not be able to rely on these techniques. * divide 22 by 23 and print result to 4 decimal places >> echo "scale=4; 22/23" | bc * subtract one decimal number from another >> echo "4.54-6.332" | bc * a floating point operation with no precision specified >> echo 7/2 | bc -l ##(prints 3.50000000) * another way but not working!!?? >> bc -l <<< s(3/5) INTEGER VARIABLES * declare variable as integer >> declare -i aa ; aa=3*8 ; echo $aa * print 'yes' if the variable 'x' is a number >> i=3; if [ "$i" -eq "$i" 2>/dev/null ]; then echo yes; fi * increment a variable >> c=4; let c=c+1; ##(c is now 5) * increment the counter by 1 >> c=4; let c+=1; ##(c is now 5) * decrement the counter by 1 >> c=4; let c-=1; ##(c is now 3) == comparing numbers, the tests are .. -eq - equal to .. -ne - not equal to .. -lt - less than < .. -le - less than or equal <= .. -gt - greater than > .. -ge - greater than or equal >= .. STRING VARIABLES * echo the text 'linux' in reverse >> echo linux | rev * reverse the order of the characters in a string >> s='tree'; s=$(echo $s | rev); echo $s * check if a the the word 'madam' is a palindrome >> s='madam'; [ "$s" == "$(echo $s | rev)" ] && echo palindrome! || echo nope ANALYSING STRINGS .... * test if a variable consists only of white space or nothing >> i=" "; if [ -z $i ]; then echo "empty"; fi >> i=" "; [ -z $i ] && echo "empty" ##(the same) * test for an empty string variable >> i=" x"; test ! -z $i && echo "not empty" ##(prints "not empty") Both of these scripts above will print 'empty' * test if the string variable $s contains the text 'tree' >> s='this tree'; if [[ $s == *tree* ]]; then echo "yes"; fi * print 'has a dot' if the string contains a full stop >> s='one.two'; if [[ $s == *.* ]]; then echo "has a dot"; fi >> s='one.two'; [[ $s == *.* ]] && echo "has a dot" * print 'no dot' if the variable 's' contains no full stop >> s='onetwo'; [[ $s != *.* ]] && echo "no dot" * display the number of characters in a string >> aa=bbbb; echo ${#aa} ##(prints 4) STRING CASE .... * Convert the word 'tree' to uppercase >> echo tree | tr '[:lower:]' '[:upper:]' * convert the string variable 's' to upper case >> s="the grass"; s=$(echo $s | tr '[:lower:]' '[:upper:]');echo $s COMPARING STRINGS .... * test if 2 strings are not the same >> if test "$a" != "$b" ; then echo 'not the same'; fi >> if [ "$a" != "$b" ]; then echo 'not the same'; fi ##(another way) * test if 2 strings are the same or different >> if test "$a" = "$b" ; then echo 'the same'; fi SPLITTING STRINGS .... * split the string 'aa bb cc' on whitespace (prints 'bb') >> ARRAY=(aa bb cc);echo ${ARRAY[1]} * split the string "aa bb cc" into 3 variables >> read v1 v2 v3 < <(echo aa bb cc); echo $v2 * split a string into an array using commas as delimiter >> s='a,b,c'; a=(${s//,/ }); echo ${a[1]} >> a=xyzxyz; a=${a//x/q}; echo $a; ##(prints qyzqyz) >> ARRAY=(aa bb cc);echo ${ARRAY[1]} WHITESPACE .... * show whitespace in a visible manner >> echo -e "\t\n" | cat -vte REPLACING AND SUBSTITUTING STRINGS .... * replace the first "x" in the string with a "q" >> a=xyzxyz; a=${a/x/q}; echo $a; ##(prints qyzxyz) * replace all occurrences of 'x' in the string 'a' >> a=xyzxyz; a=${a//x/q}; echo $a; ##(prints qyzqyz) * replace all occurrences of 'a' in a string with the word 'the' >> s="a book. a tree"; s=${s//a/the}; echo $s; * replace all occurrences of '.' in a string with a '/' slash character >> s="a.b.c"; s=${s//./\/}; echo $s; SUBSTRINGS OF STRINGS .... * extract a substring from a string >> s=abcde; echo ${s:1} ##(prints bcde, the string "s" is not changed) >> s=abcdefghi; echo ${s:3:3} ##(prints def) * delete an exact match of one variable within another >> f=tree; r=re; f=${f/$r/}; echo $f <: prints "te" :> >> f=tree; r=re; f=${f##$r}; echo $f <: the same :> * print only the first word of a sentence >> a="one two thre"; echo ${a%% *} * print only the last word in a sentence >> foo="simple forth machine"; echo ${foo##* } This prints 'machine' * strip all but the last word from a sentence >> s="simple forth machine"; s=${s##* }; echo $s The variable 's' now has the text "machine" * delete only the first match of a substring in a string >> foo="this is a test"; echo ${foo#t} ##(prints 'his is a test') * delete the shortest match starting from the left >> foo="this is a test"; echo ${foo#t*is} ##(prints 'is a test') >> s="12: this is a test"; echo ${s#t*t} * delete the longest match from the left >> foo="this is a test"; echo ${foo##t*is} ##(prints 'a test') * deletes the last match of a substring in a string >> foo="this is a test"; echo ${foo%t} ##(prints 'this is a tes') * delete the last 2 characters of a string >> foo="this is a test"; echo ${foo%??} ##(prints 'this is a te') * deletes the shortest match from the end >> foo="this is a test"; echo ${foo%t*st} ##(prints 'this is a') * deletes the longest match from the end of the string variable >> foo="this is a test"; echo ${foo%%t*st} ##(prints nothing) * remove the file name extension from a filename >> f=index.txt; f=${f%.txt}; echo $f ##(prints 'index') >> f=index.txt; mv f ${f%.txt} ##(actually renames the file, careful) See http://tldp.org/LDP/abs/html/refcards.html for more patterns CONCATENATING STRING VARIABLES .... Joining 2 or more string variables together into one big string is known as "concatenating". * append a string onto the end of a variable >> m=aa; m+=bb; m+=cc; echo $m ##(prints 'aabbcc') The above is probably a recent bash extension. * add the string "tree" to the end of a variable >> f=green; f=$f"tree"; echo $f ##(prints "greentree") >> f=green; f=${f}tree; echo $f ##(the same) >> f=green; f="${f}tree"; echo $f ##(the same again) * join a string to the end of another string. >> f="this is";g=" a tree"; f=$f$g; echo $f ##(prints "this is a tree") >> f="this is";g=" a tree"; f="$f$g"; echo $f ##(the same) * add 2 new-line characters to a string >> s=tree; s=$s$'\n'$'\n'; echo "$s" ##(the quotes on the final $s ARE needed) WRITING STRINGS TO A FILE .... * use echo -n "$data" instead of echo $data which 'squeezes' newlines >> echo -n "$data" > /home/username/data.txt HERE DOCUMENTS .... The practice of creating a long string variable between a start and an end label is called a 'here document'. This syntax exists in a number of programming languages (for example: PHP and Perl) Here documents are useful for printing or manipulating long multi-line strings. * use a 'here document', for long strings ----------------------------------------- cat << endx This is a string with several lines. The end label can be anything, and can appear in the text as long as it doesnt start the line (like this 'endhere') endx ,,, ##(the end label 'endx' must be at the start of the line) * put commands and variables in a 'here document' (use $(command)) ------------------------------------------------------------------ cat << ends The script name is $(basename $0) ends ,,, * make a 'here document' which does not expand variables or commands ------------------------------------------------------------------ cat <<'enddoc' commands such as $(pwd) will not be interpreted nor with variables such as $s enddoc ,,, ##(the trick is to use "<<'LABEL'" instead of "<> v=green; echo "The grass is $v" ##( this prints "The grass is green" ) Bash will generally 'interpolate' variables which are inserted with double-quoted (") strings. Interpolation is as idea which also occurs in perl, and other scripting languages. * stop a variable from interpolating with \$ >> v=5; echo "\$v is $v" * string interpolation works with array values as well >> area[5]=bb; echo "The contents of area[5] is ${area[5]}." * a complex example of interpolation using eval >> a=b; eval $(echo "echo \"this is xx\"" | sed 's/xx/$a/') >> a=b; eval $(echo "echo this is xx" | sed 's/xx/$a/') ##(also works) >> export a=b; echo "echo this is xx" | sed 's/xx/$a/' | bash ##(also) All 3 examples above do the same thing. But the last requires the 'export' statement to make the 'a' variable available to all subprocesses. This is because the 'bash' command begins a new process. * a simple 'templating' technique using interpolation >> echo -e 'todays \ndate \nis ' | (echo 'cat << EE';sed 's//$(date)/g'; echo 'EE') | bash The text '' in the input stream gets replaced by todays date. This is useful when the input-stream comes from a file. * use the technique above to substitute the date into the template >> cat template | (echo 'cat << EE';sed 's//$(date)/g'; echo 'EE') | bash DATE VARIABLES * check if a date is valid, even a date within a leap year- not BSD ?? >> date -d2009-05-18 > /dev/null 2>&1 ; echo $? * another date validation but not BSD unix >> date -d2009-02-33 > /dev/null 2>&1 ; [ $? ] && echo valid || echo invalid * display a millisecond time difference >> s=$(echo "scale=3; $(date +%s.%N)-$(date +%s.%N)" | bc); echo ${s%??????} * display seconds and milliseconds since 1970 (gnu date only) >> s=$(date +%s.%N); echo ${s%??????} ARRAYS Bash arrays are 0-based, that is the first element of an array 'list' is 'list[0]' not 'list[1]' (as in Ms Visual Basic) @@ http://tldp.org/LDP/abs/html/arrays.html some information about bash arrays. * display the fifth file in the current folder >> a=($(ls)); echo ${a[5]} * display the file which has been most recently modified in the folder >> a=($(ls -t)); echo ${a[0]} * display a random file in the current folder >> a=($(ls)); echo ${a[$RANDOM%${#a}]} * echo each element of a comma delimited array. >> echo one,two,three | xargs -d, -I{} echo "{}" DECLARING ARRAYS .... In the bash shell, it is normally not necessary to declare variables (that is, state what type a variable is- whether a string, an integer, or an array). But it may be advantageous in some cases. * declare an array variable 'list' explicitly >> declare -a list * explicitly un-declare an array 'aa' >> unset -v aa INITIALISING ARRAYS .... * initialize an array variable called 'list' >> list=(ant dog cat) the element list[0] is 'ant' the element list[1] is 'dog' ... * assign values to the 4th and 7th elements of the array 'area' >> area=([4]=hobart [7]=brunie) * make a new array 'files' from the current folder and display the 3rd >> files=( *.txt ); echo ${files[2]} * assign the lines of a text file to an array (1 line per element) >> ls -thor>t.txt; IFS=$'\n' aa=($(> ls -thor>t.txt; IFS=$'\n'; aa=($(> cat /usr/share/dict/words>t.txt; aa=($(> list=( *.txt ); echo ${list[@]} >> list=(*.txt); echo ${list[*]} ##(also works) * define the elements of an array over multiple lines ----------------------------------------------------- list=("a dog" cat "a tree") echo ${list[0]} ,,, * define the elements of an array over multiple lines ----------------------------------------------------- list=( "a dog" cat "a tree" ) echo ${list[0]} ,,, BIG ARRAYS .... * load a very large array from the dictionary file >> aa=($(cat /usr/share/dict/words)); echo ${aa[1000]} Since the standard unix dictionary may contain more than 200000 words the line above gives you an idea of how many elements a bash array may contain and how fast it takes to load that many elements * see how long it takes to load a big array ------------------------------------------- f=/usr/share/dict/words; time aa=($(cat $f)); echo The array has ${#aa[@]} elements ,,, INITIALIZING ARRAYS FROM FILES .... * assign the lines of a text file to an array (1 line per element) >> ls -thor > t.txt; IFS=$'\n'; aa=($( t.txt IFS=$'\n' aa=($(cat t.txt)) echo "3th element=${aa[3]}" ,,, * mapfile is also supposed to work, but not for me >> echo -e "a b\nc d\ne f" | mapfile -n 0 aa; echo ${aa[*]} >> echo -e "a b\nc d\ne f" | mapfile aa; echo ${aa[*]} >> echo -e "a b\nc d\ne f" | mapfile ; echo ${mapfile[*]} * another way to read the lines of 'test.txt' into an array --------------------------------------------------------- i=1 while read line do array[$i]="$line" echo "${array[$i]}" ((i++)) done < test.txt ,,, INITIALIZING WITH COMMAND OUTPUT .... * assign the lines of a text file to an array (1 line per element) >> IFS=$'\n' aa=($(ls -thor)); echo "4th element=${aa[3]}" The command above is the simplest way to set the elements of an array to be the lines of the output of some command. But without the IFS variable being set, the elements of the array would be initialised to the 'words' (space separated) of the command output, not the lines. Notice how there is no semi-colon after the setting the IFS variable, this means that we only change the value of the variable for this one command and not subsequent commands. * we could also do this as follows but would have to reset IFS >> IFS=$'\n'; aa=($(ls -thor)); echo "4th element=${aa[3]}" * another technique to read lines into an array ----------------------------------------------- i=1 ls -thor | while read line do array[$i]="$line"; echo ${array[$i]} ((i++)) done ,,, * read disk usage figures into an array, line by line ------------------------------------------------------- IFS=$'\n' aa=($(du -sh *)) echo "3th element=${aa[3]}" ,,, * another more complicated way to do the same thing --------------------------------------------------- du -sh * | while read line do aa[${#aa[@]}]="$line" echo "${aa[$((${#aa[*]}-1))]}" done ,,, COPYING ARRAYS .... * make a copy of array 'aa' in 'bb' >> aa=(1 2 3); bb=( "${aa[@]}" ); echo "${bb[*]}" This technique may not work of the array 'aa' is 'sparse', which means that only some values have been set. * make a copy of an array. >> array2="${array1[@]}" * Adding an element to an array. >> aa= "${aa[@]}" "new element" ) >> cc=( 1 2 ); cc[${#cc[*]}]="three"; echo ${cc[*]} These fail for sparse arrays ASSIGNING VALUES .... * set the first element of the array 'list' to the text 'horse' >> list[0]=horse * set the 2nd element of the array 'list' to the text 'fast horse' >> list[1]="fast horse" ##( use quotes to allow value white-space) * assign a list of files to an array >> list=( *.txt ); ##( list is an array and not a string ) MODIFYING ARRAYS .... * create an array and replace all 's' characters with 'S' in all elements >> list=(*.txt); echo ${list[@]//s/S} * replace all 's' characters with 'S' in an array >> list=(*.txt); list=(${list[@]//s/S}); echo ${list[@]} * replace the first character of all array elements with an '.' >> list=(*.txt); list=(${list[@]/?/.}); echo ${list[@]} ACCESSING ARRAY VALUES .... * display the first element of an array variable 'list' >> echo ${list[0]} ##( not '$list[0]' which prints 'one[0]' ) >> echo $list ##( this also displays only the 1st element ) * display the 4th element of a bash array variable 'list' >> echo ${list[3]} * use string 'interpolation' to insert the value of an array in text >> echo "The Contents of area[5] is ${area[5]}." * display all the values of the array 'list' >> echo ${list[*]} >> echo ${list[@]} ##(there is some subtle difference, but what?) ARRAY LENGTH .... * display the number of elements in an array >> list=(one 2 three four); echo ${#list[@]} * display the last element in an array >> a=(one 2 three four); echo ${a[$((${#a[@]}-1))]} * display the oldest file in the current folder >> a=($(ls -t)); echo ${a[$((${#a[@]}-1))]} SUBSETS OF ARRAYS .... * print the 2nd and remaining characters of the 1st element >> echo ${list:1} ##(if list[0]=big this prints 'ig') * display the 2nd, 3rd and 4th elements of the array 'list' >> echo ${list[@]:1:3} * display the 2nd and all remaining elements of the array 'list' >> echo ${list[@]:1} * display the 3rd, 4th and 5th characters of element 1 of 'list' >> echo ${list[0]:2:4} ##( if list[0]=quick then prints 'ick' ) >> echo ${list:2:4} ##( the same, [0] is assumed ) * display the 2rd and 3rd characters of element 4 of 'list' >> echo ${list[3]:1:2} ##( if list[3]=water then prints 'at' ) * bookmark folder for quick changing >> cdargs LOOPING THROUGH ARRAYS .... * loop through an array properly -------------------------------- names=( "a dog" cat "a tree") for (( i = 0; i < ${#names[@]}; i++ )) do echo "element $i=${names[$i]}" done ,,, note: theres no 'seq' on apple osx ! but you could use bash arithmetic expansion {1..20} etc * loop through an array properly, one element at a time, not by words --------------------------------------------------------------------- aa=( "a dog" cat "a tree") for (( i = 0; i < ${#aa[@]}; i++ )) do echo "element $i=${aa[$i]}" done ,,, * getting a dynamic array element list, tricky but works >> aa=("a b" c "d e"); eval "echo {1..${#aa[@]}}"; * a more tricky loop using bash globbing, -------------------------------- aa=( "a dog" cat "a tree") for i in $(eval "echo {0..$((${#aa[@]}-1))}") do echo "element $i=${aa[$i]}" done ,,, * a more compact loop using braces instead of do/done ----------------------------------------------------- aa=( "a dog" cat "a tree") for i in $(eval "echo {0..$((${#aa[@]}-1))}") { echo "$i=${aa[$i]}"; } ,,, * bash arithmetic, which prints 11 >> a=12; a=$((a-1)); echo $a * loop through an array properly with seq, but this is untested --------------------------------------------------------------- aa=( "a dog" cat "a tree") for i in $(seq 0 $((${#aa[@]}-1))) do echo "element $i=${names[$i]}" done ,,, LOOPING THROUGH WORDS OF ARRAYS .... * loop through all the words in the array elements ----------------------------------------------------- list=( "a dog" cat "a tree") for f in ${list[@]}; do echo $f; done echo "list has ${#list[*]} elements" ,,, * loop through all the words of elements of an array >> aa=(*.txt); for f in ${aa[@]}; do echo $f; done If the elements have spaces this may not be what you want * make an array from the disk usage and show the second element >> list=($(du -sh */)); echo ${list[1]} MORE ARRAYS .... * load a file into an array 'aa' (but this is word by word, not by line) >> aa=(`cat "$filename"`) * define an array with 3 strings, some with spaces in them >> list=("a dog" cat "a tree"); echo ${list[@]} * use 'string interpolation' to show the 2nd element of and array >> list=("a dog" cat "a tree"); echo "2nd element=${list[1]}" * add an element to an array using 'eval' and a string >> aa=(one 2 three); eval "aa[3]='a frog'"; echo ${aa[3]} * assign an array element with spaces >> aa[2]="a frog"; echo ${aa[2]} * another 2 level evaluation technique for defining an array element >> aa=(one 2 three); eval $(echo "aa[3]='a frog'"); echo ${aa[3]} * what does this do >> aa=(*.txt); s="{${aa[@]}}"; s=${s// /,}"aa"; echo $s READING LINE BY LINE @@ http://www.somebits.com/weblog/tech/bash-read-command.html good hints about reading with read @@ http://codesnippets.joyent.com/posts/show/1899 another technique for reading lines into an array a knowledgeable post. * use 'mapfile' as another way to read into arrays >> ls -thor | mapfile aa; echo ${#aa[@]} USING JOT FOR EXAMPLE DATA * make some random data, good .... >> jot * prints 21 decimal numbers increasing evenly from -1 to 1. >> jot 21 -1 1.00 * generate the ascii character set, one character per line >> jot -c 128 0 | less * generate the strings 'xaa' through 'xaz' >> jot -w xa%c 26 a | less * generate 20 random 8-letter strings >> jot -r -c 160 a z | rs -g 0 8 * generate an infinite number of yes's each on a new line >> jot -b yes 0 * generate thirty 'ed' substitution commands applying to lines 2, 7, 12, etc >> jot -w %ds/old/new/ 30 2 - 5 * The stuttering sequence 9, 9, 8, 8, 7, etc. can be produced >> jot - 9 0 -.5 * create a file containing exactly 1024 bytes >> jot -b x 512 > block * set tabs four spaces apart starting at column 10, ending in column 132 >> expand -`jot -s, - 10 132 4` * print all lines 80 characters or longer, >> grep $(jot -s "" -b . 80) PASTE * List the files in the current directory in three columns: >> ls | paste - - - ##(tab delimited) >> ls | xargs -L3 ##(space delimited) * paste corresponding lines of 'f1' & 'f2' into 2 columns separated by ',' >> jot -w tree 10 > f1; jot -w leaf 10 > f2; paste -d, f1 f2 * show the sub-folders in a 3 column output separated by commas ',' >> ls -d */ | paste -d',' - - - * Combine pairs of lines from a file into single lines: >> jot -w tree 20 > f; paste -s -d '\t\n' f * Combine pairs of lines from a file into comma delimited lines >> jot -w tree 20 > f; paste -s -d ',\n' f The jot is just used to create some example data * Number the lines in a file, similar to 'nl' >> sed = myfile | paste -s -d '\t\n' - - * Create a colon-separated list of folders named bin, suitable for use in the PATH environment variable: >> find / -name bin -type d | paste -s -d : - == other text data tools .. lam - join file lines (how is this different from 'paste'?) .. join - a text relational database tool .. END PASTE EXTENDED GLOBBING 'Globbing' is the process of expanding certain special characters (often called 'wildcards') into a list of files, or into a pattern to match against another string. * activate special wildcard 'globbing' in scripts >> shopt -s extglob * remove all files except ones with names ending in .jpg >> shopt -s extglob; rm !(*.jpg) * remove all files except *.jpg and *.gif and *.png >> shopt -s extglob; rm !(*.jpg|*.gif|*.png) * trim leading and trailing whitespace from a variable (needs extglob) >> x=${x##+([[:space:]])}; x=${x%%+([[:space:]])} * show visible the characters used to split strings in globbing >> echo "$IFS" | cat -vte == extglob extended patterns .. ?(pattern-list) - Matches zero or one occurrence of the given patterns. .. *(pattern-list) - Matches zero or more occurrences of the given patterns. .. +(pattern-list) - Matches one or more occurrences of the given patterns. .. @(pattern-list) - Matches one of the given patterns. .. !(pattern-list) - Matches anything except one of the given patterns. .. * delete the second field in a ':' delimited variable >> shopt -s extglob; a=one:two:3; a=${a/:+([^:])}; echo $a * set the ifs variable to only new lines >> IFS=$'\n' The line about seems mysterious and cryptic, but has an enormous use in many situations. If you ever want to loop through the lines of some output, using 'for' or 'select' then you need to set the IFS variable first. This is because IFS determines how input will be split into 'tokens' when iterating through some text will 'for' etc. By default 'for' and 'select' will use IFS to split all output on all whitespace (not just new-lines) >> IFS=$'\n'; PS3="Enter a number: "; select f in $(ls -la);do vim $f; break; done >> PS3="Enter a number: "; select f in $(ls -la);do vim $f; break; done STRING SUBSTITUTION see the 'extglob' shell option for better bash patterns >> If $sub matches at front end of $s, substitute $replacement for $sub >> ${s/#sub/replacement} * If $substring matches back end of $string, substitute $replacement for $substring. >> ${string/%substring/replacement} PREFIX DELETION AND SUBSTITUTION .... The ${x#..} operator is equivalent to a regular expression such as /^.../ In bash patterns '?' is equivalent to /./ in a regular expression and '*' is equivalent to /.*/ With ${x#} '*' is not greedy but with ${x##} '*' is greedy * delete up to and including the 1st colon in a string variable >> s="one:two:3:four"; s=${s#?*:}; echo $s ##(prints 'two:3:four') >> s="one:two:3:four"; s=${s#[^:]*:}; echo $s ##(the same effect) * delete up to and including the last colon in a string variable >> s="one:two:3:four"; s=${s##?*:}; echo $s ##(prints 'four') >> s="one:two:3:four"; s=${s#*:}; echo $s ##(prints 'four') SUFFIX DELETION AND SUBSTITUTION .... The ${x%...} operator is equivalent to a /...$/ regular expression pattern. In this context the '*' operator is not greedy * delete from the last ':' colon to the end of a string variable >> s="one:two:3:four"; s=${s%:?*}; echo $s ##(prints 'one:two:3') >> s="one:two:3:four"; s=${s%:[^:]*}; echo $s ##(the same effect) STRING DELETION AND SUBSTITUTION .... Modern versions of Bash allow for some very capable string substitution. In some cases these techniques may replace what was formerly done with a 'sed' command. Note the ${x/..} is a 'greedy' matching operator when used with the '*' wildcard character. It will match as much text as it can. * delete everything after the 1st colon character (not including ':') >> a=name:david; a=${a/:*/:}; echo $a ##(this prints 'name:') * delete everything after the 1st colon character (including the ':') >> a=name:ben; a=${a/:*}; echo $a ##(prints 'name') >> a=name:ben; a=${a/:*/}; echo $a ##(the same, but unnecessary) * delete all fields except the 1st and last in a ':' colon delimited variable >> a=name:alex:john:fox; a=${a/:*:/:}; echo $a ##(prints 'name:fox') >> a=6:vim:42:emacs; a=${a/:[^:]*/:}; echo $a ##(prints 'name:fox') >> a=6:b:j:fox; a=${a/:[^:]:/:}; echo $a ##(prints 'name:fox') * delete everything except the 1st word in a bash variable >> a="why is this so"; a=${a/[ ]*/}; echo $a ##(prints 'why') >> a="why is this so"; a=${a/ *}; echo $a ##(the same, prints 'why') * delete all words in a bash variable except the 1st and the last >> a="why is this so"; a=${a/[ ]*[ ]/ }; echo $a ##(prints 'why so') >> a="why is this so"; a=${a/ * / }; echo $a ##(the same) * delete everything after a '.' or a ':' (which ever comes first) >> a=google.net:8080; a=${a/[:.]*}; echo $a ##(prints 'google') >> a=google.net:8080; echo $a | sed 's/[.:].*//' ##(the same, with sed) * delete everything after the first lowercase roman letter >> a=12A:boat; a=${a/[a-z]*}; echo $a ##(prints '12A:') * delete all text after the 1st roman letter in a bash variable >> a=12A:boat; a=${a/[a-zA-Z]*}; echo $a ##(prints '12') @@ http://www.hypexr.org/bash_tutorial.php#colors some interesting notes about bash tools * suppress error messages >> ls 2>/dev/null * pipe the output of 2 commands (on separate lines) to 'less' ------------------------------------------------------------- (ls; pwd) | less ,,, * another multi-line grouping that works, note the final ';' semicolon --------------------------------------------------------------------- { ls; pwd; } | less ,,, * a function to search some book files for recipes or descriptions ------------------------------------------------------------------- function sr { if [ -z "$1" ]; then echo "usage: $FUNCNAME [] []" echo " - searches the book files for recipes with the search term(s)" echo " - the search term(s) can be in the recipe or the recipe description" echo " or multiple search terms " return 1; fi ( find ~/sf/htdocs/books -name '*-book.txt' | xargs grep -hA2 "^ *\*.*$1.*$2.*$3" ; find ~/sf/htdocs/books -name '*-book.txt' | xargs grep -hB2 "^ *>>.*$1.*$2.*$3") | less } ,,, IF ELSE * an if elif statement ---------------------- if something ; then do_stuff elif something_else ; then do_other_stuff elif [... ] ; then echo "ss" else : fi ,,, * use a ':' to do nothing in an if else clause ---------------------------------------------- if [ -z "$x" ]; then : else einfo "Not" fi ,,, A BIG LIST OF FILE TESTS -a file Exists (use -e instead) -b file Exists and is a block special file -c file Exists and is a character special file -d file Exists and is a directory -e file Exists -f file Exists and is a regular file -g file Exists and is set-group-id -h file Exists and is a symbolic link -k file Exists and its sticky bit is set -p file Exists and is a named pipe (FIFO) -r file Exists and is readable -s file Exists and has a size greater than zero -t fd Descriptor fd is open and refers to a terminal -u file Exists and its set-user-id bit is set -w file Exists and is writable -x file Exists and is executable -O file Exists and is owned by the effective user id -G file Exists and is owned by the effective group id -L file Exists and is a symbolic link -S file Exists and is a socket -N file Exists and has been modified since it was last read SUBSTRINGS .... * remove the shortest match from the front of all array elements ---------------------------------------------------------------- echo ${list[@]#f*r} if $list was 'forr five afur' before then afterwards 'r five a' ,,, * remove the longest match from front of all elements of 'list' >> echo ${list[@]##t*e} >> 'list' before: trepef four atrapee >> 'list' after: f four a * remoe the shortest match from back of all elements >> echo ${list[@]%h*e} * remove the longest match from back of all elements of the array >> echo ${list[@]%%t*e} REPLACING SUBSTRINGS IN ARRAY ELEMENTS .... * replace the first occurrence of 'fiv' with 'XYZ' in all elements >> echo ${list[@]/fiv/XYZ} * Replace all occurrences of 'iv' with 'YY' in all elements >> echo ${list[@]//iv/YY} * Delete all occurrences of 'fi' in all elements of the array >> echo ${list[@]//fi/} * replace occurrences of 'e' with the output of a function 'rep' --------------------------------------------------------------- rep() { echo -n "!!!" } #echo ${list[@]/%e/$(rep)} echo ${list[@]/e/$(rep)} ,,, CALCULATING WITH ARRAYS .... * add the 2 values of an array and assign to another >> area[5]=`expr ${area[11]} + ${area[13]}` >> area[5]=$(expr ${area[11]} + ${area[13]}) ##(the same?) ARRAY LENGTH .... * display the length of the array (or the index of last value) >> echo ${#list} >> echo ${#list[*]} ##(the same, but possibly very different) * print the length of the string of the 1st element of array 'list' >> echo ${#list[0]} ##( if list[0]=tree this prints '4') * print the length of the string of the 3st element of array 'list' >> echo ${#list[2]} ##( if list[2]=big this prints '3') JOINING ARRAYS .... * join the elements of an array with ',' commas >> a=(a b c); s=$IFS; IFS=","; new="${a[*]}"; IFS=$s; echo $new * another way to join an array ------------------------------ #/!bin/bash foo=('foo bar' 'foo baz' 'bar baz') bar=$(printf ",%s" "${foo[@]}"); bar=${bar:1} echo $bar ,,, FUNCTIONS Function definitions can be placed in the ".bashrc" file in order create new commands for the bash shell. This is more flexible than using 'aliases' because functions may have parameters. Functions may also be used within scripts to make those scripts simpler or more readable. Functions are simpler than creating a new command with a script placed in the path. @@ http://www.cyberciti.biz/faq/bash-shell-script-function-examples/ some good information * a function to add a function you've defined to .bashrc -------------------------------------------------------- addfunction () { [ -z "$1" ] && echo "usage: $FUNCNAME " && return 1 declare -f $1 >> ~/.bashrc ; } ,,, * show a usage mesage for the function if no parameters are given ----------------------------------------------------------------- function test { if [ -z "$1" ]; then echo "usage: $FUNCNAME " echo " - performs a useful task" return 1; fi ... } ,,,, * make a function called 'boo' using your last command >> boo () { !!; } * a function which prints its own name >> test () { echo "$FUNCNAME"; } * a function which prints its own name >> test () { [ -z "$1" ] && echo "$FUNCNAME: parameter needed" && return 1; } * show what each user defined function does ------------------------------------------- function fn { IFS=$'\n';for f in $(declare -F); { declare -f ${f##* }; } | \ sed -n '/^_/d;/() *$/s///p;/^ *echo "/{s// /;s/";//p}' | grep -v 'usage' | less } ,,, This recipe assumes that user bash functions are documented with 'echo "blah"' statements. The recipe above uses several tricks. Such as sed '/n/s///' which reused the matched pattern in a sed expresions. In other words >> sed '/x/s///'; is the same as >> sed '/x/s/x//'; Functions starting with '_' are usually defined by the programs, not the user. TRAPS AND GOTCHAS .... * dont use 'exit' to return from a function, use 'return' instead. * dont use '$0' for the function name use '$FUNCNAME' instead If an alias is reused as a function name, you may get strange incomprehensible errors. FUNCTION DEFINITION .... * show the definition for the bash function 'af' >> declare -f af * show help for the bash built-in command 'declare' >> help declare | less * show all defined functions which dont begin with a '_' >> declare -F | grep -v " -f _" | less * define a function with a parameter >> function s { echo $1; } >> s(){ echo $1; } ##(this is the same) * a function may span 2 or more lines ---------------------------------- function x { echo 'hello'; } ,,, * a function with one parameter ------------------------------- function say { echo $1 } say hello ##(this just prints 'hello' ) ,,, * create a read-onlyfunction >> test(){ echo "Foo"; }; readonly -f test * a recursive function (dont run this, because it is infinite) >> foo(){ foo; } EXAMPLE FUNCTIONS .... * a function to change into a folder and list its contents >> function cdls { cd $1; ls; } * Redefine the cd command's behavior using 'builtin' >> cd() { builtin cd "${@:-$HOME}" && ls; } * a function which prints a message and returns if no parameter -------------------------------------------------------- test () { [ -z "$1" ] && echo "usage: $0 file" && return 1 cat $1 } ,,, * a function which calculates the arguments (eg: mm 5*7) >> mm () { echo "$*" | bc -l; } * a function to concatenate input >> calc(){ awk "BEGIN{ print $* }" ;} * change directory up a number of levels >> function ..(){ for ((j=${1:-1},i=0;i> function mcd() { [ -n "$1" ] && mkdir -p "$@" && cd "$1"; } * make a function to search the software repositories >> function af { apt-cache search $@ | less; } * Search for the file and open in vi editor. >> vifind() { vi `find . -name "$1"` } * function to output an ascii character given its decimal equivalent >> chr () { printf \\$(($1/64*100+$1%64/8*10+$1%8)); } FUNCTION VARIABLES .... * make function variables local >> add (){ local x=$1; local y=$2; echo $(( $x + $y )); } DOING NOTHING AT ALL Despite appearances, doing nothing can be quite useful in fact even vital in certain circumstances. * do nothing for 5 seconds >> sleep 5 >> sleep 5s ##(the same) >> sleep 2 3 ##(the same) >> sleep 1 4 ##(the same again) * do nothing for half a second >> sleep 0.5 ##(gnu sleep) * do nothing for one hundredth of a second >> sleep 0.01 * do nothing for 5 minutes >> sleep 5m * do nothing but be successful >> true * print 'y' continuously and successfully >> yes ##(this is not really doing nothing, but anyway) ENVIRONMENT VARIABLES When the user logs in to a bash shell, certain variables are automatically set. == important bash environment variables .. $HOME - the users home folder .. $PATH - where executable programs are located .. $PWD - the current working folder .. $USER - the name of the user logged in .. $HOST - the name of the local computer .. @@ http://tldp.org/LDP/abs/html/internalvariables.html a good guide to the bash internal variables == more advanced bash variables .. $IFS - determines how strings are split .. * show the values of the environment variables >> env >> printenv It is possible to change the value of an environment variable for only the following command and not for any subsequent commands. This is done as follows * show what time and date it is in new york >> TZ=America/New_York date CONFIGURING THE BASH SHELL Put configuration for the bash shell in '.bashrc' or '.bashprofile' New commands for bash can be created either with aliases, functions or scripts. * List bash functions defined in .bash_profile or .bashrc >> declare -F | sed 's/^declare -f //' * show the path split into lines >> echo $PATH | tr : \\n * change default shell for all users (freebsd ??) >> cd /usr/home && for i in *;do chsh -s bash $i;done * enable change directory using variable names >> shopt -s cdable_vars THE PROMPT .... * Display screen window number in the bash prompt >> [[ "$WINDOW" ]] && PS1="\u@\h:\w[$WINDOW]\$ " * display a prompt with colours >> export PS1='\[\033[0;35m\]\h\[\033[0;33m\] \w\[\033[00m\]: ' ALIASES .... Aliases are useful for abbreviating common commands . To save an alias put it in the ~/.bashrc file so that it will always be available. The limitation of aliases are that they cannot use command line parameters. To overcome this use 'functions' instead. * make all searches withing the 'less' pager case-insensitive >> alias less='less -i' * make an alias: a new command 'dir' which calls the "ls" command >> alias dir='ls -la | less' ##(aliases cant use parameters, such as $1) * up - aliaes for moving up the directory tree >> alias up="cd .."; alias upp="cd ../.."; alias uppp="cd ../../.."; * make some convenient aliases to edit and reload .bashrc and aliases >> alias aliases='vim ~/.bash_aliases; source ~/.bash_aliases' >> alias bashrc='vim ~/.bashrc; source ~/.bashrc' * Define an alias with a correct automatic completion >> old='apt-get'; new="su-${old}"; command="sudo ${old}"; alias "${new}=${command}"; $( complete | sed -n "s/${old}$/${new}/p" ); alias ${new}; complete -p ${new} * escape any command aliases >> alias v='less'; \v ##(the alias is not executed) MISCELLANEOUS * pretty print a java code 'Co.java' 2 columns, landscape, gaudy header >> enscript -G2rE -b"Assignment 2" -p new.ps Co.java * switch virtual consoles >> alt+Fn, eg: alt+F1 to go to the first virtual terminal * add a folder to the executable path (put in ~/.bash_profile file to save it) >> export PATH=/path/to/folder:"${PATH}" ##(programs in this folder can then be executed with 'programname') THE COMMAND LINE ------------------------------------------------------- COMMAND LINE HISTORY @@ http://www.deer-run.com/~hal/UnixCommandLineKungFu.pdf good tips for the command line In Unix and Linux commands previously executed at the console are stored in a command history and are accessible and manipulable in a variety of ways. * Copy history from one terminal to another >> history -w history -r GETTING HELP FOR THE HISTORY .... * view documentation about the history file >> man bash ##(then type '/history' to search the documentation) * view the bash help for the 'history' command >> help history VIEWING THE COMMAND HISTORY .... * view the command line history >> history ##(this show the current sessions command history) * view the last 10 commands of the command history >> history 10 >> fc -nl -10 ##(?) >> history | tail -10 ##(the same) * view recent commands without numbers >> fc -nl * list the last 100 commands in reverse order with no numbers >> fc -rnl -100 SEARCHING THE COMMAND HISTORY .... * search the command history >> [control] + r >> / ##(the same if the command line is in 'vi' mode) * Insert a comment on command line as a searchable reminder >> ls -alh #mycomment * display the top 10 commands from the command history >> history | awk '{print $2}' | awk 'BEGIN {FS="|"}{print $1}' | sort | uniq -c | sort -n | tail | sort -nr EXECUTING FROM THE HISTORY .... * execute the 47th command in the command history >> !47 * substitute 'l' with 's' in the last command and execute it. >> ^l^s * remove 'ls' from the last command >> ^ls * build up command pipelines with !! >> indent | !! ##(the previous command replaces '!!') * apply a 'less' pager command to a long text file path >> cat /long/path/name/document.txt >> !! | less * run the previous command with root privileges >> vim /etc/vim/vimrc >> sudo !! ##(this executes 'sudo vim /etc/vim/vimrc') * use !$ to get the argument from the last command >> cat file.txt >> vi !$ ##(executes 'vi file.txt') * !$ - The last argument to the previous command >> svn status app/models/foo.rb; svn commit -m "Changed file" !$ * get the command argument from the second last command >> vi !-2$ * get all arguments from the previous command >> cat a.txt b.txt c.txt >> vi !* ##(executes 'vi a.txt b.txt c.txt') * print the last command which begins with the letter 'h' >> !h:p ##(':p' stops the command from being executed) * print the last command which contains the string 'hi' >> !?hi?:p * print the 507th command with any filename extension removed >> !507:r:p * print the 512nd command with a 'c' replaced by 'C' >> !512:s/c/C/:p >> !512:s/c/C/ ##(actually execute the command) >> !512:s/c/C ##(the same) >> !512:s,c,C ##(the same) >> !512:gs/c/C/ ##(replace all occurrences of 'c' with 'C') * edit the last 7 commands in a text editor (determined by $FCEDIT) >> fc -6 0 ##(watch out, the commands are executed after editing!) * after fc, to not execute the commands, clear the editor buffer >> :bd ##(in vim, assuming that $FCEDIT is set to 'vim' or 'vi') * Runs previous bash command replacing 'tree' with 'leaf' >> ^tree^leaf * Do a command but skip recording it in the bash command history >> _cd ~/nsfw; mplayer midget_donkey.mpeg CONFIGURING THE COMMAND HISTORY .... * make sure that new commands are appended to the history file >> shopt -s histappend * see in which file your command history is stored >> echo $HISTFILE ##(this normally show '~/.bash_history') * view your (or the current users) history file >> less ~/.bash_history >> less $HISTFILE ##(technically better) * see how many commands will be saved to your history file >> echo $HISTSIZE * view the maximum number of commands saveable to your history file >> echo $HISTFILESIZE * see how many commands are currently in your history file >> wc -l $HISTFILE * get rid of duplicate entries in the command history file >> sort -u $HISTFILE -o $HISTFILE * make the history file 500000 lines and write 100000 commands each time >> export HISTFILESIZE=500000 >> export HISTSIZE=100000 put these values in '~/.bashrc' so they will be permanent * include timestamps for each command in the history file >> HISTTIMEFORMAT=yes * erase duplicates for the history file >> export HISTCONTROL="erasedups:ignoreboth" * dont put the 'exit' command in the history file >> export HISTIGNORE="&:[ ]*:exit" DELETING THE COMMAND HISTORY .... * clear the command history for the current session >> history -c * delete the command history item 100 >> history -d 100 VIM AND BASH It is possible to execute bash commands from within the vim text editor using a variety of techniques. It is also possible to 'filter' parts of a document using bash commands or send (pipe) part or all of a text document from vim to a bash command. * execute the bash 'date' command from within vim >> :!date * execute the current line as a bash command (doesnt insert results) >> :.w !bash * remove the comment character and execute current line as a bash command >> :.w !sed 's/^\#//' | bash * a command to execute a fragment between '---' and ',,,' with bash >> command! Bmr ?^ *---?+1,/^ *,,,/-1w !sed 's/^ *\#//' | bash | less But the command above exits immediately even if the bash commands should read input from the user. * a vim command 'Bashr' to execute the current line in bash >> command! Bashr .w !sed 's/^ *\#//' | bash | less * a vim mapping to execute a bash command line in a document >> map ,bm :.w !sed 's/^ *\#//' \| bash * a vim command to make and show a screen shot of a bash command >> command! Bci .w !sed 's/^ *\#//' | bash; scrot -ud1 test.png; feh test.png If we are editing a vim document which has a bash command such as "tree | head -20" on a line by itself, then we can execute and create a screenshot of the executed command using the vim command which we have just made above. We can do this by executing the command ":Bci" within vim * a vim command to make and show a screen shot saving to a given file name ?? >> command! -nargs=1 Bci .w !sed 's/^ *\#//' | bash; scrot -o -ud1 test.png; feh test.png * take a delayed screen-shot of the command on the current line >> :.w !sed 's/^/scrot -ubd1 test.png \& clear; /' | bash; sleep 1.5; feh test.png If the line which the cursor is on in the current document contains the text 'top', then executing the above line will run that command in a bash shell and take a screen-shot of its output. The screen-shot will then be displayed * a command 'Scr' to take a delayed screen-shot and display it >> command! Scr .w !sed 's/^ *\#?/scrot -ubd2 test.png \& clear; /' | bash; sleep 2.5; feh test.png If the cursor is positioned on a line with 'apropos sound | less' then we can take a screen-shot of the output of that command with ':Scr' See the vim 'system' function for more advanced solutions * view the help for the vim built function 'system()' >> :h system * assign to the variable 'a' the output of the bash command 'ls' >> :let a=system("ls") SCREEN SHOTS OF BASH Sometime it is nice to make a picture of a bash terminal in order to try to impress somebody or for other reasons * create a screen shot of the 'top' command >> scrot -ud1 test.png & clear; top * make and display a screen shot of a zenity list box in 'sh.png' >> scrot -ubd2 sh.png & ls | zenity --list --column="test"; feh sh.png == scrot options .. -u - take a screen shot of the focussed window .. -d2 - delay for 2 seconds before capturing the image .. -b - when capturing a window, get the window frame as well .. EDITING THE COMMAND LINE * Enable programmable bash completion in debian linux >> aptitude install bash-completion ; source /etc/bash_completion * display all commandline editing keystrokes that use [ctrl] >> bind -p | grep -F "\C" * show what 'short-cut' keys are available on the command line >> bind -P * Go to end of current command line >> * Go to begin of current command line >> CTRL + a * set the command line editing mode to 'vi' >> set -o vi ##(the command line now seems like the 'vi' text editor) * set the command line editing mode to 'emacs' >> set -o emacs ##(this is the default mode) * view the shell edit mode variables >> set -o * once in 'vi' edit mode, to edit the command line in $EDITOR text editor >> [esc] + v ##(if $EDITOR is 'vim' then vim will open) * enable vi mode line reading in a shell script and store line in 'a' >> set -o vi; read -e a; ##(the -e switch uses the 'readline' function) * change the default text editor for the system >> export EDITOR=vim ##(put in .bash_profile) * make the vi editing mode the default >> set editing-mode vi ##(put this in the '.inputrc' file) ##(type 'man readline' for more information) Bash command line editing is determined by the 'readline' program Other programs, like 'gnu-plot' also use the same client. Put 'set editing-mode vi' in .inputrc to automatically use the 'vi' editing mode for all readline clients./ ENCANTATIONS * insert the file name of each html file at the top of the file with no extension >> for f in *.html; do i=${f/.html/}; sed -i.bak "1s/^/$f/" $f; done PERL VERSUS BASH Perl may be much faster than bash if bash is doing looping constructs, otherwise not. * loop through input with perl ------------------------------ my $state = 0; while(<>) { exit if /TIMESTAMP2/; print $_ if $state == 1; $state = 1 if /TIMESTAMP1/; } ,,, FU BASH TIPS .... * Emulate perl 'print "#" x 20, "\n"' >> printf '%*s\n' 20 | tr ' ' '#' * Getting the last argument from the previous command >> cd !$ * Diff on two variables >> diff <(echo "$a") <(echo "$b") * Get the last string of previous command with !$ >> $mkdir mydir -> mv !$ yourdir -> $cd !$ * Perform a branching conditional >> true && { echo success;} || { echo failed; } * Runs previous command replacing foo by bar every time that foo occurs >> !!:gs/foo/bar * Check command history, but avoid running it >> !whatever:p * Clear mistyped passwords from password prompt >> ^u * invoke an editor to write a long, complex, or tricky command line >> control-x e * Salvage a borked terminal >> reset LOGGING OUT * execute a command on logout >> trap cmd 0 * close shell keeping all subprocesses running >> disown -a && exit COLOURS * make folder listings always colourized >> alias ls='ls --color=always' Colours on the screen can be obtained using Ansi escape sequences and the 'echo' command * display a red 'R' and a yellow 'Y' on the screen >> echo -e "\033[1;31mR\033[1;33mY" * display hi in all the colours >> echo -e "\x1B["{0..1}";3"{0..7}"m hi" The line above uses bash exterpolation to generate a set of strings. * display hello in red (uses a hex code and no bold) >> RED="\x1B[31m"; echo -e "$RED hello" * show redish text >> echo -e "\033[47m\033[1;31m Light Red on a grey background" >> echo -e "\033[1;31m Light Red " * Remove color codes (special characters) with sed >> sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g" * make a set of colour variables to use with 'echo -e' ------------ BLACK="\x1B[0;30m"; GREY="\x1B[1;30m"; RED="\x1B[0;31m"; BRED="\x1B[1;31m" GREEN="\x1B[0;32m" BGREEN="\x1B[1;32m" YELLOW="\x1B[0;33m" BYELLOW="\x1B[1;33m" BLUE="\x1B[0;34m" BBLUE="\x1B[1;34m" MAGENTA="\x1B[0;35m" BMAGENTA="\x1B[1;35m" CYAN="\x1B[0;36m" BCYAN="\x1B[1;36m" WHITE="\x1B[0;37m" BWHITE="\x1B[1;37m" NORMAL="\x1B[0m" ,,, * display some colours --------------- echo -e " $BLACK BLACK $NORMAL x1B[0;30m $GREY GREY $NORMAL x1B[1;30m $RED RED $NORMAL x1B[0;31m $BRED BOLD RED $NORMAL x1B[1;31m $GREEN GREEN $NORMAL x1B[0;32m $BGREEN BOLD GREEN $NORMAL x1B[1;32m YELLOW=x1B[0;33m BYELLOW=x1B[1;33m BLUE=x1B[0;34m BBLUE=x1B[1;34m MAGENTA=x1B[0;35m BMAGENTA=x1B[1;35m CYAN=x1B[0;36m BCYAN=x1B[1;36m WHITE=x1B[0;37m BWHITE=x1B[1;37m NORMAL=x1B[0m " ,,, == a list of colour escape codes .. .. Black, 0;30 .. Dark Gray, 1;30 .. Red, 0;31 .. Bold Red, 1;31 .. Green, 0;32 .. Bold Green, 1;32 .. Yellow, 0;33 .. Bold Yellow, 1;33 .. Blue, 0;34 .. Bold Blue, 1;34 .. Purple, 0;35 .. Bold Purple, 1;35 .. Cyan, 0;36 .. Bold Cyan, 1;36 .. Light Gray, 0;37 .. White, 1;37 .. * colored prompt >> export PS1='\[\033[0;35m\]\h\[\033[0;33m\] \w\[\033[00m\]: ' UNICODE CHARACTERS * print some unicode characters >> /usr/bin/printf '\u201cdouble\u201D' The '/usr/bin/printf' is necessary because otherwise the bash built-in printf command is used which doesnt print unicode * install the enjoyable 'unicode' command line program on a debian box >> sudo apt-get install unicode * show all unicode characters relating to chess >> unicode chess ANSI ESCAPE SEQUENCES * save the current cursor position >> echo -n -e "\033[s" * restore the cursor to its former position >> echo -n -e "\033[u" * doesnt seem to work --------------------- # 2 is green COLOR=`tput setaf 2; tput smso` # back to normal screen colors NORMAL=`tput sgr0` # print the time-date output on the above position echo -n $COLOR$cmd$NORMAL ,,, KNOWLEDGABLE BASH PEOPLE "foonly", sydney: http://forums.whirlpool.net.au/user/44690 hoyhoy at stackoverflow.com terson at stackoverflow.com amarillion at stackoverflow.com TERMINALS The term 'terminal' is very ambiguous. I cant supply a good definition at this stage. But it has to do with similar concepts of the console, command-line, shell etc. == terminal jargon @@ line wrapping When the text of a line is too long for the computers screen then the line may be broken or 'wrapped' so that the extra text is displayed on the next line in the computer screen. This is called 'line wrapping' or just 'wrapping' @@ * clear the screen >> clear * Change the window title of the xterm terminal to 'KungFu' >> echo "^[]0;KungFu^G" * make the text on the screen 'bold' (brighter and thicker than normal) >> tput bold * use tput to save, clear and restore the terminal contents >> tput smcup; echo "Doing some things..."; sleep 2; tput rmcup The tput command has many options and switches * determine number of lines in terminal >> l=100; while [ $l -gt 1 ]; do echo $l; l=$(($l-1)); done * Change the terminal title to the last shell command executed >> trap 'echo -e "\e]0;$BASH_COMMAND\007"' DEBUG * Start a gnome terminal with three open tabs >> gnome-terminal --tab --tab --tab * print the output of a script on a different terminal >> script -f /dev/pts/3 >> script /dev/null | tee /dev/pts/3 ##(another way to do it) * stop text lines from wrapping in your terminal >> function nowrap { export COLS=`tput cols` ; cut -c-$COLS ; unset COLS ; } * Using the urxvt terminal daemon >> urxvtd -q -o -f == terminal tools .. terminfo - a database about the terminal screen .. tput - a utility for the terminal .. * change title of terminal window to verbose info useful at login >> echo -ne "\033]0;`id -un`:`id -gn`@`hostname||uname -n|sed 1q` `who -m|sed -e "s%^.* \(pts/[0-9]*\).*(\(.*\))%[\1] (\2)%g"` [`uptime|sed -e "s/.*: \([^,]*\).*/\1/" -e "s/ //g"` / `ps aux|wc -l`]\007" * Changing the terminal title to the last shell command >> [[ "x$TERM" == "xrxvt" || "x$XTERM_VERSION" == xXTerm* || "x$COLORTERM" == 'gnome-terminal' && "x$SHELL" == */bin/zsh ]] && preexec () { print -Pn "\e]0;$1\a" } RESETTING THE TERMINAL .... * Salvage a borked terminal >> echo c * Salvage a borked terminal >> stty sane BUSYBOX Busybox is a minimal set of commands including a version of the bash shell designed to be used on 'small' computers (having limited resources, such as memory, disk storage etc) CURIOSITIES * implement a daemon using bash >> echo "Starting Daemon"; ( while :; do sleep 15; echo "I am still running =]"; done ) & disown -h -ar $! A 'daemon' is a program which runs in the background waiting to provide some service or respond to some situation. * Search back through previous commands >> Ctrl-R * turn on/off keyboard leds via commandline >> xset led 3 * Let your computer lull you to sleep >> echo {1..199}" sheep," | espeak -v english -s 80 * Change newlines to space in a file by using echo >> echo $(> PS1="\`if [ \$? = 0 ]; then echo \e[33\;40m\\\^\\\_\\\^\e[0m; else echo \e[36\;40m\\\-\e[0m\\\_\e[36\;40m\\\-\e[0m; fi\` \u \w:\h)" BASH 4.0 TOPICS Bash 4.0 is said to be introducing recursive globbing with the '**' operator. * change to the home ($HOME) folder in zsh or the bash v4 shell >> ~ * arithmetic series with a 'step' value in bash version 4 >> echo {1..8..2} THINGS TO INVESTIGATE * teminator * xclip * The cli apps site * launchy for lauching scripts * sam2p image processing * SIR simple image resizer webpack for reducing websites pngout /etc/ppp/ip-up for dns makeself httrack UTAS * on apple: smb://alacritas * mount a samba share from mac osx at the university of tasmania >> mount -t smbfs //user@alacritas/user smb UNIX CULTURE Unix embodies a particular type of culture. Here are some of the things it encompasses. == unix culture .. use folders for what their designed for, put temp stuff in /tmp .. lots of pipes .. short commands .. DOCUMENT-NOTES: # this section contains information about the document and # will not normally be printed. #-- a small icon to represent the document document-icon: #-- a larger image to represent the book document-image: #-- what sort of document is this document-type: book #-- in what kind of state (good or bad) is this document document-quality: average #-- a history of work on the document document-history: @@ 2009 document begun @@ jan 2010 small revisions. added information about the bash history file and environment variables @@ 31 march 2010 some tidying up and new section names @@ 2 april 2010 classifying and tidying some recipes. Suddenly realised the importance of functions as a simple way to add new commands to the shell and the advantage over an alias @@ 4 april 2010 moved some chapters around and expanded the functions section a little. what does 'export' do. Added some 'brace expansion' material. Read some information at stackoverflow.com @@ 23 april 2010 doing some work on vim commands which will take screen-shots of of the output of bash commands. For example 'Bwti' and 'Bwi' will create images of the output of the current line in the document executed as a bash command. The problem still remains the formatting in the generated pdf document (via latex). When the image is aligned to the right it ruins the formatting of the pdf document. @@ 2 may 2010 A few notes about string variables @@ 6 may 2010 Incorporated some notes about arrays and reading things one line at a time as opposed to one word at a time. Discovered a things called 'jot' which seems a good way to make random text data, which is useful when providing examples for commands. Discovered 'rs' and 'lam' for text arrays. @@ 21 may 2010 Started writing a glossary or dictionary of technical or jargon terms. Also did some cleaning up and rewriting of recipes. Discovered a technique for using the IFS variable to initialise arrays or control automatic splitting for the bash globbing functions and for loops. @@ 25 march 2011 Some edits about lists. #-- who wrote this document-authors: mjbishop at fastmail dot fm #-- a short description of the contents, possibly used for doc lists short-description: 'recipes for bash shell scripting and the command line' #-- in what computer language are the examples written code-language: bash #-- the script which will be used to produce html (a webpage) make-html: http://bumble.sf.net/books/format-book/booktohtml.cgi #-- the script which will produce 'LaTeX' output (for printing, pdf etc) make-latex: http://bumble.sf.net/books/format-book/booktolatex.cgi NOTES * prevent an ssh server logging you off for inactivity ----- #!/usr/bin/perl $|++; while (1) { print "\e[0n"; sleep 60; } ,,, * Print all fields in the tab delimited file 'file.dat' except the first >> cut -f2- file.dat * Print all fields in a space delimited file except the first and 2nd >> cut -d' ' -f3- file.dat * Update twitter via curl as Function >> tweet(){ curl -u "$1" -d status="$2" "http://twitter.com/statuses/update.xml"; } * Preview of a picture in a terminal >> img test.jpg * Sum columns from CSV in the 3rd column >> perl -F',' -ane '$a += $F[3]; END { print $a }' test.csv * Base conversions with bc >> echo "obase=2; 27" | bc -l * use vim to get colorful diff output >> svn diff | view - * Go to parent directory of filename edited in last command >> cd !$:h * Tweet from Terminal to twitter ! >> curl -u yourusername:yourpassword -d status=?Your Message Here? * from the console, start a second X server >> xinit -- :1 * Make shell (script) low priority. Use for non interactive tasks >> renice 19 -p $$ * Pick a random line from a file >> sort -R file.txt | head -1 * ignore hidden directory in bash completion (e.g. .svn) >> bind 'set match-hidden-files off' * Recursive cat - concatenate files (filtered by extension) >> find . -type f -name *.ext -exec cat {} > file.txt \; * Display or use a random file from current directory >> $ i=(*);echo ${i[RANDOM%(${#i[@]}+1)]]} * show a scrollable colorized long file listing - hidden files sorted last >> less -Rf <( cat <(ls -l --color=always) <(ls -ld --color=always .*) ) * show disk usage using an 'ncurses' display >> ncdu directory_name The output file in postscript format will be called 'new.ps' * Go up multiple levels of directories quickly and easily. >> cd() { if [[ "$1" =~ ^\.\.+$ ]];then local a dir;a=${#1};while [ $a -ne 1 ];do dir=${dir}"../";((a--));done;builtin cd $dir;else builtin cd "$@";fi ;} * Interactively build regular expressions >> txt2regex * Go (cd) directly into a new temp folder >> cd $(mktemp -d) * Convert seconds to human-readable format >> date -d@1234567890 * Create a script of the last executed command >> echo "!!" > foo.sh * diff two unsorted files without creating temporary files >> diff <(sort file1) <(sort file2) * Set CDPATH to ease navigation >> CDPATH=:..:~:~/projects * Copy a directory recursively without data/files >> find . -type d -exec env d="$dest_root" sh -c ' exec mkdir -p -- "$d/$1"' '{}' '{}' \; * Replace spaces in filenames with underscorees >> ls | while read f; do mv "$f" "${f// /_}";done * Redirect a filehandle from a currently running process. >> yes 'Y'|gdb -ex 'p close(1)' -ex 'p creat("/tmp/output.txt",0600)' -ex 'q' -p pid * cycle through a 256 colour palette >> yes "$(seq 1 255)" | while read i; do printf "\x1b[48;5;${i}m\n"; sleep .01; done * rename files >> mmv 'banana_*_*.asc' 'banana_#2_#1.asc' * Extract a bash function >> sed -n '/^function h\(\)/,/^}/p' script.sh * search for lines not containing any of the patterns. >> grep 'test' somefile | grep -vE '(error|critical|warning)' * Shell function to exit script with error in exit status and print >> die(){ result=$1;shift;[ -n "$*" ]&&printf "%s\n" "$*" >&2;exit * show ls colors with demo >> echo $LS_COLORS | sed 's/:/\n/g' | awk -F= '!/^$/{printf("%s \x1b[%smdemo\x1b[0m\n",$0,$2)}' * silent/shh - shorthand to make commands really quiet >> silent(){ $@ > /dev/null 2>&1; }; alias shh=silent * bash shortcut: !$ !^ !* !:3 !:h and !:t >> echo foo bar foobar barfoo && echo !$ !^ !:3 !* && echo /usr/bin/foobar&& echo !$:h !$:t * List only directory names >> ls -d */ * Wrap text files on the command-line for easy reading >> fold -s * Rot13 using the tr command >> alias rot13="tr '[A-Za-z]' '[N-ZA-Mn-za-m]'" * printing barcodes >> ls /home | head -64 | barcode -t 4x16 | lpr * Search for a single file and go to it >> cd $(dirname $(find ~ -name emails.txt)) * disable history for current shell session >> unset HISTFILE * Mute xterm >> xset b off * save date and time for each command in history >> export HISTTIMEFORMAT="%h/%d-%H:%M:%S " * lotto generator >> echo $(shuf -i 1-49 | head -n6 | sort -n) * bash screensaver off >> setterm -powersave off -blank 0 * Carriage return for reprinting on the same line >> while true; do echo -ne "$(date)\r"; sleep 1; done * Just run it >> echo SSBMb3ZlIFlvdQo= | base64 -d * Open a man page as a PDF in Gnome >> TF=`mktemp` && man -t YOUR_COMMAND >> $TF && gnome-open $TF * output 'hello' when the computer starts >> echo "echo hello" >> /etc/rc.local * how long system has been running >> uptime * send Everyone on your machine some love >> w | egrep -v '(load|FROM)' | awk '{print $2}' | sed 's/^/tty/' | awk '{print "echo \"The Matrix has you...\" >> /dev/" $1}' | bash * List all system users >> ~ * logout and obliterate the bash history file >> echo 'alias q="cat /dev/null > ~/.bash_history && exit"' >> .bashrc * eject the cd rom device >> while true; do eject /dev/cdrom && eject -t /dev/cdrom; done * Duplicate a directory tree using tar and pipes >> (cd /source/dir ; tar cv .)|(cd /dest/dir ; tar xv) * how long system has been running >> last reboot * emptying a file >> cat /dev/null >filename * dd and ssh >> dd if=/dev/zero bs=1M | ssh somesite dd of=/dev/null * alias to list hidden files of a folder >> alias lh='ls -a | egrep "^\."' * renombrar un archivo que inicia con guion >> find . -name "-help" -exec mv {} help.txt \; * show current directory >> nautilus `pwd` * easiest way to get kernel version without uname >> strings /boot/kernel-file | grep 2.6 * copy files to the current folder >> locate PISTA | xargs -I{} cp {} . * alias to list hidden files of a folder >> alias lh='ls -a | egrep "^\."' * Go to the created directory after using mkdir >> mkdir() { /bin/mkdir $@ && eval cd "\$$#"; } * change your PS1 to look better :) >> newhostname=$(hostname | awk -F. '{print $1 "." $2}'); ipaddress=$(nslookup `hostname` | grep -i address | awk -F" " '{print $2}' | awk -F. '{print $3 "." $4}' | grep -v 64.142);PS1="[`id -un`.$newhostname.$ipaddress]"' (${PWD}): '; export PS1 * Clear current session history >> history -r * count occurences of each word in novel David Copperfield >> wget -q -O- http://www.gutenberg.org/dirs/etext96/cprfd10.txt | sed '1,419d' | tr "\n" " " | tr " " "\n" | perl -lpe 's/\W//g;$_=lc($_)' | grep "^[a-z]" | awk 'length > 1' | sort | uniq -c | awk '{print $2"\t"$1}' * A bash prompt which shows the bash-version >> PS1="$BLUE[$CYAN\u$BLUE@$CYAN\h$WHITE-bash \v:$GREEN\w$BLUE]$WHITE \$ " * aliases for apt-get >> alias agi="sudo apt-get install" #package_names * See smbstatus all the time >> while (( $i != 0 )) { smbstatus; sleep 5; clear } * Super Speedy Hexadecimal or Octal Calculations and Conversions to >> echo "$(( 0x10 )) - $(( 010 )) = $(( 0x10 - 010 ))" * Connect to google talk through ssh by setting your IM client to >> ssh -f -N -L 5432:talk.google.com:5222 user@home.network.com * Create a newFolder that is a symbolic link to another folder >> ln -s /destinationTarget /sourceTarget/newFolder * easily monitor memory usage >> watch -n1 --differences cat /proc/meminfo * Encoding with base64 >> echo "Hello world" | base64 * journaling directories >> mkdir `date | sed 's/[: ]/_/g'` * Show directories in the PATH, one per line >> print -l $path * run command with opposite return code >> not () { "$@" && return 1 || return 0; } * Start xterm in given directory >> xterm -e "cd /my/directory; bash" * Ignore the specified signal >> trap '' 1 2 20 24(signal number) * Compile a latex doc to generate index >> ruby -e " 3.times { system 'pdflatex mydoc.tex' } " * copies 20 most recently downloaded mp3 files (such as from Miro) >> find . -name \*.mp3 -printf "%C+ %h/%f\n" | sort -r | head -n20 | awk '{print "\""$2"\""}' | xargs -I {} cp {} ~/tmp * Adding formatting to an xml document for easier reading >> xmllint --format > * use a literal bang (exclamation point) in a command >> echo '!'whammy * find and replace tabs for spaces within files recursively >> find ./ -type f -exec sed -i 's/\t/ /g' {} \; * Quicker move to parent directory >> alias ..='cd ..' * Update your journal >> vi ~/journal/$(date +%F) * Doing some floating point calculations with rounding (e.g. at the >> echo '123/7' |bc -l |xargs printf "%.3f\n" * Go to directory or create it and go to it >> [[ -d dir ]] || mkdir dir ; cd dir * nohup that doesn't generate nohup.out >> nohup 2> /dev/null > /dev/null & * Help shell find freshly installed applications (re: PATH) >> rehash * Read almost everything (Changelog.gz, .tgz, .deb, .png, .pdf, >> less -r * Show directories >> ls -l | grep ^d * listen to an offensive fortune >> fortune -o | espeak * Run a command if today is the last day of the month >> if [[ `:$(cal);echo $_` == `date +%d` ]]; then ROTATE_MONTHLY_TABLES_SCRIPT;fi * Compare two CSV files, discarding any repeated lines >> cat foo.csv bar.csv | sort -t "," -k 2 | uniq * auto terminal title change >> echo -ne "\033]0;${USER}@${HOSTNAME}: ${PWD}\007" * apt-get via sudo >> apt-get () { [ "$1" = source ] && (command apt-get "$@";true) || sudo apt-get "$@" } * Simple example of the trap command >> trap "echo \"$0 process $$ killed on $(date).\"; exit " HUP INT QUIT ABRT TERM STOP * clear current line >> ctrl+u * ls -hog is a more compact but informative file listing >> ls -hog * Start a new command in a new screen window >> alias s='screen -X screen'; s top; s vi; s man ls; * make sure a script is run in a terminal. >> [ -t 0 ] || exit 1 * Identify long lines in a file >> awk 'length>72' file * Show directories in the PATH, one per line >> echo "${PATH//:/$'\n'}" * Find if the command has an alias >> type -all command * Share a 'screen'-session >> screen -x * Open a file at the specified line >> emacs +400 code.py * Search for matches each at start of line with egrep >> egrep '^(January|April|May)' text.log * Creating sequence of number with text >> seq 10 |xargs -n1 echo Printing line * colored tail >> tail -f FILE | grep --color=always KEYWORD * Run a command, store the output in a pastebin on the internet and >> ls | curl -F 'sprunge=<-' http://sprunge.us | xclip * find files containing text >> grep -lir "some text" * * Find recursively, from current directory down, files and >> find ./ -name '*' -exec rename 's/\s+/_/g' {} \; * cycle through a 256 colour palette >> yes "$(seq 232 255;seq 254 -1 233)" | while read i; do printf "\x1b[48;5;${i}m\n"; sleep .01; done * cycle through a 256 colour palette >> yes "$(seq 232 255;seq 254 -1 233)" | while read i; do printf "\x1b[48;5;${i}m\n"; sleep .01; done * Typing the current date ( or any string ) via a shortcut as if >> xvkbd -xsendevent -text $(date +%Y%m%d) * Amazing real time picture of the sun in your wallpaper >> curl http://sohowww.nascom.nasa.gov/data/realtime/eit_195/512/latest.jpg | xli -onroot -fill stdin * Allow to shorten the prompt. Useful when the it is taking too >> PS1='$' * Graphic mode for root >> startx -- :1 * Change default terminal emulator >> update-alternatives --config x-terminal-emulator * Nice directory listings >> alias ll="ls -lh --color=auto" * Graphically compare two directory hierarchies without Subversion >> xxdiff -r --exclude=.svn * To generate the list of dates using bash shell >> now=`date +"%Y/%m/%d" -d "04/02/2005"` ; end=`date +"%Y/%m/%d" -d "07/31/2005"`; while [ "$now" != "$end" ] ; do now=`date +"%Y/%m/%d" -d "$now + 1 day"`; echo "$now"; done * a pseudo-random coin flip in python >> echo "import random; print(random.choice(['heads', 'tails']))" | python * add a little color to your prompt >> PS1="\[\033[44;1;37m\]\u\[\033[0m\]@\h\\$ " * Make bash look like DOS >> export PS1='C:${PWD//\//\\\}>' * add a little color to your prompt >> PS1="\[\033[44;1;37m\]\u\[\033[0m\]@\h\\$ " * cd canonical (resolve any symlinks) >> alias cdc='cd `pwd -P`' * Make sure a script is run in a terminal. >> tty > /dev/null 2>&1 || { aplay error.wav ; exit 1 ;} * teatimer >> sleep 3m; play bigben.wav * Copy the full path of a file to the clipboard >> >realpath ./somefile.c | xclip -selection c * Day Date Time> Instead of $ or # at the terminal >> export PS1='\D{%a %D %T}> ' * Play music from pure data >> sudo cat /usr/share/icons/*/*/* > /dev/dsp * Quick alias for case-insensitive grep >> alias grip="grep -i" * Puts every word from a file into a new line >> < tr ' \t' '\n' | tr -s '\n' > * Creates a customized search command >> alias cr='find . 2>/dev/null -regex '\''.*\.\(c\|cpp\|pc\|h\|hpp\|cc\)$'\'' | xargs grep --color=always -ni -C2' * Prevent overwriting file when using redirection >> set -o noclobber * Poor man's ntpdate >> date -s "`curl -sI www.example.com | sed -n 's/^Date: //p'`" * get bofh excuse from a trusted source :-) >> telnet bofh.jeffballard.us 666 * run a command replacing foo with bar >> !!:gs/foo/bar