&& Web Development using Linux ------------------------------ This book seeks to provide recipes for developing web sites using the Linux operating system, with an emphasis on command line tools. This book is not about html and css since that is covered in the html-css-book.txt VISUAL PAGE CREATION WYSIWYG @@ kompozer a graphical tool for creating pages. successor of nvu @@ bluefish an html editor IMAGES @@ linux-image-book.txt more comprehensive information about managing images with tools available for the Linux operating system. IMAGE COMPRESSION .... == tools for image compression .. webpack - .. pngout - .. == image tools .. IMAGE RESIZING .... * convert a all '.html' files from iso-8859-1 to utf-8 file ------------------------------------------------------------- for x in $(find . -name '*.html') do iconv -f ISO-8859-1 -t UTF-8 $x > "$x.utf8"; rm $x; mv "$x.utf8" $x; done ,,, MANAGING IMAGES .... Feh is a flexible and fast tool for managing images which will be included in a web-site. See the linux-image-book for more details. * use feh >> feh ASCII IMAGES .... Converting images to 'ascii' (normal text characters) may in some circumstances be aethetically interesting. They could then be displayed within
 tags with an extremely small font.
 * convert an image to ascii 
 >> jp2a --size=40x20 imagefile.jpg


  @@ ffmpeg
     recode video files into different formats

  == video formats
  .. DivX
  .. Digital video - used by camcorders
  .. avi
  .. mpeg

  == video tools for web-development
  .. stills2dv - creates videos from still images
  .. kino - edit digital video data

  * install kino on linux
  >> sudo apt-get install kino


  Unicode characters can be used in web-pages but this technique
  requires that the viewer of the webpage has an adequate font 
  installed in order to be able to view the character.

  == some special chars
  .. U+260E - ☎ ☎ ☎  BLACK TELEPHONE
  .. U+260F - ☏ ☏ ☏  WHITE TELEPHONE
  .. U+265B - ♛ ♛ ♛       BLACK CHESS QUEEN
  .. U+265C - ♜ ♜ ♜       BLACK CHESS ROOK
  .. U+265D - ♝ ♝ ♝       BLACK CHESS BISHOP
  .. U+265E - ♞ ♞ ♞       BLACK CHESS KNIGHT
  .. U+265F - ♟ ♟ ♟       BLACK CHESS PAWN
  .. U+2660 - ♠ ♠ ♠       BLACK SPADE SUIT

  * The unicode page 'miscelaneous characters' runs from
  >> 9728-9865 (decimal) or x2600-x2689

  * print out 9 random entities from the miscelaneous unicode page
  >> echo "&#"{9728..9865}";" | xargs shuf -e | head -9 

  * create a little tabble of random miscelaneous entities 
  >> echo "&#"{9728..9865}";" | xargs shuf -e | head -15 | (echo ""; cat -; echo "
") ANIMATIONS @@ synfig studio a tool to create animations without 'tweening' (that is, drawing every image) FILE TRANSFER == tools for file transfer over the net .. ftp - the old file transfer tool .. rsync - transfer only changed or new files .. sftp - an interactive secure version of sftp .. scp - a non-interactive secure ftp .. sitecopy - synchronize a remote site with what is local .. WEB SITE MIRRORING * Download all images from a site >> wget -r -l1 --no-parent -nH -nd -P/tmp -A".gif,.jpg" http://example.com/images TRANSFERING FILES An important part of creating a website is the process of moving file to and from the server. One wants to do this in a simple and efficient manner, while guarding against the possibility of accidentally overwriting files with older or bad versions. Unix and Linux have a wide smorgasboard of tools for the purpose of achieving these transfers. == command line file transfer tools .. rsync - powerful and complicated, synchronise files .. scp - fast and simple, securely copy .. sftp - an interactive way to copy files across the web .. == older tools .. ftp - an insecure copy method, use sftp instead .. == graphical tools .. the firefox copy ?? .. SSH FILE SYSTEM .... Linux has a special file system available called the 'sshfs' which allows the user to mount a remote folder tree as if it was a local file system (although the accesses will seem a little slower). This may be more enjoyable than using an ssh shell since the remote file system 'seems' to be local. The only requirement is that the remote computer be running an 'ssh' secure shell server and that you have a login name and password for that system. * install the sshfs remote file system. >> sudo apt-get install sshfs * mount an ssh (remote) file system >> sshfs name@server:/path/to/folder /path/to/mount/point * unmount an ssh file system >> fusermount -u ~/sshfs_mounted_directory SCP .... Scp stands for 'secure copy program'. It is a non-interactive tool well suited for including in scripts. * bash function to upload a file via scp to the 'rpike' a/c on the server >> upfile() { scp $1 rpike@server.net:$1 } * upload and overwrite all '.txt' files to the 'doc/' folder on far.net >> scp *.txt eva@far.net:doc/ * download 2 files from the server using the account 'user' >> scp user@server.net:"chap1.txt chap2.txt" ~/books POSTING DATA * Submit data to a HTML form with POST method and save the response >> curl -sd 'rid=value&submit=SUBMIT' > out.html * post with a proxy and authentication >> curl -F name='../htdocs/notes/'$1 -F contents='<'$1 -u user:upass -x prox.net:8080 -U bob:proxpass http://serv.net/save.cgi HTML STUFF HTML LINKS .... * extract the hyperlinks ( tags) from a webpage >> lynx -dump -listonly www.server.net/page.html * find most of the urls within an html file >> egrep 'https?://([[:alpha:]]([-[:alnum:]]+[[:alnum:]])*\.)+[[:alpha:]]{2,3}(: \d+)?(/([-\w/_\.]*(\?\S+)?)?)?' ENTITIES * encode HTML entities >> perl -MHTML::Entities -ne 'print encode_entities($_)' /tmp/subor.txt * or use xmlstarlet to encode entities. BASH AND WEB DEVELOPMENT Using the bash shell to develop web-sites maybe quite efficient, if unconventional. * possibly the simplest way to create a web-page from text >> cat file.txt | (echo '
'; cat -; echo '
') * a simpler way >> echo '
'; cat file.txt; echo '
') TEMPLATING WITH BASH .... * a simple template technique with bash >> export a=b; echo -e 'one\ntwo\nand ' | (echo 'cat << EE';sed 's//$a/g'; echo 'EE') | bash * use the technique above to substitute the date into the template >> cat template | (echo 'cat << EE';sed 's//$(date)/g'; echo 'EE') | bash FOLDER LISTINGS .... * list only folders >> ls -d */ | (echo '
    '; cat -; echo '
') * make an html directory listing out of the current folder >> echo "echo -e \"{$(echo * | tr ' ' ',')}"\" | bash * list all files and folders, no links >> a=$(echo *); echo 'echo -e "\n
  • "{'${a//" "/,}'}"
  • "' | bash * list only folders, no links >> a=$(echo */); echo 'echo -e "\n
  • "{'${a//" "/,}'}"
  • "' | bash * a for loop method to list only sub-folders as an html list ------------------------------------------------------------ echo "
      " for d in $(ls -d */); do echo "
    • $d
    • " done echo "
    " ,,, * another for loop method to list only sub-folders as an html list ------------------------------------------------------------ echo "
      " for d in */; do echo "
    • $d
    • " done echo "
    " ,,, * list subfolders as html links ------------------------------- echo "
    " ,,, * list subfolders as html links using a brace loop -------------------------------------------------- echo "" ,,, BASH CGI PROGRAMMING While it is most common for Cgi web-scripts to be written in the Perl language, it is also possible to write them using the normal Bash shell scripting language. Whether this is a good idea is completely another question... @@ http://en.wikipedia.org/wiki/Internet_media_type#List_of_common_media_types A list of common "media types" (such as "text/html") which are used in the "Content-Type:" field of the Cgi script. o- use the "2>&1" idiom at the end of script lines to redirect an error message to the "standard output" (which in the case of a Cgi script is the web-browser of the script visitor). This allows you, the developer to see what is going wrong with your bash cgi script. - using "here" documents with a bash cgi script is a simple way to produce content. - THE BASH CGI GOTCHAS .... o- the content-type line has to be before /anything/ or else nothing is printed. * there must be an empty line after the "content-type" line. ------------------------------------------------------------- echo "Content-Type: text/html" echo "...the query string is $QUERY_STRING" 2>&1 ,,, * a bash cgi script indicating that the character set is "utf8" --------------------------------------------------------------- #!/bin/bash echo "Content-Type: text/html; charset=utf-8" echo echo "A Bash UTF8 Cgi Script!" 2>&1 ,,, * show error messages in the browser generated by a cgi script line >> ech "this is a mistake" 2>&1 * show error messages in the browser with output redirection >> ech "this is a mistake" 2>&1 >save.txt ##(note that the 2>&1 should come before the file redirection) * a cgi script which displays several environment variables ---------------------------------------------------------- #!/bin/bash cat << ENDxxx Content-Type: text/html; charset=utf-8 A bash cgi script
    ENDxxx ,,, GETTING AND DECODING FORM DATA .... @@ http://oinkzwurgl.org/bash_cgi bash functions for decoding cgi form data @@ http://www.fpx.de/fp/Software/ProcCGIsh.html a bash script and c program for decoding cgi form data. The c program needs to be compiled. Data sent from an html form to a web server can be sent in 2 different ways; in the querystring itself (the 'GET' method) and in the HTTP headers (the 'POST' method). If the data is 'posted' then the cgi script will receive the data on the standard input Data sent from an html form has to be url decoded. * data "posted" from an html form can be read from the standard input >> read postdata ##(the content length should first be checked) >> postdata=$(> file=$(perl -e "use CGI qw(:standard); print param('file')) ##(the CGI module takes care of url-decoding the form data) == cgi environment variables and meanings .. DOCUMENT_ROOT, The root directory of your server .. HTTP_COOKIE, The visitor's cookie, if one is set .. HTTP_HOST, The hostname of your server .. HTTP_REFERER, The URL of the page that called your script .. HTTP_USER_AGENT, The browser type of the visitor .. HTTPS, "on" if the script is being called through a secure server .. PATH, The system path your server is running under .. QUERY_STRING, The query string (see GET, below) .. REMOTE_ADDR, The IP address of the visitor .. REMOTE_HOST, The hostname of the visitor (if server has reverse-name-lookups on; otherwise this is the IP address again) .. REMOTE_PORT, The port the visitor is connected to on the web server .. REMOTE_USER, The visitor's username (for .htaccess-protected pages) .. REQUEST_METHOD, either "GET" or "POST" .. REQUEST_URI, The interpreted pathname of the requested document or CGI (relative to the document root) .. SCRIPT_FILENAME, The full pathname of the current CGI .. SCRIPT_NAME, The pathname of the cgi script(relative to the doc root) .. SERVER_ADMIN, The email address for your server's webmaster .. SERVER_NAME, Your server's fully qualified domain name (e.g. www.cgi101.com) .. SERVER_PORT, The port number your server is listening on .. SERVER_SOFTWARE, The server software you're using (such as Apache 1.3) .. * some environment variables and example values ----------------------------------------------- SERVER_SOFTWARE = Apache/2.0.54 (Fedora) SERVER_NAME = www.comp.leeds.ac.uk GATEWAY_INTERFACE = CGI/1.1 SERVER_PROTOCOL = HTTP/1.1 SERVER_PORT = 80 REQUEST_METHOD = GET HTTP_ACCEPT = 'text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5' PATH_INFO = PATH_TRANSLATED = SCRIPT_NAME = /cgi-bin/Perl/environment-example QUERY_STRING = REMOTE_HOST = REMOTE_ADDR = REMOTE_USER = CONTENT_TYPE = CONTENT_LENGTH = ,,, == cgi Content-Type: values .. .. text/html, for html content .. text/plain, for plain text .. text/plain; charset=utf-8, plain text in the utf8 encoding .. text/xml, xml text .. CONVERTING HTML PLAIN TEXT CONVERSION .... * Display command lines visible on commandlinefu.com homepage >> ruby -ropen-uri -e 'require "hpricot";(Hpricot(open("http://commandlinefu.com"))/".command").each{| c| puts c.to_plain_text}' HTTP http is the protocol apon which the web is based. All web-servers must implement at least a part of this protocol. HTTP HEADERS .... * Dump HTTP header using lynx or w3m >> lynx -dump -head http://www.example.com/ XML AND HTML xml and html are closely related mark-up languages. XHTML == tools .. xmlstarlet - queries and edits xml from the command line .. VALIDATION Validation is the process of confirming that a particular html, xhtml, or css document actually conforms to the technical specifications for that document. Historically browsers have been designed to display more or less correctly even html and xhtml pages which contain 'mistakes' or syntax which does not conform to the technical specification for that type of document. @@ http://validator.w3.org validates xhtml @@ http://jigsaw.w3.org/css-validator a css validator @@ tidy.sourceforge.net A command line html validator and pretty printer. * install the html tidy program and docs on a debian type linux >> sudo apt-get tidy tidy-doc * view the html tidy documentation >> ls /usr/share/doc/tidy-doc/htmldoc >> xdg-open /usr/share/doc/tidy-doc/htmldoc/Overview.html * test out tidy printing errors and warnings >> echo "" | tidy -e - 2>&1 | less CURL STUFF * getting a page via an authenticating proxy server >> curl -x proxy.utas.edu.au:8080 -U bobj http://www.server.net * get a page via authenticating proxy server as user 'bob' password 'sec' >> curl -x proxy.net:8080 -U bob:sec http://www.server.net Supplying the password in this manner is possibly not a good idea from a security point of view * download a text file through a proxy and edit it with vim ----------------------------------------------------------- function edn { curl -x proxy.org.au:8080 -U bob:pass www.serv.net/a.txt -o ~/notes.txt vim ~/notes.txt } ,,, * upload a file to webserver w cgi script with http authententication, -------------------------------------------------------------------- function up { [ -z "$1" ] && echo 'no parameter' && return 1; curl -F name='../htdocs/notes/'$1 -F contents='<'$1 -u user:upass -x prox.net:8080 -U bob:proxpass http://serv.net/save.cgi } ,,, PERL TRICKS == useful modules .. www::mechanize .. lwp .. * a perl mechanize example -------------------------- # navigate to the main page $mech->get('http://www.somesite.com/'); # follow a link that contains the text 'download this' $mech->follow_link( text_regex => qr/download this/i ); # submit a POST form, to log into the site $mech->submit_form( with_fields => { username => 'mungo', password => 'lost-and-alone', } ); # save the results as a file $mech->save_content('somefile.zip'); ,,, PHP * Testing php configuration >> php -r "phpinfo\(\);" * get the urls from a webpage >> $browser->getUrls() CRAWLING .. ScriptableBrowser (simpletest) LOAD TESTING == tools .. funkload - web testing .. STUFF USED IN GOOGLE CHROME == tools used in google chrome .. bsdiff, .. bspatch, .. bzip2, .. dtoa, .. hunspell, .. ICU, .. JSCRE, libjpeg, libpng, libxml, libxslt, LZMA SDK, modp_b64, .. Mozilla interface to Java Plugin APIs, .. npapi, nspr, nss, Pthreads for win32, sqlite, .. tlslite, V8 assembler, WebKit, WTL, zlib .. FORUM SITES @@ ask.metafilter.com visited by knowledgable people @@ SERVING PAGES * share the current directory tree (via http) at http://$HOSTNAME:8000/ >> python -m SimpleHTTPServer * create a webserver to share all files in /tmp/mydocs on port 8081 >> wbox servermode webroot /tmp/mydocs * create a webserver to share all files in /tmp/mydocs on port 8080 >> wbox servermode serverport 8080 webroot /tmp/mydocs * Sharing file through http 80 port >> nc -w 5 -v -l -p 80 < file.ext CGI SERVERS * a simple cgi server >> python -m CGIHTTPServer 8080 * some kind of perl cgi server >> HTTP::Simple::PSGI SMALL WEBSERVERS @@ http://ask.metafilter.com/65481/Help-me-find-a-cool-little-unix-http-utility-I-cant-remember some good things about mini web servers @@ http://hping.org/wbox/ site for wbox == small quick and easy webservers .. wbox - http server .. thttpd - small web server .. mini_httpd - same author as thttpd but smaller .. webfs - serves a file system from the web .. busybox httpd command - small webservery thing .. WBOX * show how long each part of a webpage takes to generate >> wbox nowhere.net/page.html timesplit 1 * show the http header information for a page >> wbox www.google.it/notexistingpage.html 1 showhdr MAC OSX Installed by default are php, curl, TEMPLATING @@ http://www.perl.com/pub/a/2001/08/21/templating.html an article about using templating systems with perl == tools .. Template Toolkit - almost active development, perl, and python .. HTML::Mason - ? callback style, active as of 2010 .. Embperl - embedd perl into webpages, stopped 2006 .. HTML::Template - perl template module .. Text::Template - a general purpose templater .. Apache::ASP - use asp with apache, stopped 2004 .. CGI::FastTemplate - another one .. @@ http://template-toolkit.org/ the site for the template toolkit TEMPLATE TOOLKIT * example statement using dot notation. >> How are things in [% customer.address.city %]? * a for loop [% FOREACH list %] [% name %] [% END %] ,,, HTML TEMPLATE * example loop with html::template ----- ,,, MASON .... Seems to be in active development. It can be run without a webserver @@ http://www.masonhq.com/ the official site * install mason using apt-get >> apt-get install libmason-perl ##(??? unchecked) PERL STUFF == some interesting perl web modules .. Dancer - perl web apps with good examples .. Web::Simple - simple web apps .. CGI::Application .. Catalyst - big web apps .. Mason .. Mojolicious .. WEB SIMPLE .... @@ http://search.cpan.org/~mstrout/Web-Simple-0.002/lib/Web/Simple.pm the documentation for web simple Developed in 2009. Can create a webapplication without a webserver @@ hobbs at stackoverflow.com knowledgable perl web person CPAN TOOL CRASH COURSE * start a cpan shell to install mason >> perl -MCPAN -e 'shell' == simple cpan shell .. h - show help .. get - get the source for a module .. make - compile ? the module .. test - test a module .. install - do all of get, make, test .. clean - get rid of a badly installed module .. look - see whats happening .. readme - see whats going on .. @@ http://www.livejournal.com/doc/server/lj.install.perl_setup.modules.html a list of potentially useful modules used with livejournal * upgrade cpan but could cause problems ??? ------------------------------------------- perl -MCPAN -e shell cpan> install Bundle::CPAN cpan> reload cpan ,,, GETTING WEB PAGES FROM THE COMMAND LINE * http get of a web page via proxy server with login credentials >> curl -U username[:password] -x proxy:proxyport webpage * use netcat to get a webpage >> echo "GET / HTTP/1.0\r\r" | nc -v www.somewebsite.com 80 * get a webpage with the console php tool >> php -r "file('http://metafilter.com/');" * get a webpage with perl >> perl -MHTTP::Client -e 'print HTTP::Client->new()->get("http://localhost/path")' * a simpler way to do the same >> perl -MLWP::Simple -e 'get("http://www.metafilter.com/")' LOAD TESTING * perform 4 queries per second (with 4 processes) on the local webserver >> wbox http://localhost clients 4 BLOG ENGINES == .. serendipity - a blog engine with database backend .. bilboblog - a simple engine .. CMS SYSTEMS == a small summary .. redmine - popular small software orientated (ruby NOTES google uses python @@ http://unixmages.com/ an unrelated but interesting site GOOGLE CLI Google provides a command line interface to its services such as the 'picasa' photo album service. This may be useful for scripting etc. * post to a blogger blog >> google blogger post --tags "GoogleCL, awesome" --title "Test Post" "I'm posting from the command line" * post to a blogger blog using the file 'post.txt' as the text >> google blogger post post.txt >> google blogger list title,url-site # List posts >> google blogger delete --title "Test Post" >> google delete --title "Silly post number [0-9]*" # Delete posts matching regex >> google tag --title "Dev post" --tags "Python, software" # label an existing post * examples of putting photos in picasa from the command line ------------------------------------------------------------ google picasa create --title "Vermont Test" --tags Vermont vermont.jpg google picasa get --title "Vermont Test" /path/to/download/folder google picasa list title,url-direct --query "A tag" google picasa post --title "Vermont Test" ~/old_photos/*.jpg # Add to an album google picasa tag --title "Vermont Test" --tags "places" google picasa delete --title "Vermont Test" # delete entire album ,,, * examples of posting to youtube from command line -------------------------------------------------- google youtube post --category Education --devtags GoogleCL killer_robots.avi google youtube delete --title "killer_robots.avi" google youtube list # list my videos google youtube tag -n ".*robot.*" --tags robot ,,, MISCELANEOUS * query wikipedia via dns by requesting a 'txt' record >> dig +short txt .wp.dg.cx * query wikipedia via dns for the term 'bash' >> dig +short txt bash.wp.dg.cx JARGON == web jargon @@ the net and the web The net is the internet and all its protocols such as ftp, ssh, http etc where as the web consists of just the http and https protocol and therefor webpages. @@ download transfer files from a net server to your local computer @@ markup a way of embedding extra information in a document with the use of 'tags' or 'codes'. Html is an example of markup. @@ minimalist mark-up minimalist mark-up is a markup language which attempts to use the minimum number of tags possible to embedd in the document. The purpose of this system is not to interfere psychologically with the human readers and writers of the document who may be distracted and confused by bulky and numerous tags. An example of minimalist markup is 'markdown' or most 'wiki' markup languages @@ wiki Is a web site or document which many or most web-visitors are able to edit. @@ web 2.0 The term encapsulates the idea of the web as a collaborative medium for the compilation and consumption of information, rather that a 'uni-directional' information source. The television is an example of a uni-directional information source since the viewer is unable to effect its transmissions, apart from changing channel or switching off. The web 2.0 is multi-directional because the user can create and modify the content (through the use of wikis, and collaborative sites). @@ tags mark-up languages often use 'tags' in order to embedd information in text documents. Usually there is a start tag and an end tag which surrounds the information to which it pertains. For example in the text "tree" the word 'tree' is surrounded by the start and end html tag "". This system is used by various mark-up languages such as html, xml and sgml. @@ semantic markup the use of mark-up codes or tags to indicate 'semantic' information rather than 'visual display or layout' information. The phrase 'semantic information' may be a tautology. An example of 'visual display' markup are the html and tags which indicate to the document rendering software the text should be displayed bigger or small than the normal sized text. Semantic markup is considered to be 'good' in the w3c world and among ideologists, since information is not lost in a forest of meaningless layout tags. @@ plain text plain text is the basis of the internet and web. Plain text is any data which represents a stream of characters in a human writing system which is encoded electronically according to a standard 'text encoding'. An example of these encoding are utf8, ascii, or latin1. Modern unicode text encodings (such as utf8) can represent any character in any human language which has an established writing system @@ separation of content and presentation a much bandied about phrase. @@ style sheets documents which indicate how other documents should be displayed. A number of style sheet languages exist but the most common is 'css' cascading style sheets. @@ cms - content management system A content management system is supposedly a way to quickly build websites focussing on the content of that site rather than web-development issues and design issues. There are a dizzying array of free and open-source cms systems available @@ version control system is a way of managing changes to files made by potentially many people, simultaneously @@ database a way of storing data which is supposedly secure and convenient. many database systems exist with the simplest being a text file database, and the most command being a 'relational' database which stores information in a series of linked tables. Databases are often used to store the content of web-sites @@ content This is a much used web-term which relates to the information which a web-site contains as opposed to the visual or technical design of the site @@ documentroot DOCUMENT-NOTES: # this section contains information about the document and # will not normally be printed. # A small (16x16) icon image to identify the book document-icon: # A larger image to identify or illustrate the title page document-image: # what sort of document is this document-type: book # in what kind of state (good or bad) is this document document-quality: just begun # who wrote this document-history: @@ march 2010 This book was begun in order to tackle issues such as the batch resizing of images, techniques for quickly creating websites, file transfer, templating, etc. This book doesnt cover html and css. @@ 1 april 2010 some notes inserted @@ 7 april 2010 included some more notes. @@ 11 april 2010 added a small amount of bash scripting information, including a simple technique for templating. @@ 19 may 2010 added a jargon section to explain certain common terms. Also added a reference to the 'redmine' cms system. # who wrote this authors: mjbishop at utas dot edu dot au # a short description of the contents, possible used for doc lists short-description: Techniques for developing web-sites # A computer language which is contained in the document, if any code-language: various # the script which will be used to produce html (a webpage) make-html: ./booktohtml.cgi # the script which will produce 'LaTeX' output (for printing, pdf etc) make-latex: ./booktolatex.cgi NOTES * convert a web page into a png >> touch $2;firefox -print $1 -printmode PNG -printfile $2 >> urls=('www.ubuntu.com' 'google.com'); for i in ${urls[@]}; do http_code=$(curl -I -s $i -w %{http_code}); echo $i status: ${http_code:9:3}; done * Check availability of Websites based on HTTP_CODE >> urls=('www.ubuntu.com' 'google.com'); for i in ${urls[@]}; do http_code=$(curl -I -s $i -w %{http_code}); echo $i status: ${http_code:9:3}; done * Watch for when your web server returns >> watch -n 15 curl -s --connect-timeout 10 http://www.google.com/ >> gsed -e :a -e 's/\(<\/[^>]*>\)/\1\n/g;s/\(
    \)/\1\n/g' page2.txt | sed -n '//p;s/\(.*\)<\/cite>/\1/g' >> output * Check which files are opened by Firefox then sort by largest size. >> FFPID=$(pidof firefox-bin) && lsof -p $FFPID | awk '{ if($7>0) print ($7/1024/1024)" MB -- "$9; }' | grep ".mozilla" | sort -rn * use wget to check if a remote file exists >> wget --spider -v http://www.server.com/path/file.ext * Poke a Webserver to see what it's powered by. >> wget -S -O/dev/null "INSERT_URL_HERE" 2>&1 | grep Server >> for i in `ps aux | grep httpd | awk '{print $2}'`; do lsof -n -p $i | grep ESTABLISHED; done; * Regex to remove HTML-Tags from a file >> sed -e :a -e 's/<[^>]*>//g;/> firefox -no-remote -P * search for a place with google maps >> http://maps.google.com.au/maps?q=lindeith%20sandy%20bay%2018%20map&oe=utf-8&rls=com.ubuntu:en-US:official&client=firefox-a&um=1&ie=UTF-8&sa=N&hl=en&tab=wl * Update twitter via curl as Function >> tweet(){ curl -u "$1" -d status="$2" "http://twitter.com/statuses/update.xml"; } * Testing php configuration >> php -r phpinfo(); * Search and replace text in all php files with ruby >> ruby -i.bkp -pe "gsub(/search/, 'replace')" *.php * extract all urls from firefox sessionstore >> sed -e "s/\[{/\n/g" -e "s/}, {/\n/g" sessionstore.js | grep url | awk -F"," '{ print $1 }'| sed -e "s/url:\"\([^\"]*\)\"/\1/g" -e "/^about:blank/d" > session_urls.txt * single line twitter-tracker >> WRDS="word1 word2 wordN"; while [ 1 ];do curl -s http://twitter.com/statuses/public_timeline.rss |grep '' |cut -d '>' -f 2 |cut -d '<' -f 1 > .twitt.tmp && for word in $WRDS;do grep --color=auto -i $word .twtt.tmp;done;sleep 300;done * Twit Amarok "now playing" song >> curl -u : -d status="Amarok, now playing: $(dcop amarok default nowPlaying)" http://twitter.com/statuses/update.json * Extract tarball from internet without local saving >> wget -qO - "http://www.tarball.com/tarball.gz" | tar zxvf - * Copy your SSH public key on a remote machine for passwordless >> ssh-copy-id username@hostname * Wget Command to Download Full Recursive Version of Web Page >> wget -p --convert-links http://www.foo.com * Speak the top 6 lines of your twitter timeline every 5 >> while [ 1 ]; do curl -s -u username:password http://twitter.com/statuses/friends_timeline.rss|grep title|sed -ne 's/<\/*title>//gp' | head -n 6 |festival --tts; sleep 300;done * printing barcodes >> ls /home | head -64 | barcode -t 4x16 | lpr * Command Line to Get the Stock Quote via Yahoo >> curl -s 'http://download.finance.yahoo.com/d/quotes.csv?s=csco&f=l1' * Speak your horoscope with the command line >> curl -s 'http://www.trynt.com/astrology-horoscope-api/v2/?m=2&d=23' | xmlstarlet sel -t -m '//horoscope' -v 'horoscope' | festival --tts * Resume an aborted scp file transfers >> rsync --partial --progress --rsh=ssh SOURCE DESTINATION * download a specific file >> curl -f -O http://pcbsd.fastbull.org/7.0.2/i386/PCBSD7.0.2-x86-DVD.iso * single line twitter-tracker >> WRDS="word1 word2 wordN"; while [ 1 ];do curl -s http://twitter.com/statuses/public_timeline.rss |grep '' |cut -d '>' -f 2 |cut -d '<' -f 1 > .twitt.tmp && for word in $WRDS;do grep --color=auto -i $word .twtt.tmp;done;sleep 300;done * Get your public ip address >> python -c "import socket; s=socket.socket(socket.AF_INET, socket.SOCK_DGRAM); s.connect(('google.com', 80)); print s.getsockname()[0]" * count the occurences of each word in novel David Copperfield >> wget -q -O- http://www.gutenberg.org/dirs/etext96/cprfd10.txt | sed '1,419d' | tr "\n" " " | tr " " "\n" | perl -lpe 's/\W//g;$_=lc($_)' | grep "^[a-z]" | awk 'length > 1' | sort | uniq -c | awk '{print $2"\t"$1}' * Connect to google talk through ssh by setting your IM client to >> ssh -f -N -L 5432:talk.google.com:5222 user@home.network.com * find all references to a server in web.config files with >> ls \\someserver\c$\inetpub\wwwroot -r -i web.config | Select-String "SomeMachineName" * Download and extract a *tar.gz file with curl. >> curl http://domain.com/file.tar.gz | tar zx * rsync + find >> rsync -avz -e ssh --files-from=<(find -mtime +30 -mtime -60) source dest * PlayTweets from the command line >> vlc $(curl -s http://twitter.com/statuses/user_timeline/18855500.rss|grep play|sed -ne '//s/^.*\(http.*\)<\/title/\1/gp'|awk '{print $1}') * Proxy all web traffic via ssh >> Putty -d 8080 [server] * Twitter update from terminal (pok3's snipts ?) >> curl -u YourUsername:YourPassword -d status="Your status message go here" http://twitter.com/statuses/update.xml * Run a command, store the output in a pastebin on the internet and >> ls | curl -F 'sprunge=<-' http://sprunge.us | xclip * Another Curl your IP command >> curl -s http://checkip.dyndns.org | sed 's/[a-zA-Z<>/ :]//g' * Improvement of curl + Twitter >> echo "Set Twitter Status" ; read STATUS; curl -u user:pass -d status="$STATUS" http://twitter.com/statuses/update.xml * bkup the old files to an ssh host >> find <dir> -type f -mtime +<days> -exec scp -r {} user@backuphost:/data/bkup \; * extract the majority of urls from a file >> egrep -ie "<*HREF=(.*?)>" index.html | awk -F\" '{print $2}' | grep :// * Retrieve a list of all webpages on a site >> URL=www.example.com && wget -rq --spider --force-html "http://$URL" && find $URL -type d > url-list.txt && rm -rf $URL * Get your external IP address >> wget http://checkip.dyndns.org && clear && echo && echo My IP && egrep -o '([[:digit:]]{1,3}\.){3}[[:digit:]]{1,3}' index.html && echo && rm index.html * Get your external IP address >> wget -O - http://checkip.dyndns.org|sed 's/[^0-9.]//g' * Get your external IP address >> fetch -q -o - http://ipchicken.com | egrep -o '([[:digit:]]{1,3}\.){3}[[:digit:]]{1,3}' * See the top 10 IP addresses in a web access log >> # cut -d ' ' -f1 /var/log/nginx/nginx-access.log | sort | uniq -c | sort -nr | head -10 | nl * Adhoc tar backup >> tar -cvzf - /source/path | ssh <targethostname> -l <username> dd of=/destination/path/backupfile.tgz * The simplest way to transport information over a network >> (on destination machine) nc -l 1234 > whatever; (on source machine) nc destination 1234 < whatever; * Ping a URL sending output to file and STDOUT >> ping google.com | tee ping-output.txt