% latex generated by the script '/home/project-web/bumble/cgi-bin/text2latex.cgi' 
% with help from the sed script 'text2latex.sed' 
% query string: 'books/linux-web/linux-web-book' 
% sFileName= 'linux-web-book' 
% sDirName = 'books/linux-web' 
% sName = '' -->
% latex by http://bumble.sourceforge.net/scripts/text2latex.sed 
% this script is a work in progress 
\documentclass[11pt]{article} 
\usepackage[hmargin=2cm, vmargin=2cm]{geometry} 
\usepackage{url} 
\usepackage{hyperref} 
\begin{document}

 \\&\\& Web Development using Linux
 ------------------------------

 This book seeks to provide recipes for developing web sites using the Linux
 operating system, with an emphasis on command line tools. This book is not
 about html and css since that is covered in the html-css-book.txt

VISUAL PAGE CREATION WYSIWYG

 @@ kompozer
 a graphical tool for creating pages. successor of nvu
 @@ bluefish
 an html editor

IMAGES

 @@ linux-image-book.txt
 more comprehensive information about managing images with tools
 available for the Linux operating system.

IMAGE COMPRESSION ....
 

\title{} 
\author{bumble.sourceforge.net} 
\maketitle 
\tableofcontents tools for image compression
 .. webpack - 
 .. pngout -
 ..


\title{} 
\author{bumble.sourceforge.net} 
\maketitle 
\tableofcontents image tools
 .. 

IMAGE RESIZING ....

 * convert a all '.html' files from iso-8859-1 to utf-8 file
 -------------------------------------------------------------
 for x in \\$(find . -name '*.html')
 do iconv -f ISO-8859-1 -t UTF-8 \\$x > "\\$x.utf8"; rm \\$x;
 mv "\\$x.utf8" \\$x; 
 done 
 ,,,

MANAGING IMAGES ....

 Feh is a flexible and fast tool for managing images which will
 be included in a web-site. See the linux-image-book for more
 details.

 * use feh
 >> feh

ASCII IMAGES ....
 
 Converting images to 'ascii' (normal text characters) may in some 
 circumstances be aethetically interesting. They could then be displayed
 within <pre> tags with an extremely small font.
 
 * convert an image to ascii 
 >> jp2a --size=40x20 imagefile.jpg

SVG SCALABLE VECTOR GRAPHICS ....
 
VIDEO

 @@ ffmpeg
 recode video files into different formats


\title{} 
\author{bumble.sourceforge.net} 
\maketitle 
\tableofcontents video formats
 .. DivX
 .. Digital video - used by camcorders
 .. avi
 .. mpeg
 ..


\title{} 
\author{bumble.sourceforge.net} 
\maketitle 
\tableofcontents video tools for web-development
 .. stills2dv - creates videos from still images
 .. kino - edit digital video data
 .. 

 * install kino on linux
 >> sudo apt-get install kino

USING UNICODE CHARACTERS 

 Unicode characters can be used in web-pages but this technique
 requires that the viewer of the webpage has an adequate font 
 installed in order to be able to view the character.


\title{} 
\author{bumble.sourceforge.net} 
\maketitle 
\tableofcontents some special chars
 .. U+260E - \\&\\#x260E; \\&\\#9742; ☎ BLACK TELEPHONE
 .. U+260F - \\&\\#x260F; \\&\\#9743; ☏ WHITE TELEPHONE
 .. U+265B - \\&\\#x265B; \\&\\#9819; ♛ BLACK CHESS QUEEN
 .. U+265C - \\&\\#x265C; \\&\\#9820; ♜ BLACK CHESS ROOK
 .. U+265D - \\&\\#x265D; \\&\\#9821; ♝ BLACK CHESS BISHOP
 .. U+265E - \\&\\#x265E; \\&\\#9822; ♞ BLACK CHESS KNIGHT
 .. U+265F - \\&\\#x265F; \\&\\#9823; ♟ BLACK CHESS PAWN
 .. U+2660 - \\&\\#x2660; \\&\\#9824; ♠ BLACK SPADE SUIT
 ..

 * The unicode page 'miscelaneous characters' runs from
 >> 9728-9865 (decimal) or x2600-x2689

 * print out 9 random entities from the miscelaneous unicode page
 >> echo "\\&\\#"\{9728..9865\}";" | xargs shuf -e | head -9 

 * create a little tabble of random miscelaneous entities 
 >> echo "<td>\\&\\#"\{9728..9865\}";</td>" | xargs shuf -e | head -15 | (echo "<table><tr>"; cat -; echo "</tr></table>")

ANIMATIONS 

 @@ synfig studio
 a tool to create animations without 'tweening'
 (that is, drawing every image)

FILE TRANSFER


\title{} 
\author{bumble.sourceforge.net} 
\maketitle 
\tableofcontents tools for file transfer over the net
 .. ftp - the old file transfer tool
 .. rsync - transfer only changed or new files
 .. sftp - an interactive secure version of sftp
 .. scp - a non-interactive secure ftp
 .. sitecopy - synchronize a remote site with what is local
 ..

WEB SITE MIRRORING

 * Download all images from a site
 >> wget -r -l1 --no-parent -nH -nd -P/tmp -A"http://example.com/images" .gif,.jpg

TRANSFERING FILES

 An important part of creating a website is the process of moving 
 file to and from the server. One wants to do this in a simple and 
 efficient manner, while guarding against the possibility of accidentally
 overwriting files with older or bad versions. 

 Unix and Linux have a wide smorgasboard of tools for the purpose of
 achieving these transfers. 


\title{} 
\author{bumble.sourceforge.net} 
\maketitle 
\tableofcontents command line file transfer tools
 .. rsync - powerful and complicated, synchronise files
 .. scp - fast and simple, securely copy
 .. sftp - an interactive way to copy files across the web
 ..


\title{} 
\author{bumble.sourceforge.net} 
\maketitle 
\tableofcontents older tools
 .. ftp - an insecure copy method, use sftp instead
 .. 


\title{} 
\author{bumble.sourceforge.net} 
\maketitle 
\tableofcontents graphical tools
 .. the firefox copy ??
 ..
 
SSH FILE SYSTEM ....

 Linux has a special file system available called the 'sshfs'
 which allows the user to mount a remote folder tree as if
 it was a local file system (although the accesses will seem a little
 slower). This may be more enjoyable than using an ssh shell since
 the remote file system 'seems' to be local. The only requirement is that
 the remote computer be running an 'ssh' secure shell server and that
 you have a login name and password for that system.

 * install the sshfs remote file system.
 >> sudo apt-get install sshfs

 * mount an ssh (remote) file system
 >> sshfs name@server:/path/to/folder \texttt{/path/to/mount/point}

 * unmount an ssh file system
 >> fusermount -u \\$\\verb|~|/sshfs\\_mounted\\_directory

SCP ....

 Scp stands for 'secure copy program'. It is a non-interactive tool
 well suited for including in scripts.

 * bash function to upload a file via scp to the 'rpike' a/c on the server
 >> upfile() \{ scp \\$1 rpike@server.net:\\$1 \}

 * upload and overwrite all '.txt' files to the 'doc/' folder on far.net
 >> scp *.txt eva@far.net:doc/

 * download 2 files from the server using the account 'user' 
 >> scp user@server.net:"chap1.txt chap2.txt" \\$\\verb|~|/books

POSTING DATA

 * Submit data to a HTML form with POST method and save the response
 >> curl -sd 'rid=value\\&submit=SUBMIT' <URL> > out.html

 * post with a proxy and authentication
 >> curl -F name='../htdocs/notes/'\\$1 -F contents='<'\\$1 -u user:upass -x prox.net:8080 -U bob:proxpass \url{http://serv.net/save.cgi}
HTML STUFF

HTML LINKS ....

 * extract the hyperlinks (<a> tags) from a webpage
 >> lynx -dump -listonly \url{http://www.server.net/page.html}

 * find most of the urls within an html file 
 >> egrep 'https?://([[:alpha:]]([-[:alnum:]]+[[:alnum:]])*\\$\\backslash\\$.)+[[:alpha:]]\{2,3\}(: \\$\\backslash\\$d+)?(/([-\\$\\backslash\\$w/\\_\\$\\backslash\\$.]*(\\$\\backslash\\$?\\$\\backslash\\$S+)?)?)?'

ENTITIES

 * encode HTML entities
 >> perl -MHTML::Entities -ne 'print encode\\_entities(\\$\\_)' \texttt{/tmp/subor.txt}

 * or use xmlstarlet to encode entities.

BASH AND WEB DEVELOPMENT

 Using the bash shell to develop web-sites maybe quite efficient, if
 unconventional. 

 * possibly the simplest way to create a web-page from text
 >> cat file.txt | (echo '<html><body><pre>'; cat -; echo '</pre></body></html>') 

 * a simpler way
 >> echo '<html><body><pre>'; cat file.txt; echo '</pre></body></html>') 

TEMPLATING WITH BASH ....

 * a simple template technique with bash
 >> export a=b; echo -e 'one\\$\\backslash\\$ntwo\\$\\backslash\\$nand <x>' | (echo 'cat << EE';sed 's/<x>/\\$a/g'; echo 'EE') | bash 

 * use the technique above to substitute the date into the template
 >> cat template | (echo 'cat << EE';sed 's/<date>/\\$(date)/g'; echo 'EE') | bash

FOLDER LISTINGS ....
 
 * list only folders
 >> ls -d */ | (echo '<ul class="fol">'; cat -; echo '</ul>')

 * make an html directory listing out of the current folder
 >> echo "echo -e \\$\\backslash\\$<li\\$\\backslash\\$>"\{\\$(echo * | tr ' ' ',')\}"\\$\\backslash\\$</li\\$\\backslash\\$>" | bash

 * list all files and folders, no links
 >> a=\\$(echo *); echo 'echo -e "\\$\\backslash\\$n<li>"\{'\\$\{a//" "/,\}'\}"</li>"' | bash

 * list only folders, no links
 >> a=\\$(echo */); echo 'echo -e "\\$\\backslash\\$n<li>"\{'\\$\{a//" "/,\}'\}"</li>"' | bash

 * a for loop method to list only sub-folders as an html list
 ------------------------------------------------------------
 echo "<ul class=fol>"
 for d in \\$(ls -d */); do
 echo "<li>\\$d</li>"
 done
 echo "</ul>"
 ,,,

 * another for loop method to list only sub-folders as an html list
 ------------------------------------------------------------
 echo "<ul class=fol>"
 for d in */; do
 echo " <li>\\$d</li>"
 done
 echo "</ul>"
 ,,,

 * list subfolders as html links 
 -------------------------------
 echo "<ul class=fol>"
 for d in */; do
 echo "<li><a href='\\$d'>\\$d</a></li>"
 done
 echo "</ul>"
 ,,,

 * list subfolders as html links using a brace loop
 --------------------------------------------------
 echo "<ul class=fol>"
 for d in */
 \{ echo " <li><a href='\\$d'>\\$d</a></li>"; \} 
 echo "</ul>"
 ,,,

BASH CGI PROGRAMMING

 While it is most common for Cgi web-scripts to be written in 
 the Perl language, it is also possible to write them using the 
 normal Bash shell scripting language. Whether this is a good idea
 is completely another question...

 @@ \url{http://en.wikipedia.org/wiki/Internet\\_media\\_type\\#List\\_of\\_common\\_media\\_types}
 A list of common "media types" (such as "text/html") which are used
 in the "Content-Type:" field of the Cgi script.

 \begin{enumerate}
  \item 
 use the "2>\\&1" idiom at the end of script lines to
 redirect an error message to the "standard output" (which in the 
 case of a Cgi script is the web-browser of the script visitor).
 This allows you, the developer to see what is going wrong with
 your bash cgi script.
 \item 
 using "here" documents with a bash cgi script is a simple way
 to produce content.
 \item
\end{enumerate}
  
THE BASH CGI GOTCHAS ....

 \begin{enumerate}
  \item the content-type line has to be before \texttt{/anything/} or else
 nothing is printed.
\end{enumerate}
  
 * there must be an empty line after the "content-type" line.
 -------------------------------------------------------------
 echo "Content-Type: text/html"
 echo "<html>...</html"
 \\# This is Incorrect because there is no blank line after the content type
 ,,,

 * trying to call the perl CGI module twice Doesnt work
 ------------------------------------------------------
 file=\\$(perl -e "use CGI qw(:standard); print param('file'))
 name=\\$(perl -e "use CGI qw(:standard); print param('name'))

 \\# Incorrect, the "name" variable is not set because the 
 \\# 'cgi' module reads all the form posted data from standard input
 \\# in one go, leaving nothing to read the second time around.
 ,,, 

CGI BASH EXAMPLES ....

 * make a cgi script to display the query string
 -----------------------------------------------
 \\#!/bin/sh
 echo "Content-Type: text/html"
 echo 
 echo "<html><body>the query string is \\$QUERY\\_STRING</body></html>" 2>\\&1
 ,,, 


 * a bash cgi script indicating that the character set is "utf8"
 ---------------------------------------------------------------
 \\#!/bin/bash
 echo "Content-Type: text/html; charset=utf-8"
 echo
 echo "<html><body>A Bash UTF8 Cgi Script!</body></html>" 2>\\&1
 ,,,

 * show error messages in the browser generated by a cgi script line
 >> ech "this is a mistake" 2>\\&1

 * show error messages in the browser with output redirection
 >> ech "this is a mistake" 2>\\&1 >save.txt
 \\#\\#(note that the 2>\\&1 should come before the file redirection)

 * a cgi script which displays several environment variables
 ----------------------------------------------------------
 \\#!/bin/bash
 cat << ENDxxx 
 Content-Type: text/html; charset=utf-8 

 <!-- The empty line above this one is essential 
 <html>
 <head><title>A bash cgi script</title></head>
 <body>
 <dl>
 <dt>DOCUMENT\\_ROOT<dd>\\$DOCUMENT\\_ROOT
 <dt>REMOTE\\_ADDRESS<dd>\\$REMOTE\\_ADDRESS
 <dt>SCRIPT\\_NAME<dd>\\$SCRIPT\\_NAME
 </dl>
 </body>
 </html>
ENDxxx
 ,,,

GETTING AND DECODING FORM DATA ....

 @@ \url{http://oinkzwurgl.org/bash\\_cgi}
 bash functions for decoding cgi form data
 @@ \url{http://www.fpx.de/fp/Software/ProcCGIsh.html}
 a bash script and c program for decoding cgi form data. The 
 c program needs to be compiled.

 Data sent from an html form to a web server can be sent in 2
 different ways; in the querystring itself (the 'GET' method) and
 in the HTTP headers (the 'POST' method). If the data is 'posted'
 then the cgi script will receive the data on the standard input
 Data sent from an html form has to be url decoded.

 * data "posted" from an html form can be read from the standard input 
 >> read postdata \\#\\#(the content length should first be checked)
 >> postdata=\\$(</dev/stdin) \\#\\#(the same)

 * use perl to get and decode data posted from an html form
 >> file=\\$(perl -e "use CGI qw(:standard); print param('file'))
 \\#\\#(the CGI module takes care of url-decoding the form data)


\title{} 
\author{bumble.sourceforge.net} 
\maketitle 
\tableofcontents cgi environment variables and meanings
 .. DOCUMENT\\_ROOT, The root directory of your server
 .. HTTP\\_COOKIE, The visitor's cookie, if one is set
 .. HTTP\\_HOST, The hostname of your server
 .. HTTP\\_REFERER, The URL of the page that called your script
 .. HTTP\\_USER\\_AGENT, The browser type of the visitor
 .. HTTPS, "on" if the script is being called through a secure server
 .. PATH, The system path your server is running under
 .. QUERY\\_STRING, The query string (see GET, below)
 .. REMOTE\\_ADDR, The IP address of the visitor
 .. REMOTE\\_HOST, The hostname of the visitor (if server has reverse-name-lookups on; otherwise this is the IP address again)
 .. REMOTE\\_PORT, The port the visitor is connected to on the web server
 .. REMOTE\\_USER, The visitor's username (for .htaccess-protected pages)
 .. REQUEST\\_METHOD, either "GET" or "POST"
 .. REQUEST\\_URI, The interpreted pathname of the requested document or CGI (relative to the document root)
 .. SCRIPT\\_FILENAME, The full pathname of the current CGI
 .. SCRIPT\\_NAME, The pathname of the cgi script(relative to the doc root)
 .. SERVER\\_ADMIN, The email address for your server's webmaster
 .. SERVER\\_NAME, Your server's fully qualified domain name (e.g. \url{http://www.cgi101.com)}
 .. SERVER\\_PORT, The port number your server is listening on
 .. SERVER\\_SOFTWARE, The server software you're using (such as Apache 1.3)
 ..

 * some environment variables and example values
 -----------------------------------------------
 SERVER\\_SOFTWARE = Apache/2.0.54 (Fedora)
 SERVER\\_NAME = \url{http://www.comp.leeds.ac.uk}
 GATEWAY\\_INTERFACE = CGI/1.1
 SERVER\\_PROTOCOL = HTTP/1.1
 SERVER\\_PORT = 80
 REQUEST\\_METHOD = GET
 HTTP\\_ACCEPT = 'text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5'
 PATH\\_INFO = 
 PATH\\_TRANSLATED = 
 SCRIPT\\_NAME = \texttt{/cgi-bin/Perl/environment-example}
 QUERY\\_STRING = 
 REMOTE\\_HOST = 
 REMOTE\\_ADDR = 200.93.167.25
 REMOTE\\_USER = 
 CONTENT\\_TYPE = 
 CONTENT\\_LENGTH = 
 ,,,


\title{} 
\author{bumble.sourceforge.net} 
\maketitle 
\tableofcontents cgi Content-Type: values
 ..
 .. text/html, for html content
 .. text/plain, for plain text
 .. text/plain; charset=utf-8, plain text in the utf8 encoding
 .. text/xml, xml text
 ..

CONVERTING HTML


PLAIN TEXT CONVERSION ....

 * Display command lines visible on commandlinefu.com homepage
 >> ruby -ropen-uri -e 'require "hpricot";(Hpricot(open("http://commandlinefu.com"))/".command").each\{| c| puts c.to\\_plain\\_text\}'

HTTP

 http is the protocol apon which the web is based. All web-servers
 must implement at least a part of this protocol.

HTTP HEADERS ....

 * Dump HTTP header using lynx or w3m
 >> lynx -dump -head \url{http://www.example.com/}

XML AND HTML
 
 xml and html are closely related mark-up languages.
 
XHTML


\title{} 
\author{bumble.sourceforge.net} 
\maketitle 
\tableofcontents tools
 .. xmlstarlet - queries and edits xml from the command line
 ..

VALIDATION 

 Validation is the process of confirming that a particular html, xhtml, or
 css document actually conforms to the technical specifications for that
 document. Historically browsers have been designed to display more or
 less correctly even html and xhtml pages which contain 'mistakes' or
 syntax which does not conform to the technical specification for that
 type of document.

 @@ \url{http://validator.w3.org} 
 validates xhtml
 @@ \url{http://jigsaw.w3.org/css-validator}
 a css validator
 @@ tidy.sourceforge.net
 A command line html validator and pretty printer.

 * install the html tidy program and docs on a debian type linux 
 >> sudo apt-get tidy tidy-doc

 * view the html tidy documentation
 >> ls \texttt{/usr/share/doc/tidy-doc/htmldoc}
 >> xdg-open \texttt{/usr/share/doc/tidy-doc/htmldoc/Overview.html}

 * test out tidy printing errors and warnings
 >> echo "<html></html>" | tidy -e - 2>\\&1 | less

CURL STUFF

 * getting a page via an authenticating proxy server
 >> curl -x proxy.utas.edu.au:8080 -U bobj \url{http://www.server.net} 

 * get a page via authenticating proxy server as user 'bob' password 'sec'
 >> curl -x proxy.net:8080 -U bob:sec \url{http://www.server.net}

 Supplying the password in this manner is possibly not a good idea from a
 security point of view 
 
 * download a text file through a proxy and edit it with vim
 -----------------------------------------------------------
 function edn 
 \{
 curl -x proxy.org.au:8080 -U bob:pass \url{http://www.serv.net/a.txt} -o \\$\\verb|~|/notes.txt
 vim \\$\\verb|~|/notes.txt
 \}
 ,,,

 * upload a file to webserver w cgi script with http authententication, 
 --------------------------------------------------------------------
 function up \{
 [ -z "\\$1" ] \\&\\& echo 'no parameter' \\&\\& return 1;
 curl -F name='../htdocs/notes/'\\$1 -F contents='<'\\$1 -u user:upass -x prox.net:8080 -U bob:proxpass \url{http://serv.net/save.cgi}
\}
,,, 

PERL TRICKS


\title{} 
\author{bumble.sourceforge.net} 
\maketitle 
\tableofcontents useful modules
 .. www::mechanize
 .. lwp
 ..

 * a perl mechanize example
 --------------------------

 \\# navigate to the main page
 \\$mech->get('http://www.somesite.com/'); 

 \\# follow a link that contains the text 'download this'
 \\$mech->follow\\_link( text\\_regex => qr/download this/i );

 \\# submit a POST form, to log into the site
 \\$mech->submit\\_form(
 with\\_fields => \{
 username => 'mungo',
 password => 'lost-and-alone',
 \}
 );

 \\# save the results as a file
 \\$mech->save\\_content('somefile.zip');
 ,,,

PHP

 * Testing php configuration
 >> php -r "phpinfo\\$\\backslash\\$(\\$\\backslash\\$);"

 * get the urls from a webpage
 >> \\$browser->getUrls()

CRAWLING

 .. ScriptableBrowser (simpletest)

LOAD TESTING 


\title{} 
\author{bumble.sourceforge.net} 
\maketitle 
\tableofcontents tools
 .. funkload - web testing
 ..

STUFF USED IN GOOGLE CHROME


\title{} 
\author{bumble.sourceforge.net} 
\maketitle 
\tableofcontents tools used in google chrome
 .. bsdiff,
 .. bspatch, 
 .. bzip2,
 .. dtoa, 
 .. hunspell, 
 .. ICU, 
 .. JSCRE, libjpeg, libpng, libxml, libxslt, LZMA SDK, modp\\_b64, 
 .. Mozilla interface to Java Plugin APIs,
 .. npapi, nspr, nss, Pthreads for win32, sqlite, 
 .. tlslite, V8 assembler, WebKit, WTL, zlib
 ..

FORUM SITES

 @@ ask.metafilter.com
 visited by knowledgable people
 @@

SERVING PAGES

 * share the current directory tree (via http) at \url{http://\\$HOSTNAME:8000/}
 >> python -m SimpleHTTPServer

 * create a webserver to share all files in \texttt{/tmp/mydocs} on port 8081
 >> wbox servermode webroot \texttt{/tmp/mydocs}

 * create a webserver to share all files in \texttt{/tmp/mydocs} on port 8080
 >> wbox servermode serverport 8080 webroot \texttt{/tmp/mydocs}

 * Sharing file through http 80 port
 >> nc -w 5 -v -l -p 80 < file.ext

CGI SERVERS

 * a simple cgi server
 >> python -m CGIHTTPServer 8080

 * some kind of perl cgi server
 >> HTTP::Simple::PSGI

SMALL WEBSERVERS

 @@ \url{http://ask.metafilter.com/65481/Help-me-find-a-cool-little-unix-http-utility-I-cant-remember} 
 some good things about mini web servers
 @@ \url{http://hping.org/wbox/}
 site for wbox


\title{} 
\author{bumble.sourceforge.net} 
\maketitle 
\tableofcontents small quick and easy webservers
 .. wbox - http server
 .. thttpd - small web server
 .. mini\\_httpd - same author as thttpd but smaller
 .. webfs - serves a file system from the web
 .. busybox httpd command - small webservery thing
 ..

WBOX
 
 * show how long each part of a webpage takes to generate
 >> wbox nowhere.net/page.html timesplit 1

 * show the http header information for a page
 >> wbox \url{http://www.google.it/notexistingpage.html} 1 showhdr

MAC OSX

 Installed by default are php, curl, 

TEMPLATING

 @@ \url{http://www.perl.com/pub/a/2001/08/21/templating.html}
 an article about using templating systems with perl


\title{} 
\author{bumble.sourceforge.net} 
\maketitle 
\tableofcontents tools
 .. Template Toolkit - almost active development, perl, and python
 .. HTML::Mason - ? callback style, active as of 2010
 .. Embperl - embedd perl into webpages, stopped 2006
 .. HTML::Template - perl template module
 .. Text::Template - a general purpose templater
 .. Apache::ASP - use asp with apache, stopped 2004
 .. CGI::FastTemplate - another one
 ..
 
 @@ \url{http://template-toolkit.org/} 
 the site for the template toolkit

TEMPLATE TOOLKIT

 * example statement using dot notation.
 >> How are things in [\\% customer.address.city \\%]?

 * a for loop
 [\\% FOREACH list \\%]
 <a href="[\\% url \\%]"><b>[\\% name \\%]</a></a>
 [\\% END \\%]
 ,,,

HTML TEMPLATE

 * example loop with html::template
 -----
 <TMPL\\_LOOP list>
 <a href="<TMPL\\_VAR url>"><b><TMPL\\_VAR name></b></A>
 </TMPL\\_LOOP>
 ,,,

MASON ....

 Seems to be in active development. It can be run
 without a webserver

 @@ \url{http://www.masonhq.com/}
 the official site

 * install mason using apt-get
 >> apt-get install libmason-perl \\#\\#(??? unchecked)

PERL STUFF


\title{} 
\author{bumble.sourceforge.net} 
\maketitle 
\tableofcontents some interesting perl web modules
 .. Dancer - perl web apps with good examples
 .. Web::Simple - simple web apps
 .. CGI::Application
 .. Catalyst - big web apps
 .. Mason
 .. Mojolicious
 ..

WEB SIMPLE ....

 @@ \url{http://search.cpan.org/\\$\\verb|~|mstrout/Web-Simple-0.002/lib/Web/Simple.pm}
 the documentation for web simple

 Developed in 2009. Can create a webapplication without
 a webserver

 @@ hobbs at stackoverflow.com
 knowledgable perl web person

CPAN TOOL CRASH COURSE

 * start a cpan shell to install mason
 >> perl -MCPAN -e 'shell'


\title{} 
\author{bumble.sourceforge.net} 
\maketitle 
\tableofcontents simple cpan shell
 .. h - show help
 .. get - get the source for a module
 .. make - compile ? the module
 .. test - test a module
 .. install - do all of get, make, test
 .. clean - get rid of a badly installed module
 .. look - see whats happening
 .. readme - see whats going on
 ..
 
 @@ \url{http://www.livejournal.com/doc/server/lj.install.perl\\_setup.modules.html}
 a list of potentially useful modules used with livejournal

 * upgrade cpan but could cause problems ???
 -------------------------------------------
 perl -MCPAN -e shell
 cpan> install Bundle::CPAN
 cpan> reload cpan
 ,,,

GETTING WEB PAGES FROM THE COMMAND LINE

 * http get of a web page via proxy server with login credentials
 >> curl -U username[:password] -x proxy:proxyport webpage

 * use netcat to get a webpage
 >> echo "GET / HTTP/1.0\\$\\backslash\\$r\\$\\backslash\\$r" | nc -v \url{http://www.somewebsite.com} 80

 * get a webpage with the console php tool
 >> php -r "file('http://metafilter.com/');"

 * get a webpage with perl
 >> perl -MHTTP::Client -e 'print HTTP::Client->new()->get("http://localhost/path")'

 * a simpler way to do the same
 >> perl -MLWP::Simple -e 'get("http://www.metafilter.com/")'

LOAD TESTING
 
 * perform 4 queries per second (with 4 processes) on the local webserver
 >> wbox \url{http://localhost} clients 4

BLOG ENGINES


\title{} 
\author{bumble.sourceforge.net} 
\maketitle 
\tableofcontents 
 .. serendipity - a blog engine with database backend
 .. bilboblog - a simple engine
 ..

CMS SYSTEMS

 

\title{} 
\author{bumble.sourceforge.net} 
\maketitle 
\tableofcontents a small summary
 .. redmine - popular small software orientated (ruby

NOTES

 google uses python

 @@ \url{http://unixmages.com/}
 an unrelated but interesting site

GOOGLE CLI

 Google provides a command line interface to its services such
 as the 'picasa' photo album service. This may be useful for
 scripting etc.

 * post to a blogger blog
 >> google blogger post --tags "GoogleCL, awesome" --title "Test Post" "I'm posting from the command line"

 * post to a blogger blog using the file 'post.txt' as the text
 >> google blogger post post.txt

 >> google blogger list title,url-site \\# List posts
 >> google blogger delete --title "Test Post"

 >> google delete --title "Silly post number [0-9]*" \\# Delete posts matching regex
 >> google tag --title "Dev post" --tags "Python, software" \\# label an existing post

 * examples of putting photos in picasa from the command line
 ------------------------------------------------------------
 google picasa create --title "Vermont Test" --tags Vermont vermont.jpg
 google picasa get --title "Vermont Test" \texttt{/path/to/download/folder}
 google picasa list title,url-direct --query "A tag"
 google picasa post --title "Vermont Test" \\$\\verb|~|/old\\_photos/*.jpg \\# Add to an album
 google picasa tag --title "Vermont Test" --tags "places"
 google picasa delete --title "Vermont Test" \\# delete entire album
 ,,,

 * examples of posting to youtube from command line
 --------------------------------------------------

 google youtube post --category Education --devtags GoogleCL killer\\_robots.avi
 google youtube delete --title "killer\\_robots.avi"
 google youtube list \\# list my videos
 google youtube tag -n ".*robot.*" --tags robot
 ,,,

MISCELANEOUS
 
 * query wikipedia via dns by requesting a 'txt' record
 >> dig +short txt <keyword>.wp.dg.cx

 * query wikipedia via dns for the term 'bash' 
 >> dig +short txt bash.wp.dg.cx

JARGON 


\title{} 
\author{bumble.sourceforge.net} 
\maketitle 
\tableofcontents web jargon
 @@ the net and the web
 The net is the internet and all its protocols such as 
 ftp, ssh, http etc where as the web consists of just the 
 http and https protocol and therefor webpages.
 @@ download
 transfer files from a net server to your local computer
 @@ markup
 a way of embedding extra information in a document with
 the use of 'tags' or 'codes'. Html is an example of markup.
 @@ minimalist mark-up
 minimalist mark-up is a markup language which attempts to
 use the minimum number of tags possible to embedd in the 
 document. The purpose of this system is not to interfere 
 psychologically with the human readers and writers of the 
 document who may be distracted and confused by bulky and 
 numerous tags. An example of minimalist markup is 'markdown'
 or most 'wiki' markup languages
 @@ wiki
 Is a web site or document which many or most web-visitors
 are able to edit.
 @@ web 2.0
 The term encapsulates the idea of the web as a collaborative
 medium for the compilation and consumption of information, 
 rather that a 'uni-directional' information source. The television
 is an example of a uni-directional information source since the 
 viewer is unable to effect its transmissions, apart from changing
 channel or switching off. The web 2.0 is multi-directional because
 the user can create and modify the content (through the use
 of wikis, and collaborative sites).
 @@ tags
 mark-up languages often use 'tags' in order to embedd information
 in text documents. Usually there is a start tag and an end tag
 which surrounds the information to which it pertains. For example
 in the text "<em>tree</em>" the word 'tree' is surrounded by the 
 start and end html tag "<em>". This system is used by various
 mark-up languages such as html, xml and sgml. 
 @@ semantic markup
 the use of mark-up codes or tags to indicate 'semantic' 
 information rather than 'visual display or layout' information.
 The phrase 'semantic information' may be a tautology. An 
 example of 'visual display' markup are the html <big> and 
 <small> tags which indicate to the document rendering software 
 the text should be displayed bigger or small than the normal
 sized text. Semantic markup is considered to be 'good' in the 
 w3c world and among ideologists, since information is not
 lost in a forest of meaningless layout tags.
 @@ plain text
 plain text is the basis of the internet and web. Plain text is any
 data which represents a stream of characters in a human writing 
 system which is encoded electronically according to a standard 'text
 encoding'. An example of these encoding are utf8, ascii, or latin1.
 Modern unicode text encodings (such as utf8) can represent any
 character in any human language which has an established writing
 system
 @@ separation of content and presentation
 a much bandied about phrase.
 @@ style sheets
 documents which indicate how other documents should be displayed.
 A number of style sheet languages exist but the most common is
 'css' cascading style sheets.
 @@ cms - content management system
 A content management system is supposedly a way to quickly build
 websites focussing on the content of that site rather than
 web-development issues and design issues. There are a dizzying array
 of free and open-source cms systems available
 @@ version control system
 is a way of managing changes to files made by potentially
 many people, simultaneously
 @@ database
 a way of storing data which is supposedly secure and convenient.
 many database systems exist with the simplest being a text file
 database, and the most command being a 'relational' database which
 stores information in a series of linked tables. Databases are often
 used to store the content of web-sites
 @@ content
 This is a much used web-term which relates to the information
 which a web-site contains as opposed to the visual or 
 technical design of the site
 @@ documentroot
 

DOCUMENT-NOTES:

 \\# this section contains information about the document and
 \\# will not normally be printed.

 \\# A small (16x16) icon image to identify the book
 document-icon:
 \\# A larger image to identify or illustrate the title page
 document-image:

 \\# what sort of document is this
 document-type: book

 \\# in what kind of state (good or bad) is this document 
 document-quality: just begun

 \\# who wrote this
 document-history:
 @@ march 2010
 This book was begun in order to tackle issues such as the batch
 resizing of images, techniques for quickly creating websites, file
 transfer, templating, etc. This book doesnt cover html and css.
 @@ 1 april 2010
 some notes inserted
 @@ 7 april 2010
 included some more notes.
 @@ 11 april 2010
 added a small amount of bash scripting information,
 including a simple technique for templating.
 @@ 19 may 2010
 added a jargon section to explain certain common terms. 
 Also added a reference to the 'redmine' cms system.

 \\# who wrote this
 authors: mjbishop at utas dot edu dot au

 \\# a short description of the contents, possible used for doc lists
 short-description: Techniques for developing web-sites

 \\# A computer language which is contained in the document, if any
 code-language: various

 \\# the script which will be used to produce html (a webpage)
 make-html: ./booktohtml.cgi

 \\# the script which will produce 'LaTeX' output (for printing, pdf etc)
 make-latex: ./booktolatex.cgi

NOTES

 * convert a web page into a png
 >> touch \\$2;firefox -print \\$1 -printmode PNG -printfile \\$2
 >> urls=('www.ubuntu.com' 'google.com'); for i in \\$\{urls[@]\}; do http\\_code=\\$(curl -I -s \\$i -w \\%\{http\\_code\}); echo \\$i status: \\$\{http\\_code:9:3\}; done

 * Check availability of Websites based on HTTP\\_CODE
 >> urls=('www.ubuntu.com' 'google.com'); for i in \\$\{urls[@]\}; do http\\_code=\\$(curl -I -s \\$i -w \\%\{http\\_code\}); echo \\$i status: \\$\{http\\_code:9:3\}; done

 * Watch for when your web server returns
 >> watch -n 15 curl -s --connect-timeout 10 \url{http://www.google.com/}
 >> gsed -e :a -e 's/\\$\\backslash\\$(<\\$\\backslash\\$/[\\^>]*>\\$\\backslash\\$)/\\$\\backslash\\$1\\$\\backslash\\$n/g;s/\\$\\backslash\\$(<br>\\$\\backslash\\$)/\\$\\backslash\\$1\\$\\backslash\\$n/g' page2.txt | sed -n '/<cite>/p;s/<cite>\\$\\backslash\\$(.*\\$\\backslash\\$)<\\$\\backslash\\$/cite>/\\$\\backslash\\$1/g' >> output

 * Check which files are opened by Firefox then sort by largest size.
 >> FFPID=\\$(pidof firefox-bin) \\&\\& lsof -p \\$FFPID | awk '\{ if(\\$7>0) print (\\$7/1024/1024)" MB -- "\\$9; \}' | grep ".mozilla" | sort -rn

 * use wget to check if a remote file exists
 >> wget --spider -v \url{http://www.server.com/path/file.ext}

 * Poke a Webserver to see what it's powered by.
 >> wget -S -O/dev/null "INSERT\\_URL\\_HERE" 2>\\&1 | grep Server
 >> for i in `ps aux | grep httpd | awk '\{print \\$2\}'`; do lsof -n -p \\$i | grep ESTABLISHED; done;
 
 * Regex to remove HTML-Tags from a file
 >> sed -e :a -e 's/<[\\^>]*>//g;/</N;//ba' index.html

 * Run the Firefox Profile Manager
 >> firefox -no-remote -P

 * search for a place with google maps
 >> \url{http://maps.google.com.au/maps?q=lindeith\\%20sandy\\%20bay\\%2018\\%20map\\&oe=utf-8\\&rls=com.ubuntu:en-US:official\\&client=firefox-a\\&um=1\\&ie=UTF-8\\&sa=N\\&hl=en\\&tab=wl}

 
 * Update twitter via curl as Function
 >> tweet()\{ curl -u "\\$1" -d status="\\$2" "http://twitter.com/statuses/update.xml"; \}
 
 * Testing php configuration
 >> php -r phpinfo();
 
 * Search and replace text in all php files with ruby
 >> ruby -i.bkp -pe "gsub(/search/, 'replace')" *.php
 
 * extract all urls from firefox sessionstore
 >> sed -e "s/\\$\\backslash\\$[\{/\\$\\backslash\\$n/g" -e "s/\}, \{/\\$\\backslash\\$n/g" sessionstore.js | grep url | awk -F"," '\{ print \\$1 \}'| sed -e "s/url:\\$\\backslash\\$"\\$\\backslash\\$([\\^\\$\\backslash\\$"]*\\$\\backslash\\$)\\$\\backslash\\$"/\\$\\backslash\\$1/g" -e "/\\^about:blank/d" > session\\_urls.txt
 
 * single line twitter-tracker
 >> WRDS="word1 word2 wordN"; while [ 1 ];do curl -s \url{http://twitter.com/statuses/public\\_timeline.rss} |grep '<description>' |cut -d '>' -f 2 |cut -d '<' -f 1 > .twitt.tmp \\&\\& for word in \\$WRDS;do grep --color=auto -i \\$word .twtt.tmp;done;sleep 300;done
 
 * Twit Amarok "now playing" song
 >> curl -u <user>:<password> -d status="http://twitter.com/statuses/update.json" Amarok, now playing: \\$(dcop amarok default nowPlaying)
 
 * Extract tarball from internet without local saving
 >> wget -qO - "http://www.tarball.com/tarball.gz" | tar zxvf -
 
 * Copy your SSH public key on a remote machine for passwordless
 >> ssh-copy-id username@hostname
 
 * Wget Command to Download Full Recursive Version of Web Page
 >> wget -p --convert-links \url{http://www.foo.com}
 
 * Speak the top 6 lines of your twitter timeline every 5
 >> while [ 1 ]; do curl -s -u username:password \url{http://twitter.com/statuses/friends\\_timeline.rss|grep} title|sed -ne 's/<\\$\\backslash\\$/*title>//gp' | head -n 6 |festival --tts; sleep 300;done
 
 * printing barcodes
 >> ls /home | head -64 | barcode -t 4x16 | lpr
 
 * Command Line to Get the Stock Quote via Yahoo
 >> curl -s 'http://download.finance.yahoo.com/d/quotes.csv?s=csco\\&f=l1'
 
 * Speak your horoscope with the command line
 >> curl -s 'http://www.trynt.com/astrology-horoscope-api/v2/?m=2\\&d=23' | xmlstarlet sel -t -m '//horoscope' -v 'horoscope' | festival --tts

 * Resume an aborted scp file transfers
 >> rsync --partial --progress --rsh=ssh SOURCE DESTINATION

 * download a specific file
 >> curl -f -O \url{http://pcbsd.fastbull.org/7.0.2/i386/PCBSD7.0.2-x86-DVD.iso}

 * single line twitter-tracker
 >> WRDS="word1 word2 wordN"; while [ 1 ];do curl -s \url{http://twitter.com/statuses/public\\_timeline.rss} |grep '<description>' |cut -d '>' -f 2 |cut -d '<' -f 1 > .twitt.tmp \\&\\& for word in \\$WRDS;do grep --color=auto -i \\$word .twtt.tmp;done;sleep 300;done
 

 * Get your public ip address
 >> python -c "import socket; s=socket.socket(socket.AF\\_INET, socket.SOCK\\_DGRAM); s.connect(('google.com', 80)); print s.getsockname()[0]"


 * count the occurences of each word in novel David Copperfield
 >> wget -q \emph{O} \url{http://www.gutenberg.org/dirs/etext96/cprfd10.txt} | sed '1,419d' | tr "\\$\\backslash\\$n" " " | tr " " "\\$\\backslash\\$n" | perl -lpe 's/\\$\\backslash\\$W//g;\\$\\_=lc(\\$\\_)' | grep "\\^[a-z]" | awk 'length > 1' | sort | uniq -c | awk '\{print \\$2"\\$\\backslash\\$t"\\$1\}'


 * Connect to google talk through ssh by setting your IM client to
 >> ssh -f -N -L 5432:talk.google.com:5222 user@home.network.com


 * find all references to a server in web.config files with
 >> ls \\$\\backslash\\$\\$\\backslash\\$someserver\\$\\backslash\\$c\\$\\$\\backslash\\$inetpub\\$\\backslash\\$wwwroot -r -i web.config | Select-String "SomeMachineName"

 * Download and extract a *tar.gz file with curl.
 >> curl \url{http://domain.com/file.tar.gz} | tar zx

 * rsync + find
 >> rsync -avz -e ssh --files-from=<(find -mtime +30 -mtime -60) source dest


 * PlayTweets from the command line
 >> vlc \\$(curl -s \url{http://twitter.com/statuses/user\\_timeline/18855500.rss|grep} play|sed -ne '/<title>/s/\\^.*\\$\\backslash\\$(http.*\\$\\backslash\\$)<\\$\\backslash\\$/title/\\$\\backslash\\$1/gp'|awk '\{print \\$1\}')


 * Proxy all web traffic via ssh
 >> Putty -d 8080 [server]

 * Twitter update from terminal (pok3's snipts ?)
 >> curl -u YourUsername:YourPassword -d status="http://twitter.com/statuses/update.xml" Your status message go here

 * Run a command, store the output in a pastebin on the internet and
 >> ls | curl -F 'sprunge=<-' \url{http://sprunge.us} | xclip


 * Another Curl your IP command
 >> curl -s \url{http://checkip.dyndns.org} | sed 's/[a-zA-Z<>/ :]//g'

 * Improvement of curl + Twitter
 >> echo "Set Twitter Status" ; read STATUS; curl -u user:pass -d status="http://twitter.com/statuses/update.xml" \\$STATUS

 * bkup the old files to an ssh host
 >> find <dir> -type f -mtime +<days> -exec scp -r \{\} user@backuphost:/data/bkup \\$\\backslash\\$;

 * extract the majority of urls from a file
 >> egrep -ie "<*HREF=(.*?)>" index.html | awk -F\\$\\backslash\\$" '\{print \\$2\}' | grep ://

 * Retrieve a list of all webpages on a site
 >> URL=www.example.com \\&\\& wget -rq --spider --force-html "http://\\$URL" \\&\\& find \\$URL -type d > url-list.txt \\&\\& rm -rf \\$URL


 * Get your external IP address
 >> wget \url{http://checkip.dyndns.org} \\&\\& clear \\&\\& echo \\&\\& echo My IP \\&\\& egrep -o '([[:digit:]]\{1,3\}\\$\\backslash\\$.)\{3\}[[:digit:]]\{1,3\}' index.html \\&\\& echo \\&\\& rm index.html


 * Get your external IP address
 >> wget -O - \url{http://checkip.dyndns.org|sed} 's/[\\^0-9.]//g'


 * Get your external IP address
 >> fetch -q -o - \url{http://ipchicken.com} | egrep -o '([[:digit:]]\{1,3\}\\$\\backslash\\$.)\{3\}[[:digit:]]\{1,3\}'

 * See the top 10 IP addresses in a web access log
 >> \\# cut -d ' ' -f1 \texttt{/var/log/nginx/nginx-access.log} | sort | uniq -c | sort -nr | head -10 | nl

 * Adhoc tar backup
 >> tar -cvzf - \texttt{/source/path} | ssh <targethostname> -l <username> dd of=/destination/path/backupfile.tgz

 * The simplest way to transport information over a network
 >> (on destination machine) nc -l 1234 > whatever; (on source machine) nc destination 1234 < whatever;

 * Ping a URL sending output to file and STDOUT
 >> ping google.com | tee ping-output.txt

\end{document}
%end generated latex