% -------------------------------------------
  %  latex generated by: booktolatex.cgi   
  %   from source file : ../htdocs/books/linux-web/linux-web-book.txt       
  %                  on: 19 April 2024,  7:14pm         
  %         querystring: books/linux-web/linux-web-book.txt  
  %       document-root: /var/www/html
  %         script-name: /cgi-bin/booktolatex.cgi
  %         Server-name: bumble.sourceforge.net
  %          Sed-script: booktolatex.sed
  % ------------------------------------------- 

  \documentclass[a4paper,12pt]{article} 
  \usepackage[margin=0.4cm,noheadfoot]{geometry} 
  \usepackage{color}                   %% to use colours, use "xcolor" for more
  \usepackage{multicol}                %% for multiple columns 
  \usepackage{keystroke}               %% for keyboard key images 
  \usepackage[toc]{multitoc}           %% for multi column table of contents
  \usepackage{tocloft}                 %% to customize the table of contents
  \setcounter{tocdepth}{2}             %% only display 2 levels in the contents
  \setlength{\cftbeforesecskip}{0cm}   %% make the toc more compact
  \usepackage{listings}                %% for nice code listings
  %\lstset{language={},
  \lstset{language=various,                 
    %% define special comment delimiters '##(' and ')' 
    moredelim=[s][\color{grey}\itshape\footnotesize\ttfamily]{~(}{)},
    basicstyle=\ttfamily,           %% fixed pitch font
    xleftmargin=1cm,                %% margin on the left outside the frames
    breaklines=true,                %% break long code lines
    breakatwhitespace=false,        %% break long code lines anywhere
    breakindent=10pt,               %% reduce the indent from 20pt to 10
    postbreak=\mbox{{\color{blue}\small$\Rightarrow$\space}},  %% mark with arrow
    showstringspaces=false,            %% dont show spaces within strings 
    framerule=5pt,                     %% thickness of the frames
    rulecolor=\color{lightgrey}, frame=l}   %% source code settings
  \usepackage{graphicx}                %% to include images
  \usepackage{fancybox}                %% boxes with rounded corners 
  \usepackage{wrapfig}                 %% flow text around tables, images 
  \usepackage{tabularx}                %% change width of tables 
  \usepackage[table]{xcolor}           %% alternate row colour tables
  \usepackage{booktabs}                %% for heavier rules in tables 
  \usepackage[small,compact]{titlesec} %% sections more compact, less space
  \usepackage{enumitem}                %% more compact and better lists 
  \setlist{noitemsep}                  %% reduce list item spacing 
  \usepackage{hyperref}     %% make urls into hyperlinks
  \hypersetup{              %% add "pdftex," if only pdf output is required
     colorlinks=true,       %% set up the colours for the hyperlinks 
     linkcolor=black,       %% internal document links black 
     urlcolor=black,        %% url links black 
     filecolor=red,       
     citecolor=red, 
     bookmarks=true, pdfpagemode=UseOutlines} 

  % define some colours to use
  \definecolor{lightgrey}{gray}{0.70}
  \definecolor{grey}{gray}{0.30}

  \titleformat{\section}[frame]    %% titlesec: create framed section headings
     {\normalfont} 
     {\filleft \footnotesize \enspace Section \thesection\enspace\enspace} 
     {3pt} {\bfseries\itshape\filright} 
     
  \title{Web Development using Linux} 
  \author{}
  \date{27 October 2011,  6:33pm} 
  \setlength{\parindent}{0pt}
  % \setlength{\parskip}{1ex}

  % label lists with stars
  \renewcommand{\labelitemi}{$\star$}

  \begin{document} 

  \centerline{\Large \bf Web Development using Linux} \medskip
  \begin{center}
    {\huge ``}\textit{}{\huge ''}
    \textsc{}
  \end{center}
  % -----------------------------------
  % the toc should be 2 columns because of the \multitoc package 
  \tableofcontents


  \begin{lstlisting}

 This book seeks to provide recipes for developing web sites using the Linux
 operating system, with an emphasis on command line tools. This book is not
 about html and css since that is covered in the html-css-book.txt

VISUAL PAGE CREATION WYSIWYG

  @@ kompozer
     a graphical tool for creating pages. successor of nvu
  @@ bluefish
     an html editor

IMAGES

  @@ linux-image-book.txt
     more comprehensive information about managing images with tools
     available for the Linux operating system.

IMAGE COMPRESSION ....
  
  == tools for image compression
  .. webpack - 
  .. pngout -
  ..

  == image tools
  .. 

IMAGE RESIZING  ....

  * convert a all '.html' files from iso-8859-1 to utf-8 file
  \begin{lstlisting}
   for x in $(find . -name '*.html')
     do iconv -f ISO-8859-1 -t UTF-8 $x > "$x.utf8"; rm $x;
     mv "$x.utf8" $x; 
   done 
  \end{lstlisting}

\subsection{Managing Images}

  Feh is a flexible and fast tool for managing images which will
  be included in a web-site. See the linux-image-book for more
  details.

\emph{ Use feh }
\begin{lstlisting}
  feh
\end{lstlisting}

\subsection{Ascii Images}
 
 Converting images to 'ascii' (normal text characters) may in some 
 circumstances be aethetically interesting. They could then be displayed
 within $<$pre$>$ tags with an extremely small font.
 
\emph{ Convert an image to ascii  }
\begin{lstlisting}
  jp2a --size=40x20 imagefile.jpg
\end{lstlisting}

\subsection{Svg Scalable Vector Graphics}
 
\section{Video}

\begin{description}[labelindent=1cm, leftmargin=2cm, style=nextline]
 \item[\url{ffmpeg}]
     recode video files into different formats
\end{description}


  \arrayrulecolor{gray}
  \begin{center}
  \begin{tabular}{ |rl| }
    \multicolumn{2}{c}{\textbf{ video formats }} \\
  \hline

  \texttt{ DivX } \\ 

  \texttt{ Digital video } & Used by camcorders \\ 

  \texttt{ avi } \\ 

  \texttt{ mpeg } \\ 

    \hline
    \end{tabular}
    \end{center}

  \arrayrulecolor{gray}
  \begin{center}
  \begin{tabular}{ |rl| }
    \multicolumn{2}{c}{\textbf{ video tools for web-development }} \\
  \hline

  \texttt{ stills2dv } & Creates videos from still images \\ 

  \texttt{ kino } & Edit digital video data \\ 

    \hline
    \end{tabular}
    \end{center}
\emph{ Install kino on linux }
\begin{lstlisting}
  sudo apt-get install kino
\end{lstlisting}

\section{Using Unicode Characters}

  Unicode characters can be used in web-pages but this technique
  requires that the viewer of the webpage has an adequate font 
  installed in order to be able to view the character.


  \arrayrulecolor{gray}
  \begin{center}
  \begin{tabular}{ |rl| }
    \multicolumn{2}{c}{\textbf{ some special chars }} \\
  \hline

  \texttt{ U+260E } & \&\#x260E; \&\#9742; ☎  BLACK TELEPHONE \\ 

  \texttt{ U+260F } & \&\#x260F; \&\#9743; ☏  WHITE TELEPHONE \\ 

  \texttt{ U+265B } & \&\#x265B; \&\#9819; ♛       BLACK CHESS QUEEN \\ 

  \texttt{ U+265C } & \&\#x265C; \&\#9820; ♜       BLACK CHESS ROOK \\ 

  \texttt{ U+265D } & \&\#x265D; \&\#9821; ♝       BLACK CHESS BISHOP \\ 

  \texttt{ U+265E } & \&\#x265E; \&\#9822; ♞       BLACK CHESS KNIGHT \\ 

  \texttt{ U+265F } & \&\#x265F; \&\#9823; ♟       BLACK CHESS PAWN \\ 

  \texttt{ U+2660 } & \&\#x2660; \&\#9824; ♠       BLACK SPADE SUIT \\ 

    \hline
    \end{tabular}
    \end{center}
\emph{ The unicode page 'miscelaneous characters' runs from }
\begin{lstlisting}
  9728-9865 (decimal) or x2600-x2689
\end{lstlisting}

\emph{ Print out 9 random entities from the miscelaneous unicode page }
\begin{lstlisting}
  echo "&#"{9728..9865}";" | xargs shuf -e | head -9 
\end{lstlisting}

\emph{ Create a little tabble of random miscelaneous entities  }
\begin{lstlisting}
  echo "<td>&#"{9728..9865}";</td>" | xargs shuf -e | head -15 | (echo "<table><tr>"; cat -; echo "</tr></table>")
\end{lstlisting}

\section{Animations}

\begin{description}[labelindent=1cm, leftmargin=2cm, style=nextline]
 \item[\url{synfig}] studio
     a tool to create animations without 'tweening'
     (that is, drawing every image)
\end{description}

\section{File Transfer}


  \arrayrulecolor{gray}
  \begin{center}
  \begin{tabular}{ |rl| }
    \multicolumn{2}{c}{\textbf{ tools for file transfer over the net }} \\
  \hline

  \texttt{ ftp } & The old file transfer tool \\ 

  \texttt{ rsync } & Transfer only changed or new files \\ 

  \texttt{ sftp } & An interactive secure version of sftp \\ 

  \texttt{ scp } & A non-interactive secure ftp \\ 

  \texttt{ sitecopy } & Synchronize a remote site with what is local \\ 

    \hline
    \end{tabular}
    \end{center}
\section{Web Site Mirroring}

\emph{ Download all images from a site }
\begin{lstlisting}
  wget -r -l1 --no-parent -nH -nd -P/tmp -A".gif,.jpg" http://example.com/images
\end{lstlisting}

\section{Transfering Files}

  An important part of creating a website is the process of moving 
  file to and from the server. One wants to do this in a simple and 
  efficient manner, while guarding against the possibility of accidentally
  overwriting files with older or bad versions. 

  Unix and Linux have a wide smorgasboard of tools for the purpose of
  achieving these transfers. 


  \arrayrulecolor{gray}
  \begin{center}
  \begin{tabular}{ |rl| }
    \multicolumn{2}{c}{\textbf{ command line file transfer tools }} \\
  \hline

  \texttt{ rsync } & Powerful and complicated, synchronise files \\ 

  \texttt{ scp } & Fast and simple, securely copy \\ 

  \texttt{ sftp } & An interactive way to copy files across the web \\ 

    \hline
    \end{tabular}
    \end{center}

  \arrayrulecolor{gray}
  \begin{center}
  \begin{tabular}{ |rl| }
    \multicolumn{2}{c}{\textbf{ older tools }} \\
  \hline

  \texttt{ ftp } & An insecure copy method, use sftp instead \\ 

    \hline
    \end{tabular}
    \end{center}

  \arrayrulecolor{gray}
  \begin{center}
  \begin{tabular}{ |rl| }
    \multicolumn{2}{c}{\textbf{ graphical tools }} \\
  \hline

  \texttt{ the firefox copy ?? } \\ 
\subsection{Ssh File System}

 Linux has a special file system available called the 'sshfs'
 which allows the user to mount a remote folder tree as if
 it was a local file system (although the accesses will seem a little
 slower). This may be more enjoyable than using an ssh shell since
 the remote file system 'seems' to be local. The only requirement is that
 the remote computer be running an 'ssh' secure shell server and that
 you have a login name and password for that system.

\emph{ Install the sshfs remote file system. }
\begin{lstlisting}
  sudo apt-get install sshfs
\end{lstlisting}

\emph{ Mount an ssh (remote) file system }
\begin{lstlisting}
  sshfs name@server:/path/to/folder /path/to/mount/point
\end{lstlisting}

\emph{ Unmount an ssh file system }
\begin{lstlisting}
  fusermount -u ~/sshfs_mounted_directory
\end{lstlisting}

\subsection{Scp}

  Scp stands for 'secure copy program'. It is a non-interactive tool
  well suited for including in scripts.

\emph{ Bash function to upload a file via scp to the 'rpike' a/c on the server }
\begin{lstlisting}
  upfile() { scp $1 rpike@server.net:$1 }
\end{lstlisting}

\emph{ Upload and overwrite all '.txt' files to the 'doc/' folder on far.net }
\begin{lstlisting}
  scp *.txt eva@far.net:doc/
\end{lstlisting}

\emph{ Download 2 files from the server using the account 'user'  }
\begin{lstlisting}
  scp user@server.net:"chap1.txt chap2.txt" ~/books
\end{lstlisting}

\section{Posting Data}

\emph{ Submit data to a HTML form with POST method and save the response }
\begin{lstlisting}
  curl -sd 'rid=value&submit=SUBMIT' <URL> > out.html
\end{lstlisting}

\emph{ Post with a proxy and authentication }
\begin{lstlisting}
  curl -F name='../htdocs/notes/'$1 -F contents='<'$1 -u user:upass -x prox.net:8080 -U bob:proxpass http://serv.net/save.cgi
\end{lstlisting}
\section{Html Stuff}

\subsection{Html Links}

\emph{ Extract the hyperlinks ($<$a$>$ tags) from a webpage }
\begin{lstlisting}
  lynx -dump -listonly www.server.net/page.html
\end{lstlisting}

\emph{ Find most of the urls within an html file  }
\begin{lstlisting}
  egrep 'https?://([[:alpha:]]([-[:alnum:]]+[[:alnum:]])*\.)+[[:alpha:]]{2,3}(: \d+)?(/([-\w/_\.]*(\?\S+)?)?)?'
\end{lstlisting}

\section{Entities}

\emph{ Encode HTML entities }
\begin{lstlisting}
  perl -MHTML::Entities -ne 'print encode_entities($_)' /tmp/subor.txt
\end{lstlisting}

\emph{ Or use xmlstarlet to encode entities. }

\section{Bash And Web Development}

  Using the bash shell to develop web-sites maybe quite efficient, if
  unconventional. 

\emph{ Possibly the simplest way to create a web-page from text }
\begin{lstlisting}
  cat file.txt | (echo '<html><body><pre>'; cat -; echo '</pre></body></html>')  
\end{lstlisting}

\emph{ A simpler way }
\begin{lstlisting}
  echo '<html><body><pre>'; cat file.txt; echo '</pre></body></html>')  
\end{lstlisting}

\subsection{Templating With Bash}

\emph{ A simple template technique with bash }
\begin{lstlisting}
  export a=b; echo -e 'one\ntwo\nand <x>' | (echo 'cat << EE';sed 's/<x>/$a/g'; echo 'EE') | bash 
\end{lstlisting}

\emph{ Use the technique above to substitute the date into the template }
\begin{lstlisting}
  cat template | (echo 'cat << EE';sed 's/<date>/$(date)/g'; echo 'EE') | bash
\end{lstlisting}

\subsection{Folder Listings}
  
\emph{ List only folders }
\begin{lstlisting}
  ls -d */ | (echo '<ul class="fol">'; cat -; echo '</ul>')
\end{lstlisting}

\emph{ Make an html directory listing out of the current folder }
\begin{lstlisting}
  echo "echo -e \<li\>"{$(echo * | tr ' ' ',')}"\</li\>" | bash
\end{lstlisting}

\emph{ List all files and folders, no links }
\begin{lstlisting}
  a=$(echo *); echo 'echo -e "\n<li>"{'${a//" "/,}'}"</li>"' | bash
\end{lstlisting}

\emph{ List only folders, no links }
\begin{lstlisting}
  a=$(echo */); echo 'echo -e "\n<li>"{'${a//" "/,}'}"</li>"' | bash
\end{lstlisting}

\emph{ A for loop method to list only sub-folders as an html list }
  \begin{lstlisting}
    echo "<ul class=fol>"
    for d in $(ls -d */); do
      echo "<li>$d</li>"
    done
    echo "</ul>"
  \end{lstlisting}

\emph{ Another for loop method to list only sub-folders as an html list }
  \begin{lstlisting}
    echo "<ul class=fol>"
    for d in */; do
      echo " <li>$d</li>"
    done
    echo "</ul>"
  \end{lstlisting}

\emph{ List subfolders as html links   }
  \begin{lstlisting}
    echo "<ul class=fol>"
    for d in */; do
      echo "<li><a href='$d'>$d</a></li>"
    done
    echo "</ul>"
  \end{lstlisting}

\emph{ List subfolders as html links using a brace loop }
  \begin{lstlisting}
    echo "<ul class=fol>"
    for d in */
      { echo " <li><a href='$d'>$d</a></li>"; } 
    echo "</ul>"
  \end{lstlisting}

\section{Bash Cgi Programming}

  While it is most common for Cgi web-scripts to be written in 
  the Perl language, it is also possible to write them using the 
  normal Bash shell scripting language. Whether this is a good idea
  is completely another question...

\begin{description}[labelindent=1cm, leftmargin=2cm, style=nextline]
 \item[\url{http://en.wikipedia.org/wiki/Internet_media_type\#List_of_common_media_types}]
     A list of common ``media types'' (such as ``text/html'') which are used
     in the ``Content-Type:'' field of the Cgi script.
\end{description}

  \begin{itemize}
  \item  
     use the ``2$>$\&1'' idiom at the end of script lines to
     redirect an error message to the ``standard output'' (which in the 
     case of a Cgi script is the web-browser of the script visitor).
     This allows you, the developer to see what is going wrong with
     your bash cgi script.
   \item  
     using ``here'' documents with a bash cgi script is a simple way
     to produce content.
   \item 
\end{itemize}
\subsection{The Bash Cgi Gotchas}

  \begin{itemize}
  \item   the content-type line has to be before /anything/ or else
      nothing is printed.
\end{itemize}
\emph{ There must be an empty line after the ``content-type'' line. }
  \begin{lstlisting}
    echo "Content-Type: text/html"
    echo "<html>...</html"
    # This is Incorrect because there is no blank line after the content type
  \end{lstlisting}

\emph{ Trying to call the perl CGI module twice Doesnt work }
  \begin{lstlisting}
      file=$(perl -e "use CGI qw(:standard); print param('file'))
      name=$(perl -e "use CGI qw(:standard); print param('name'))

      # Incorrect, the "name" variable is not set because the 
      # 'cgi' module reads all the form posted data from standard input
      # in one go, leaving nothing to read the second time around.
  \end{lstlisting}

\subsection{Cgi Bash Examples}

\emph{ Make a cgi script to display the query string }
  \begin{lstlisting}
   #!/bin/sh
   echo "Content-Type: text/html"
   echo 
   echo "<html><body>the query string is $QUERY_STRING</body></html>" 2>&1
  \end{lstlisting}


\emph{ A bash cgi script indicating that the character set is ``utf8'' }
  \begin{lstlisting}
  #!/bin/bash
  echo "Content-Type: text/html; charset=utf-8"
  echo
  echo "<html><body>A Bash UTF8 Cgi Script!</body></html>" 2>&1
  \end{lstlisting}

\emph{ Show error messages in the browser generated by a cgi script line }
\begin{lstlisting}
  ech "this is a mistake" 2>&1
\end{lstlisting}

\emph{ Show error messages in the browser with output redirection }
\begin{lstlisting}
  ech "this is a mistake" 2>&1 >save.txt
\end{lstlisting}
  (note that the 2$>$\&1 should come before the file redirection)

\emph{ A cgi script which displays several environment variables }
  \begin{lstlisting}
     #!/bin/bash
     cat << ENDxxx 
       Content-Type: text/html; charset=utf-8 

       <!-- The empty line above this one is essential -->
       <html>
         <head><title>A bash cgi script</title></head>
         <body>
           <dl>
             <dt>DOCUMENT_ROOT<dd>$DOCUMENT_ROOT
             <dt>REMOTE_ADDRESS<dd>$REMOTE_ADDRESS
             <dt>SCRIPT_NAME<dd>$SCRIPT_NAME
           </dl>
         </body>
       </html>
ENDxxx
  \end{lstlisting}

\subsection{Getting And Decoding Form Data}

\begin{description}[labelindent=1cm, leftmargin=2cm, style=nextline]
 \item[\url{http://oinkzwurgl.org/bash_cgi}]
     bash functions for decoding cgi form data
 \item[\url{http://www.fpx.de/fp/Software/ProcCGIsh.html}]
     a bash script and c program for decoding cgi form data. The 
     c program needs to be compiled.
\end{description}

  Data sent from an html form to a web server can be sent in 2
  different ways; in the querystring itself (the 'GET' method) and
  in the HTTP headers (the 'POST' method). If the data is 'posted'
  then the cgi script will receive the data on the standard input
  Data sent from an html form has to be url decoded.

\emph{ Data ``posted'' from an html form can be read from the standard input  }
\begin{lstlisting}
  read postdata   ~(the content length should first be checked)
\end{lstlisting}
\begin{lstlisting}
  postdata=$(</dev/stdin)  ~(the same)
\end{lstlisting}

\emph{ Use perl to get and decode data posted from an html form }
\begin{lstlisting}
  file=$(perl -e "use CGI qw(:standard); print param('file'))
\end{lstlisting}
  (the CGI module takes care of url-decoding the form data)


  \arrayrulecolor{gray}
  \begin{center}
  \begin{tabular}{ |rl| }
    \multicolumn{2}{c}{\textbf{ cgi environment variables and meanings }} \\
  \hline

  \texttt{ DOCUMENT\_ROOT } & The root directory of your server \\ 

  \texttt{ HTTP\_COOKIE } & The visitor's cookie, if one is set \\ 

  \texttt{ HTTP\_HOST } & The hostname of your server \\ 

  \texttt{ HTTP\_REFERER } & The URL of the page that called your script \\ 

  \texttt{ HTTP\_USER\_AGENT } & The browser type of the visitor \\ 

  \texttt{ HTTPS } & ``on'' if the script is being called through a secure server \\ 

  \texttt{ PATH } & The system path your server is running under \\ 

  \texttt{ QUERY\_STRING } & The query string (see GET, below) \\ 

  \texttt{ REMOTE\_ADDR } & The IP address of the visitor \\ 

  \texttt{ REMOTE\_HOST } & The hostname of the visitor (if server has reverse-name-lookups on; otherwise this is the IP address again) \\ 

  \texttt{ REMOTE\_PORT } & The port the visitor is connected to on the web server \\ 

  \texttt{ REMOTE\_USER } & The visitor's username (for .htaccess-protected pages) \\ 

  \texttt{ REQUEST\_METHOD } & Either ``GET'' or ``POST'' \\ 

  \texttt{ REQUEST\_URI } & The interpreted pathname of the requested document or CGI (relative to the document root) \\ 

  \texttt{ SCRIPT\_FILENAME } & The full pathname of the current CGI \\ 

  \texttt{ SCRIPT\_NAME } & The pathname of the cgi script(relative to the doc root) \\ 

  \texttt{ SERVER\_ADMIN } & The email address for your server's webmaster \\ 

  \texttt{ SERVER\_NAME } & Your server's fully qualified domain name (e.g. www.cgi101.com) \\ 

  \texttt{ SERVER\_PORT } & The port number your server is listening on \\ 

  \texttt{ SERVER\_SOFTWARE } & The server software you're using (such as Apache 1.3) \\ 

    \hline
    \end{tabular}
    \end{center}
\emph{ Some environment variables and example values }
  \begin{lstlisting}
     SERVER_SOFTWARE = Apache/2.0.54 (Fedora)
     SERVER_NAME = www.comp.leeds.ac.uk
     GATEWAY_INTERFACE = CGI/1.1
     SERVER_PROTOCOL = HTTP/1.1
     SERVER_PORT = 80
     REQUEST_METHOD = GET
     HTTP_ACCEPT = 'text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5'
     PATH_INFO = 
     PATH_TRANSLATED = 
     SCRIPT_NAME = /cgi-bin/Perl/environment-example
     QUERY_STRING = 
     REMOTE_HOST = 
     REMOTE_ADDR = 200.93.167.25
     REMOTE_USER = 
     CONTENT_TYPE = 
     CONTENT_LENGTH = 
  \end{lstlisting}


  \arrayrulecolor{gray}
  \begin{center}
  \begin{tabular}{ |rl| }
    \multicolumn{2}{c}{\textbf{ cgi Content-Type: values }} \\
  \hline

  \texttt{ text/html } & For html content \\ 

  \texttt{ text/plain } & For plain text \\ 

  \texttt{ text/plain; charset=utf-8 } & Plain text in the utf8 encoding \\ 

  \texttt{ text/xml } & Xml text \\ 

    \hline
    \end{tabular}
    \end{center}
\section{Converting Html}


\subsection{Plain Text Conversion}

\emph{ Display command lines visible on commandlinefu.com homepage }
\begin{lstlisting}
  ruby -ropen-uri -e 'require "hpricot";(Hpricot(open("http://commandlinefu.com"))/".command").each{| c| puts c.to_plain_text}'
\end{lstlisting}

\section{Http}

  http is the protocol apon which the web is based. All web-servers
  must implement at least a part of this protocol.

\subsection{Http Headers}

\emph{ Dump HTTP header using lynx or w3m }
\begin{lstlisting}
  lynx -dump -head http://www.example.com/
\end{lstlisting}

\section{Xml And Html}
 
 xml and html are closely related mark-up languages.
  
\section{Xhtml}


  \arrayrulecolor{gray}
  \begin{center}
  \begin{tabular}{ |rl| }
    \multicolumn{2}{c}{\textbf{ tools }} \\
  \hline

  \texttt{ xmlstarlet } & Queries and edits xml from the command line \\ 

    \hline
    \end{tabular}
    \end{center}
\section{Validation}

  Validation is the process of confirming that a particular html, xhtml, or
  css document actually conforms to the technical specifications for that
  document. Historically browsers have been designed to display more or
  less correctly even html and xhtml pages which contain 'mistakes' or
  syntax which does not conform to the technical specification for that
  type of document.

\begin{description}[labelindent=1cm, leftmargin=2cm, style=nextline]
 \item[\url{http://validator.w3.org}]  
     validates xhtml
 \item[\url{http://jigsaw.w3.org/css-validator}]
     a css validator
 \item[\url{tidy.sourceforge.net}]
     A command line html validator and pretty printer.
\end{description}

\emph{ Install the html tidy program and docs on a debian type linux  }
\begin{lstlisting}
  sudo apt-get tidy tidy-doc
\end{lstlisting}

\emph{ View the html tidy documentation }
\begin{lstlisting}
  ls /usr/share/doc/tidy-doc/htmldoc
\end{lstlisting}
\begin{lstlisting}
  xdg-open /usr/share/doc/tidy-doc/htmldoc/Overview.html
\end{lstlisting}

\emph{ Test out tidy printing errors and warnings }
\begin{lstlisting}
  echo "<html></html>" | tidy -e - 2>&1 | less
\end{lstlisting}

\section{Curl Stuff}

\emph{ Getting a page via an authenticating proxy server }
\begin{lstlisting}
  curl -x proxy.utas.edu.au:8080 -U bobj http://www.server.net 
\end{lstlisting}

\emph{ Get a page via authenticating proxy server as user 'bob' password 'sec' }
\begin{lstlisting}
  curl -x proxy.net:8080 -U bob:sec http://www.server.net
\end{lstlisting}

  Supplying the password in this manner is possibly not a good idea from a
  security point of view 
  
\emph{ Download a text file through a proxy and edit it with vim }
  \begin{lstlisting}
  function edn 
  {
    curl -x proxy.org.au:8080 -U bob:pass www.serv.net/a.txt -o ~/notes.txt
    vim ~/notes.txt
  }
  \end{lstlisting}

\emph{ Upload a file to webserver w cgi script with http authententication,  }
 \begin{lstlisting}
 function up {
  [ -z "$1" ] && echo 'no parameter' && return 1;
  curl -F name='../htdocs/notes/'$1 -F contents='<'$1 -u user:upass -x prox.net:8080 -U bob:proxpass http://serv.net/save.cgi
}
\end{lstlisting}

\section{Perl Tricks}


  \arrayrulecolor{gray}
  \begin{center}
  \begin{tabular}{ |rl| }
    \multicolumn{2}{c}{\textbf{ useful modules }} \\
  \hline

  \texttt{ www::mechanize } \\ 

  \texttt{ lwp } \\ 

    \hline
    \end{tabular}
    \end{center}
\emph{ A perl mechanize example }
  \begin{lstlisting}

    # navigate to the main page
    $mech->get('http://www.somesite.com/'); 

    # follow a link that contains the text 'download this'
    $mech->follow_link( text_regex => qr/download this/i );

    # submit a POST form, to log into the site
    $mech->submit_form(
    with_fields      => {
    username    => 'mungo',
    password    => 'lost-and-alone',
    }
    );

    # save the results as a file
    $mech->save_content('somefile.zip');
   \end{lstlisting}

\section{Php}

\emph{ Testing php configuration }
\begin{lstlisting}
  php -r "phpinfo\(\);"
\end{lstlisting}

\emph{ Get the urls from a webpage }
\begin{lstlisting}
  $browser->getUrls()
\end{lstlisting}

\section{Crawling}

  .. ScriptableBrowser (simpletest)

\section{Load Testing}


  \arrayrulecolor{gray}
  \begin{center}
  \begin{tabular}{ |rl| }
    \multicolumn{2}{c}{\textbf{ tools }} \\
  \hline

  \texttt{ funkload } & Web testing \\ 

    \hline
    \end{tabular}
    \end{center}
\section{Stuff Used In Google Chrome}


  \arrayrulecolor{gray}
  \begin{center}
  \begin{tabular}{ |rl| }
    \multicolumn{2}{c}{\textbf{ tools used in google chrome }} \\
  \hline

  \texttt{ bsdiff } & \\ 

  \texttt{ bspatch } & \\ 

  \texttt{ bzip2 } & \\ 

  \texttt{ dtoa } & \\ 

  \texttt{ hunspell } & \\ 

  \texttt{ ICU } & \\ 

  \texttt{ JSCRE } & Libjpeg, libpng, libxml, libxslt, LZMA SDK, modp\_b64,  \\ 

  \texttt{ Mozilla interface to Java Plugin APIs } & \\ 

  \texttt{ npapi } & Nspr, nss, Pthreads for win32, sqlite,  \\ 

  \texttt{ tlslite } & V8 assembler, WebKit, WTL, zlib \\ 

    \hline
    \end{tabular}
    \end{center}
\section{Forum Sites}

\begin{description}[labelindent=1cm, leftmargin=2cm, style=nextline]
 \item[\url{ask.metafilter.com}]
     visited by knowledgable people
  @@
\end{description}

\section{Serving Pages}

\emph{ Share the current directory tree (via http) at http://\$HOSTNAME:8000/ }
\begin{lstlisting}
  python -m SimpleHTTPServer
\end{lstlisting}

\emph{ Create a webserver to share all files in /tmp/mydocs on port 8081 }
\begin{lstlisting}
  wbox servermode webroot /tmp/mydocs
\end{lstlisting}

\emph{ Create a webserver to share all files in /tmp/mydocs on port 8080 }
\begin{lstlisting}
  wbox servermode serverport 8080 webroot /tmp/mydocs
\end{lstlisting}

\emph{ Sharing file through http 80 port }
\begin{lstlisting}
  nc -w 5 -v -l -p 80 < file.ext
\end{lstlisting}

\section{Cgi Servers}

\emph{ A simple cgi server }
\begin{lstlisting}
  python -m CGIHTTPServer 8080
\end{lstlisting}

\emph{ Some kind of perl cgi server }
\begin{lstlisting}
  HTTP::Simple::PSGI
\end{lstlisting}

\section{Small Webservers}

\begin{description}[labelindent=1cm, leftmargin=2cm, style=nextline]
 \item[\url{http://ask.metafilter.com/65481/Help-me-find-a-cool-little-unix-http-utility-I-cant-remember}] 
    some good things about mini web servers
 \item[\url{http://hping.org/wbox/}]
    site for wbox
\end{description}


  \arrayrulecolor{gray}
  \begin{center}
  \begin{tabular}{ |rl| }
    \multicolumn{2}{c}{\textbf{ small quick and easy webservers }} \\
  \hline

  \texttt{ wbox } & Http server \\ 

  \texttt{ thttpd } & Small web server \\ 

  \texttt{ mini\_httpd } & Same author as thttpd but smaller \\ 

  \texttt{ webfs } & Serves a file system from the web \\ 

  \texttt{ busybox httpd command } & Small webservery thing \\ 

    \hline
    \end{tabular}
    \end{center}
\section{Wbox}
  
\emph{ Show how long each part of a webpage takes to generate }
\begin{lstlisting}
  wbox nowhere.net/page.html timesplit 1
\end{lstlisting}

\emph{ Show the http header information for a page }
\begin{lstlisting}
  wbox www.google.it/notexistingpage.html 1 showhdr
\end{lstlisting}

\section{Mac Osx}

  Installed by default are php, curl, 

\section{Templating}

\begin{description}[labelindent=1cm, leftmargin=2cm, style=nextline]
 \item[\url{http://www.perl.com/pub/a/2001/08/21/templating.html}]
     an article about using templating systems with perl
\end{description}


  \arrayrulecolor{gray}
  \begin{center}
  \begin{tabular}{ |rl| }
    \multicolumn{2}{c}{\textbf{ tools }} \\
  \hline

   \texttt{ Template Toolkit } & Almost active development, perl, and python \\ 

   \texttt{ HTML::Mason } & ? callback style, active as of 2010 \\ 

   \texttt{ Embperl } & Embedd perl into webpages, stopped 2006 \\ 

   \texttt{ HTML::Template } & Perl template module \\ 

   \texttt{ Text::Template } & A general purpose templater \\ 

   \texttt{ Apache::ASP } & Use asp with apache, stopped 2004 \\ 

   \texttt{ CGI::FastTemplate } & Another one \\ 

    \hline
    \end{tabular}
    \end{center}
\begin{description}[labelindent=1cm, leftmargin=2cm, style=nextline]
 \item[\url{http://template-toolkit.org/}] 
   the site for the template toolkit
\end{description}

\section{Template Toolkit}

\emph{ Example statement using dot notation. }
\begin{lstlisting}
  How are things in [% customer.address.city %]?
\end{lstlisting}

\emph{ A for loop }
\includegraphics[width=0.5\textwidth]{\% FOREACH list \%}
      $<$a href=``[\% url \%]''$>$$<$b$>$[\% name \%]$<$/a$>$$<$/a$>$
\includegraphics[width=0.5\textwidth]{\% END \%}
    ,,,

\section{Html Template}

\emph{ Example loop with html::template }
    -----
    $<$TMPL\_LOOP list$>$
      $<$a href=``$<$TMPL\_VAR url$>$''$>$$<$b$>$$<$TMPL\_VAR name$>$$<$/b$>$$<$/A$>$
    $<$/TMPL\_LOOP$>$
    ,,,

\subsection{Mason}

  Seems to be in active development. It can be run
  without a webserver

\begin{description}[labelindent=1cm, leftmargin=2cm, style=nextline]
 \item[\url{http://www.masonhq.com/}]
     the official site
\end{description}

\emph{ Install mason using apt-get }
\begin{lstlisting}
  apt-get install libmason-perl  ~(??? unchecked)
\end{lstlisting}

\section{Perl Stuff}


  \arrayrulecolor{gray}
  \begin{center}
  \begin{tabular}{ |rl| }
    \multicolumn{2}{c}{\textbf{ some interesting perl web modules }} \\
  \hline

  \texttt{ Dancer } & Perl web apps with good examples \\ 

  \texttt{ Web::Simple } & Simple web apps \\ 

  \texttt{ CGI::Application } \\ 

  \texttt{ Catalyst } & Big web apps \\ 

  \texttt{ Mason } \\ 

  \texttt{ Mojolicious } \\ 

    \hline
    \end{tabular}
    \end{center}
\subsection{Web Simple}

\begin{description}[labelindent=1cm, leftmargin=2cm, style=nextline]
 \item[\url{http://search.cpan.org/~mstrout/Web-Simple-0.002/lib/Web/Simple.pm}]
      the documentation for web simple
\end{description}

  Developed in 2009. Can create a webapplication without
  a webserver

\begin{description}[labelindent=1cm, leftmargin=2cm, style=nextline]
 \item[\url{hobbs}] at stackoverflow.com
     knowledgable perl web person
\end{description}

\section{Cpan Tool Crash Course}

\emph{ Start a cpan shell to install mason }
\begin{lstlisting}
  perl -MCPAN -e 'shell'
\end{lstlisting}


  \arrayrulecolor{gray}
  \begin{center}
  \begin{tabular}{ |rl| }
    \multicolumn{2}{c}{\textbf{ simple cpan shell }} \\
  \hline

  \texttt{ h } & Show help \\ 

  \texttt{ get } & Get the source for a module \\ 

  \texttt{ make } & Compile ? the module \\ 

  \texttt{ test } & Test a module \\ 

  \texttt{ install } & Do all of get, make, test \\ 

  \texttt{ clean } & Get rid of a badly installed module \\ 

  \texttt{ look } & See whats happening \\ 

  \texttt{ readme } & See whats going on \\ 

    \hline
    \end{tabular}
    \end{center}
\begin{description}[labelindent=1cm, leftmargin=2cm, style=nextline]
 \item[\url{http://www.livejournal.com/doc/server/lj.install.perl_setup.modules.html}]
  a list of potentially useful modules used with livejournal
\end{description}

\emph{ Upgrade cpan but could cause problems ??? }
  \begin{lstlisting}
    perl -MCPAN -e shell
    cpan> install Bundle::CPAN
    cpan> reload cpan
  \end{lstlisting}

\section{Getting Web Pages From The Command Line}

\emph{ Http get of a web page via proxy server with login credentials }
\begin{lstlisting}
  curl -U username[:password] -x proxy:proxyport webpage
\end{lstlisting}

\emph{ Use netcat to get a webpage }
\begin{lstlisting}
  echo "GET / HTTP/1.0\r\r" | nc -v www.somewebsite.com 80
\end{lstlisting}

\emph{ Get a webpage with the console php tool }
\begin{lstlisting}
  php -r "file('http://metafilter.com/');"
\end{lstlisting}

\emph{ Get a webpage with perl }
\begin{lstlisting}
  perl -MHTTP::Client -e 'print HTTP::Client->new()->get("http://localhost/path")'
\end{lstlisting}

\emph{ A simpler way to do the same }
\begin{lstlisting}
  perl -MLWP::Simple -e 'get("http://www.metafilter.com/")'
\end{lstlisting}

\section{Load Testing}
 
\emph{ Perform 4 queries per second (with 4 processes) on the local webserver }
\begin{lstlisting}
  wbox http://localhost clients 4
\end{lstlisting}

\section{Blog Engines}


  \texttt{ serendipity } & A blog engine with database backend \\ 

  \texttt{ bilboblog } & A simple engine \\ 

    \hline
    \end{tabular}
    \end{center}
\section{Cms Systems}

  

  \arrayrulecolor{gray}
  \begin{center}
  \begin{tabular}{ |rl| }
    \multicolumn{2}{c}{\textbf{ a small summary }} \\
  \hline

  \texttt{ redmine } & Popular small software orientated (ruby \\ 

    \hline
    \end{tabular}
    \end{center}
\section{Notes}

 google uses python

\begin{description}[labelindent=1cm, leftmargin=2cm, style=nextline]
 \item[\url{http://unixmages.com/}]
    an unrelated but interesting site
\end{description}

\section{Google Cli}

  Google provides a command line interface to its services such
  as the 'picasa' photo album service. This may be useful for
  scripting etc.

\emph{ Post to a blogger blog }
\begin{lstlisting}
  google blogger post --tags "GoogleCL, awesome" --title "Test Post" "I'm posting from the command line"
\end{lstlisting}

\emph{ Post to a blogger blog using the file 'post.txt' as the text }
\begin{lstlisting}
  google blogger post post.txt
\end{lstlisting}

\begin{lstlisting}
  google blogger list title,url-site # List posts
\end{lstlisting}
\begin{lstlisting}
  google blogger delete --title "Test Post"
\end{lstlisting}

\begin{lstlisting}
  google delete --title "Silly post number [0-9]*" # Delete posts matching regex
\end{lstlisting}
\begin{lstlisting}
  google tag --title "Dev post" --tags "Python, software" # label an existing post
\end{lstlisting}

\emph{ Examples of putting photos in picasa from the command line }
  \begin{lstlisting}
    google picasa create --title "Vermont Test" --tags Vermont vermont.jpg
    google picasa get --title "Vermont Test" /path/to/download/folder
    google picasa list title,url-direct --query "A tag"
    google picasa post --title "Vermont Test" ~/old_photos/*.jpg # Add to an album
    google picasa tag --title "Vermont Test" --tags "places"
    google picasa delete --title "Vermont Test" # delete entire album
  \end{lstlisting}

\emph{ Examples of posting to youtube from command line }
  \begin{lstlisting}

    google youtube post --category Education --devtags GoogleCL killer_robots.avi
    google youtube delete --title "killer_robots.avi"
    google youtube list # list my videos
    google youtube tag -n ".*robot.*" --tags robot
  \end{lstlisting}

\section{Miscelaneous}
 
\emph{ Query wikipedia via dns by requesting a 'txt' record }
\begin{lstlisting}
  dig +short txt <keyword>.wp.dg.cx
\end{lstlisting}

\emph{ Query wikipedia via dns for the term 'bash'  }
\begin{lstlisting}
  dig +short txt bash.wp.dg.cx
\end{lstlisting}

\section{Jargon}


  \arrayrulecolor{gray}
  \begin{center}
  \begin{tabular}{ |rl| }
    \multicolumn{2}{c}{\textbf{ web jargon }} \\
  \hline
\begin{description}[labelindent=1cm, leftmargin=2cm, style=nextline]
 \item[\url{the}] net and the web
     The net is the internet and all its protocols such as 
     ftp, ssh, http etc where as the web consists of just the 
     http and https protocol and therefor webpages.
 \item[\url{download}]
     transfer files from a net server to your local computer
 \item[\url{markup}]
     a way of embedding extra information in a document with
     the use of 'tags' or 'codes'. Html is an example of markup.
 \item[\url{minimalist}] mark-up
     minimalist mark-up is a markup language which attempts to
     use the minimum number of tags possible to embedd in the 
     document. The purpose of this system is not to interfere 
     psychologically with the human readers and writers of the 
     document who may be distracted and confused by bulky and 
     numerous tags. An example of minimalist markup is 'markdown'
     or most 'wiki' markup languages
 \item[\url{wiki}]
     Is a web site or document which many or most web-visitors
     are able to edit.
 \item[\url{web}] 2.0
     The term encapsulates the idea of the web as a collaborative
     medium for the compilation and consumption of information, 
     rather that a 'uni-directional' information source. The television
     is an example of a uni-directional information source since the 
     viewer is unable to effect its transmissions, apart from changing
     channel or switching off. The web 2.0 is multi-directional because
     the user can create and modify the content (through the use
     of wikis, and collaborative sites).
 \item[\url{tags}]
     mark-up languages often use 'tags' in order to embedd information
     in text documents. Usually there is a start tag and an end tag
     which surrounds the information to which it pertains. For example
     in the text ``$<$em$>$tree$<$/em$>$'' the word 'tree' is surrounded by the 
     start and end html tag ``$<$em$>$''. This system is used by various
     mark-up languages such as html, xml and sgml. 
 \item[\url{semantic}] markup
     the use of mark-up codes or tags to indicate 'semantic' 
     information rather than 'visual display or layout' information.
     The phrase 'semantic information' may be a tautology. An 
     example of 'visual display' markup are the html $<$big$>$ and 
     $<$small$>$ tags which indicate to the document rendering software 
     the text should be displayed bigger or small than the normal
     sized text. Semantic markup is considered to be 'good' in the 
     w3c world and among ideologists, since information is not
     lost in a forest of meaningless layout tags.
 \item[\url{plain}] text
     plain text is the basis of the internet and web. Plain text is any
     data which represents a stream of characters in a human writing 
     system which is encoded electronically according to a standard 'text
     encoding'. An example of these encoding are utf8, ascii, or latin1.
     Modern unicode text encodings (such as utf8) can represent any
     character in any human language which has an established writing
     system
 \item[\url{separation}] of content and presentation
     a much bandied about phrase.
 \item[\url{style}] sheets
     documents which indicate how other documents should be displayed.
     A number of style sheet languages exist but the most common is
     'css' cascading style sheets.
 \item[\url{cms}] - content management system
     A content management system is supposedly a way to quickly build
     websites focussing on the content of that site rather than
     web-development issues and design issues. There are a dizzying array
     of free and open-source cms systems available
 \item[\url{version}] control system
     is a way of managing changes to files made by potentially
     many people, simultaneously
 \item[\url{database}]
     a way of storing data which is supposedly secure and convenient.
     many database systems exist with the simplest being a text file
     database, and the most command being a 'relational' database which
     stores information in a series of linked tables. Databases are often
     used to store the content of web-sites
 \item[\url{content}]
     This is a much used web-term which relates to the information
     which a web-site contains as opposed to the visual or 
     technical design of the site
 \item[\url{documentroot}]
\end{description}
     

    \hline
    \end{tabular}
    \end{center}
   \end{document}