#*

  mark.latex.simple.pss

  This script attempts to parse a "markdown"-like syntax (this script header is
  an example of the document format) and to transform it into LaTeX source
  code. The script runs on on the "pep" parsing machine which is implemented at
  http://bumble.sf.net/books/pars/ . There are other ways, which may seem
  better or more straight-forward of achieving this. The ANTLR parsing system
  can be used to right grammars that can parse markdown like structures, or
  just regular expressions can be used. But this is a good exercise for the
  pep/nom machine, and more complex structures can be recognised than with
  plain regular expressions.

  Required tokens are, at least: 
    >>* ---* codeline* codeblock* link* file* quoted* emline* uutext* 
    uuword* nl* word* text* [[* (for images)
    We dont actually need heading* and subheading* tokens because they get
    transpiled (into LaTeX) as soon as they are seen in the document.
    Also, dont need link/file/quoted/star

STATUS

  Producing nice readable output with pdflatex.

  Would like to include date lists with descriptions.
  eg 
  24 july 2022
    did something
  26 jul 2022
    more stuff

  The script is now generating compilable tex output with:
  >> pep -f eg/mark.latex.pss pars-book.txt > test.tex
  >> pdflatex test.tex; pdflatex test.tex
 
  This produces a pdf file "test.pdf" and the formatting includes
  lists.

  A work in progress. The document parses as text, but not all structures
  are recognised. This file contains an interesting way of resolving or 
  removing parse tokens when they are no longer significant- it uses a 
  negative approach such as...

  ----
    # codeblocks with no caption (description) 
    !"codeblock*".E"codeblock*".!B"emline*" {
      clear; get; add " "; ++; get; --; put; clear;
      add "text*"; push; .reparse
    }
  ,,,,

  Another interesting idea: use the accumulator as a state marker when
  parsing in codeblocks*. So if acc>0 then dont create uuword or other 
  tokens.

TOKEN LIST

  * tokens currently used by this script
  >> --- >> 4dots codeblock codeline emline nl text uutext uuword word
  
  For lists need: - dash, bl, list
  eg:  o/- -> olist, 
  olist, text, nl, dash -> list
  list, text, bl -> 
  But why have bl???? just used nl nl
  because nl/nl is reduced immediately.

  star* has been eliminated by parsing immediately, and >> and ---
  could also be eliminated. Probably need to add bl*=blankline
  dash* and ulist/olist/dlist for unordered lists, ordered lists
  and definition lists

  Useful grammar analysis.

  * get a unique list of tokens used during parsing
  >> pep -f eg/mark.latex.simple.pss pars-book.txt | sed '/%% ---/q;' | sed 's/^[^:]*: *//;s/\* *$//' | tr '*' '\n' | sort | uniq


BUGS

  Strange pep/nom segmentation fault with multiline comment.

NOTES

  Add images, datelists, inline code?
  Convert this grammar to generate html/markdown etc

  Need to tidy up description lists.

  Nested lists may just work- out of the box! But a different
  list terminator (not blankline) would be handy.

  What about inline code?
  This script needs to parse *any* text successfully! Even text
  that is not in any particular format.

  May need to add "quoted" to handle quoted text, but not really
  necessary at the moment.
  
  Using o/- O/- u/- U/- d/- D/- to start ordered/unordered/description
  lists!

DOCUMENT FORMAT

  
  This is a document format I have used in many code and 
  booklet files. It is a kind of markdown, with even less markup than
  markdown. It would be useful to try to transform it to markdown.
  
  A Description of the format
  - ALL UPPER CASE LINE
      Is a heading
  or UPPER CASE WITH "QUOTES" 

 d/- ALL UPPER CASE LINE WITH FOUR DOTS ....
      is a subheading
  - asterixes before and after a word eg *this*
      emphasises the word.
  - >> a line starting with 2 >
      is a code line 
  - a block surrounded by --- and ,,,, on their own lines
      is a code block
  - * a line starting with a star
      describes the code line or block which follows
  - urls and filenames should get a special format.


  d/- d/- : makes a description list
    - term : end with a colon
    - machine
        or with a newline
    - end the whole list with a blank line
    
  o/- or O/- or 0/- makes and ordered
    - list
    - end with a blank line
    - u/- makes an unordered list.

  Key names are rendered as keys.
  eg [return] is rendered as a keyname.
  urls are turned into links. Filenames are made into fixed-pitch
  font.

LATEX 

  Need to escape all these chars in latex
  >> & % $ # _ { } ~ ^ \
  The first 7 add a backslash eg \&
  The last 3 do
    \textasciitilde, \textasciicircum, and \textbackslash

  \begin{verbatim}...\end{verbatim}

  Look at books/format-book/booktocgi.sed for lots of latex tricks and tips

HISTORY 

   9 July 2022
     ordered and unordered and description lists appear to work well.
     Made emphasis lines (starting with star) appear on a line by 
     themselves. This means they can be used for as a simple list.
     Also, introduced a bl* blankline token which will be used with
     lists (to terminate a list). Made the parser more permissive.
   8 July 2022
     Removed the star* token and parsed immediately in the word block.
     But didnt remove >>* because nl* tokens cause problems.
   7 july 2022
     A lot of progress. Had the idea to use the accumulator to 
     count words on a line and so be able to tell what is the 1st word.
     This could allow to displense with a number of tokens, and simplify
     the grammar.

     Output now seems acceptable apart from lists not working.
     Revisiting this to try to make a nice pep/nom book. Cant use 
     a table in a figure environment. This version mark.latex.simple.pss
     is actually better than mark.latex.pss

   17 June 2021
     Adding emphasis words such as *this* . Need to do lists with - * and
     maybe dates, images with floating and resizing. Better code listings

   14 June 2021
     Looked at how to escape chars. Done. We also escape chars that 
     are in star lines and maybe codelines, but the \verbatim and 
     \lstlisting words seem to take care of special characters in
     those blocks.

    I should try
      -----
        ">>*star*" --> "starline*"
        "starline*word*" --> "starline*"
        "starline*uuword*" --> "starline*"
      ,,,,

    This is in contrast to the technique of immediately reading to the 
    end of the line with "until" or "whilenot"

   3 june
    more work. proceeding well.

   2 june 2021
    Much progress, a basic latex doc has been formed. Now to refine
    and introduce new tokens. Have done sections, subsections, and 
    codeblocks. Found an elegant way to eliminate insignificant tokens
    using a "negative" logic.

    Continuing to write this. The unstack;print;stack trick is 
    very useful. Parsing and printing text with just upper case headings.
    This seems a promising incremental way to parse

   1 June 2021
    Try the unstack trick at the parse label to debug token
    reductions. I fixed the "stack" command in object/machine.interp.c
    so that it updates the tape pointer.

  23 April 2021
    Continuing to work on the script and look for ways to 
    simply debug it. I have come to the idea that ".reparse"
    should also print the stack if a particular switch is set.
    This would allow the stack reductions to be easily followed.

  21 April 2021
    Script begun. This is a second attempt.

vvvvvvv
*#
  
  begin {
    # create a dummy newline so that doc structures work even
    # on the first line of the file/stream.
    add "nl*"; push;
  }

  read;

  ![:space:] {
    # count words per line with the accumulator
    a+;
    whilenot [:space:]; put;
    clear; add "word*"; push; .reparse 
  }

  # keep leading space in newline token?
  [\n] { 
    # set accumulator == 0 so that we can count words 
    # per line (and know which is the first word)
    zero; nochars;
    while [ ]; put; clear; add "nl*"; 
    push; .reparse
  }
  [\r\t ] { clear; !(eof){.restart} }

parse>
  # for debugging, add % as a latex comment.
   add "%%> line "; lines; add " char "; chars; add ": "; print; clear; 
   unstack; print; stack; add "\n"; print; clear;

  # -------------
  # 1 token
  pop;

  "nl*" { nop; }

  # here we classify words into other tokens
  # we can use accumulator with a+ a- to determine if current
  # word is the first word of the line, or even count number of 
  # words per line. This should simplify grammar items such as
  # nl/---  and nl/star/ etc
  # another advantage, is that we can dispense with tokens such as 
  # ---, >> etc and not have to get rid of them later.
  "word*" {
    clear; get; 
    # no numbers in headings!
    [A-Z] { clear; add "uuword*"; push; .reparse }

    # allow uppercase quoted words in headings
    B'"'.E'"'.!'"' { 
      clip; clop; put;
      [A-Z] {
        # add LaTeX curly quotes to the heading word
        clear; add "``"; get; add "''"; put; clear;
        add "uuword*"; push; .reparse 
      }
      # otherwise handle this in quotes.
      clear; add "``"; get; add "''"; put; 
    }

    # at least three --- on a newline marks a code block start
    # use 'count;' here to simplify. The token --- probably doesnt
    # need to exist.
    B"---".[-] { clear; add "---*"; push; .reparse }

    # >> on a newline marks a code line start
    ">>" { add "*"; push; .reparse }

    # subheading marker
    B"....".[.] { clear; add "4dots*"; push; .reparse }

    # dash is used for lists 
    # only make a dash token if it is first word on the line
    "-" { 
      clear; count;
      "1" { clear; add "dash*"; push; .reparse }
      clear; get;
    }

    # ordered list start token 
    # only make token if it is first word on the line
    "o/-","O/-","0/-" { 
      clear; count;
      "1" { clear; put; add "olist*"; push; .reparse }
      clear; get;
    }

    # unordered list start token 
    "u/-","U/-" { 
      clear; count;
      "1" { clear; put; add "ulist*"; push; .reparse }
      clear; get;
    }

    # definition/description list start token 
    # need to parse a bit differently because of the desc
    "d/-","D/-" { 
      clear; count;
      "1" { 
        clear;
        # read description here, but have to escape special
        # or put in verb
        add "\n \\item["; whilenot [\n:]; add "]"; put;
        clear; add "dlist*"; push; .reparse
      }
      clear; get;
    }

    # star on newline marks emphasis, list or code description 
    # probably dont need star token.
    "*" { 
      # check that * is 1st 'word' on line using accumulator
      clear; count; 
      !"1" { clear; add "*"; }
      "1" {
        clear; while [ \t\f]; clear;
        whilenot [\n]; cap; put; clear;
        # this is a trick, because we want special LaTeX chars to
        # be escaped. So, will add \\emph{} after next replace code. 
        add "::EMPH::"; get; put;
        #add "emline*"; push; .reparse
      }
    }

    # need to escape % # } \ and others
    # & % $ # _ { } ~ ^ \   
    # \textasciitilde, \textasciicircum, and \textbackslash
    replace '\\' "\\textbackslash ";
    replace "&" "\\&";
    replace "%" "\\%";
    replace "$" "\\$";
    replace "#" "\\#";
    replace "_" "\\_";
    replace "{" "\\{";
    replace "}" "\\}";
    replace "~" "\\textasciitilde";
    replace "^" "\\textasciicircum";
    replace "LaTeX" "\\LaTeX{}";

    # now make the emphasis line token, after special chars have 
    # been escaped.
    B"::EMPH::" { 
      replace "::EMPH::" " \\emph{"; add "}";
      put; clear;
      add "emline*"; push; .reparse
    }

    # If a previous test has matched, then the workspace should
    # be clear, and so none of the following will match.

    # graphical key representations
    B"[".E"]" {
      replace "[esc]" "\\Esc";
      replace "[enter]" "\\Enter";
      replace "[return]" "\\Enter";
      replace "[insert]" "\\Ins";
      replace "[shift]" "\\Shift";
      replace "[delete]" "\\Del";
      replace "[home]" "\\Home";
      # keys defined by 'keystroke' package, can make new ones.
      # \Enter \Del  \Ins    \Esc   \Shift  \Ctrl  \Home
      # \End   \PgUp \PgDown \PrtSc \Scroll \Break
    }

    #replace '\\n' "\\textbackslash n";
    #replace '\\f' "\\textbackslash f";
    #replace '\\r' "\\textbackslash r";
    #replace '\\t' "\\textbackslash t";

    put;     
    
    # urls, not so important for LaTex (and pdf) output.
    # Dont really need a token because we can render immediately
    # Could maybe render them as footnotes
    B"http://",B"https://",B"www.",B"ftp://",B"sftp://" { 
      # clear; add "url*"; push; .reparse
      # render as fixed pitch font
      clear; add "\\url{"; get; add "}"; put; clear; 
    }

    # filenames, could be elided with quoted filenames
    E".c",E".a",E".txt",E".doc",E".py",E".rb",E".rs",E".java",E".class",
    E".tcl",E".tk",E".sw",E".js",E".go",E".pp",E".pss",E".cpp",E".pl",
    E".html",E".pdf",E".tex",E".sh",E".css",E".out",E".log",
    E".png",E".jpg",E".jpeg",E".bmp",
    E".mp3",E".wav",E"aux",
    E".tar",E".gz",E"/" {
      clear; add "\\texttt{"; get; add "}"; put; clear;
    }

    # paths and directories ? 
    B"../".!"../" {
      clear; add "\\texttt{"; get; add "}"; put; clear;
    }

    B'"'.E'"'.!'""' {
      # filenames in quotes
      clip; clop; put;
      E".c",E".a",E".txt",E".doc",E".py",E".rb",E".rs",E".java",E".class",
      E".tcl",E".tk",E".sw",E".js",E".go",E".pp",E".pss",E".cpp",E".pl",
      E".html",E".pdf",E".tex",E".sh",E".css",E".out",E".log",
      E".png",E".jpg",E".jpeg",E".bmp",
      E".mp3",E".wav",E"aux",
      E".tar",E".gz"
        { clear; add "``\\texttt{"; get; add "}''"; put; clear; }
      # everything else in quotes (but only words without spaces!)
      !"" { clear; add "``\\textit{"; get; add "}''"; put; clear; }
    }

    # filenames 
    # crude pattern checking.
    B"/".!"/" {
      clip; E"." { clear; add "\\texttt{"; get; add "}"; put; clear; }
      clip; E"." { clear; add "\\texttt{"; get; add "}"; put; clear; }
      clip; E"." { clear; add "\\texttt{"; get; add "}"; put; clear; }
    }

    # emphasis is *likethis* (only words, not phrases) 
    B"*".E"*".!"**" {
      clip; clop; put; clear; 
      add "\\textbf{\\emph{"; get; add "}}"; put; clear;
    }

    # && starting a line marks the document title 

    # the document 'title' after && or first heading, & has already 
    # been escaped
    "\\&\\&" { 
      clear; count; 
      "1" {
        clear; while [ \t\f]; clear;
        whilenot [\n]; put; clear;
        add "\\centerline{\\Large \\bf "; get;
        add "} \\medskip \n"; put; clear;
      }
    }

   # A quote, starting the line
   "quote:" { 
      clear; count; 
      "1" {
        # \begin{center}
        #    {\huge \`\`}\textit{$quote}{\huge ''}
        #    \textsc{$quoteauthor}
        #  \end{center}
        clear; while [ \t\f]; clear;
        whilenot [\n]; put; clear;
        add "\\begin{center}{\\huge ``}\\textit{"; get;
        add "}{\\huge ''}\\end{center} \n"; put; clear;
      }
    }

    clear; add "word*";
  }

  pop;
  # -------------
  # 2 tokens

  # ellide text
  "text*text*","word*text*",
  "word*word*","text*word*",
  "word*uuword*","text*uuword*","uutext*word*","uuword*word*" {
    clear; get; add " "; ++; get; --; put; clear;
    add "text*"; push; .reparse
  }

  # tokenlist:
  # --- >> 4dots codeblock codeline emline nl text uutext uuword word
  # codeblock,
  # remove pesky newline tokens, 4dots handled elsewhere
  # not really working
  #*
  "nl*text*","nl*word*","nl*emline*","nl*codeline*",
  "nl*codeblock*" {
    # delete nl token
    clop; clop; clop; push; clear;
    # ignore newline
    get; --; put; clear;
    .reparse
  }
  *#

  "nl*text*","nl*word*", "bl*text*","bl*word*" {
    clear; get; ++; get; --; put; clear;
    add "text*"; push; .reparse
  }

  "nl*dash*" {
    clear; get; ++; get; --; put; clear;
    add "dash*"; push; .reparse
  }

  "nl*emline*","bl*emline*" {
    clear; ++; get; --; put; clear;
    add "emline*"; push; .reparse
  }

  # We are using a dummy nl* token at the start of the doc, so the 
  # codeblock* codeline* etc tokens are not able to be the first token
  # of the document. So we can remove the !"codeblock*". clause.

  # multiline codeblocks with no caption 
  E"codeblock*".!B"emline*" {
    clear; get; 
    add "\n\n \\begin{tabular}{l}\n  ";
    ++; get; --; 
    add " \\end{tabular} \n";
    put; clear;
    add "text*"; push; .reparse
  }

  # single line code with no caption 
  E"codeline*".!B"emline*" {
    clear; get; 
    add "\n\n \\begin{tabular}{l}\n  ";
    ++; get; --; 
    add " \\end{tabular} \n";
    put; clear;
    add "text*"; push; .reparse
  }

  # eliminate emline* tokens (not followed by codeblock/line)
  # the logic is slightly different because emline* is significant before
  # other tokens, not after.
  # also, consider emline*text*nl*
  B"emline*".!E"nl*".!E"codeline*".!E"codeblock*" {
    replace "emline*" "text*"; push; push; 
    # make emline display on its own line, even when not
    # followed by codeline/codeblock. LaTeX will treat a blank line 
    # as a paragraph break, but \newline or \\ could be used.
    --; --; add "\n\n"; get; add "\n\n"; put; clear;
    .reparse
  }

  # remove insignificant 4dots* tokens, 
  # 4 dots (....) marks a subheading and always comes at the end of 
  # all capitals line. Just replacing the 4dots token with a text
  # token is safer and more logical.
  E"4dots*".!B"uutext*".!B"uuword*" {
    replace "4dots*" "text*"; push; push; .reparse
  }

  # remove insignificant ---* tokens
  E"---*".!B"nl*".!B"bl*" {
    clear; get; add " "; ++; get; --; put; clear;
    add "text*"; push; .reparse
  }

  # remove insignificant >>* tokens
  # lets assume that codelines cant start a document? Or lets
  # generate a dummy nl* token at the start of the document to 
  # make parsing easier.
  # !">>*".E">>*".!B"nl*" {
  E">>*".!B"nl*".!B"bl*" {
    clear; get; add " "; ++; get; --; put; clear;
    add "text*"; push; .reparse
  }

  # ellide upper case text 
  "uuword*uuword*","uutext*uuword*" {
    clear; get; add " "; ++; get; --; put; 
    clear; add "uutext*"; push; .reparse
  }

  # a blank line token for terminating lists etc 
  # bl/bl should not happen really
  "nl*nl*","bl*nl*","bl*bl*" {
    clear; get; ++; get; --; put; clear;
    add "bl*"; push; .reparse
  }

  # ellide multiple newlines 
  #"nl*nl*" {
  #  clear; get; ++; get; --; put; clear;
  #  add "nl*"; push; .reparse
  #}

  
  # code line (starts with >>) 
  "bl*>>*","nl*>>*" { 
    # ignore leading space.
    clear; while [ \t\f]; clear;
    whilenot [\n]; put; clear;
    add " \\verb|"; get;
    add "|\n"; put; clear;
    add "codeline*"; push; .reparse
  }

  # code block marker 
  "bl*---*","nl*---*" { 
    clear; until ",,,"; clip; clip; clip;
    # remove excessive indentation.
    replace "\n   " "\n";
    put; while [,]; clear;
    add "\n \\begin{lstlisting}[breaklines]"; get;
    add "\n \\end{lstlisting} \n"; put; clear;
    add "codeblock*"; push; .reparse
  }

  # a code block with its preceding description
  "emline*codeblock*" {
    clear; 
    add "\n\n \\begin{tabular}{l}\n  ";
    get; add " \\\\ "; ++; get; --; 
    add " \\end{tabular} \n";
    put; clear;
    add "text*"; push; .reparse
  }

  # a code line with its preceding description
  # add some tabular LaTeX markup here.
  "emline*codeline*" {
    clear; 
    add "\n\n \\begin{tabular}{l}\n  ";
    get; add " \\\\ \n";
    ++; get; --; 
    add " \\end{tabular} \n";
    #add " \\end{figure}";
    put; clear;
    add "text*"; push; .reparse
  }

  # probably indicates an empty - at the end of a list
  # add a dummy text token
  "olist*bl*","ulist*bl*","dlist*bl*" {
    push; clear; add "empty"; put; 
    clear; add "\n\n"; ++; put; --;
    clear; add "text*bl*"; push; push; .reparse
  }
  # also ulist*dash* what to do.??
  "olist*dash*","ulist*dash*","dlist*dash*" {
    push; clear; add "empty"; put; 
    clear; add "text*dash*"; push; push; .reparse
  }

  pop;
  # -------------
  # 3 tokens

  "olist*word*dash*","ulist*word*dash*","dlist*word*dash*",
  "olist*word*bl*","ulist*word*bl*","dlist*word*bl*" {
    replace "word*" "text*"; 
    # or dont reparse
    # push; push; push; .reparse
  }

  # eliminate dashes that are not part of a list
  # eg: ulist*dash* olist*text*dash* dlist*word*dash*
  # the logic is tricky, how do we know there are really 3 tokens 
  # here, and not 2. This is the problem with negative tests.
  # doesnt matter because not altering attributes here.
  E"dash*" {
    !B"ulist*text*".!B"olist*text*".!B"dlist*text*" {
      replace "dash*" "text*"; push; push; push; .reparse
    }
  }

  "olist*text*dash*" {
    clear;
    get; add "\n \\item "; ++; get; --; put; clear;
    add "olist*"; push; .reparse
  }

  # could be ellided, but for readability, no
  "ulist*text*dash*" {
    clear;
    get; add "\n \\item "; ++; get; --; put; clear;
    add "ulist*"; push; .reparse
  }

  # 
  "dlist*text*dash*" {
    clear;
    # already have \item start
    get; add " "; ++; get; --; 
    # also, put a \verbatim in [] because text is not escaped??
    add "\n \\item["; whilenot [\n:]; add "] ";
    put; clear;
    add "dlist*"; push; .reparse
  }

  # finish off the ordered list, also could finish it off with 
  # ulist*dash* ??
  "olist*text*bl*" {
    clear; 
    add "\n \\begin{enumerate}\n"; get;
    add "\n \\item "; ++; get; --; 
    add "\n \\end{enumerate}\n\n"; 
    put; clear;
    # insert the blankline attribute
    add "\n\n"; ++; put; --; clear;
    add "text*bl*"; push; push; .reparse
  }

  # finish off the unordered list
  "ulist*text*bl*" {
    clear; 
    add "\n \\begin{itemize}\n"; get;
    add "\n \\item "; ++; get; --; 
    add "\n \\end{itemize}\n\n"; 
    put; clear; 
    # insert the blankline attribute
    add "\n\n"; ++; put; --; clear;
    add "text*bl*"; push; push; .reparse
  }

  # finish off the description list
  "dlist*text*bl*" {
    clear; 
    add "\n \\begin{description}\n"; get;
    add "\n "; ++; get; --; 
    add "\n \\end{description}\n\n"; 
    put; clear; 
    # insert the blankline attribute
    add "\n\n"; ++; put; --; clear;
    add "text*bl*"; push; push; .reparse
  }

  # top level headings, all upper case on the line in the source document.
  # dont need a "heading" token because we dont parse the document as a 
  # heirarchy, we just render things as we find them in the stream.
  "nl*uutext*nl*","nl*uuword*nl*",
  "bl*uutext*nl*","bl*uuword*nl*" {
    clear; 
    # Check that heading is at least 4 chars
    ++; get; --; clip; clip; clip; 
    "" { 
      add "nl*text*nl*"; push; push; push; .reparse
    }
    clear;
    # make headings capital case
    ++; get; 
    # capitalise even 1st word in latex curly quotes
    # add "<<heading\n"; print; replace "<<heading\n" "";
    B"``" { clop; clop; }
    cap; put; replace "''" "";
    # add open curly quotes if there before.
    !(==) {
      clear; add "``"; get;
    }
    put; --; clear; 
    get; # newline
    add '\\section{'; ++; get; --; add "}"; put; 
    clear;
    # transfer nl value
    ++; ++; get; --; put; clear; --;
    add "text*nl*"; push; push; .reparse
  }

  # simple reductions 
  "nl*text*nl*","nl*word*nl*", "bl*text*nl*","bl*word*nl*",
  "text*text*nl*","emline*text*nl*" {
    clear; get; ++; get; --; put; clear;
    ++; ++; get; --; put; --; clear; # transfer newline value
    add "text*nl*"; push; push; .reparse
  }

  pop;
  # -------------
  # 4 tokens

  # sub headings, 
  "nl*uutext*4dots*nl*","nl*uuword*4dots*nl*",
  "bl*uutext*4dots*nl*","bl*uuword*4dots*nl*" {
    clear; 

    # Check that sub heading text is at least 4 chars ?
    # yes but need to transfer 4dots and nl
    # ++; get; --; clip; clip; clip; 
    # "" { add "nl*text*nl*"; push; push; push; .reparse }

    clear;
    # make subheadings capital case
    ++; get; 
    # capitalise even 1st word in latex curly quotes
    B"``" { clop; clop; }
    cap; put; replace "''" "";
    # add open curly quotes if there before.
    !(==) {
      clear; add "``"; get;
    }
    put; --; clear; 
    get; # newline
    add '\\subsection{'; ++; get; --; add "}"; put; clear;
    # transfer nl value
    ++; ++; ++; get; --; --; put; clear; --;
    add "text*nl*"; push; push; .reparse
  }

  push; push; push; push;

  (eof) {
    # or use 'unstack' but does it adjust the tape pointer?
    pop; pop; pop; pop; pop; pop;

    # "nl*word*","nl*text*" have already been dealt with.

    # we would like "permissive" parsing, because this is just
    # a document format, not code, so will just check for starting
    # text token
    #"text*nl*","text*bl","text*" {

    B"text*",B"word*" {
      # show the token parse stack at the top of the document
      ++; put; clear; 
      add "%% Document parse-stack is: "; get; add "\n"; --;
      clear; 
      # make a valid LaTeX document
      add "
  %% <start-document>
  %% -------------------------------------------
  %%  latex generated by: mark.latex.pss 
  %%   from source file : 
  %%                  on: 
  %% -------------------------------------------

  \\documentclass[a4paper,12pt]{article}
  \\usepackage[margin=4pt,noheadfoot]{geometry}
  \\usepackage{color}                   %% to use colours, use 'xcolor' for more
  \\usepackage{multicol}                %% for multiple columns
  \\usepackage{keystroke}               %% for keyboard key images
  \\usepackage[toc]{multitoc}           %% for multi column table of contents
  \\usepackage{tocloft}                 %% to customize the table of contents
  \\setcounter{tocdepth}{2}             %% only display 2 levels in the contents
  \\setlength{\\cftbeforesecskip}{0cm}   %% make the toc more compact
  \\usepackage{listings}                %% for nice code listings
  \\usepackage{caption}                 %% 
  \\lstset{
    captionpos=t,
    language=bash,
    basicstyle=\\ttfamily,           %% fixed pitch font
    xleftmargin=0pt,                %% margin on the left outside the frames
    framexleftmargin=0pt,
    framexrightmargin=0pt,
    framexbottommargin=5pt,
    framextopmargin=5pt,
    breaklines=true,                %% break long code lines
    breakatwhitespace=false,        %% break long code lines anywhere
    breakindent=10pt,               %% reduce the indent from 20pt to 10
    postbreak=\\mbox{{\\color{blue}\\small$\\Rightarrow$\\space}},  %% mark with arrow
    showstringspaces=false,            %% dont show spaces within strings
    framerule=2pt,                     %% thickness of the frames
    frame=top,frame=bottom,
    rulecolor=\\color{lightgrey}, 
    % frame=l
    % define special comment delimiters '##(' and ')'
    % moredelim=[s][\\color{grey}\\itshape\\footnotesize\\ttfamily]{~(}{)},
  }   %% source code settings
  \\usepackage{graphicx}                %% to include images
  \\usepackage{fancybox}                %% boxes with rounded corners
  \\usepackage{wrapfig}                 %% flow text around tables, images
  \\usepackage{tabularx}                %% change width of tables
  \\usepackage[table]{xcolor}           %% alternate row colour tables
  \\usepackage{booktabs}                %% for heavier rules in tables
  \\usepackage[small,compact]{titlesec} %% sections more compact, less space
  \\usepackage{enumitem}                %% more compact and better lists
  \\setlist{noitemsep}                  %% reduce list item spacing
  \\usepackage{hyperref}     %% make urls into hyperlinks
  \\hypersetup{              %% add pdftex if only pdf output is required
     colorlinks=false,       %% set up the colours for the hyperlinks
     linkcolor=black,        %% internal document links black
     urlcolor=black,        %% url links black
     frenchlinks=true,
     bookmarks=true, pdfpagemode=UseOutlines}

  \\geometry{ left=1.0in,right=1.0in,top=1.0in,bottom=1.0in }
  %% define some colours to use
  \\definecolor{lightgrey}{gray}{0.70}
  \\definecolor{grey}{gray}{0.30}

  %% titlesec: create framed section headings
  %% \\titleformat{\\section}[frame]{\\normalfont}
  %%   {\\filleft \\footnotesize \\enspace Section \\thesection\\enspace\\enspace}
  %%   {3pt} {\\bfseries\\itshape\\filright}

  \\title{The Pep/nom parsing language and machine}
  \\author{m.j.bishop}
  \\date{\\today}
  \\setlength{\\parindent}{0pt}
  %% \\setlength{\\parskip}{1ex}

  %% label lists with stars
  \\renewcommand{\\labelitemi}{$\\star$}

  \\parindent=0pt
  \\parskip=6pt
  \\begin{document}

  ";

      get; 
      add "\n\\end{document} \n";
      add "\n\n %% Document parsed as text*!\n"; 
      # show parse-stack at end of doc as well
      ++; add " %% Document parse-stack is: "; get; add "\n"; --;
      print; quit;
    }

    stack; 
    add "Document parsed unusually!\n";
    add "Stack at line "; lines; add " char "; chars; add ": "; print; clear; 
    unstack; print; stack; add "\n"; print; clear;
    quit;

  }
