
===========
pure-pandoc
===========

Version @version@, |today|

Albert Gräf <aggraef@gmail.com>

.. pandoc title block for the manpage:
.. % pure-pandoc(1) | Version @version@

.. Note: This document is written in RST syntax, but you can easily convert
.. it to Markdown format using the pure-pandoc script itself! For
.. convenience, the Makefile offers a target for this, just run ``make
.. markdown``. Similarly, ``make html`` and ``make pdf`` will format the
.. manual as an html or pdf file, respectively.

Synopsis
========

pure-pandoc [*options* ...] [*files* ...]

Description
===========

This script acts as a frontend to Pandoc_ which allows you to convert plain
RST_ documents and some Sphinx_ elements to Markdown format (or any other
output format supported by Pandoc). Its primary purpose is to convert the
Sphinx-formatted Pure documentation files. It might also work with other
simple RST-based documents, but the recognition of the Sphinx elements is
currently tailored to Pure-related markup, so the script will surely need some
work if you want to use it for other Sphinx domains such as C/C++ and Python.

.. _Pandoc: http://johnmacfarlane.net/pandoc/
.. _RST: http://docutils.sourceforge.net
.. _Sphinx: http://sphinx-doc.org

The script goes to some lengths to preserve RST link targets and Sphinx
cross-references to program options, functions, variables, etc. For the rest
of the conversion we rely on Pandoc. The script certainly isn't perfect, but
it does a reasonably good job on the RST and Sphinx elements that Pandoc
doesn't handle, and thereby gives you a quick way to obtain usable Markdown
documents from the Pure manuals which can then be edited and formatted in
various ways.

Working with Markdown and Pandoc offers many benefits. For most purposes,
Markdown markup is even gentler and less obtrusive than RST, so it is easier
to both write and read. Markdown also has very good tool support, including
preview tools and editors both on "big" computers and on mobile devices, which
render Markdown documents on the fly and show you the beautifully formatted
results without any further ado. Markdown is also readily supported on code
hosting sites such as Bitbucket and GitHub. And, last but not least, Pandoc
makes it very easy to convert Markdown to a plethora of other document
formats, if the need arises. So there are good reasons that Markdown has
become the prevalent format for plain text documents.

Markdown also has some drawbacks; most notably, it doesn't support
cross-referencing between documents in large collections very well. This is
also the main reason that the Pure project still relies on RST and Sphinx. The
Pure documentation consists of dozens of manuals which are created from
various sources, including "docstrings" scraped from a large collection of
Pure scripts. RST/Sphinx works very well for that, so Pure's documentation
format isn't likely to change anytime soon. But we'd still like to leverage
Markdown's advantages, by making it easy to convert the Pure manuals to
Markdown. This is what the pure-pandoc script is for.

Installation
============

To run this script, you need to have Pandoc and GNU awk (as ``gawk``)
installed. The script probably needs a fairly recent Pandoc version (1.12.3
has been tested). To install the script, simply run ``make install`` in the
source directory (the installation prefix defaults to ``/usr/local``; you can
set this with the ``prefix`` make variable).

Recent pandoc versions should have syntax highlighting for Pure already built
into them. If you're still running an older pandoc version, you may have to
build your own versions of the pandoc and highlighting-kate Haskell modules to
get this. To do this, drop the Kate syntax-highlighting for Pure (available as
etc/pure.xml in the Pure sources) into the highlighting-kate/xml folder. Then
rebuild and install highlighting-kate and pandoc (in that order). This is
described in more detail (for the J language, but the steps are analogous) in
John Baker's blog at
http://bakerjd99.wordpress.com/2012/09/20/pandoc-based-j-syntax-highlighting/.

You can then set the default language for code sections and the desired
highlighting style with Pandoc's ``--indented-code-classes`` and
``--highlight-style`` options when creating the target document. For
instance::

  pure-pandoc pure.txt -o pure.md
  pandoc --indented-code-classes=pure --highlight-style=kate pure.md -o pure.html

Options
=======

General Options
---------------

.. option:: -h
.. option:: --help

   Print a short help message, including a brief summary of the available
   options.

.. option:: -v
.. option:: --verbose

   Print some progress information useful for debugging purposes.

Pandoc Options
--------------

The recognized pandoc options are listed below. They are simply passed
through. Please see the pandoc documentation for a closer description of
these.

.. option:: -o output-file
.. option:: --output output-file

   Set the name of the pandoc output file.

.. option:: -r
.. option:: --reference-links

   Use reference rather than inline links.

.. option:: -s
.. option:: --standalone

   Produce a standalone document.

.. option:: -S
.. option:: --smart

   Produce smart quotes, dashes, etc. in the output.

.. option:: -t output-format
.. option:: --to output-format

   Set the pandoc output format.

The following option can be used to pass any other pandoc option which isn't
directly supported by the script.

.. option:: -O pandoc-options
.. option:: --options pandoc-options

   Additional options to be passed to pandoc. This is cumulative, so each
   argument given with :option:`-O` is added to the pandoc options.

Special Options
---------------

The script also understands some additional pre-, post-processing and
formatting options which affect the generated output.

.. option:: -d
.. option:: --date
.. option:: -V
.. option:: --version

   These options allow you to substitute values for the most commonly used
   meta variables (:verb:`\|today\|`, :verb:`\|version\|` and
   :verb:`\@version\@`) in RST source files. By default, the current date is
   used, and the script also tries to detect the version number from the
   ``version`` variable in a Makefile in the current directory, if present.

.. option:: -C
.. option:: --callouts

   Use a somewhat fancier formatting for RST note blocks (``.. note::``),
   which makes the Markdown rendering look a bit more like in the
   Sphinx-formatted docs.

.. option:: -c[max-items]
.. option:: --collapse-items[=max-items]

   Collapse adjacent item headers in Sphinx markup for options, functions,
   variables, etc., if possible. This gives a denser format if there are
   multiple items for the same description text, but only works if the items
   are rendered as Markdown definition lists (which is the default, but can be
   disabled with the :option:`-H` / :option:`--headers` option, see below).
   This option takes an optional argument which, if specified, denotes the
   desired maximum number of items per header line.

.. option:: -D[/pattern/format/]
.. option:: --describe[=/pattern/format/]

   This lets you adjust the format of generic Sphinx description items
   (``.. describe::``). The optional argument takes the form of a regex
   replacement which is applied to the item text; the result should be valid
   Markdown text. The default format is :verb:`/.*/\`\\0\`/` which formats the
   entire item text as literal (typewriter) text. Something like, e.g.,
   ``/(\w+(\s+\w+)*):\s+(.*)/\1: `\3`/`` might be used to format a description
   of the form ``some text: foo bar`` so that the text up to the colon is set
   in regular text font (you can use :option:`-D` / :option:`--describe`
   without argument to give this specific example a try). This option is not
   straightforward to use (it's much too easy to mess up the replacement and
   end up with invalid Markdown), but may be useful for scripted conversions
   if you're not content with the default formatting of these items.

.. option:: -H
.. option:: --headers

   Uses Markdown section headers rather than definition lists for Sphinx
   descriptions of options, functions, variables, etc., which potentially
   makes links to these items work in some output formats other than html.
   Please check the notes under Usage_ below for details.

.. option:: -i index-file
.. option:: --index index-file

   Creates an index file in order to make cross references between separate
   documents work (see `Multiple Documents`_ below for details). The argument
   denotes the name of the index file to be created or updated.

.. option:: -N template
.. option:: --name template

   Document name template for cross-document links. This must be a printf
   template with exactly one ``%s`` in it. Please check `Multiple Documents`_
   below for details.

.. option:: -n
.. option:: --no-links

   Strips all inline and reference links from the Markdown output, leaving
   just the link text in single quotes. This is useful when generating Pandoc
   output formats such as manpages (``-tman``) which don't support links and
   where the resulting output looks confusing if they are included.

.. option:: -R
.. option:: --raw

   Includes raw sections from the RST source (``.. raw::``). Pandoc seems to
   unconditionally include these sections, which usually messes up the output,
   so pure-pandoc will by default strip them from the RST source during
   pre-processing, so that Pandoc never sees them. But if you really want to
   have this material included in the output then this option will give you
   that.

.. option:: -T
.. option:: --title-block

   Scrapes a title block from the RST source. The title block should be
   formatted as follows (please check the Pandoc documentation for a
   description of the title block syntax)::

     .. % Title
     .. % Author
     .. % Date

   This will be treated as a comment by RST processors. If the :option:`-T`
   option is used, then any additional frontmatter (i.e., everything leading
   up to the title block) will be stripped from the RST source, so that the
   output document begins with the title block. (Note that this means that the
   resulting document will be empty if there's no title block in the source at
   all, so beware!)

   Instead of a Pandoc title block, the script also recognizes a block with
   YAML metadata beginning with three dashes. In fact, the "title block" can
   be any contiguous block of RST comment lines beginning with either ``.. %``
   or ``.. ---``. The title block will be copied verbatim, after removing the
   ``..`` prefixes (the :verb:`\|today\|`, :verb:`\|version\|` and
   :verb:`\@version\@` meta variables will be substituted, however).

Usage
=====

pure-pandoc is invoked as follows: pure-pandoc [*options* ...] [*files* ...].
For simple stand-alone documents, this usually boils down to: pure-pandoc
*in-file* -o *out-file*. Use ``pure-pandoc -h`` to get a brief summary of the
available options.  Here's a typical invocation, which translates the RST
source document ``pure.txt`` to a Markdown document ``pure.md``::

  pure-pandoc pure.txt -o pure.md

The input files must all be in RST format. If none are specified, input is
read from stdin. Output goes to stdout, unless an output filename is given
with :option:`-o` / :option:`--output`.

The output format is usually auto-detected by pandoc if an output filename is
specified with :option:`-o` / :option:`--output`, otherwise you can specify
the desired target format with the :option:`-t` / :option:`--to` option. The
script understands a number of other pandoc options which are simply passed
through, and the :option:`-O` / :option:`--options` option lets you pass any
other options not directly supported by the script. The script also offers a
number of special pre-, post-processing and formatting options which affect
the generated output, please check the Options_ section above for details.

Note that Pandoc produces its own Markdown flavour by default. This is usually
what you want if further processing of the output document will be done with
Pandoc. However, many 3rd party tools work with other Markdown variants, such
as "strict" a.k.a. "basic" Markdown, or MultiMarkdown. Using the :option:`-t`
/ :option:`--to` option, you can specify the target Markdown dialect like
this::

  # for basic Markdown:
  pure-pandoc pure.txt -o pure.md -tmarkdown_strict
  # for MultiMarkdown:
  pure-pandoc pure.txt -o pure.md -tmarkdown_mmd

With the target format specification you can also selectively enable or
disable certain Markdown elements (called "extensions" in the Pandoc
documentation). E.g., to make the Marked_ previewer on the Mac happy, you may
want to produce MultiMarkdown documents with pipe tables, which can be done as
follows::

  pure-pandoc pure.txt -o pure.md -tmarkdown_mmd+pipe_tables

.. _Marked: http://marked2app.com/

Math elements (LaTeX inline and display formulas, ``:math:`` in RST) are fully
supported (as far as Pandoc supports them in the target document format). Note
that depending on the complexity of your formulas, you may have to use a
suitable math rendering engine for html output, such as ``--latexmathml`` or
``--mathjax``; please check the Pandoc documentation for details. For
instance::

  pure-pandoc pure.txt -o pure.html -O --mathjax

As already mentioned, the script goes to some lengths to preserve links to
explicit RST link targets and Sphinx definitions. Unfortunately, due to
limitations in Pandoc and the Markdown format itself, most of these links will
only work in html output. However, references to section and document titles
('``:ref:``' and '``:doc:``' in Sphinx) should work in other document formats
such as pdf, even across different source documents if the appropriate
measures are taken (see `Multiple Documents`_ below).

Sphinx definitions are by default rendered as Markdown definition lists, but
the :option:`-H` / :option:`--headers` option can be used to have them
translated to Markdown section titles instead. pure-pandoc will then use
Pandoc's special PHP Markdown-style syntax for the section link targets. This
syntax isn't widely supported by 3rd party tools, but supposedy it works at
least with a few of Pandoc's own writer backends, so you *might* have better
luck getting these links to work with some non-html document types.

Multiple Documents
==================

Links across different documents are ubiquitous in the Pure documentation, so
pure-pandoc offers reasonably good support for them. However, since Pandoc has
no knowledge about Sphinx's cross-reference syntax, you need to take some
special measures to have pure-pandoc resolve the cross-document links for you.
In particular, you need to create an *index file* in which pure-pandoc records
the RST and Sphinx link targets in each document and other needed information
such as the document titles. This is done by invoking pure-pandoc with the
:option:`-i` / :option:`--index` option, e.g.::

  pure-pandoc pure.txt -o pure.md -i puredoc.index

When working with big collections of documents, entering all the required
commands can become quite tedious; you will want to automatize the process
with a Makefile. Here are some simple rules (using GNU make) which will
translate a collection of ``.txt`` (RST source) files in the current directory
to Markdown and generate the index file along the way::

  sources = $(wildcard *.txt)
  targets = $(patsubst %.txt, %.md, $(sources))

  all: $(targets)

  index: $(sources)
      touch $@

  %.md: %.txt index
      pure-pandoc $< -o $@ -i index

Note that the rule to generate the ``.md`` files also has a dependency on the
index file so that *all* targets will be regenerated whenever the index
changes. Usually it will be necessary to run ``make`` twice so that the index
information used to generate the target documents is up-to-date. We also need
a rule to create an empty index file if it doesn't exist yet, and to ensure
that the index file is at least as new as any of the sources.

Finally, we can generate output in pdf format from the Markdown documents as
follows::

  pdf: mydoc.pdf

  mydoc.pdf: $(targets)
      pandoc -s $^ -o $@

As a convenience, we also added a phony ``pdf`` target so that running just
``make pdf`` will create the pdf document. Similar targets can be used to
create html output or any other output format supported by Pandoc.

Note that this creates one big file with all the source documents in it. This
is most appropriate when creating document formats like pdf which are intended
for offline viewing or printing. For html output it is often preferable to
create separate output documents for each source document instead. In this
case the procedure is slightly different. Specifically, pure-pandoc needs to
know about the names of the output documents for each source file, which are
specified as a printf-style pattern with the :option:`-N` / :option:`--name`
option. The pattern must contain exactly one '``%s``' where the basename of
the source file is to be inserted. In most cases you can just use
``"%s.html"``. So the rule to generate the Markdown files becomes::

  %-html.md: %.txt index
      pure-pandoc $< -o $@ -i index -N "%s.html"

There's a little gotcha here, namely that the contents of the files generated
with the ``-N`` option will be slightly different from the Markdown files we
created earlier in order to produce the pdf document. In order not to clobber
these, we added a special ``-html`` suffix here, so that the new rule can be
combined with the rules for creating pdf output from above. (The index
information is the same in either case, however, so there's no need to create
a separate index file for the html targets.)

The ``-html.md`` files will contain proper cross-document links of the form
``name.html#id``, so that the links are resolved correctly if we create
individual html files as the final output::

  html: $(patsubst %.txt, %.html, $(sources))

  %.html: %-html.md
      pandoc -s $< -o $@

Here's the complete Makefile with targets for both pdf and html output as it
stands now::

  sources = $(wildcard *.txt)
  targets = $(patsubst %.txt, %.md, $(sources))

  all: $(targets)

  index: $(sources)
      touch $@

  %.md: %.txt index
      pure-pandoc $< -o $@ -i index

  pdf: mydoc.pdf

  mydoc.pdf: $(targets)
      pandoc -s $^ -o $@

  %-html.md: %.txt index
      pure-pandoc $< -o $@ -i index -N "%s.html"

  html: $(patsubst %.txt, %.html, $(sources))

  %.html: %-html.md
      pandoc -s $< -o $@

To use this Makefile, just run ``make`` twice to make sure that the index is
up-to-date, then run ``make pdf`` and/or ``make html`` in any order to produce
the final output documents. If you need additional output formats such as
EPUB, DOC, ODT, Texinfo etc., they can easily be created in an analogous
fashion.

The above procedure can be simplified considerably if you only need to create
a single output format. In this case you may just skip the generation of the
intermediate Markdown files and have pure-pandoc create the target format
directly. E.g., creating just the individual html documents can be done simply
as follows::

  sources = $(wildcard *.txt)
  targets = $(patsubst %.txt, %.html, $(sources))

  all: $(targets)

  index: $(sources)
      touch $@

  %.html: %.txt index
      pure-pandoc -s $< -o $@ -i index -N "%s.html"

Or, to create a single pdf document from all sources::

  sources = $(wildcard *.txt)

  all: mydoc.pdf

  index: $(sources)
      touch $@

  mydoc.pdf: $(sources) index
      pure-pandoc -s $(sources) -o $@ -i index

Note that even with this simplified setup you generally have to run ``make``
twice so that the index information is updated before the final output is
produced. This will be inefficient with output formats such as pdf which
involve a time-consuming formatting step. Therefore it is usually preferable
to use the more elaborate procedure above which employs Markdown as an
intermediate representation.

Bugs
====

At present, the script only recognizes a subset of the RST and Sphinx
cross-referencing syntax, so arbitrary complex documents will most likely give
funky results or not work at all. The script has been tested on the entire
Pure documentation, however, and should give proper results there.

Markdown requires links to random locations inside a document to be encoded as
html link anchors which generally works in html output only. This means that
most of the special RST and Sphinx link targets will not work in other output
formats either. However, references to section and document titles should
still work in document formats such as pdf, even across different source
documents if you follow the procedure described in `Multiple Documents`_
above.

Another limitation of Markdown and Pandoc is that if you compile a collection
of source files to a single output document, then *all* referenced link
targets and section names in the source documents must be unique; otherwise
you'll get duplicate link targets and cross-references may point to the wrong
locations in the output document. There's no easy way to work around this with
the current Pandoc version, so if you run into this issue then you'll either
have to fix the duplicate links manually or create separate output documents
instead.

Author
======

Albert Gräf <aggraef@gmail.com>

See Also
========

- pandoc(1): http://johnmacfarlane.net/pandoc/

- Docutils/RST: http://docutils.sourceforge.net

- Sphinx: http://sphinx-doc.org

- The source code of this program can be downloaded at
  http://purelang.bitbucket.org/.
