The automation of documentation
A simple introduction to the automation of the documentation process
--
Documenting with high flexibility, combined with version control, continuous delivery and a variety of output formats — how to do that is best seen from software development. A simple tool set based on Markdown, git, Pandoc and mkdocs offers excellent possibilities.
German Translation: click here
The real life
Whether SGML, Docbook, HTML, LaTeX, Word texts, Illustrator or any other documentation format — every format, every application and every documentation system for writing and managing texts has its advantages and disadvantages. First of all probably the entry step to learn the syntax and/or the system for writing the texts. If you have perhaps learned LaTeX a little in your own studies, you will find in the company that Docbook or SGML is used. Or your own HTML knowledge is handed out at the entrance because you use Word company-wide and thus share the lowest common denominator with all colleagues and external persons.
It becomes difficult when the versions of the software are not compatible with each other and texts have to be exchanged in Word 97 format. In addition, documentations must not only be distributed as PDF documents, but perhaps also as HTML pages or eBooks. The versioning may then also be noted in the file name or something like a sharepoint server may be used.
If you look at the entire process for creating, maintaining and providing documentation, you will quickly notice that not everything always runs as perfectly as the business process has defined it. You can use a different specialist area and not only simplify the process, but also secure it and make it more transparent.
The tools described in this article can be illustrated in a simple process diagram. Here you can see that the manual effort in the overall process is low and after the setup of all tools involved many process steps can run automatically. Each of the steps shown here will be explained below.
Looking beyond the end of the nose
In software development, versioning, separation of content and layout, and continuous delivery have been known for a long time. What works there can also be used for documentation.
If you look at the topic of documentation, the individual steps can be broken down very well. In simplified form, texts are written, corrected and edited, stored in different versions and converted into an output format. To simplify matters, we take proofreading and editing out of the process at this point. Writing, versioning and output of the documentation then remain.
The Writing
Do you remember how you learned how to write? Surely you scribbled on a sheet of paper with a pencil and at some point an adult showed you how to make characters out of the scribble, then single words out of it and then longer texts out of it. Substantially later you learned what a layout is and how helpful spell-checking can be; what a word processor is used for and how to write documentation. Have you ever wondered if it could be easier and if you need it all at all?
When writing text, it’s often helpful to just focus on the text. Without distractions and without layout specifications. Simple text editors for so-called “plain text” are suitable for this. Here there are only the system fonts and no style sheets. In case of doubt, the spelling correction comes from the operating system, but not from the program. These texts are stored in the simplest format that any system can read without exception.
So it’s about the writing of text itself. The layout is separated from this text and defined later. This gives you the freedom to think about the layout and the content separately and to edit them separately. Especially in the case of extensive documentation, the authors do not have to deal with the design, but can leave it to other specialists. And even if you are the only one who takes care of everything, the layout is simply another process step that does not change the content, but only the appearance. This offers the flexibility to store the text only once and still be able to offer it in different output formats. Copying the text (for further processing) is ideally no longer necessary.
The only thing that is needed is a text editor that meets the simplest and own requirements. Here you should make sure that simple text files (*.txt file extension) can be created. Editors like Notepad++ or BBEdit are ideal. Editors from software development can also be used, such as Brackets, Atom or Visual Studio Code.
You may already be wondering how the layout will be defined later if the texts are only written in the simplest text format. This happens with just as simple formatting in the text that you write yourself. So you define and type these formats yourself.
Why is that better and sensible? Also quite simple: You define “what” should be considered in the layout, but not “how” it should be considered. Separate the process mentally here as well. The content is defined in the text and which content is displayed differently at the end. However, it is not defined how this content is displayed. This is the layout question at the end.
If you use the simplest form for creating texts, there must also be a simple form for formatting. This can be found in the “markdown” format.
The format
Markdown is a very simplified markup language. In the standard version a few details are enough to write a documentation. This means that texts created with Markdown are interpreted to display the texts according to the used information.
If it is not necessary to use exactly the same form of presentation, it is sufficient to use an editor that converts and displays the corresponding formats (e.g. by extensions in the editors Brackets and Atom or by independent applications for viewing markdown files).
As a rule, however, a markdown file should be transformed into a document that always looks the same for every user. For this purpose, the markdown files must be “translated”. For example, into a (static) web page or into another output format, such as PDF. This will be explained in one of the next steps.
With the help of Markdown data in simple texts, you not only have simplification, but also flexibility in later processing. To achieve this, markdown provides a few simple syntax rules.
For headings, use the hash character (#). A `#` is placed before the main heading (first order heading). Second order headings are prefixed with two `#`, third order headings with three of these characters, and so on. Up to six subsections are possible.
# Title
## First Chapter
### Sub chapter to the first
## Second Chapter
### First sub chapter to the second
### Second sub chapter to the second
Bold and italic text formatting are marked with `*`. The corresponding text is simply enclosed in `*`. One asterisk applies to italic text, two asterisks to bold text, and three asterisks to italic and bold text.
*italic*
**bold**
***italic and bold***
There are more format specifications, which are shown in the following table.
For these listed items there are also items for tables, footnotes and many more. As a general rule, markdown only defines that a text is given a layout, but not the layout itself. How certain specifications are interpreted and displayed depends on the output system (the layout definition). In principle, however, the format specifications standardized here should always be considered.
For a complete help it is recommended to have a look at a Cheat Sheet:
https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet
https://www.markdownguide.org/cheat-sheet/
https://www.heise.de/mac-and-i/downloads/65/1/1/6/7/1/0/3/Markdown-CheatSheet-Deutsch.pdf
If the standard markdown formats are not sufficient, they can be extended by HTML and LaTeX specifications. However, both should only be used if the output format allows it. For example, HTML data can only be used for web pages; in PDF files these values are probably output as text.
In addition to the format specifications, it depends on the output format and the layout definition whether and how tables of contents are displayed. For example, a web page does not require a table of contents, but a navigation structure; in a PDF file, this is not necessary, but a table of contents. The manually creation of a table of contents is therefore not necessary and does not make sense. This belongs in a later, automated step.
Additional information about the document (so-called meta information) can be created directly at the beginning of the file (without previous blank lines). This information is taken into account depending on the output system (e.g. PDF export) and also serves as information for other authors or other participants.
Different data can be used as meta-information. The title, authors and creation date should always be listed. The information is introduced with three hyphens and ended with three dots. All information in between are meta information.
- — -
title: ‘The Title’
author:
- Person 1
- Person 2
- …
copyright:
- Value 1
- Value 2
- …
date: 2019–07–30
keywords: [ ]
abstract: …..
fontsize: 12px
papersize: a4
…
When working with multiple files, e.g. to separate chapters from each other or to subdivide areas for a web page, each file can contain such meta data.
Information for page breaks can be helpful for later output in a print format (e.g. PDF). In a later process step, the text files (markdown files) are converted to another format (PDF file). Certain meta data are taken into account here. For a page break before a new chapter, the corresponding LaTeX information can be specified here:
- — -
include-before:
- ‘`\newpage{}`{=latex}’
…
When a text file with markdown information has been created, it is saved with the file extension “md”. For example, “myFile.md” instead of “myFile.txt”. The file extension “md” indicates to other programs that this is a markdown file that should be displayed and edited accordingly. The file extension has no influence on the content. It is a simple text file with markdown formatting, which describes the later layout.
Before we look at the possibilities of output and layout, the question of tracking changes (versioning) should be addressed. On the one hand, changed content should not be lost, on the other hand, the changes of all authors involved should be traceable.
Change tracking (versioning)
In software development, as in documentation, it is necessary to be able to track and, if necessary, undo changes. In addition, various developments should take place at the same time, without influencing the current status and other ongoing developments. Corrections must also be able to be implemented without changing ongoing developments.
In cooperation with other persons, it is also necessary for everyone to be able to integrate and, if necessary, change the modifications made by the other persons. Competing changes must therefore also be taken into account.
In the last decades, versioning systems have been established for the mentioned use case to cover the above mentioned requirements. In addition to freely available systems such as Subversion or git (or older systems such as cvs), commercial systems also exists. All of them usually offer the possibility to work on files locally at a workstation and to synchronize these files with a certain version (the so-called development line). Additional information from other people is then merged and made available centrally for all participants.
The git system has established as one of the best-known and most frequently used systems in recent years. It is considered to be very stable, reliable and easy to use. Git is freely available and can be used locally on a single workstation as well as in a network of workstations and one (or more) servers. If required, free providers can also be used via the Internet (e.g. Github, BitBucket, etc.). The flexibility of git therefore allows the system to be used in almost any working environment.
The use of git can be confusing for new users at first. Like any unknown system, an initial learning curve has to be passed through. Therefore only the basics are explained here. Git is a very powerful tool that offers many features. For deeper questions it is recommended to read a detailed git documentation (e.g. git Basics).
In addition to the versioning functions, git offers the possibility of automatically running processes if something has changed within a file structure. Such automatisms are controlled by so-called hooks and can be used, for example, to create a pdf file (or any other output format) from a markdown file or to start other processes.
For documentation, git can be used by installing it on the local workstation (and all other workstations involved). The installation programs for the various operating systems are ideal for this purpose. The various applications are available from the git website (https://git-scm.com/downloads).
The local installation ensures that a version control can be created for any directory. If a directory is managed by git, the git functions can be used.
Note: If it is already clear at the beginning of a documentation that it should also be available on a central server, the corresponding git directory should first be managed there. The local workstations then “clone” the server directory. This is one of the main differences to the previous storage of files on a server. A user no longer writes directly to the server, but clones a copy to his local workstation and compares it with the server (with so-called commits and push).
In order to manage a local directory via git, various editors offer the corresponding functions. Basically, a directory is initialized for git. This can be done manually by changing to the directory (in the console, in Windows with the command line cmd and under Mac/Linux with the terminal) and executing the command `git init`. With `git init` a so called repository is created. All changes in the directory can then be versioned.
If a repository is to be adopted from a server, this is called cloning. Then all settings and files are taken over and each change is also traceable on the local computer.
To clone an existing repository, the corresponding functions are available in the various editors or you can work manually by executing the command `git clone` with the respective server address (e.g.: `git clone git://github.com/git/htmldocs.git`).
Note: To execute the individual git commands, it is a good idea to use separate applications. These applications provide an interface to manage and synchronize local git repositories. The best known are Sourcetree (https://www.sourcetreeapp.com/) and GitKraken (https://www.gitkrake).
When a local directory has been created (or cloned from a server), the directory can be used as a normal file repository (e.g. for markdown files). Please note, however, that new and changed files are only versioned correctly by git when git executes the corresponding command.
In order for git to take over new and changed files into the version management, a so-called commit is necessary. This saves the state of the files at this moment (so-called snapshot).
If you are working with servers, the local repository must be synchronized with the server. For this purpose, a so-called push must be executed. The local changes are then communicated to the server and are available to all other participants.
If several participants work on the same repository (i.e. with the help of one server), it is necessary that the changes of all others are downloaded. This requires a pull.
The individual commands illustrate the basic process in version management and in collaboration with others. There is a server available to synchronize all participants. Changes are transferred to the local repository on each workstation (commit) and then communicated to the server (push). Changes of all other participants have to be downloaded and integrated (pull).
Note: As long as different parties are working on different files, there is no indication of competing changes. If different people are working on the same file within the same period of time, there may be corresponding notifications that have to be resolved (e.g. by a merge). These and other cases are explained in the git documentation and should be considered at an early stage. Changes should therefore be transferred to the server’s repository as soon as possible so that all participants get changes early and competing changes are minimized.
By creating documentation as a markdown file and transferring it to a git repository, versioning was set up and the output prepared for various formats.
Output formats and layout
Especially for documentations web pages are used as output format. These are often based on “Content Management Systems” (CMS), which make it possible to map an editorial process. Such CMS use database systems and a software infrastructure that must be installed, maintained, updated and secured. The effort often lies in the hands of the IT department. Both the content management system and the database behind must be protected against unauthorized access.
CMS have also been developed for regular maintenance of new content by different users. The scheduling of publications, a rights management and the connection of external services such as Twitter, Facebook etc. are often part of this. For a documentation that only generates most changes at the beginning of creation, but then remains relatively static, it must be questioned whether the “technology stack” is necessary for the operation of a CMS. It could also work with a generated, static website, which is maintained on the basis of markdown files and generated by a git automatism.
A website should therefore be considered as an output format that is generated automatically. Various systems are suitable for this, through which static web pages are generated. One of them is “MkDocs” — a freely available and very simple system developed with the Python programming language. Generating the website can be done with a simple program call, which is executed by a so called “Git-Hook”.
The exact procedure and the setup of a “Git-Hook” can be found in the Git documentation (https://git-scm.com/book/de/v1/Git-individuell-einrichten-Git-Hooks). In principle, hooks can be integrated and executed at any point within the git process. A hook can contain a command to execute other programs. For example, calling “MkDocs” to generate a static web page based on just updated markdown files.
Web page
A website usually consists of HTML, CSS and possibly some JavaScript. If current content is loaded and displayed from a database each time it is accessed (e.g. when using a CMS), this is referred to as dynamic content (e.g. daily newspapers, etc.). In contrast, static pages are created or generated once and stored as individual files. These are simply loaded and displayed during access, without the need for a further server request. This is sufficient for most online documentations.
A variety of different tools are available for generating static web pages based on Markdown. Here we refer to the list of StaticGen (https://www.staticgen.com/), which shows different systems and further related information.
At this point, MkDocs and its basic functions are described. Alternatively, any other system can be used to create static web pages.
MkDocs
MkDocs (https://www.mkdocs.org/) is based on the Python programming language and can also be extended with Python libraries. Detailed documentation can be found on the website, as well as installation instructions.
Basically, MkDocs requires a directory structure containing the configuration file for MkDocs (the file “mkdocs.yml”) and a subdirectory named “docs”. All markdown files are stored in the subdirectory, which are then pointed to in the configuration file.
A simple structure for the website is therefore defined by the file “mkdocs.yml”. Here at least the name and the markdown files to be displayed are defined.
site_name: My Online-Doc
nav:
- Home: index.md
- About: about.md
Note: The main file must always be named “index.md” and must be included in the navigation.
Once the configuration has been established, MkDocs can be started in the root directory of the documentation (where the configuration file is located). For local use it is sufficient to execute the command `mkdocs serve`. This starts the integrated web server and the documentation can be called with the help of a browser by entering `http://localhost:8000` in the address line.
If the static website is to be generated on a server, only the command `mkdocs build` has to be executed from the main directory. The output location (the target directory) must be defined in the configuration file `mkdocs.yml`, otherwise the static output is generated in a subdirectory and would be stored in the git repository.
For the local work on the documentation, `mkdocs serve` is executed. Only when a `push` uploads the markdown documents to the server, the server executes the command `mkdocs build` with the help of a hook and generates the static web page in the configured target directory (usually the directory for the “real” web server for public access or internal approval).
MkDocs allows the use of themes — templates for the layout of the website. There are several ready-made themes available, you can download more and create your own. The adaptation and the use of an own “Corporate Design” is therefore possible.
The interaction of Markdown files, Git and MkDocs shows how online documentations can be created and updated. In addition to providing online documentation, there is often a need for PDF files or other formats for further processing. Here, too, further output formats can be created from the Markdown files.
Other output formats
To convert Markdown files to other formats a conversion program is required. The “Pandoc” application is currently considered one of the best and most comprehensive conversion tools. With this application it is possible to transfer files with a certain format into another format (e.g. Markdown files into PDF, Word or other formats).
Pandoc offers many parameters that can be found in the documentation of this program (https://pandoc.org). Therefore, only the call to convert a Markdown file to PDF is shown here as an example.
Note: The upper part describes how to configure the structure for MkDocs. Here several (often many) Markdown files are included. This must be considered during the conversion so that a single output file is created from all individual files.
To create a single output file with Pandoc, either all the individual files must be passed to Pandoc as parameters, or all the Markdown files must be merged into a single file and then passed to Pandoc. This process is not described here.
The easiest way to call Pandoc to convert Markdown files into a PDF file is to use the command:
pandoc source.md -o target.pdf
The same applies to other output formats, such as Word files. In addition, a large number of parameters are available which can be used to define the handling of the source and target files in more detail.
Since pandoc can also be executed via parameters and a simple system call, it is also suitable for calling it via a git hook. Modified documentation can thus be automatically converted into other formats after each change.
Pandoc also offers the option of using so-called templates, which define the layout of the output format more closely. By separating layout and content, it is possible for all converted documents to have the same layout.
Conclusion
The Markdown format has limits. Some of them can be extended by HTML or latex information. The Markdown format will certainly continue to evolve over the next few years.
By the simple use of Markdown and the combination with other tools extensive documentations can be provided, versioned and published in different formats. Whether this is sufficient for one’s own specialist area must be clarified individually.
All in all, it can be said that there are very good possibilities if you think outside the box and are willing to take a step back and concentrate on the essentials. Not every process step has to be covered by one system. Sometimes complicated issues can be solved better if you look at smaller parts separately and use a special tool for a partial task.
