Switching to reStructuredText and ReadTheDocs for GitHub Project Documentation

Chris Wahl · Posted on

Documentation is never something I’ve really cared much about writing. It’s tedious and often boring to craft, requires constant updates as things change, and, well … it’s just not fun. As a user, however, I love documentation!

I recently set about solving this conflict between creating documentation and reading documentation by browsing around open source projects that I respect to see what they are doing. This search bore fruit: by embedding most of the documentation directly into the code, and by using a fancy parser that was above and beyond what Markdown could do, I saw the potential for making better documentation that wasn’t quite so labor intensive.

This discovery lead to exploring the world of reStructuredText (RST) and ReadTheDocs. RST is a rather powerful markup language that can be parsed by ReadTheDocs to create some really fancy stuff, including automatic generation of a PDF, Epub, and HTML pages. By including RST files in my GitHub projects and creating an integration with ReadTheDocs, I’m able to make documentation a part of the project files. This is in contrast to using a Wiki or some other data collection system, which is a bit more constraining on options and portability.

Making this all work really boils down to three steps: learning how to write RST files, writing a bit of code to generate RST files for the project, and then making sure the RST files live in the correct location so that ReadTheDocs can use them for a build. Before you go on, here’s the results of my labor.

Writing reStructuredText (RST) Files

The first step in using RST is learning the syntax. Here are the three pages I tend to float around for reference:

I’ll admit that the format took me a little while to absorb. The syntax for links and other reference pointers were especially goofy at first blush. However, I now really enjoy how clean it all looks in both the raw edit mode and when it’s published. Not having hyperlinks directly in the text is actually quite crisp, and commonly used links can be re-used throughout the documentation.

To illustrate this, take a look at part of the index.rst file I’m using for the Rubrik PowerShell Module.

Welcome to the Rubrik PowerShell Module
========================
.. image:: https://ci.appveyor.com/api/projects/status/52cv3jshak2w7624?svg=true
   :target: https://ci.appveyor.com/project/chriswahl/powershell-module
.. image:: http://readthedocs.org/projects/powershell-module/badge/?version=latest
   :target: http://powershell-module.readthedocs.io/en/latest/?badge=latest
This is a community project that provides a Windows PowerShell module for managing and monitoring Rubrik's Converged Data Management fabric by way of published RESTful APIs. If you're looking to perform interactive automation, setting up scheduled tasks, leverage an orchestration engine, or need ad-hoc operations, this module is intended to be valuable to your needs. The code is open source, and `available on GitHub`_.
Below is a quick YouTube video that explains how to begin using the module. 
.. image:: http://i.imgur.com/MGHfunv.png
   :target: https://www.youtube.com/watch?v=XJ6IaVhYWWY
.. _available on GitHub: https://github.com/rubrikinc/PowerShell-Module

Notice the use of two periods to escape the text and send a command, such as the .. image:: or .. _available on GitHub: portions. The fact that I can reference a link by simply adding accent marks and an underscore anywhere in the text is snazzy. It certainly takes a little bit of getting used to when compared to Markdown, which uses the [text](link) style format, but is really clean looking and easy to spot when reading the raw file.

Also, GitHub can render the RST files natively, which is nice.

Programmatically Creating Documentation

As I eluded to earlier, the ability to dynamically create documentation files is really what I was after for most of the work. To do this, I decided to plant all of the detailed help information into the PowerShell functions that are part of the Module. This is a good idea in any scenario – the Get-Help cmdlet is extremely helpful (pun intended) and should be loaded with information. Therefore, why not just output the detailed help directly into RST files and use a bit of additional markup to craft fancy documentation?

The code to do this is small and simple. A PowerShell script named BuildDocs.ps1 is given the task of making the RST files. Let’s start by examining the code.

$verbs = (Get-Command -Module Rubrik |
  ForEach-Object -Process {
    $_.Name.Split('-')[0]
  } |
Select-Object -Unique)
 
foreach ($verb in $verbs)
{
  $data = @()  
  $data += "$verb Commands"
  $data += '========================='
  $data += ''
  $data += "This page contains details on **$verb** commands."
  $data += ''
  foreach ($help in (Get-Command -Module Rubrik | Where-Object -FilterScript {
        $_.name -match $verb
  }))
  {
    $data += $help.Name
    $data += '-------------------------'
    $data += ''
    $data += Get-Help -Name $help.name -Detailed
    $data += ''
  }
  $data | Out-File -FilePath "$PSScriptRoot\cmd_$($verb.ToLower()).rst" -Encoding utf8
}
  • L1-5: Orders the documentation by the PowerShell verb, such as Get, Set, and New. It starts by grabbing all of the functions in the module and seeing how many unique verbs are being used.
  • L7: Walk through each verb type.
  • L9-14: A $data variable will hold all of the page content. The top of the RST page header is created to fancy things up a bit.
  • L15-23: Using the Get-Help cmdlet with the -Detailed parameter, the code creates a second level header with the function’s name and then outputs the help data.
  • L26-27: Results are sent from $data to an RST file using the verb’s name. The UTF8 encoding parameter is required for proper parsing by ReadTheDocs.

Parsing RST Files with ReadTheDocs

The last step is to save all of the RST files into a folder named docs within the project. Note that the case is sensitive – using Docs won’t work. Place an index.rst file in the root of the directory, which also describes the table of contents (TOC) for Sphinx, which is used by ReadTheDocs. Here’s another sample from the Rubrik PowerShell Module:

.. toctree::
   :maxdepth: 2
   :hidden:
   :caption: User Documentation
   requirements
   installation
   getting_started
   support
   contribution
   licensing  
   faq

All of the document names below the .. toctree:: section correspond to RST files. Therefore, by including the word requirements I must also have a file named requirements.rst in the docs folder. The TOC name for the requirements page is then derived from the title within requirements.rst. It makes the update process quite painless – a simple update in one place trickles down elsewhere with ease. Every time a commit is made to the “master” branch, the docs rebuild themselves.

In order to figure most of this out, I ended up just seeing how ReadTheDocs created their own documentation and reverse engineering the efforts. I still have further tuning I want to do, and I need to get more documentation online for the Vester project – you can see a preview here. But, I’m quite happy with how simple the workflow has become and how much less work is required to maintain help for the PowerShell functions.