Rendering Sphinx Documentation with React
This is another in what seems to be becoming a series of articles about mashing up different technologies. In most cases, I’m trying to demonstrate how to use TypeScript with some existing framework. But this article is a bit different. In this article, I demonstrate how easily you can use React components to statically render Sphinx documentation and even sprinkle in interactivity once the pages are loaded in the DOM.
TL;DR
With a pretty minimal setup, you can forget about Jinja and use React to render your Sphinx documentation. You get the benefits of JSX, CSS-in-JS, hot module reloading and even types when iterating on the layout and style of your documentation. The pages are fully elaborated as static HTML just as with Sphinx but they can be augmented with interactivity once loaded into the browser.
How It All Began
Back in 2014, I ran a successful Kickstarter to write my second book. At the time, I looked at several different documentation tools and chose Sphinx. For those who don’t know, Sphinx is a system for writing documentation in reStructured Text. The system is written in Python and it can transform properly organized reStructured Text documentation into HTML, PDF, ePub, etc. I chose Sphinx because I wanted to support all these different targets. I also had some pretty involved requirements like good support for typesetting math, ability to create extensions to custom render certain bits, etc. Sphinx was a great choice, and I was pleased with how the system worked.
However, one of my goals was to integrate interactivity into the web version of the book. What I ended up having to do was use Jinja and jQuery to achieve what I wanted. Jinja is the templating engine for Sphinx. I have used it for a couple of different projects in the past. But as of today, I would always choose JSX over Jinja if I had the choice (even more so when using TypeScript because I get static type checking of my templates). But at the time I wrote the book, I used Jinja to do as much templating (think server-side rendering) as I could get away with and jQuery to manipulate the DOM once the document was loaded.
Seeing the Light
Revisiting all this recently to deal with some lingering issues in the book, I found myself having to modify some of the Jinja code. Frankly, it stopped me dead in my tracks. It was a mess. Part of that was, admittedly, my fault. I was in a hurry to get the book out and so I didn’t organize things as well as I could have. But there is only so much I could have done in this approach because there is no real module system for Jinja (especially if using it with Sphinx for templating and theming) and it is stringly typed (think C preprocessor). The fact that I had some code written for statically generating HTML and other code that worked on the DOM also struck me as inconsistent.
I have had my eye on React based static site generators like Gatsby and react-static for some time. I even bumped into Kyle Mathews (developer of Gatsby) at React Europe 2017. I was pretty interested to try out some of these systems. Although I was under a bit of time pressure, sitting there looking at all that Jinja code there was a debate raging in my head. One part of my brain was saying “this could be a great chance to try out one of these static site generators!” while the other one was saying “Ain’t nobody got time for that!”. The compromise I arrived at was “Ok, spend just a little bit of time just to see how far we can get”.
I was surprised at how quickly I made progress and how much better the solution I ended up with was…
My Goals
The problem, as I saw it, with the current site was that it was this sloppy mess of frontend and backend rendering using older technologies. It wasn’t just that the technologies (and my knowledge of them) were fading it was also that it made it quite difficult to modify the site. So my goal was to see if there was a way to unify all this rendering in a consistent way using JSX. When I say unify, I mean that it would handle rendering the same across frontend and backend so I really only had to write React components and the static site generator would take care of any “hydration” that had to happen on the frontend side. I’d never done any server side rendering before and I was pretty blown away by how seamless it actually turned out…
Data
I chose to try out react-static although I suspect Gatsby would have been just as good a choice (and I may use that for a future project just to contrast and compare). In getting started, my first thought was “how am I going to get the content of the book into this thing?”. The static generators I’m familiar with are all pretty data agnostic. They don’t really care where the data comes from, whether it be GraphQL, headless CMSs, files, etc. Needless to say, none of them list Sphinx.
But it turns out that Sphinx has a special target called json. So I tried it. Sure enough, it spits out a bunch of JSON data in .fjson files (why the non-standard suffix…no idea). I was at first a bit disappointed to see that the .fjson files included raw HTML as properties of the JSON object. My concern was that too many upfront assumptions may have already been made here about how the site would be rendered. But in looking carefully, I found that they had actually chosen precisely the correct boundary here. They focused only on the structure and content of the book and nothing on the presentation (no styles, etc).
So then the question was how to import this stuff. In react-static, you have a configuration file called static.config.js where you define all the routes to your site. It turns out that Sphinx generates a file called index.fjson that includes all the pages in the documentation. So I just wrote a getRoutes method in static.config.js that looped over all pages and added a route for each one. Along the way, I had to take care that the URLs used by Sphinx to reference each page matched exactly the URLs I was giving them on the static site. Surprisingly, it was almost trivial to ensure that.
Rendering
Content
When you establish a route in react-static you also need to establish a component to render it and any props you want to pass to that component. The only prop I needed to include was the contents of the .fjson file. It contained everything I needed from the page content to navigational information (previous page, next page, parent pages). As a result, the React component necessary to render a basic page started off as simply:
(props) => (<div>
<h1>{props.title}</h1>
<div dangerouslySetInnerHTML={{ __html: props.body }}/>
</div>);Wow, that was a lot easier than I thought. Of course, that only includes the body of the page, not the navigation, breadcrumbs, etc. But all that stuff is in props so it is just a question of elaborating it all out in the JSX. With just this and a special root level route (i.e. /), I already had a complete site and all the hyperlinks worked. Damn!
With the content taken care of, I could then turn my attention to layout, styles, etc. Keep in mind that react-static supports hot module reloading. So as I changed the component for rendering the page, the browser instantly updated. This meant I could really quickly iterate on the layout, styles, etc.
HTML
The rendering above only covers the content on the page. A big part of this exercise is actually putting together the template for all the HTML (CSS, scripts, head metadata, etc). Fortunately, react-static supports that with a method they called Document. Again, it is a JSX template (specified in static.config.js) for how to render the entirety of the HTML. So I had to go to work including everything I needed there. But since I had an existing site, most of that was just copying in the stuff I already had and then running HTML to JSX on it.
Math
One wrinkle I faced was rendering of math. For the original site, I used MathJax. MathJax’s normal mode of operation is to wait until the page loads and then scan the page for inline equations and render them. I went this route with the original release of the book. But this tends to be a little slow and I was a bit nervous about the churn in CDN solutions for serving the MathJax assets. So I wanted to see if there was a way to deal with the typesetting of equations in the static site generator itself.
Fortunately, there is a tool called mathjax-node-page (which leverages mathjax-node) that allows me to pass it HTML and it returns to me HTML with all the equations replaced by inline SVG. As a result, I could also statically render all the math on the page. This meant that the page loads were optimized and I didn’t need to worry about loading all the MathJax stuff into the browser (no CDN, no self hosting, nothing). Another big improvement and almost trivial to integrate.
Dynamic Content
You may remember I originally talked about jQuery and DOM manipulation. It turns out there are lots of plots in the book. One of my original ideas was to make these plots interactive by allowing people to change the parameters were used as input to the associated simulations and see how the plot would change as they changed parameters (see bottom of this page for an example). But, this only applies to the web version of the book. The HTML rendering from Sphinx also supports the PDF and eBook versions of the book and I didn’t want to mess with that. So the interactivity had to be introduced only in the web site.
Previously, I just used jQuery to “rewrite” the DOM to add interactive widgets when the web page loaded. But this was tedious because I wasn’t using any kind of UI framework (e.g., Angular, React, etc). It was pure vanilla DOM + jQuery. So an open question was still how to incorporate these dynamic interactions. More specifically, could I use statically rendered React components in react-native and have them be interactive when reconstituted in the browser? In a word…Yes.
From react-static's point of view, all components are rehydrated in the browser. But mine had a twist. You see, I already had the page content generated from Sphinx as HTML in props.body. This HTML was in the form of a string, not JSX. That is where the plots themselves were. So I still needed to “rewrite” that HTML to inject my React components. But remember that the HTML was a string and was being written to the DOM with something like:
(props) => (<div>
<h1>{props.title}</h1>
<div dangerouslySetInnerHTML={{ __html: props.body }}/>
</div>);So where does my interactive React plotting component go?!? It belongs inside the contents of props.body. So what I ended up doing (and React seemed ok with this?!) was to make this into a proper class based component and add a componentDidMount method. The componentDidMount method only gets called when there is an actual DOM around (i.e., it isn’t invoked server side). My componentDidMount implementation used a reference to the <div> (see ref attribute) where my page HTML was stored and then found all the plots and switched them with React components using ReactDOM.render. It ended up looking something like this:
class PageView extends React.Component {
content = null;
componentDidMount() {
let figures = this.content.getElementsByClassName("plot");
figures.forEach((plot) => {
ReactDOM.render(<Plot .../>, someExistingDivInDOM);
});
}
render() {
let props = this.props;
return (
<div>
<h1>{props.title}</h1>
<div ref={(content) => this.content = content}
dangerouslySetInnerHTML={{ __html: props.body }}/>
</div>);
}
}In other words, I “dangerously” rendered the Sphinx generated HTML into a <div> and when that <div> was mounted into the DOM I then went through and replaced the existing DOM elements for the plots with interactive React components (inside a DOM that was already managed by React!). It seems a bit crazy and I suspect it would not work if PageView decided to re-render itself (since React is managing overlapping parts of the DOM). But fortunately, the page content never changes and React doesn’t even spit out so much as a warning about this craziness.
Other Wrinkles
TypeScript
I’m a big TypeScript user and I try to use it as much as possible for anything Javascript related (especially when it comes to JSX and the oh so sweet static checks it provides). I started my react-native project using the TypeScript template. Frankly, I didn’t really understand what they were doing there with aliases in tsconfig.json. But it did bundle ts-loader and some useful type definitions. So it was a reasonable start.
What is tricky about all this is that the static.config.js file is in JavaScript. This code is evaluated by react-static and there isn’t even a way to specify an alternative configuration file (as far as I could find). So if you want to use TypeScript to write your configuration, you’d really have to then transpile it and generate static.config.js from your TypeScript. That was a bit more trouble than I was willing to go to. Note that in static.config.js you define a bunch of methods. So you either write those in Javascript or transpile some TypeScript and import the transpiled code. Again, a bit of trouble.
Fortunately, the amount of JavaScript code you have to write for configuration is pretty small and it doesn’t change much. It is mostly concerned with finding and parsing the .fjson files from Sphinx and loading their contents up to be passed on to the components as props. It turns out that you can specify your components as strings not as actual React components. This is because those strings are then passed to webpack to import the components. Since webpack is actually using ts-loader, those strings can reference TypeScript modules. So the bottom line here is that while I wrote all the configuration code in Javascript (with some JSX for the Document method), I was able to write all my components in TypeScript. Hooray!
Static Assets
Another wrinkle is with the static assets generated by Sphinx. This includes things like images to be used in figures. Fortunately, this was also quite easy to handle in react-static because react-static provides a folder called public. Anything in the public folder will be served from the root of the site. Sphinx uses a similar approach. So all I had to do was add the _images, _sources and _static directories created during the json build in Sphinx to the public folder. In fact, I just used symbolic links so that they stayed in constant sync with each other.
Search
Personally, I’ve never gotten the search capabilities of Sphinx working. The compilation process generates an index that you can use for search (although it looks corrupted to me…but I haven’t looked that carefully). A next step here would be to leverage that data to create a React component that could provide interactive documentation search. I haven’t done it yet, but if the search index data is correctly provided by Sphinx, it should be very easy.
Current Status
I just did this work and I’m writing it up now before I forget it all when I move on to my next project. All the source for the book is hosted in a public GitHub repository, but the code I’m talking about here is loaded as a submodule from my book-generator repository. Note that src/setup is concerned mainly with translating the index.fjson data into the routes needed by static.config.js, src/containers are some simple functions that return JSX, src/sphinx contains type definitions for the Sphinx .fjson data and src/components is where all the real React components are stored.
The version of the book rendered using the method described in this article is currently in “beta” and is hosted at http://beta.book.xogeny.com (the old book is, at the time I wrote this, still the “current” release and can be found at http://book.xogeny.com although the former will hopefully soon replace latter). There are still some rough edges to smooth out, but as a demonstration of what is possible, it seems pretty reasonable.
Conclusion
Overall, I was quite impressed with how quickly I was able to get a react-static site running that rendered Sphinx data. I only started this effort a little over 24 hours ago and I’ve already converted the entire book site over. The basics were almost trivial and iterating on the basic approach was very fast thanks to hot module reloading. Furthermore, I’m really quite thrilled to have all the rendering done in a consistent way leveraging React, JSX and TypeScript.
I continue to be intrigued by how these tools can fully elaborate the content of the site as static HTML and then rehydrate it on the client side. My sense is that there are many use cases where it would make sense to go this route.
