The future of Scaladoc

Scala 3, like all modern languages, needs a tool for generating documentation. Scaladoc, the documentation engine used with Scala 2, is stable and powerful, but it cannot be used ‘as is’ in the Scala 3 world. The old Scaladoc is built on top of the Scala 2 compiler; essentially, you can think of it as a compiler that produces .html instead of .class files.

Scala 3 brings in a new compiler, so we decided to build a brand new Scaladoc as a standalone tool that uses artefacts produced by the compiler to generate documentation. This does not mean that we took the decision to replace the old Scaladoc (or its architecture) lightly. There were multiple reasons for this, perhaps the most important of which was that Scaladoc was part of the Scala 2 compiler, which means there is a heavy coupling between the two. The new approach is also more principled, but we had no real issues documenting all the definitions, types and features of Scala 3.

This article will discuss the new Scaladoc’s future development areas rather than provide a rationale for why it looks the way it does. We can do that in another article if there is enough interest.

Scala has a vibrant community that has created many innovative and high-quality libraries and tools, such as Akka and Scala Steward. Documentation tools are no exception. Our first step in planning the future of the new Scaladoc was to identify features that, when integrated into the doctool itself, will provide more benefits than using sbt plugins or other external tools.

Unifying the features described below with Scaladoc means creating top-notch documentation for any project will be as simple as setting a few bits in the buildtool and writing the content itself. It will also mean that all these features will interact with and enhance each other.

Unified documentation (sbt-unidoc)

The Scala compiler (and, by extension, the old Scaladoc) operates on the level of compilation units: a set of source files with a common classpath and a set of compiler options, which are compiled together into a single artefact. Sometimes libraries are split into many compilation units even though users should consume given library as a single compilation unit. Why is that? Performance, macro definitions, and project history are among the many reasons. As you have probably guessed, in these cases the old Scaladoc will generate multiple documentation instances, which makes the documentation inconvenient and misleading.

Akka was the first project in the Scala community that solved this problem. The solution was a special tool for merging APIs, which Eugene Yokota (the hero of sbt world) later transformed into the sbt-unidoc plugin. Sbt-unidoc can combine Scala and Java documentation from multiple sbt projects.

The new Scaladoc is a stand-alone tool that consumes .tasty files: fully typed ASTs serialized into binary format. Simply by providing extra tasty files or excluding parts of the compiler output, this approach gives us more control in terms of what we will document. Yes, at this very moment, users of the new Scaladoc can already generate unified documentation for multiple projects by merely adding all generated .tasty files and inputs and properly configuring the classpath (`doc/sources` and `doc/libraryClasspath` settings in sbt).

In the future, we plan to provide a set of settings and tasks to make configuration and maintenance as straightforward as possible.

In Scala 3, .tasty files are present by default in published jars, leading us to another possible feature: we can use this information to generate a single aggregated piece of documentation for all the dependencies in a project. Having a single, aggregated piece of documentation that is always up to date may be helpful, especially when dealing with monorepos or bigger applications.

Static Site (microsite, mdoc)

Most users start learning a new library by looking for its documentation site. The Scala community currently uses various tools to build such sites: sbt-microsite, mdoc, Docusaurus or Jekyll (mostly in Github Pages). In order to replace dottydoc in building the dotty.epfl.ch website, support for building static sites was the main requirement for the new Scaladoc from day 1. We invite anyone to give it a try (you can find more in our documentation).

For now, the new Scaladoc supports only the basic features that are known from Jekyll (we are using similar formats and conventions), extended with some Scala-specific things like automatic linking to API pages, linking to source files (including an ‘edit’ option) or basic variable injection (version, project name, revision etc.)

We can see several benefits of having static site generation in a doctool:

• Unified styles. Using a single tool to generate both the API and static documentation will blend both of them nicely in terms of configuration (versions etc.), styles, or look and feel. Of course, we do not want to enforce any style; we want to provide sensible defaults.
• Verified linking between APIs and static documentation
• Static documentation will be a part of `-javadoc.jar` artefacts, so users will be able to access documentation that was shipped with the version used in their projects.
• The generation of a static site will benefit from existing Scaladoc features (like source mapping)
• Errors/warnings that occur during documentation generation will be reported using reporting from the compiler. This will ensure proper integration with build tools (essential for snippets; see below)

Snippet validation and results (mdoc)

Good documentation requires examples, and without proper tooling these snippets can get rapidly outdated. This is where mdoc (or tut in the past) comes into play by compiling examples against the latest code.

Integration of such functionality into Scaladoc will provide some new benefits. First, we will be able to validate snippets from static docs and docstrings. Furthermore, we can analyse compilation artefacts to enhance the API by, e.g., providing links to all examples where a given class is used, or by enhancing the HTML that snippets with links to API pages.

Runnable snippets: (Scastie / Scala fiddle)

Making code snippets interactive will make them even more useful. The ability to play with the code within the documentation directly in the browser should make crude prototyping much easier and in the long term reduce the frustration of spending a significant amount of time only to realise that a given library does not fill one’s needs.

Let us know if your projects or libraries can benefit from any of the described features.

You can reach us out using Twitter, gitter or GitHub issues.