Jackson: A Growing User Base Presents New Challenges

Oct 20, 2020 · 7 min read

Jackson is a mature and feature-rich open source project that we use, support, and contribute to here at Indeed. In my previous post, I introduced Jackson’s core competency as a JSON library for Java. I went on to describe the additional data formats, data types, and JVM languages Jackson supports. In this post, I will present challenges resulting from Jackson’s expansion, in my view, as Jackson’s creator and primary maintainer. I’ll also share our plans to address these challenges as a community.

A running river surrounded by lush tress in Cape Flattery in Washington
“The other side of Cape Flattery” by Tatu Saloranta

Growing pains

Because of its ever-expanding functionality and user base, Jackson is experiencing growing pains and could use help with:

  • Improving the documentation
  • Enhancing the project’s website and branding
  • Managing and prioritizing large-scale changes and features
  • Refining the details of reported issues
  • Testing the compatibility of new Jackson versions against downstream dependencies

Documentation’s structure and organization

The problem is not so much a lack of documentation per se. A lot of content exists already:

3rd-party tutorials

Jackson project repos


Jackson also has a Javadoc reference that documents most classes of interest. However, Jackson is a vast library. The Javadoc reference can seem overwhelming, especially to new users. And the Javadoc reference does not include usage examples.

Our first priority is creating How-To guides that walk library consumers through the most typical use cases, such as:

  • How to convert the f.ex JSON structure to match Java objects.
  • How to format other data types such as XML.
  • How to use Jackson within frameworks, such as Spring/Spring Boot, DropWizard, JAX-RS/Jersey, Lombok, and Immutables.

We also want to improve the Javadoc reference. Some classes and methods have no descriptions, while descriptions for others are incomplete. In addition to the auto-generated Javadoc reference, we publish manually created wiki pages that detail the most used features and objects. We’d like to find ways to auto-update these wiki pages.

To implement these modifications, we need the following documentation structure, tooling, and process changes:

  • A super structure for adding inline content and links to external content
  • Access permissions for contributors to add and correct documentation
  • A documentation feedback mechanism to help focus improvement and maintenance efforts
  • Documentation templates to make it easier to add information

Project website and branding

For Hacktoberfest 2020, we created an issue for a Jackson logo design. Examples we like:

Managing large-scale changes

The most valuable type of contribution for the Jackson project has historically been user bug reports filed as GitHub issues. We get a dozen issues each week, leading to fixes and quality improvements. Without these reports, Jackson would not be half as good as it is. The new feature requests we receive regularly are another valuable source of improvements. New feature requests also help us prioritize new development work.

Better management of larger changes and features is an area for improvement. These efforts take a long time to execute and benefit from careful planning and collaboration. While it’s possible to handle bigger changes under a single issue, this can get unwieldy quickly.

The Jackson project could use help with setting the priority of new feature requests.

  • Which issues should we focus on improving?
  • Can we gauge the consensus of the whole community instead of relying on a small number of active and vocal participants? This helps to avoid “squeaky wheel problems” getting priority.
  • What’s the best way to obtain early feedback on API and implementation plans? Using existing mailing lists, issue trackers, and chat all have their challenges.

To address the need for more community feedback, I introduced the idea of “mini-specifications” called Jackson STrategic Enhancement Plan: JSTEP. Similar to KIPs in Kafka, JSTEPs are intended to foster collaboration. So far this has had limited success, partly due to the limitations of the tool used: GitHub wiki pages do not have the same feature set as GitHub issues or Google documents.

I also started keeping a personal but public todo or work-in-progress (WIP) list — Jackson WIP — as a GitHub wiki page. I wanted a low-maintenance mechanism for me to track and share my own near-term plans.

Refining reported issues, collaboratively

Basic issue triage has become time consuming and includes the following steps:

  • Validating the reported issues.
  • Determining if the reported issues resulted from documentation that is missing or unclear.
  • Asking for more information.

The triage time takes away from the limited time we have to work on functionality and documentation. It can also slow down responses to issue reports, which in turn leads to a poor reporter experience.

We would like to find ways to train and include volunteer issue triagers. They can help with refining issues in a timely manner and improving user engagement by:

  • Adding labels
  • Asking questions to get more information
  • Adding findings of their own
  • Getting module and domain experts engaged where applicable

Starting with jackson-databind issues, we improved the workflow and refined GitHub issues by creating:

  • Issue templates to help users submit better issue reports
  • The “to-evaluate” label — set automatically for new issues
  • The “most-wanted” label — which indicates issues that have consistently come up on mailing lists or up-voted using the GitHub thumbs-up emoticon

If these changes are successful, we’ll propagate them to other project repos. Similarly, we can make other incremental changes along these lines.

Compatibility testing of new versions with dependencies

One challenge is the difference between the development lifecycles of Jackson and the libraries and frameworks that use Jackson. Existing libraries and frameworks depend on the stable Jackson minor version and test everything only against that version. Version 2.11 is the current stable version. They report any issues they find. We may fix the issues in a patch release, like 2.11.1.

Simultaneously, the Jackson project is developing a new minor version, 2.12, which will be released and published as version 2.12.0. Working on versions 2.11.1 and 2.12 concurrently should not be an issue. We use the standard semantic versioning scheme for the Jackson public API, which should guard against creating new issues.

In practice, standard semantic versioning cannot protect against all actual issues. Semantic versioning is used for Jackson’s public API, but internal APIs across core components do not have the same level of backwards compatibility. While Jackson maintainers consider the compatibility between the published and documented API, users tend to rely on the API’s actual observed behavior rather than its intended behavior. Sometimes what maintainers consider bugs or unexpected behaviors, users consider the behavior an actual feature or working as expected.

That said, we can catch and address such issues during the development cycle of the new version if we test downstream systems using the under-development SNAPSHOT version of Jackson. At the time of writing this article, the current version of Spring Boot ships with a dependency on Jackson 2.11.2, the latest patch version. We can create an alternative set of Spring Boot unit and integration tests that instead rely on Jackson 2.12.0-SNAPSHOT. The snapshot version is published regularly, and the Jackson project provides automatic snapshot builds.

Such cross-project integration tests would benefit all involved projects. This prevents the release of some (or perhaps most) regression issues, reducing the need for patch versions. Over time it should also significantly improve compatibility across Jackson versions, closing the gap between expected and actual behavior of the API.

How to get involved

About the author

Cross-posted on Indeed Engineering Blog.

Indeed Engineering

Stories from Indeed Engineering