Zephyr Badges: The Open Matrix (OMX) Standard

Jeff Newman
Jun 6, 2020 · 8 min read

If you think of standardization as the best that you know today, but which is to be improved tomorrow; you get somewhere. -Henry Ford

The Zephyr Software Badgers have been hard at work, burning the midnight oil in the infamous badge mines (BTW — did you know that midnight oil fumes are toxic?), and we’re ready to share with you the first badge we’ve awarded: the Open Matrix (OMX) Standard.

I said “a matrix file”, not “The Matrix”.

Transport Models Love Matrix Data

A problem arises in heterogeneous model workflows when a model developer wants to share matrix data between distinct modeling tools, each of which may not expose a common interface. In this case, model developers could end up spending quite a bit of time brokering out the differences. At least, this was the case until the emergence of the Open Matrix (OMX) standard.

The OpenMatrix (OMX) Standard

It is also useful to note that the OMX standard is supported by more than just a serializable file format definition; implementations of the standard are also available as open-source APIs for Python, R, C#, C++, Java and… FreePascal, if you are that type. In particular, the Python API serves as the reference implementation and includes recently introduced validation tools (more on that below). Also because the openmatrix library for Python makes it possible to access OMX directly with numpy, it is relatively easy for Python workflows to integrate directly with OMX in-memory without requiring more expensive file-based I/O.

In addition to proposing a common interface to matrix data, OMX also introduces facilities to address some of the most traditionally ambiguous and problematic aspects of matrix data exchange, including:

  • TAZ numbering — in most travel models, the zones representing the rows/columns of a matrix are not contiguous; OMX proposes a way to transfer TAZ numbering along with the matrix values
  • Shape — OMX provides explicit specification to express and discover the shape of the data with support for non-square matrices if required
  • Null values — OMX provides an optional declaration for representing N/A or Null values, a convention which may vary substantially between modeling tools and application
  • Compression — Matrix data may be voluminous, and users are free to choose optional file compression using the ubiquitous zlib filters for their application depending on whether they prefer fast read/write or compressed storage

Since its inception the OMX format has been adopted by planning agencies, consultancies, university researchers and students as well as software developers who have collectively introduced support for the standard in a growing list of modeling tools. In this sense, OMX has become a great example of industry-wide contribution and collaboration, in part because a wide variety of stakeholders find a value proposition to the work.

Will the Real OMX Please Stand Up?

As part of the software badging process, Zephyr’s software badging project management group identified this lack of an independent external validation process as an important oversight, and something that would contribute to the badging goals to promote tools that are useful, that foster collaboration, and that are easy to use. We asked the originators of the OMX standard to build such a tool, and we provided some support to actually build and test it.

The result? A new OMX Validator tool which allows independent verification of any OMX data by end-users to diagnose standards-compliance issues on their own. The validator is developed and provided by the OMX project team and tested by Zephyr, so now you can take any OMX data and get an authoritative, neutral-party validation report on your own. If you are concerned about the OMX support in any tool you are using, you can simply take sample OMX data, import/export it through a tool, and then validate the end result. You’ll get a report that describes both required and optional aspects of OMX compliance.

Validating OMX Data

Now, say you have an OMX file to validate. All you need to do is click on the OMX Validator link and you will be taken to a Google Colab page. From there you can open the notebook on free computing resources provided by Google, as long as you are willing to upload the OMX file you want to validate. You don’t even need a Google account to do so — simple hit the “Open in playground” button:

The page will reload, and a server will be initialized and connected to the validator notebook after a few seconds. This server, and any data you copy to it, will disappear when you close your browser window or leave it inactive for a few minutes. (If you want a more permanent validation process, try installing the tools locally on your own computer instead of relying on free cloud computing!)

Now you can upload your OMX data. Below the Table of Contents at the top-left of the page, select the Folder icon.

You’ll see a “sample_data” folder, and you can simply drag-and-drop your OMX file next to this directory, or click the Upload button to open a File browse dialog, where you can select your OMX file and click Open.

You will get a pop-up warning you that this data will get recycled when the session ends. Dismiss it, wait for your upload to complete, and confirm that the file is visible, as shown here.

Next, we need to install the OMX libraries. Find the section titled Validator Functions and run the cell by clicking the Run (play icon) button on the left. When complete, it should look like this:

Now the moment we’ve been waiting for, let’s validate the OMX. In order to do this you’ll need to replace the filename example.omx with the path to the file you uploaded. Mouse over your uploaded OMX file in the Files browser, click the menu options and select Copy path.

Then, in the Validate section of the notebook, paste the path into the cell that runs the OMX validation as follows.

Now run the cell by clicking the play button. You’ll see a detailed validation reply, like this, which tells you whether your OMX data or application is standards-compliant, and which contains important information to help diagnose any issues you may be facing.

After the detailed report there is also a simplified scorecard which you may prefer if you find the details a bit verbose.

A Zephyr Badge for OMX

Look ma! It’s a shiny new badge!

Please let us know whether you’ve had a chance to put OMX to work in your own models and applications, whether you agree with us that the tools are useful to the broader transport planning community, or with any other related comments you may have.

And the Next Badge goes to…

The Zephyr Foundation’s mission is to advance rigorous transportation and land use decision-making for the public good by advocating for and supporting improved travel analysis and facilitating its implementation.

zephyrfoundation

Commentary insights on making travel analysis useful to…