Zephyr Badges: The Open Matrix (OMX) Standard
If you think of standardization as the best that you know today, but which is to be improved tomorrow; you get somewhere. -Henry Ford
- Co-authored by Jeff Newman and Dan Florian
The Zephyr Software Badgers have been hard at work, burning the midnight oil in the infamous badge mines (BTW — did you know that midnight oil fumes are toxic?), and we’re ready to share with you the first badge we’ve awarded: the Open Matrix (OMX) Standard.
Transport Models Love Matrix Data
Computational transport modeling tools rely on I/O to exchange data about socioeconomics, land-use, transport networks, accessibility measures and many other mobility-related data. In practice, it is often necessary for modelers to exchange information between different modeling tools — and in transport planning arguably the most commonly required data exchange involves matrix data. Matrices are generally used to store travel demand, network accessibilities, costs or skims or other “level of service” characteristics representing various aspects of a travel experience between some set of places, most commonly between travel analysis zones (TAZs). This is perhaps best exemplified when interfacing a transport network model, which produces network accessibilities, with a travel demand model which produces, well, travel demand. But if you are reading this article you probably already knew that, and could probably come up with plenty of other reasons you’d need to exchange matrix data.
A problem arises in heterogeneous model workflows when a model developer wants to share matrix data between distinct modeling tools, each of which may not expose a common interface. In this case, model developers could end up spending quite a bit of time brokering out the differences. At least, this was the case until the emergence of the Open Matrix (OMX) standard.
The OpenMatrix (OMX) Standard
The goal of OMX is simple — to create an industry standard matrix format for storing and transferring matrix data, using an open, common format for matrix data across models and software to make model development and application easier. The specification for OMX version 0.2 codifies this standard into a relatively compact set of rules and conventions. It is easy to understand what is required, what is permitted, and what is explicitly not permitted in an OMX file, making the development of an implementation to read and write OMX files from any given tool relatively straightforward.
It is also useful to note that the OMX standard is supported by more than just a serializable file format definition; implementations of the standard are also available as open-source APIs for Python, R, C#, C++, Java and… FreePascal, if you are that type. In particular, the Python API serves as the reference implementation and includes recently introduced validation tools (more on that below). Also because the openmatrix library for Python makes it possible to access OMX directly with numpy, it is relatively easy for Python workflows to integrate directly with OMX in-memory without requiring more expensive file-based I/O.
In addition to proposing a common interface to matrix data, OMX also introduces facilities to address some of the most traditionally ambiguous and problematic aspects of matrix data exchange, including:
- TAZ numbering — in most travel models, the zones representing the rows/columns of a matrix are not contiguous; OMX proposes a way to transfer TAZ numbering along with the matrix values
- Shape — OMX provides explicit specification to express and discover the shape of the data with support for non-square matrices if required
- Null values — OMX provides an optional declaration for representing N/A or Null values, a convention which may vary substantially between modeling tools and application
- Compression — Matrix data may be voluminous, and users are free to choose optional file compression using the ubiquitous zlib filters for their application depending on whether they prefer fast read/write or compressed storage
Since its inception the OMX format has been adopted by planning agencies, consultancies, university researchers and students as well as software developers who have collectively introduced support for the standard in a growing list of modeling tools. In this sense, OMX has become a great example of industry-wide contribution and collaboration, in part because a wide variety of stakeholders find a value proposition to the work.
Will the Real OMX Please Stand Up?
So great, interchange problems solved, right? Well, not so fast. There were still known instances in the community, especially from multi-agency project work, where OMX interchange wasn’t working as expected. The questions were similar — “Help, my OMX data didn’t transfer properly between tool A and tool B, even though tool A and tool B both claim OMX support.” Evidently there are at least three possibilities: either Tool A has a bug, or Tool B has a bug, or OMX itself has a bug, or perhaps some combination of these. In practice, when working remote and cross-tool, it can be difficult for an end-user to be sure they are receiving clear and accurate assertions from all parties. Indeed, it is possible that the makers of Tool A or Tool B may have not even known that there was a problem with their tools, as there was no well established independent method to test that purported OMX files were indeed OMX-compliant.
As part of the software badging process, Zephyr’s software badging project management group identified this lack of an independent external validation process as an important oversight, and something that would contribute to the badging goals to promote tools that are useful, that foster collaboration, and that are easy to use. We asked the originators of the OMX standard to build such a tool, and we provided some support to actually build and test it.
The result? A new OMX Validator tool which allows independent verification of any OMX data by end-users to diagnose standards-compliance issues on their own. The validator is developed and provided by the OMX project team and tested by Zephyr, so now you can take any OMX data and get an authoritative, neutral-party validation report on your own. If you are concerned about the OMX support in any tool you are using, you can simply take sample OMX data, import/export it through a tool, and then validate the end result. You’ll get a report that describes both required and optional aspects of OMX compliance.
Validating OMX Data
The easiest way to validate your OMX is to use the online OMX Validator notebook as described below. This avoids the need to download and install your own development environment. (If you do have a local Python environment, however, you can install OMX using `pip install openmatrix` and accomplish similar using the omx-validate script. You can find these and other details in the OMX Python API Documentation.)
Now, say you have an OMX file to validate. All you need to do is click on the OMX Validator link and you will be taken to a Google Colab page. From there you can open the notebook on free computing resources provided by Google, as long as you are willing to upload the OMX file you want to validate. You don’t even need a Google account to do so — simple hit the “Open in playground” button:
The page will reload, and a server will be initialized and connected to the validator notebook after a few seconds. This server, and any data you copy to it, will disappear when you close your browser window or leave it inactive for a few minutes. (If you want a more permanent validation process, try installing the tools locally on your own computer instead of relying on free cloud computing!)
Now you can upload your OMX data. Below the Table of Contents at the top-left of the page, select the Folder icon.
You’ll see a “sample_data” folder, and you can simply drag-and-drop your OMX file next to this directory, or click the Upload button to open a File browse dialog, where you can select your OMX file and click Open.
You will get a pop-up warning you that this data will get recycled when the session ends. Dismiss it, wait for your upload to complete, and confirm that the file is visible, as shown here.
Next, we need to install the OMX libraries. Find the section titled Validator Functions and run the cell by clicking the Run (play icon) button on the left. When complete, it should look like this:
Now the moment we’ve been waiting for, let’s validate the OMX. In order to do this you’ll need to replace the filename example.omx with the path to the file you uploaded. Mouse over your uploaded OMX file in the Files browser, click the menu options and select Copy path.
Then, in the Validate section of the notebook, paste the path into the cell that runs the OMX validation as follows.
Now run the cell by clicking the play button. You’ll see a detailed validation reply, like this, which tells you whether your OMX data or application is standards-compliant, and which contains important information to help diagnose any issues you may be facing.
After the detailed report there is also a simplified scorecard which you may prefer if you find the details a bit verbose.
A Zephyr Badge for OMX
Now that the OpenMatrix standard is independently verifiable, those of us working on the Zephyr badging committee agree that the standard is simultaneously i) useful to the community, ii) contributes to a common problem space in a manner that encourages progress, and, finally, iii) is easy and clear to use across a variety of practical applications. As such, we are glad to award the OMX standard an official Zephyr software badge.
Please let us know whether you’ve had a chance to put OMX to work in your own models and applications, whether you agree with us that the tools are useful to the broader transport planning community, or with any other related comments you may have.
And the Next Badge goes to…
Now that we’ve handed out our first badge, we’re ready to hand out more. But we need your help to find the next tools that are badge-worthy. If you’ve got an idea, submit it and we’ll take a look!
The Zephyr Foundation’s mission is to advance rigorous transportation and land use decision-making for the public good by advocating for and supporting improved travel analysis and facilitating its implementation.
