When You Can’t Read Your Own Writing
Metamorphic File Formats
I once worked on a development team contracted with the Air Force Research Laboratory to design and deploy a wind-tunnel diagnostic system that could measure two-dimensional, on-body pressure profiles. At the heart of the system was a 16-bit, scientific-grade CCD camera that captured images of the pressure-sensitive coating we had sprayed on the model. To verify the results calculated by our latest-generation PC-based system, we wanted to process our data using the Air Force’s UNIX-based image-analysis software with the six-figure price tag. As we expected, the software package did not recognize the native format of our image files. We were writing the control software for our system so it seemed a simple matter to hack out an image-export routine. Listed among the translators for various CCD manufacturers’ file formats recognized by the UNIX-based software was an importer for files written to the Flexible Image Transport System (FITS) specification. FITS is the standard data interchange and archival format of the worldwide astronomy community — the same community that spurred the development of CCD camera technology.
A brief examination of the FITS specification revealed that it only supported pixel intensities up to 16-bit, signed integers. FITS had been adopted when 14-bit CCD cameras were the rage and did not anticipate the storage of the 16-bit, unsigned integers that our CCD camera acquired. There was a suggested work-around in the FITS documentation but the import routine of the UNIX-based software did not recognize it. We eventually developed a fix, but it felt as if the complexity of the file format problem had eclipsed the difficulty of the original measurement.
Whatis.com maintains a document titled “Every File Format in the World” that currently lists the names of thousands of file formats along with their file extensions. Compound this number with release version, fixes, and quarterly updates and the actual number of different file formats is difficult to quantitate. In the late 1980s, when laboratory instrumentation was beginning to sprout connected personal computer data stations, several organizations recognized how intractable the “automated laboratory of the future” would be without standard methods for reading and archiving acquired data. One such standard initiative was undertaken by the American Society for Testing and Materials (ASTM), known as ANDI — ANalytical Data Interchange, for the handling of chromatographic data. ANDI is an extension of the NetCDF (Network Common Data Form) specification maintained by the Unidata Program Center in Boulder, Colorado. In addition to chromatograms, this format affords the storage of instrument settings, sample characteristics, and operator information. About the same time, the International Union of Pure and Applied Chemistry (IUPAC) sanctioned the standard file format developed by the Joint Committee on Atomic and Molecular Physical Data (JCAMP) known as JCAMP-DX. This standard can be used to record spectroscopic data including IR, Raman, UV/VIS, NMR, mass spectra, X-Ray powder patterns, thermograms, and chemical structure information in addition to chromatograms.
Although it is one matter to develop a standard data file format, it is quite another to convince instrument manufacturers that it is in their best interest to implement the standard. A natural consequence of standardization is that it is much easier and more convenient for a laboratory to switch from one vendor to another. In the past, the balance of the vendor’s effort was placed on the initial sale and customers could be retained by “holding their data hostage” in proprietary file formats. After the adoption of file-format standards, manufacturers are required to follow the sale with exemplary service and support to maintain and grow their customer base.
The data-acquisition community is not alone in its quest for file-format standards. The Collaborative Electronic Notebook Systems Association (CENSA) has members representing the software, high-tech, consumer products, chemical, pharmaceutical, biotech, healthcare, and government sectors. Beyond the storage of individual data files, CENSA is striving to replace paper laboratory notebooks with all-digital electronic notebook systems. These systems would not only obviate the “cut-and-tape” method of entering data into the notebook but would create online documents that are searchable, sharable, and less subject to errors made during transcription and re-keying of values. Sounding more like science fiction than fact, the technology exists to develop collaborative electronic notebooks capable of placing orders for the materials needed for an experiment, searching the scientific literature for related research, calculating reaction rates and structure configurations, receiving analysis results back from the QA/QC laboratory, and date-time stamping the work for the patent department — all automatically. In graduate school my research advisor jokingly suggested that I add a feature to all of my data acquisition code wherein pressing the F1 key would analyze the data and e-mail the finished manuscript to the journal editor. Perhaps we are heading in that direction.
This material originally appeared as a Contributed Editorial in Scientific Computing and Instrumentation 17:6 May 2000, pg. 14.
William L. Weaver is an Associate Professor in the Department of Integrated Science, Business, and Technology at La Salle University in Philadelphia, PA USA. He holds a B.S. Degree with Double Majors in Chemistry and Physics and earned his Ph.D. in Analytical Chemistry with expertise in Ultrafast LASER Spectroscopy. He teaches, writes, and speaks on the application of Systems Thinking to the development of New Products and Innovation.
