Shared Storage Part 1 — File Constructs
When we think of the foundations of General Compute we often think of the store program processors like x86 or ARM. And while this truly enables General Compute, it’s not what really made it practical. For that, we need to look at files and file systems, and in this post, we will explore how they accomplish this.
I am not sure why it has been so difficult for me to write this post about something that seems so simple in my head, but this has to be my 20th time in as many days to do so. I think it is because explaining the concepts of a file construct is easy, but explaining why we would do this is extremely complicated. So please bear with me as I give it one more try.
Storing data can be as simple in concept as copying what’s in active system memory to persistent storage like a hard drive or flash drive. But how this is done is extremely complex. For example, some code in a program knows that some byte in active memory represents 256 shades of red that, along with two other bytes for shades of green and blue, represent the color of a single pixel of an image. We could just store those bytes as is in persistent storage, but what happens when the computer restarts and active memory is cleared? How do we know what those bytes represent or even where it is?
The where it is part is even more complex as unused bytes in the present storage device, let’s call it a flash drive, are all zeros or randomly set. This means that even if only one image is stored on our flash drive, we don’t know which or how many of the bytes on the flash drive are for our image and which are just random. Addressing these issues is the point of a file construct.
This is also a good place for me to mention that a file construct is more of a concept of how data is arranged for storage and is different from how it is stored, such as a file or object. A file construct provides a framework for what the bytes are, how many there are, and a meaningful ordering. Let’s use our RGB image example to understand this and look like a GIF file.
The first bytes in most files are called magic bytes, representing how the data is organized (the file type). In the case of a GIF file, these magic bytes are 47 49 46 38 37 61. They tell the computer that the file is compressed using LZW compression and has four parts separated by specific byte sequences. The first part is the header which gives information like how many pixels wide and tall the image is. The second part is the image itself, where every three bytes is a pixel to be loaded from the top left to the bottom right in the grid defined in the first part. The third part is how the image should be handled; the last part is the trailer and represents the end of the file. With this information, not only can the original program “load” the image, but so can any other application running on any other system which has knowledge of the file type.
Sometimes, this is a double or recursive process, such as JSON, CSV, and XML. The file construct is actually a plain old text file in these specific cases. The application must first interpret the bytes of the file as ASCII characters, such as the number 49 representing the character “1”. On another pass of the data, it then has to recognize the string value of “1” as the integer value 1. Another pass is needed to then organize it into some form of data formats like key and value.
If you are like most of us, you are probably asking yourself why would we do this if it is so complex and the answer is fairly simple. There are two main reasons why this is a good idea. The first is that it gives us some context to the data. Any program can open the file up and, either by being told or using the magic word at the begging of the file have an idea of where the break points in the long blob of bytes between values, how it should be interpreted, and what it should do with them. Think back on our image file example. We know every set of three bytes is an RGB pixel color.
The other, in the case of formats like JSON offers more than one layer. As a text file, any program can open and properly marshal the bytes to show the encoded bytes, like 49 means display the character “1” even if it can’t recognize it as a dataset. Besides making it human-readable and editable, as text, it can pass through firewalls, proxies, and other network elements easily.
The most significant drawback to storing data in file structures is what is called the Serialisation / Deserialization process, in which data is marshaled into and out of active memory and the file’s format. This process takes a significant amount of computing and is the actual cause of what even many programmers incorrectly think of as I/O speed. This delay can be compounded by the fact that, in most cases, a drive can only access one file at a time, and only one CPU core can be used to marshal the file back into the data.
Sidestepping the Serialisation / Deserialization process is the main driver behind projects like Apache Arrow, ProtoBuf, and many others. However, because of the nature of persistent shared storage, there is no easy way to avoid this process. Even “raw memory dumps” using something called memmap are, in fact, marshaled into a file format.
In my next posts, I will discuss how these files work with block storage and how they fit with file systems and object stores. So stay tuned for those posts. In the meantime, if you have any questions or comments, please feel free to let me know.
