JS Buffers matter! Sometimes.
Its hard to believe how young the field of computer science is, when we have already come so far! But as far as we’ve come, computers still rely on binary for storing and retrieving information. As I’ve been looking into this process a little more I ran into the concept of buffers. As it turns out, buffers provide a specific purpose in working with streams of binary. I had a difficult time understanding why they important in JavaScript though, because it is such a high level language, and so far I haven’t run into the need for buffers in my experience with JavaScript. Before jumping into JS buffers, lets take a deeper look at the purpose of buffers in general.
Buffers are essentially storage units for binary code, often used to temporarily store the binary before or after it is transferred. This is actually really handy because all data on our computers is sent and stored as binary! Without buffers, sending data would consist of breaking down some data into binary and then immediately sending that binary. The receiver would then start receiving binary chunks of code and immediately start trying to use it. This process would cause some issues that buffers help prevent. The actual process of transferring the data is more related to streams. Streams and buffers are somewhat related, but NOT the same thing. Click here to more about streams!
As mentioned above, buffers will provide a temporary storage for the binary code. Specifically, buffers will allow for binary code to congregate until all the binary has finished transferring, or at least until enough binary has transferred to allow a smooth transition to the receiver. This is crucial because we don’t know the rate at which binary will be transferred! Depending on the type of data and intended use of the data, we may want a consistent transfer of binary. For example, when streaming a movie no one wants to watch a choppy movie that skips a lot. We may also need all of the data before we start processing it, and don’t want to risk data corruption just because the last little bits of binary (pun intended) didn’t finish transferring. Buffers will help mediate both of these scenarios!
Lets break it down with a real word scenario. Think back to your most recent field trip. You get to your school and wait for everyone else to show up before the buses leave (take for granted there are a lot of people, and several buses are needed). For this example lets say that as soon as enough people arrive to fill one bus, that bus can leave. When that bus arrives at its destination, the students must wait there for the remaining buses to arrive. In this example the loading area around the bus would be considered a buffer, allowing people (representing bits of information) to congregate until enough is actually ready to be transported without fear of losing someone. Similarly you will wait at a designated area upon arrival until everyone else arrives. This will ensure a bus is not forgotten. This may not exactly be how school trips happen, but it is a good representation of the services buffers provide with loading areas representing buffers, people representing bits of data, and buses representing binary chunks.
As for buffers in JS, before es6 there were none. JS did not really support the manipulation of raw binary code at all until typed arrays were introduced in es6. Now we have access to TypedArrays, which are array like objects that offer a view into a buffer. Interestingly enough, buffers by themselves cannot directly be viewed or manipulated in JS as demonstrated on lines 8-10 in the code snippet above. To access them we create a view called bufferView, which is an instance of a TypedArray. The specific type we are instantiating is a Uint8Array, seen on line 13. This is saying we expect the array to contain UTF-8 characters. UTF-8 stands for Uniform Transformation Format that uses between one and four 8-bit blocks to encode each character. This is just one specific way to encode data and has actually been the predominant method of encoding data to send on the web since 2009. In bufferView we can add acceptable character codes into our buffer and we can decode them to see our actual data. If we wanted to create a buffer from some data instead of creating and filling in an empty buffer, we can make use of JS’s TextEncoder. If we already have a buffer to work on, we just need to create a view on that buffer using a typed array of the desired encoding.
Node.js allows us to manipulate buffers as well, and actually adds some abstraction for us. After a quick npm install buffer , we can require buffer as seen on line 20 in the snippet above (though as the snippet says, you may not have to). We can then create a new instance of a buffer, which immediately has several useful methods, including a toString() method which will convert the content of a buffer to a usable string. The way the buffer was instantiated on line 23 is just one of three main ways to create a new buffer. In this case we gave the data to be encoded, then the desired encoding method. We could also create a new buffer passing in binary chunks. Node.js buffers use utf-8 standard, because it is actually a subclass of JS’s Uint8Array, but it is given a different prototype to add functionality.
We have covered the basics of implementing and using buffers in JS and Node.js, but don’t go rushing into manipulating binary just yet! Buffers can be useful in JS if you really need to manipulate raw binary. That being said, JS already handles most if not all of the necessary binary manipulation behind the scenes. In fact, so does Node.js. There are not many cases you absolutely need to manually set up buffers in JS, but they are still useful. Sometimes storing, accessing, and transferring data using buffers — instead of converting it to non-binary data, then converting it back to binary — can actually save quite a lot of time in large operations. In addition to this, buffers allow for a little more flexibility by facilitating lower level computation. Buffers can also be useful when you are expecting binary chunks, as in Node.js servers. While you can typically convert the chunks to a string, it could save time and computation to allow it to stay in binary until it is needed in another form.
Perhaps you’re thinking: ‘If it saves time, why don’t we always use buffers?’ For starters, creating and using buffers adds complexity to your code that JS intentionally abstracts away for you. For small computations, the difference won’t be all that noticeable anyways. Besides this you need to be aware of buffer overflow. If you aren’t careful, it can be very easy to try to fill your buffer with more data than it can actually hold, allowing data to spill into other memory than what you have set aside for the buffer. If this sounds really scary, its because it is. Buffer overflow is actually incredibly dangerous for your code and often leads to easy access for hackers.
We have covered the general use of buffers, the basics of implementing them in Node.js and JS, and the pros and cons of manually implementing buffers. Take this information and explore buffers for yourself! With a little practice they could really impact your method of working with code.
