Buffers in Deno

Buffers are one of the commonly used data structures in programming. A buffer is an in-memory storage of any data. In Deno, buffers are variable sized i.e. they can grow as needed. This makes the buffers attractive for unknown/variable sized data handling. Deno’s buffers expand themselves as more data is added to them. Deno’s buffers implements read and write functions, thereby making them compatible with Reader and Writer interfaces.

In this article, we’ll go over the basics of Deno’s buffers, some useful functions, and using them for/as Reader and Writer .

Basic functions

From release 2.0, Deno would move the buffers to the standard library. Till now, they’re part of the Deno’s core runtime. As the movement is near, we’ll go over the implementation of buffer present in the standard library.

To use the buffer, import it from the io module:

import { Buffer } from "https://deno.land/std/io/mod.ts";

Deno’s buffer has two important parts:

• Bytes: This is the raw data held by the buffer
• Offset: This is the read offset that moves after every read operation (like a cursor)

The buffer keeps track of how much data has been read using the offset variable. The next read operation would always continue where the last one finished.

Buffer write operations always append new data at the end of the buffer.

Initialization

A buffer can be created/initialized in two ways:

• Empty: An empty buffer with no storage (the storage can expand later)
• ArrayBuffer: A buffer that’s copies the size and data from the given ArrayBuffer (like Uint8Array)

const b=new Buffer();
const b2=new Buffer(new ArrayBuffer(100));
const b3=new Buffer(new Uint8Array(100).fill(65));

Capacity

The capacity accessor function returns the total size of the buffer. This is different from offset .

const b=new Buffer();
b.capacity;
//0
const b2=new Buffer(new ArrayBuffer(100));
b2.capacity;
//100

Grow

A buffer can grow in two ways:

• Automatic: Grows as data gets added to the buffer
• Explicit: Increase buffer size by given number

We’ll see automatic increase later. Let’s see explicit grow :

const b=new Buffer();
b.grow(1000);
//b.capacity = 1000
const b=new Buffer(new ArrayBuffer(1000));
b.grow(1000);
//b.capacity = 3000

In the first case, the buffer grew to the expected size of 1000. However, in the second example, the buffer grew more than the expected size of 2000 (it grew to 3000). How did that happen? The reason is that, grow adds what is requested plus some extra bytes using this logic:

2 * currentCapacity + requestedSize

In the second example, the new size gets calculated as: 2 * 1000 + 1000 => 3000.

Also, it’s important to note that there is a maximum size to which Deno’s buffer can grow. The maximum size is 4294967294 bytes or ~4.2G. A request to grow the buffer beyond maximum size gets converted to maximum size.

empty

The empty function return true if there are any bytes in the buffer that hasn’t been read. Contrary to it’s name, empty function doesn’t mean that the buffer is empty (or capacity=0).

const b=new Buffer();
const b2=new Buffer(new ArrayBuffer(1000));
b.empty();
//true
b2.empty();
//false

The buffer b2 is empty because it was initialized with 1000 bytes and nothing hasn’t been read.

length

The length accessor function is similar to empty , except for the fact that, length returns the number of bytes present in the unread portion of the buffer, while empty returns true or false.

const b=new Buffer();
const b2=new Buffer(new ArrayBuffer(1000));
b.length;
//0
b2.length;
//1000

truncate

The truncate function is used to truncate the unread portion of the buffer in two ways:

• Full truncation: If input is 0, the complete unread portion is discarded

const b=new Buffer(new ArrayBuffer(1000));
b.length;
//1000
b.truncate(0);
b.length;
//0

• Some truncation: If input is greater than 0, the unread portion is truncated after the given number of bytes

const b=new Buffer(new ArrayBuffer(1000));
b.length;
//1000
b.truncate(300);
b.length;
//700

After truncating at 300, there are still 700 more bytes in the unread portion of the buffer.

reset

The reset function empties the buffer and moves the offset to 0. The capacity of the buffer is still maintained i.e. it won’t shrink the buffer.

const b=new Buffer(new ArrayBuffer(1000));
b.reset();
b.capacity; 
/1000
b.length;
//0

readFrom/readFromSync

This is one of the most useful functions to fill a buffer. The readFrom function takes a Reader as input and copies all the data from the Reader into the buffer. Any resource implementing Reader interface can be used.

• Fill buffer from a file

const file=await Deno.open('/var/tmp/a.txt');
const b=new Buffer();
await b.readFrom(file);
b.capacity;
//12
b.length;
//12

• Fill buffer with HTTP request body

for await(const r of serve(':5000')) {
    const b=new Buffer();
    await b.readFrom(r.body);
}
//--
curl http://localhost:5000 -d '1234567890'
//b.capacity = 10
//b.length = 10

• Fill buffer from two sources (file, request body)

for await(const r of serve(':5000')) {
    const b=new Buffer();
    await b.readFrom(r.body);
    //b.capacity = 10
    //b.length = 10
    const file=await Deno.open('/var/tmp/a.txt');    
    await b.readFrom(file);
    //b.capacity = 32
    //b.length = 32
}
//-
cat /var/tmp/a.txt
hello world

write/writeSync

The write function appends bytes to the buffer. If there isn’t enough space, the buffer would grow. The data is always written at the end of the buffer.

const b=new Buffer();
const d=new Uint8Array(100).fill(1);
await b.write(d);
//b.capacity = 100
//b.length = 100
const d2=new Uint8Array(50).fill(1);
await b.write(d2);
//b.capacity = 250
//b.length = 150

read/readSync

The read function an Uint8Array as input and fills up to its length by reading bytes from the buffer. At the end of the read operation, offset is moved by the number of bytes read. The values/output of offset , length , emtpy would change after every call to the read function.

const b=new Buffer(new Uint8Array(100).fill(1));
//b.capacity = 100
//b.length = 100
const t1=new Uint8Array(10);
await b.read(t1);
//b.capacity = 100
//b.length = 90
const t2=new Uint8Array(50);
await b.read(t2);
//b.capacity = 100
//b.length = 40
const t3=new Uint8Array(100);
await b.read(t3);
//b.capacity = 100
//b.length = 0

In the first step, 10 bytes are read from the buffer. There are 90 unread bytes. In the second step, 50 bytes are read from the buffer. There are 40 unread bytes. In the final step, only 40 bytes are read from the buffer as it reached EOF.

bytes

The bytes function is used to get a copy of or the actual unread portion of the buffer.

const b=new Buffer(new Uint8Array(100).fill(1));
//b.length = 100
const t1=new Uint8Array(10);
await b.read(t1);
//b.length = 90
const t2=b.bytes();
//b.length = 90
//t2.length = 90

Buffer as Reader and Writer

Buffers implement read and write interfaces similar to Reader and Writer , therefore they can be used in all places where Readers and Writers are expected.

• Write buffer on console (buffer as reader)

const b=new Buffer(new Uint8Array(10).fill(65));
await Deno.copy(b, Deno.stdout);
//AAAAAAAAAA

• Send buffer in HTTP response (buffer as writer)

for await(const r of serve(':5000')) {
    const b=new Buffer(new Uint8Array(10).fill(65));
    r.respond({status: 200, body: b});
}
//--
curl http://localhost:5000
AAAAAAAAAA

Example

Now that we’ve gone through all the basic functions, let’s see a full example (hypothetical of course!):

//Initialize with 100 bytes
const b=new Buffer(new Uint8Array(100).fill(1));
//b.capacity =  100 , b.length =  100
//Add 1337 bytes from /var/tmp/a.1 to buffer
await b.readFrom(await Deno.open('/var/tmp/a.1'));
//b.capacity =  1537 , b.length =  1437
//Read 1000 bytes from buffer
const t1=new Uint8Array(1000);
await b.read(t1);
//b.capacity =  1537 , b.length =  437
//Add 13337 bytes from /var/tmp/b.1 to buffer
await b.readFrom(await Deno.open('/var/tmp/b.1'));
//b.capacity =  16411 , b.length =  13774
//Read 5000 bytes from buffer
const t2=new Uint8Array(5000);
await b.read(t2);
//b.capacity =  16411 , b.length =  8774
//Get the unread portion of the buffer
const t3=b.bytes();
//b.capacity =  16411 , b.length =  8774 , t3.length =  8774
b.reset();
//b.capacity =  16411 , b.length =  0

This story is a part of the exclusive medium publication on Deno: Deno World.