Reading line-by-line in Deno

Purpose

Time and again, a lot of applications face the issue of reading data line-by-line i.e. get a line, process it, and repeat till it’s done. Some of the common use cases are reading a file line-by-line, reading a large string line-by-line, reading CSV data line-by-line, reading HTTP response line-by-line, etc.

There are two ways to read line-by-line in Deno:

• Reader’s readLines: This is a generator function that returns AsyncIterator which can be looped upon
• TextProtoReader’s readLine: This is a normal async function that returns a promise for the next line

Reader’s readLines would be discussed in another article where we’ll go over Deno buffers in detail.

Deno’s standard library comes with a hidden utility called TextProtoReader that provides functions to read data line-by-line. The usage is simple, but the process is a bit lengthy. The data needs to be converted to the right format for TextProtoReader to be able to process it. The unique point of TextProtoReader is that it returns a promise containing the next line, and therefore, is more suitable for non-looped reading.

The TextProtoReader is already in use by the HTTP server, WebSocket server, File server, etc.

Classes and functions

To use TextProtoReader, import it from Deno’s standard library:

import { TextProtoReader } from "https://deno.land/std/textproto/mod.ts"

Like TextEncoder and TextDecoder, TextProtoReader is a class, so an object needs to be created before using it.

const tp=new TextProtoReader(r: BufReader);

TextProtoReader is fixed to a data source from the start. The data source can’t be changed in the same object. To use a new data source, a new TextProtoReader needs to be created. The data source must always be a buffer reader. This makes TextProtoReader generic enough to handle strings, files, stream, etc. Regardless of the source or type of data, it must be converted to buffer reader or BufReader before attaching it to TextProtoReader.

There are several functions in TextProtoReader, but we’ll only focus on the readline function.

readLine(): Promise<string | null>

• readline: This is the only function required to read data line-by-line. This is an async function. It would keep returning a promise containing a string till there is nothing more to read.

Pipeline

As mentioned earlier, the TextProtoReader is attached to the data source at the time of creation. The data sources could be strings, files, streams, etc. Regardless of the type of data source, TextProtoReader expects a generic buf reader. All the data sources need to be converted into buf reader before attaching them to the TextProtoReader.
This is similar to a pipeline that takes a data source as input and produces a TextProtoReader as output. The pipeline would roughly go through the following stages:

• Stage 1: Convert the source to an object that implements Reader
• Stage 2: Convert the Reader to BufReader
• Stage 3: Convert the BufReader to TextProtoReader

In short:

Source -> Reader -> BufReader -> TextProtoReader

The conversion from source to reader varies by the data source. Some data sources might already be implementing the reader interface, while others won’t. Stage 1 is specific to the data source. Stages 2 and 3 are generic.

Sources to Reader

Let’s see go over some common data sources and see how they can be converted to the reader.

String

A string can be converted to a reader using the StringReader:

import { StringReader } from "https://deno.land/std/io/readers.ts"
const str='A\nB\nC\n';
const sr=new StringReader(str);

sr is a Reader object and can be given as input to the BufReader.

File

A file, when opened using Deno.open, returns an object that already implements the reader interface. Therefore, for files, there is no need to use another reader.

await Deno.open("/var/tmp/a.log", {read: true});

The output of Deno.open can be given to BufReader.

Streams

Streams are the toughest among strings, files, and streams. Understanding streams, their interfaces, readers, etc. is a candidate for a separate discussion. As the purpose of this article is to show how to read line-by-line, we’ll use a ready readable stream that comes out of the fetch API response.

Deno’s fetch API returns a response object which contains a body that is of type ReadableStream. The response body can be fed into the pipeline to get a TextProtoReader.

import { readerFromStreamReader } from "https://deno.land/std/io/streams.ts"
const fetchRes=await fetch("https://google.com");
const streamReader=readerFromStreamReader(fetchRes.body!.getReader());
  //streamReader can be given to the next stage in pipeline i.e. BufReader
}

The response body is a ReadableStream that provides a function getReader which could be used to read the stream. This stream reader needs to be converted to an object that implements the reader interface. To make this conversion, readerFromStreamReader is used.

We’ve seen how data sources can be converted to readers, the next stage of the pipeline is to give them to buf reader.

As mentioned at the start, Reader’s readLines could be used at this point to get an iterator that can be used to loop over the lines. This is good enough for some use cases.

Reader to BufReader

This step is quite simple. Just give the reader to BufReader. Here are the three cases:

//STRING
import { BufReader } from "https://deno.land/std/io/bufio.ts"
import { StringReader } from "https://deno.land/std/io/readers.ts"
const str='A\nB\nC\n';
const sr=new StringReader(str);
new BufReader(sr);
//FILE
import { BufReader } from "https://deno.land/std/io/bufio.ts"
new BufReader(await Deno.open("/var/tmp/a.log", {read: true}));
//STREAM
import { readerFromStreamReader } from "https://deno.land/std/io/streams.ts"
const fetchRes=await fetch("https://google.com");
const streamReader=readerFromStreamReader(fetchRes.body!.getReader());
new BufReader(streamReader);

BufReader to TextProtoReader

The final step is to convert BufReader to TextProtoReader. This is generic i.e. not related to any data source.

new TextProtoReader(new BufReader(/*Reader from the source*/));

That’s all! The TextProtoReader is ready for use! It certainly looks like a lot of steps, but that isn’t the case. Especially file and string reading is quite easy. The only thing remaining is to use readline till the source is done.

Examples

Let’s start with a string example and read it line-by-line:

import { TextProtoReader } from "https://deno.land/std/textproto/mod.ts"
import { BufReader } from "https://deno.land/std/io/bufio.ts"
import { StringReader } from "https://deno.land/std/io/readers.ts";
const str='A\nB\nC\n';
const tp=new TextProtoReader(new BufReader(new StringReader(str)));
await tp.readLine();
//A
await tp.readLine();
//B
await tp.readLine();
//C

In the example above, the lines are being read one-by-one. They can be processed independently. There is no loop. The TextProtoReader is attached to the source, so the readLine keeps track of where the next reading point is.

Next, let’s see an example of reading a file line-by-line:

import { TextProtoReader } from "https://deno.land/std/textproto/mod.ts"
import { BufReader } from "https://deno.land/std/io/bufio.ts"
const file='/var/tmp/a.log';
const tp=new TextProtoReader(new BufReader(await Deno.open(file, {read: true})));
//read first three lines only
console.log(await tp.readLine());
//DEBUG JS - args []
//ignore this line
console.log(await tp.readLine());
//DEBUG TS - ">>> exec start" {"rootNames":["https://deno.land/x/doze@1.0/mod.ts","file:///Users/mayankc/Work/source/deno-vs-nodejs/delayworker.ts","https://deno.land/x/doze@1.0/doze.ts"]}
//do something with this line
console.log(await tp.readLine());
//DEBUG TS - {"allowJs":true,"esModuleInterop":true,"experimentalDecorators":true,"incremental":true,"isolatedModules":true,"lib":["deno.window"],"module":"esnext","strict":true,"target":"esnext","tsBuildInfoFile":"deno:///.tsbuildinfo","emitDecoratorMetadata":false,"jsx":"react","inlineSourceMap":true,"outDir":"deno://","removeComments":true}
//do something different with this line
//No need to read more

It is also possible to read the lines in a loop till the readLine returns null. However, that would be the same as using Reader’s readLines function.

Next, let’s see the example of line-by-line reading from a stream:

import { TextProtoReader } from "https://deno.land/std/textproto/mod.ts"
import { BufReader } from "https://deno.land/std/io/bufio.ts"
import { readerFromStreamReader } from "https://deno.land/std/io/streams.ts"
const fetchRes=await fetch("https://abc.com");
const tp=new TextProtoReader(new BufReader(readerFromStreamReader(fetchRes.body!.getReader())));
await tp.readLine();
//<!doctype html>
await tp.readLine();
//<html lang="en">
await tp.readLine();
//<head>

Finally, here is the example of reading line-by-line in a loop:

import { TextProtoReader } from "https://deno.land/std/textproto/mod.ts"
import { BufReader } from "https://deno.land/std/io/bufio.ts"
import { readerFromStreamReader } from "https://deno.land/std/io/streams.ts"
const fetchRes=await fetch("https://raw.githubusercontent.com/mayankchoubey/deno-doze/main/doze.ts");
const tp=new TextProtoReader(new BufReader(readerFromStreamReader(fetchRes.body!.getReader())));
let line;
while(line=await tp.readLine())
    console.log(line);

This story is a part of the exclusive medium publication on Deno: Deno World.