The JS runtimes
Published in

The JS runtimes

A beginner’s guide to streams in Deno

Introduction

Processing large amount of data

A simple test

const file = await Deno.readTextFile("./readings100M.txt");
const numReadings = file.split(",").length;
console.log(`Total readings found=${numReadings}`);
> time deno run --allow-read app.ts 
Total readings found=20971521
real 0m1.994s
user 0m2.423s
sys 0m0.572s
Physical footprint (peak):  641.1M

Streams to the rescue

Types of streams

ReadableStream

WritableStream

TransformStream

Stream interfaces

Stream pipes

PipeTo

PipeThrough

Use cases of streams

Save HTTP request body in a file

import { serve } from "https://deno.land/std/http/mod.ts";const reqHandler = async (req: Request) => {
const destFile = await Deno.open(`./uploadedFile`, {
create: true,
write: true,
truncate: true,
});
await req.body?.pipeTo(destFile.writable);
return new Response(null, { status: 201 });
};
serve(reqHandler, { port: 8082 });
> deno run --allow-write=./  --allow-net=:8082 app.ts
Listening on http://localhost:8082/
> ls -l readings100M.txt
-rw-r--r-- 1 mayankc staff 104857600 Apr 6 2021 readings100M.txt
> curl http://localhost:8082 --data-binary "@./readings100M.txt" -v
> POST / HTTP/1.1
> Content-Length: 104857600
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< HTTP/1.1 201 Created

Serve a file

import { serve } from "https://deno.land/std/http/mod.ts";const reqHandler = async (req: Request) => {
const fileName = `./${new URL(req.url).pathname}`;
const srcFile = await Deno.open(fileName);
return new Response(srcFile.readable);
};
serve(reqHandler, { port: 8082 });
> deno run --allow-read=./  --allow-net=:8082 app.ts 
Listening on http://localhost:8082/
> cat testdata/sample.txt
Learning Deno Is Fun!
> curl http://localhost:8082/testdata/sample.txt
Learning Deno Is Fun!

Zip a file

const srcFile = await Deno.open("./testdata/sample.pdf");
const dstFile = await Deno.open("./sample.gz", {
create: true,
write: true,
truncate: true,
});
srcFile.readable
.pipeThrough(new CompressionStream("gzip"))
.pipeTo(dstFile.writable);
> deno run --allow-read=./ --allow-write=./ app.ts> ls -ltr ./testdata/sample.pdf
-rw-r--r--@ 1 mayankc staff 69273 Apr 3 2021 ./testdata/sample.pdf
> ls -ltr sample.gz
-rw-r--r-- 1 mayankc staff 64602 May 29 22:42 sample.gz

Save output of a child process in a file

const child = Deno.spawnChild("./someScript.sh", {
args: [
"THIS SHOULD BE ON STDOUT",
"THIS SHOULD BE ON STDERR",
],
});
const dstFileOut = await Deno.open("./childOutput", {
create: true,
write: true,
});
const dstFileErr = await Deno.open("./childErr",
{ create: true, write: true });
child.stdout.pipeTo(dstFileOut.writable);
child.stderr.pipeTo(dstFileErr.writable);
> deno run --allow-read=./ --allow-write=./ --allow-run --unstable app.ts> cat someScript.sh 
echo "Hello world to $1"
echo "Hello world to $2" >&2
> cat childOutput
Hello world to THIS SHOULD BE ON STDOUT
> cat childErr
Hello world to THIS SHOULD BE ON STDERR

Unzip & process HTTP request line-by-line

import { serve } from "https://deno.land/std/http/mod.ts";
import { TextLineStream } from "https://deno.land/std/streams/mod.ts";
const reqHandler = async (req: Request) => {
if (!req.body) {
return new Response(null, { status: 415 });
}
const lines = req.body
.pipeThrough(new DecompressionStream("gzip"))
.pipeThrough(new TextDecoderStream())
.pipeThrough(new TextLineStream());
let numWords = 0;
for await (const line of lines) {
numWords += line.split(" ").length;
}
return new Response("Total words:" + numWords);
};
serve(reqHandler, { port: 8082 });
> curl http://localhost:8082 --data-binary "@./500K.txt.gz"
Total words:75830
import { DelimiterStream } from "https://deno.land/std/streams/mod.ts";const delim = new TextEncoder().encode(",");
const srcFile = await Deno.open("./readings100M.txt");
const readings = srcFile.readable
.pipeThrough(new DelimiterStream(delim));
let numReadings = 0;
for await (const reading of readings) {
numReadings++;
}
console.log(`Total readings found=${numReadings}`);
> time deno run --allow-read=./ app.ts 
Total readings found=20971521
real 0m21.813s
user 0m21.433s
sys 0m0.277s
Physical footprint (peak):  111.7M
> time deno run --allow-read=./ app.ts 
Check file:///Users/mayankc/Work/source/denoExamples/app.ts
Total readings found=200000001
real 3m11.084s
user 3m10.007s
sys 0m2.164s
// --Physical footprint (peak): 158.7M

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store