Internal code structure — Deno under the hood series

This is a part of the series: Deno under the hood.

This series explores the nuts and bolts that are part of the internals of Deno. The purpose of this series is to provide an easy overview of the internals of Deno. None of the articles in the series are in a particular order, therefore any of the article can be read anytime.

For a high level overview of Deno’s architecture, visit the article here.

The scope of this article is Deno’s core runtime. We’ll cover standard library in some other article. Though Tokio & V8 are the foundations of Deno, they aren’t a part of the Deno’s code base, so aren’t considered here.

Code structure

The Deno’s core runtime is divided in four components:

CLI: CLI is the orchestrator & contains the main program of Deno. It also contains the implementation for the commands (aka tool chain), utility functions, etc. CLI uses core, extensions, and runtime to offer Deno’s functionality.

Core: Core provides a small set of core services like V8 bindings, OPS dispatch, loading of extensions, module handling, resource handling, V8 module loading/callbacks, etc.

Extensions: Each module in extensions implements a web API. It contains everything needed for a web API (high level to low level).

Runtime: The runtime has JS code for Deno’s runtime, implementation of all the low-level OPS, and workers.

Let’s go through the code structure inside each of the components. We’ll work in this order: core, runtime, extensions, and CLI.

Core

The core component of Deno offers a small set of core functionality. This is the module that also bridges between JS space & Rust space.

Here is the complete list of files present in core:

Let’s go through some important files:

• bindings.rs: This binds external functions into V8. These are the functions that are used to offer external services to V8 like opcall, encode, decode, serialize, deserialize, etc.
• core.js: This contains low-level JavaScript functions to dispatch sync/async ops, process results, print data on console, etc. This is the first JS code that gets loaded into V8.
• extensions.rs: This is the extension loader. It goes through the modules present in extensions directory, and loads them properly.
• modules.rs: This contains all the module loading functionality, like recursively processing a module, module loader, dynamic imports, etc. It uses runtime.rs to load modules into V8.
• ops.rs: This maintains a map of all the ops. The requests from JS space are routed to the appropriate op handler.
• resources.rs: This contains a resource table that maintains all the open resources. This includes built-in resources like stdin, stdout, stderr, and additional ones like open files, sockets, etc.
• runtime.rs: This provides the V8 interfacing by loading & instantiating modules into V8, resolving module callbacks, etc. This also contains the famous event loop.

Runtime

The runtime component of Deno contains all the JavaScript APIs offered by Deno along with their low-level implementations (ops). It also contains the implementation of main & web worker.

• JS: The JS part of the runtime contains all the JS code present in the Deno’s core runtime. This gets loaded into V8 before the application gets loaded. All the APIs present here are available to user without any imports. It includes all the core modules like files, fs, io, net, buffer, os, metrics, http, etc.

• OPS: The ops part of the runtime contains all the low-level Rust code that supports the high level APIs present in JS part of the runtime. Usually there is an ops file for each of the high-level JS modules. Additionally, it contains the ops required for main & web worker. The ops code is completely written in Rust.

• worker.rs & web_worker.rs: This contains the complete implementation of the main worker & web worker. The web worker is a bit different from the main worker.

Extensions

The extensions offer the standard web APIs. Each module in the extensions is self-contained i.e. it contains all parts like JS, interfaces, ops, etc. All the modules in the extension implement web specification.

Here is the structure of one of the extension: webstorage. There is a fixed set of files:

• 01_webstorage.js: This is the part of extension that runs in JS space
• lib.deno_webstorage.d.ts: The interface file (.d.ts) contains all the interface definitions for this module
• lib.rs: This contains the implementation of the ops required by this module

Here is another extension called ‘crypto’ (note that it follows the same structure):

CLI

CLI is the biggest component of Deno (of course, not including V8 that runs over 1M lines of code). The CLI is the starting point of Deno. This is where all the commands are implemented. As mentioned earlier, CLI uses core, runtime, and extensions to carry out the work (i.e. executing the application).

Here is the structure of CLI:

There are many source files & directories in CLI. Let’s go over some important ones:

• dts: The dts directory contains all the interface definitions (includes Deno specific and standard definitions)
• lsp: The lsp directory contains the implementation of the lsp command (language server protocol)
• tsc: The TSC directory contains the Microsoft’s TSC compiler (150K lines of code) & some utilities
• tools: The tools directory contains supporting implementation for tools related commands like doc, fmt, lint, repl, installer, and upgrade.
• build.rs: This loads the pre-defined modules into V8 & then create a snapshot
• colors.rs: To write with colors on the console
• disk_cache.rs: This is used to store the downloaded code on the disk
• file_fetcher.rs: This is used to fetch a file (local or remote)
• flags.rs: This parses the command-line options supported for each of the commands
• info.rs: This contains the supporting implementation of the deno info command
• main.rs: The main program of Deno. This also contains the basic implementation of the commands (of course supported by individual files)
• module_graph.rs: The module graph to maintain the modules and their dependencies. It’s a simple graph.
• module_loader.rs: This loads the main module & all its dependencies
• program_state.rs: The program state consists of deno directory, command line options, file fetcher, cache, etc.