V8 JavaScript Engine: Compiling with GN and Ninja

Published in

Compilers

12 min readAug 30, 2020

I’m a compiler enthusiast, who has been learning how the V8 JavaScript Engine works. Of course, the best way to learn something is to write about it, so that’s why I’m sharing my experiences here. I hope this might be interesting to others too.

This first blog post is an overview of how V8 is compiled. As you can see from the V8 source code repository, the V8 Engine is mostly written in C++, requiring source code to be compiled into executable files. This should be no surprise, given that V8’s primary purpose is fast compilation and execution of JavaScript programs.

I’ll be discussing three main topics related to compiling the V8 executables:

The gm.py wrapper script, providing a convenient approach to compile V8 from source, and for invoking the test suites.
The GN meta-build system (invoked by gm.py) taking an easy-to-read description of the software components, then auto-generating a machine-readable build description suitable for the Ninja build tool.
Finally, the Ninja build tool uses that same machine-readable build description to analyze inter-file dependencies and invoke the relevant compilers.

The earlier diagram (at the top of this blog post) illustrates the overall flow of tool invocation, and shows which files are read, generated, and invoked.

Let’s examine each step in detail. If you’re new to this type of compilation process, I’ll put in a shameless plug for this book on Software Build Systems. It’s been almost ten years since I wrote that book, but the underlying concepts remain the same.

The gm.py Script

The first time you compile V8, you should use the recommended gm.py script to fully compile all the object files, libraries, and executables.

$ ./tools/dev/gm.py x64.release.check

This is described in the V8 documentation as a convenience script because it’s a one-step solution for all the steps you need to get started. It takes about 20 minutes to run to completion (on my MacBook). Here’s what it’s doing:

1. Creating and Configuring the Build Output Directories

Using the best practice that object and executable files should be stored separately from the source code, the gm.py script creates the v8/out/x64.release directory. In this example, we’ve asked for V8 to be compiled for the x64.release target (Intel x86 64-bit for release images), although if you also want to compile for different targets (such as x64.debug or arm64.debug), separate directories would be created for those.

This step also generates the args.gn file in the v8/out/x64.release directory, specifying the build options for this configuration (most notable are the is_debug and target_cpu options).

is_component_build = false
is_debug = false
target_cpu = "x64"
use_goma = false
goma_dir = "None"
v8_enable_backtrace = true
v8_enable_disassembler = true
v8_enable_object_print = true
v8_enable_verify_heap = true

2. Auto-Generating N`inja` files from the `GN` Build Specification

The next step in the build process is for gm.py to invoke the GN tool to translate the human-readable BUILD.gn file into lower-level files for the Ninja tool (with .ninja suffix).

The BUILD.gn file contains easy-to-read directives specifying the content of each build target. In the following example, the d8 executable is constructed from a small number of C++ source files, linked together with additional libraries that contain the core JavaScript engine.

v8_executable("d8") {
  sources = [
    "src/d8/async-hooks-wrapper.cc",
    "src/d8/async-hooks-wrapper.h",
    "src/d8/d8-console.cc",
    ...
    "src/d8/d8.cc",
    "src/d8/d8.h",
  ]  ...  deps = [
    ":v8",
    ":v8_libbase",
    ":v8_libplatform",
    ...
  ]
}

Later in this blog post, there’ll be more detail about this file format. For now, let’s look at what happens when the gn gen command generates all the .ninja files from the hand-written BUILD.gn file.

$ gn gen out/x64.release

This results in a collection of roughly 100 .ninja files in the out/x64.release directory. Each .ninja file corresponds to one of the build targets described in the BUILD.gn file.

./toolchain.ninja
./build.ninja
./obj/d8.ninja
./obj/v8_libbase.ninja
./obj/v8_simple_wasm_compile_fuzzer.ninja
./obj/v8_libplatform.ninja
./obj/v8_simple_multi_return_fuzzer.ninja
...
./obj/test/unittests/cppgc_unittests_sources.ninja
./obj/test/unittests/unittests_sources.ninja
./obj/test/wasm-api-tests/wasm_api_tests.ninja
./obj/test/common_test_headers.ninja
./obj/test/cctest/generate-bytecode-expectations.ninja
./obj/test/cctest/cctest_sources.ninja
./obj/test/cctest/cctest_headers.ninja
./obj/test/cctest/cctest.ninja

As an example, here’s the content of the d8.ninja file. At first glance, this output is quite similar to an old-style Makefile — that is, not very readable!.

defines = -D_LIBCPP_HAS_NO_ALIGNED_ALLOCATION -DCR_XCODE_VERSION=1160 -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D_FORTIFY_SOURCE=2 -D_LIB
CPP_ABI_UNSTABLE -D_LIBCPP_DISABLE_VISIBILITY_ANNOTATIONS -D_LIBCXXABI_DISABLE_VISIBILITY_ANNOTATIONS -D_LIBCPP_ENABLE_NODISCARD -DCR_LIBCXX_REVISION=375504 ...include_dirs = -I../.. -Igen -I../.. -I../../include -Igen -I../../include -Igen/include -I../../third_party/icu/source/common -I../../third_party/icu/source/i18n -I../../include
cflags = -fno-strict-aliasing -fstack-protector -fcolor-diagnostics -fmerge-all-constants -fcrash-diagnostics-dir=../../tools/clang/crashreports -mllvm -instcombine-lower-dbg-declare=0 -fcomplete-member-pointers -arch x86_64 -Wno-builtin-macro-redefined ...build obj/d8/async-hooks-wrapper.o: cxx ../../src/d8/async-hooks-wrapper.cc || obj/d8.inputdeps.stamp
build obj/d8/d8-console.o: cxx ../../src/d8/d8-console.cc || obj/d8.inputdeps.stamp
build obj/d8/d8-js.o: cxx ../../src/d8/d8-js.cc || obj/d8.inputdeps.stamp
build obj/d8/d8-platforms.o: cxx ../../src/d8/d8-platforms.cc || obj/d8.inputdeps.stamp
build obj/d8/d8.o: cxx ../../src/d8/d8.cc || obj/d8.inputdeps.stamp
build obj/d8/d8-posix.o: cxx ../../src/d8/d8-posix.cc || obj/d8.inputdeps.stamp
...

Now that we have all the .ninja files, we can start to compile the source code.

3. Using the Ninja Build Tool to Compile the Objects and Executables

The next step in the build process is for gm.py to invoke the C++ compiler (amongst other tools). This is done by invoking the autoninja command, which itself is a wrapper for the ninja command.

$ autoninja -C out/x64.release d8

This command reads the relevant .ninja files, determines which object files are missing (or out of date), then invokes the C++ compiler to create them. This process is familiar to anyone who has used the Make build tool (or similar).

After roughly 20 minutes (on my MacBook), we end up with a fully populated build tree of roughly 2700 files, including auto-generated source files ( .cc and .h suffix), object files (.o suffix), library files (.a suffix), and a small number of executable files:

out/x64.release/obj
out/x64.release/obj/v8_libbase/time.o
out/x64.release/obj/v8_libbase/semaphore.o
out/x64.release/obj/v8_libbase/platform-macos.o
out/x64.release/obj/v8_libbase/condition-variable.o
out/x64.release/obj/v8_libbase/ieee754.o
out/x64.release/obj/v8_libbase/file-utils.o
...
out/x64.release/obj/v8_compiler/effect-control-linearizer.o
out/x64.release/obj/v8_compiler/js-native-context-specialization.o
out/x64.release/obj/v8_compiler/store-store-elimination.o
out/x64.release/obj/v8_compiler/code-assembler.o
...
out/x64.release/gen/torque-generated/src/wasm/wasm-objects-tq-csa.h
out/x64.release/gen/torque-generated/src/wasm/wasm-objects-tq-csa.cc
out/x64.release/gen/torque-generated/src/objects/map-tq-csa.h
out/x64.release/gen/torque-generated/src/objects/code-tq-csa.h
...
out/x64.release/obj/libv8_libplatform.a
out/x64.release/obj/libwee8.a
out/x64.release/obj/third_party/zlib/libchrome_zlib.a
out/x64.release/obj/third_party/icu/libicui18n.a
out/x64.release/obj/third_party/icu/libicuuc.a
out/x64.release/obj/libv8_libbase.a
...
out/x64.release/obj/d8/d8.o
out/x64.release/obj/d8/d8-posix.o
out/x64.release/obj/d8/d8-console.o
out/x64.release/obj/d8/async-hooks-wrapper.o
out/x64.release/obj/d8/d8-js.o
out/x64.release/obj/d8/d8-platforms.o
out/x64.release/d8
...

Everything is now compiled, so I can run the d8 program to execute some JavaScript code:

$ ./out/x64.release/d8 
V8 version 8.6.0 (candidate)
d8> console.log(2 + 2);
4
undefined
d8>

Looking at this example output, you might think that d8 is actually the same thing as NodeJS (and the node command), but it’s actually a simple wrapper around the core V8 libraries. It doesn’t add any of the additional functionality that NodeJS provides, but instead just supports the core JavaScript language. It’s this core library that’s linked into NodeJS, the Chrome browser, and any other software that needs to compile JavaScript.

4. Executing the Unit Tests

The final step of the gm.py wrapper script is to execute the unit tests. This is done using the run-tests.py script.

$ ./tools/run-tests.py --outdir=out/x64.release \
       debugger intl mjsunit cctest message unittests

I plan to talk about V8 testing in another blog post, so I won’t give more detail here. Let’s instead dig deeper into both the GN build tool, and the Ninja build tool.

The GN Meta Build Tool

The GN Build Tool is classified as a “meta build” tool in that it doesn’t actually invoke the C++ compiler directly, but instead converts a human-readable build description into a lower-level format suitable for the Ninja build tool. This concept was popularized by CMake, which (amongst other things) is capable of auto-generating a tree ofMakefile files, to be used by the Make tool

GN Command Line Options

We’ve already seen how GN is used (with the gn gen option) to generate all the .ninja files, but what else can it do? Here are some interesting examples:

First, we can list all the possible build targets for V8:

$ gn ls out/x64.release//:bytecode_builtins_list_generator
//:cppgc
//:cppgc_base
//:cppgc_for_testing
//:cppgc_for_v8_embedders
//:cppgc_standalone
//:d8
//:fuzzer_support
//:gen-regexp-special-case
//:generate_bytecode_builtins_list
//:gn_all
//:json_fuzzer
//:lib_wasm_fuzzer_common
...

Next, we can show all the compilation flags, input files, and dependent libraries for one of these targets:

$ gn desc out/x64.release //:d8type: executable
toolchain: //build/toolchain/mac:clang_x64...sources
  //src/d8/async-hooks-wrapper.cc
  //src/d8/async-hooks-wrapper.h
  //src/d8/d8-console.cc
  //src/d8/d8-console.h
  //src/d8/d8-js.cc
  //src/d8/d8-platforms.cc
  //src/d8/d8-platforms.h
  //src/d8/d8.cc
  //src/d8/d8.h
  //src/d8/d8-posix.cc
  ...cflags
  -fno-strict-aliasing
  -fstack-protector
  -fcolor-diagnostics
  -fmerge-all-constants
  ...defines
  _LIBCPP_HAS_NO_ALIGNED_ALLOCATION
  CR_XCODE_VERSION=1160
  CR_CLANG_REVISION="llvmorg-12-init-1771-g1bd7046e-3"
  __STDC_CONSTANT_MACROS
  __STDC_FORMAT_MACROS
  ...Direct dependencies
  //:v8
  //:v8_dump_build_config
  //:v8_libbase
  //:v8_libplatform
  //:v8_tracing
  //build/config:executable_deps
  //build/win:default_exe_manifest

Finally, the reverse operation is to show which targets will be built from a specific source file.

$ gn refs out/x64.release //src/d8/d8-platforms.cc//:d8
//tools/gcmole:v8_run_gcmole

Note that none of these commands actually tell you which targets are currently out of date. As we’ll see later, that’s the responsibility of the Ninja tool.

Understanding BUILD.gn for the “d8” Target

The commands shown above are very useful, but how does GN know about the targets and their dependencies? Let’s spend some time looking at the highlights of the v8/BUILD.gn file. If you want more information on the BUILD.gn syntax, an excellent introductory presentation is also available.

We’ll be looking at how the d8 executable is constructed. This following is the code starting at line 4744 of my copy of BUILD.gn (it’s a long file!). I’ve added the “section” comments to make the code easier to refer to.

# Section 1 - The v8_executable Template
v8_executable("d8") {  # Section 2 - Defining the Sources
  sources = [
    "src/d8/async-hooks-wrapper.cc",
    "src/d8/async-hooks-wrapper.h",
    "src/d8/d8-console.cc",
    "src/d8/d8-console.h",
    "src/d8/d8-js.cc",
    "src/d8/d8-platforms.cc",
    "src/d8/d8-platforms.h",
    "src/d8/d8.cc",
    "src/d8/d8.h",
  ]  # Section 3 - Optional Sources
  if (v8_fuzzilli) {
    sources += [
      "src/d8/cov.cc",
      "src/d8/cov.h",
    ]
  }  # Section 4 - Compilation Configuration
  configs = [
    ":internal_config_base",
    ":v8_tracing_config",
  ]  # Section 5 - Additional Dependencies
  deps = [
    ":v8",
    ":v8_libbase",
    ":v8_libplatform",
    ":v8_tracing",
    "//build/win:default_exe_manifest",
  ]  ...
}

Let’s learn some of the main concepts of GN by walking through this example.

Section 1 — The v8_executable Template:

Out of the box, GN provides the executable command for describing how to construct an executable program. For V8, we actually use the v8_executable “template” (a GN concept) that wraps the basic executable command, providing some additional functionality for compiling V8 executables. This template is defined by including import("gni/v8.gni") at the top of the BUILD.gn file. The v8.gni file itself contains this snippet of code:

...template("v8_executable") {
   executable(target_name) {
     ...
   }
   ...
}...

This file also contains similar templates for v8_static_library, v8_shared_library, and v8_source_set that build upon the corresponding GN standard commands. In addition, the main BUILD.gn file also contains some template definitions, making the build description more concise by abstracting away the complexity.

Section 2 — Defining the Sources:

To specify the C++ source files to be included in the d8 executable, we define a variable that contains a list of file paths. The GN tool supports a simple programming language, including the concept of variables and values, as well as lists of values.

  sources = [
    "src/d8/async-hooks-wrapper.cc",
    "src/d8/async-hooks-wrapper.h",
    ...
  ]

Note that unlike many build tools, we’re only required to list the file paths. We don’t need to construct file name pattern matching, or specify dependencies between files. The mechanism for doing that is hidden from you in the auto-generated .ninja files.

Section 3 — Optional Sources:

There are numerous build variants for V8, supporting a wide range of host platforms, target CPUs, optimization choices, JavaScript language-level selection, and additional feature libraries. To support all these variants, GN provides an if statement for us to test variables and conditionally modify the list of sources (using sources +=)

  if (v8_fuzzilli) {
    sources += [
      "src/d8/cov.cc",
      "src/d8/cov.h",
    ]
  }

In this particular example, we’re adding support for the Fuzzilli fuzzing tool which requires additional code-coverage functionality.

Section 4 — Compilation Configuration:

To specify additional compilation flags for the d8 target, we make reference to a couple of “configs”:

configs = [
  ":internal_config_base",
  ":v8_tracing_config",
]

Here’s the definition of internal_config_base that appears earlier in the BUILD.gn file.

config("internal_config_base") {
  visibility = [ ":*" ]  configs = [ ":v8_tracing_config" ]  include_dirs = [
    ".",
    "include",
    "$target_gen_dir",
  ]
}

A “config” is a way to package together include paths, C++ symbol definitions, compiler flags, and additional libraries. These configs can obviously become quite complex, especially with support for multiple host platforms. But luckily, build targets simply need to reference the config by name, rather than worrying about all of those details.

Section 5 — Additional Dependencies:

Finally, to specify additional source files, or libraries to be linked into the d8 executable, we define the deps variable. Each entry in the list specifies a V8 build target, which itself provides a static/shared library, or a set of source files to include.

deps = [
    ":v8",
    ":v8_libbase",
    ":v8_libplatform",
    ":v8_tracing",
    "//build/win:default_exe_manifest",
  ]

That’s it! A relatively simple way of specifying how to construct the d8 executable, without burdening the developer with the complexities of compilation flags, dependencies, and file pattern matching. There are plenty of other GN commands/directives that we haven’t discussed, but the GN documentation shows them all.

The Ninja Build Tool

The last step in the V8 build process (with the exception of running tests) is to invoke the Ninja Build Tool to generate the object files, libraries, and executables. Given that users aren’t expected to look at the auto-generated .ninja files, there’s no need to look at further examples. However, it’s interesting to learn more about invoking Ninja, and the various command-line options available.

Speed is Everything

One of the interesting selling points of Ninja is its raw speed. Given my extensive history of using build tools like Make, I was very curious about what makes Ninja so responsive. When dealing with hundreds (or thousands) of source files, a lot of build tools will “pause” for 20–30 seconds as they determine which files are out of date. With Ninja, incremental builds seem to start instantly.

Here are some interesting facts about Ninja:

First, the build description files (with .ninja suffix) are very simplistic. There is no complicated language to be parsed, and no advanced features requiring time to execute. For this reason, the documentation describes the syntax as “machine code”. The .ninja files are also very compact, often with minimal white space. Keeping them small and simple makes them fast to read into memory, and to parse.
Second, implicit dependencies are stored in a single cache file, the .ninja_deps file (a 2MB binary file on my computer). In a Make-based environment, it’s common to have a unique .d file corresponding to each .cc file to store the list of C++ header files depended-on by the main .cc file. As a result, the build tool parses a very large number of files each time an incremental build is invoked. However, for Ninja, reading the one-and-only .ninja_deps file is extremely fast!

In the case of V8, the ninja tool starts reading build.ninja, which then imports the toolchain.ninja file. It’s this second file that recursively imports all the other .ninja files in the out/x64.release directory (shown earlier). Despite having roughly 100 .ninja files, reading and processing them is very fast.

Ninja Command-Line Options

To finish off, let’s show some of the commonly-used Ninja commands. To list all the available built targets, use the targets command:

$ ninja -t targetsbuild.ninja: gn
obj/test/common_test_headers.inputdeps.stamp: stamp
obj/test/unittests/unittests.inputdeps.stamp: stamp
cppgc: phony
cppgc_base: phony
cppgc_for_testing: phony
fuzzer_support: phony
generate_bytecode_builtins_list: phony
...
:d8: phony
:fuzzer_support: phony
:gen-regexp-special-case: phony
:generate_bytecode_builtins_list: phony
:gn_all: phony

Naturally, these build targets are similar to what was shown in the BUILD.gn file, and reported by the gn ls command.

To compile a specific target, just mention it on the command line, optionally with the -v flag if you want to see the underlying C++ compiler invocations.

$ ninja -v d8[1/1506] ../../third_party/llvm-build/Release+Asserts/bin/clang++ -MMD -MF ...
[2/1506] ../../third_party/llvm-build/Release+Asserts/bin/clang++ -MMD -MF ......[1506/1506] ...

To show all of the compilation commands required to build a target, without actually invoking the compiler, use the commands option:

$ ninja -t commands d8...

Finally, to show where a particular file is used (that is, which libraries or executables depend on it), use the query command:

$ ninja -t query ./obj/v8_libbase/mutex.oobj/v8_libbase/mutex.o:
  input: cxx
    ../../src/base/platform/mutex.cc
  outputs:
    obj/libv8_libbase.a
    obj/libwee8.a

These are the basics, but for more advanced options, see the Ninja documentation for further detail.

Summary

The V8 JavaScript engine has an excellent build system, comprised of a top-level convenience script (gm.py), which invokes the GN meta-build tool to generate lower-level build description files to be executed by the Ninja build tool. This combination of tools allows developers to work with the human-readable BUILD.gn file format, while allowing for a fast execution of the build steps, especially for incremental build.