V8 JavaScript Engine: Compiling with GN and Ninja
I’m a compiler enthusiast, who has been learning how the V8 JavaScript Engine works. Of course, the best way to learn something is to write about it, so that’s why I’m sharing my experiences here. I hope this might be interesting to others too.
This first blog post is an overview of how V8 is compiled. As you can see from the V8 source code repository, the V8 Engine is mostly written in C++, requiring source code to be compiled into executable files. This should be no surprise, given that V8’s primary purpose is fast compilation and execution of JavaScript programs.
I’ll be discussing three main topics related to compiling the V8 executables:
- The
gm.py
wrapper script, providing a convenient approach to compile V8 from source, and for invoking the test suites. - The GN meta-build system (invoked by
gm.py
) taking an easy-to-read description of the software components, then auto-generating a machine-readable build description suitable for the Ninja build tool. - Finally, the Ninja build tool uses that same machine-readable build description to analyze inter-file dependencies and invoke the relevant compilers.
The earlier diagram (at the top of this blog post) illustrates the overall flow of tool invocation, and shows which files are read, generated, and invoked.
Let’s examine each step in detail. If you’re new to this type of compilation process, I’ll put in a shameless plug for this book on Software Build Systems. It’s been almost ten years since I wrote that book, but the underlying concepts remain the same.
The gm.py Script
The first time you compile V8, you should use the recommended gm.py
script to fully compile all the object files, libraries, and executables.
$ ./tools/dev/gm.py x64.release.check
This is described in the V8 documentation as a convenience script because it’s a one-step solution for all the steps you need to get started. It takes about 20 minutes to run to completion (on my MacBook). Here’s what it’s doing:
1. Creating and Configuring the Build Output Directories
Using the best practice that object and executable files should be stored separately from the source code, the gm.py
script creates the v8/out/x64.release
directory. In this example, we’ve asked for V8 to be compiled for the x64.release
target (Intel x86 64-bit for release images), although if you also want to compile for different targets (such as x64.debug
or arm64.debug
), separate directories would be created for those.
This step also generates the args.gn
file in the v8/out/x64.release
directory, specifying the build options for this configuration (most notable are the is_debug
and target_cpu
options).
is_component_build = false
is_debug = false
target_cpu = "x64"
use_goma = false
goma_dir = "None"
v8_enable_backtrace = true
v8_enable_disassembler = true
v8_enable_object_print = true
v8_enable_verify_heap = true
2. Auto-Generating Ninja
files from the GN
Build Specification
The next step in the build process is for gm.py
to invoke the GN tool to translate the human-readable BUILD.gn
file into lower-level files for the Ninja tool (with .ninja
suffix).
The BUILD.gn
file contains easy-to-read directives specifying the content of each build target. In the following example, the d8
executable is constructed from a small number of C++ source files, linked together with additional libraries that contain the core JavaScript engine.
v8_executable("d8") {
sources = [
"src/d8/async-hooks-wrapper.cc",
"src/d8/async-hooks-wrapper.h",
"src/d8/d8-console.cc",
...
"src/d8/d8.cc",
"src/d8/d8.h",
] ... deps = [
":v8",
":v8_libbase",
":v8_libplatform",
...
]
}
Later in this blog post, there’ll be more detail about this file format. For now, let’s look at what happens when the gn gen
command generates all the .ninja
files from the hand-written BUILD.gn
file.
$ gn gen out/x64.release
This results in a collection of roughly 100 .ninja
files in the out/x64.release
directory. Each .ninja
file corresponds to one of the build targets described in the BUILD.gn
file.
./toolchain.ninja
./build.ninja
./obj/d8.ninja
./obj/v8_libbase.ninja
./obj/v8_simple_wasm_compile_fuzzer.ninja
./obj/v8_libplatform.ninja
./obj/v8_simple_multi_return_fuzzer.ninja
...
./obj/test/unittests/cppgc_unittests_sources.ninja
./obj/test/unittests/unittests_sources.ninja
./obj/test/wasm-api-tests/wasm_api_tests.ninja
./obj/test/common_test_headers.ninja
./obj/test/cctest/generate-bytecode-expectations.ninja
./obj/test/cctest/cctest_sources.ninja
./obj/test/cctest/cctest_headers.ninja
./obj/test/cctest/cctest.ninja
As an example, here’s the content of the d8.ninja
file. At first glance, this output is quite similar to an old-style Makefile — that is, not very readable!.
defines = -D_LIBCPP_HAS_NO_ALIGNED_ALLOCATION -DCR_XCODE_VERSION=1160 -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D_FORTIFY_SOURCE=2 -D_LIB
CPP_ABI_UNSTABLE -D_LIBCPP_DISABLE_VISIBILITY_ANNOTATIONS -D_LIBCXXABI_DISABLE_VISIBILITY_ANNOTATIONS -D_LIBCPP_ENABLE_NODISCARD -DCR_LIBCXX_REVISION=375504 ...include_dirs = -I../.. -Igen -I../.. -I../../include -Igen -I../../include -Igen/include -I../../third_party/icu/source/common -I../../third_party/icu/source/i18n -I../../include
cflags = -fno-strict-aliasing -fstack-protector -fcolor-diagnostics -fmerge-all-constants -fcrash-diagnostics-dir=../../tools/clang/crashreports -mllvm -instcombine-lower-dbg-declare=0 -fcomplete-member-pointers -arch x86_64 -Wno-builtin-macro-redefined ...build obj/d8/async-hooks-wrapper.o: cxx ../../src/d8/async-hooks-wrapper.cc || obj/d8.inputdeps.stamp
build obj/d8/d8-console.o: cxx ../../src/d8/d8-console.cc || obj/d8.inputdeps.stamp
build obj/d8/d8-js.o: cxx ../../src/d8/d8-js.cc || obj/d8.inputdeps.stamp
build obj/d8/d8-platforms.o: cxx ../../src/d8/d8-platforms.cc || obj/d8.inputdeps.stamp
build obj/d8/d8.o: cxx ../../src/d8/d8.cc || obj/d8.inputdeps.stamp
build obj/d8/d8-posix.o: cxx ../../src/d8/d8-posix.cc || obj/d8.inputdeps.stamp
...
Now that we have all the .ninja
files, we can start to compile the source code.
3. Using the Ninja Build Tool to Compile the Objects and Executables
The next step in the build process is for gm.py
to invoke the C++ compiler (amongst other tools). This is done by invoking the autoninja
command, which itself is a wrapper for the ninja
command.
$ autoninja -C out/x64.release d8
This command reads the relevant .ninja
files, determines which object files are missing (or out of date), then invokes the C++ compiler to create them. This process is familiar to anyone who has used the Make build tool (or similar).
After roughly 20 minutes (on my MacBook), we end up with a fully populated build tree of roughly 2700 files, including auto-generated source files ( .cc
and .h
suffix), object files (.o
suffix), library files (.a
suffix), and a small number of executable files:
out/x64.release/obj
out/x64.release/obj/v8_libbase/time.o
out/x64.release/obj/v8_libbase/semaphore.o
out/x64.release/obj/v8_libbase/platform-macos.o
out/x64.release/obj/v8_libbase/condition-variable.o
out/x64.release/obj/v8_libbase/ieee754.o
out/x64.release/obj/v8_libbase/file-utils.o
...
out/x64.release/obj/v8_compiler/effect-control-linearizer.o
out/x64.release/obj/v8_compiler/js-native-context-specialization.o
out/x64.release/obj/v8_compiler/store-store-elimination.o
out/x64.release/obj/v8_compiler/code-assembler.o
...
out/x64.release/gen/torque-generated/src/wasm/wasm-objects-tq-csa.h
out/x64.release/gen/torque-generated/src/wasm/wasm-objects-tq-csa.cc
out/x64.release/gen/torque-generated/src/objects/map-tq-csa.h
out/x64.release/gen/torque-generated/src/objects/code-tq-csa.h
...
out/x64.release/obj/libv8_libplatform.a
out/x64.release/obj/libwee8.a
out/x64.release/obj/third_party/zlib/libchrome_zlib.a
out/x64.release/obj/third_party/icu/libicui18n.a
out/x64.release/obj/third_party/icu/libicuuc.a
out/x64.release/obj/libv8_libbase.a
...
out/x64.release/obj/d8/d8.o
out/x64.release/obj/d8/d8-posix.o
out/x64.release/obj/d8/d8-console.o
out/x64.release/obj/d8/async-hooks-wrapper.o
out/x64.release/obj/d8/d8-js.o
out/x64.release/obj/d8/d8-platforms.o
out/x64.release/d8
...
Everything is now compiled, so I can run the d8
program to execute some JavaScript code:
$ ./out/x64.release/d8
V8 version 8.6.0 (candidate)
d8> console.log(2 + 2);
4
undefined
d8>
Looking at this example output, you might think that d8
is actually the same thing as NodeJS (and the node
command), but it’s actually a simple wrapper around the core V8 libraries. It doesn’t add any of the additional functionality that NodeJS provides, but instead just supports the core JavaScript language. It’s this core library that’s linked into NodeJS, the Chrome browser, and any other software that needs to compile JavaScript.
4. Executing the Unit Tests
The final step of the gm.py
wrapper script is to execute the unit tests. This is done using the run-tests.py
script.
$ ./tools/run-tests.py --outdir=out/x64.release \
debugger intl mjsunit cctest message unittests
I plan to talk about V8 testing in another blog post, so I won’t give more detail here. Let’s instead dig deeper into both the GN build tool, and the Ninja build tool.
The GN Meta Build Tool
The GN Build Tool is classified as a “meta build” tool in that it doesn’t actually invoke the C++ compiler directly, but instead converts a human-readable build description into a lower-level format suitable for the Ninja build tool. This concept was popularized by CMake, which (amongst other things) is capable of auto-generating a tree ofMakefile
files, to be used by the Make tool
GN Command Line Options
We’ve already seen how GN is used (with the gn gen
option) to generate all the .ninja
files, but what else can it do? Here are some interesting examples:
First, we can list all the possible build targets for V8:
$ gn ls out/x64.release//:bytecode_builtins_list_generator
//:cppgc
//:cppgc_base
//:cppgc_for_testing
//:cppgc_for_v8_embedders
//:cppgc_standalone
//:d8
//:fuzzer_support
//:gen-regexp-special-case
//:generate_bytecode_builtins_list
//:gn_all
//:json_fuzzer
//:lib_wasm_fuzzer_common
...
Next, we can show all the compilation flags, input files, and dependent libraries for one of these targets:
$ gn desc out/x64.release //:d8type: executable
toolchain: //build/toolchain/mac:clang_x64...sources
//src/d8/async-hooks-wrapper.cc
//src/d8/async-hooks-wrapper.h
//src/d8/d8-console.cc
//src/d8/d8-console.h
//src/d8/d8-js.cc
//src/d8/d8-platforms.cc
//src/d8/d8-platforms.h
//src/d8/d8.cc
//src/d8/d8.h
//src/d8/d8-posix.cc
...cflags
-fno-strict-aliasing
-fstack-protector
-fcolor-diagnostics
-fmerge-all-constants
...defines
_LIBCPP_HAS_NO_ALIGNED_ALLOCATION
CR_XCODE_VERSION=1160
CR_CLANG_REVISION="llvmorg-12-init-1771-g1bd7046e-3"
__STDC_CONSTANT_MACROS
__STDC_FORMAT_MACROS
...Direct dependencies
//:v8
//:v8_dump_build_config
//:v8_libbase
//:v8_libplatform
//:v8_tracing
//build/config:executable_deps
//build/win:default_exe_manifest
Finally, the reverse operation is to show which targets will be built from a specific source file.
$ gn refs out/x64.release //src/d8/d8-platforms.cc//:d8
//tools/gcmole:v8_run_gcmole
Note that none of these commands actually tell you which targets are currently out of date. As we’ll see later, that’s the responsibility of the Ninja tool.
Understanding BUILD.gn for the “d8” Target
The commands shown above are very useful, but how does GN know about the targets and their dependencies? Let’s spend some time looking at the highlights of the v8/BUILD.gn
file. If you want more information on the BUILD.gn
syntax, an excellent introductory presentation is also available.
We’ll be looking at how the d8
executable is constructed. This following is the code starting at line 4744 of my copy of BUILD.gn
(it’s a long file!). I’ve added the “section” comments to make the code easier to refer to.
# Section 1 - The v8_executable Template
v8_executable("d8") { # Section 2 - Defining the Sources
sources = [
"src/d8/async-hooks-wrapper.cc",
"src/d8/async-hooks-wrapper.h",
"src/d8/d8-console.cc",
"src/d8/d8-console.h",
"src/d8/d8-js.cc",
"src/d8/d8-platforms.cc",
"src/d8/d8-platforms.h",
"src/d8/d8.cc",
"src/d8/d8.h",
] # Section 3 - Optional Sources
if (v8_fuzzilli) {
sources += [
"src/d8/cov.cc",
"src/d8/cov.h",
]
} # Section 4 - Compilation Configuration
configs = [
":internal_config_base",
":v8_tracing_config",
] # Section 5 - Additional Dependencies
deps = [
":v8",
":v8_libbase",
":v8_libplatform",
":v8_tracing",
"//build/win:default_exe_manifest",
] ...
}
Let’s learn some of the main concepts of GN by walking through this example.
Section 1 — The v8_executable
Template:
Out of the box, GN provides the executable
command for describing how to construct an executable program. For V8, we actually use the v8_executable
“template” (a GN concept) that wraps the basic executable
command, providing some additional functionality for compiling V8 executables. This template is defined by including import("gni/v8.gni")
at the top of the BUILD.gn
file. The v8.gni
file itself contains this snippet of code:
...template("v8_executable") {
executable(target_name) {
...
}
...
}...
This file also contains similar templates for v8_static_library
, v8_shared_library
, and v8_source_set
that build upon the corresponding GN standard commands. In addition, the main BUILD.gn
file also contains some template definitions, making the build description more concise by abstracting away the complexity.
Section 2 — Defining the Sources:
To specify the C++ source files to be included in the d8
executable, we define a variable that contains a list of file paths. The GN tool supports a simple programming language, including the concept of variables and values, as well as lists of values.
sources = [
"src/d8/async-hooks-wrapper.cc",
"src/d8/async-hooks-wrapper.h",
...
]
Note that unlike many build tools, we’re only required to list the file paths. We don’t need to construct file name pattern matching, or specify dependencies between files. The mechanism for doing that is hidden from you in the auto-generated .ninja
files.
Section 3 — Optional Sources:
There are numerous build variants for V8, supporting a wide range of host platforms, target CPUs, optimization choices, JavaScript language-level selection, and additional feature libraries. To support all these variants, GN provides an if
statement for us to test variables and conditionally modify the list of sources (using sources +=
)
if (v8_fuzzilli) {
sources += [
"src/d8/cov.cc",
"src/d8/cov.h",
]
}
In this particular example, we’re adding support for the Fuzzilli fuzzing tool which requires additional code-coverage functionality.
Section 4 — Compilation Configuration:
To specify additional compilation flags for the d8
target, we make reference to a couple of “configs”:
configs = [
":internal_config_base",
":v8_tracing_config",
]
Here’s the definition of internal_config_base
that appears earlier in the BUILD.gn
file.
config("internal_config_base") {
visibility = [ ":*" ] configs = [ ":v8_tracing_config" ] include_dirs = [
".",
"include",
"$target_gen_dir",
]
}
A “config” is a way to package together include paths, C++ symbol definitions, compiler flags, and additional libraries. These configs can obviously become quite complex, especially with support for multiple host platforms. But luckily, build targets simply need to reference the config by name, rather than worrying about all of those details.
Section 5 — Additional Dependencies:
Finally, to specify additional source files, or libraries to be linked into the d8
executable, we define the deps
variable. Each entry in the list specifies a V8 build target, which itself provides a static/shared library, or a set of source files to include.
deps = [
":v8",
":v8_libbase",
":v8_libplatform",
":v8_tracing",
"//build/win:default_exe_manifest",
]
That’s it! A relatively simple way of specifying how to construct the d8
executable, without burdening the developer with the complexities of compilation flags, dependencies, and file pattern matching. There are plenty of other GN commands/directives that we haven’t discussed, but the GN documentation shows them all.
The Ninja Build Tool
The last step in the V8 build process (with the exception of running tests) is to invoke the Ninja Build Tool to generate the object files, libraries, and executables. Given that users aren’t expected to look at the auto-generated .ninja
files, there’s no need to look at further examples. However, it’s interesting to learn more about invoking Ninja, and the various command-line options available.
Speed is Everything
One of the interesting selling points of Ninja is its raw speed. Given my extensive history of using build tools like Make, I was very curious about what makes Ninja so responsive. When dealing with hundreds (or thousands) of source files, a lot of build tools will “pause” for 20–30 seconds as they determine which files are out of date. With Ninja, incremental builds seem to start instantly.
Here are some interesting facts about Ninja:
- First, the build description files (with
.ninja
suffix) are very simplistic. There is no complicated language to be parsed, and no advanced features requiring time to execute. For this reason, the documentation describes the syntax as “machine code”. The.ninja
files are also very compact, often with minimal white space. Keeping them small and simple makes them fast to read into memory, and to parse. - Second, implicit dependencies are stored in a single cache file, the
.ninja_deps
file (a 2MB binary file on my computer). In a Make-based environment, it’s common to have a unique.d
file corresponding to each.cc
file to store the list of C++ header files depended-on by the main.cc
file. As a result, the build tool parses a very large number of files each time an incremental build is invoked. However, for Ninja, reading the one-and-only.ninja_deps
file is extremely fast!
In the case of V8, the ninja
tool starts reading build.ninja
, which then imports the toolchain.ninja
file. It’s this second file that recursively imports all the other .ninja
files in the out/x64.release
directory (shown earlier). Despite having roughly 100 .ninja
files, reading and processing them is very fast.
Ninja Command-Line Options
To finish off, let’s show some of the commonly-used Ninja commands. To list all the available built targets, use the targets
command:
$ ninja -t targetsbuild.ninja: gn
obj/test/common_test_headers.inputdeps.stamp: stamp
obj/test/unittests/unittests.inputdeps.stamp: stamp
cppgc: phony
cppgc_base: phony
cppgc_for_testing: phony
fuzzer_support: phony
generate_bytecode_builtins_list: phony
...
:d8: phony
:fuzzer_support: phony
:gen-regexp-special-case: phony
:generate_bytecode_builtins_list: phony
:gn_all: phony
Naturally, these build targets are similar to what was shown in the BUILD.gn
file, and reported by the gn ls
command.
To compile a specific target, just mention it on the command line, optionally with the -v
flag if you want to see the underlying C++ compiler invocations.
$ ninja -v d8[1/1506] ../../third_party/llvm-build/Release+Asserts/bin/clang++ -MMD -MF ...
[2/1506] ../../third_party/llvm-build/Release+Asserts/bin/clang++ -MMD -MF ......[1506/1506] ...
To show all of the compilation commands required to build a target, without actually invoking the compiler, use the commands
option:
$ ninja -t commands d8...
Finally, to show where a particular file is used (that is, which libraries or executables depend on it), use the query
command:
$ ninja -t query ./obj/v8_libbase/mutex.oobj/v8_libbase/mutex.o:
input: cxx
../../src/base/platform/mutex.cc
outputs:
obj/libv8_libbase.a
obj/libwee8.a
These are the basics, but for more advanced options, see the Ninja documentation for further detail.
Summary
The V8 JavaScript engine has an excellent build system, comprised of a top-level convenience script (gm.py
), which invokes the GN meta-build tool to generate lower-level build description files to be executed by the Ninja build tool. This combination of tools allows developers to work with the human-readable BUILD.gn
file format, while allowing for a fast execution of the build steps, especially for incremental build.