Setting up a build system for a full-stack web application presents some unique challenges. Builds for these projects not only include compiling and testing executables for the back-end, but the build system also needs to be flexible enough to integrate with various tools for processing web assets, packaging the build, etc. Since these systems can get relatively large, build times are also important.
A Phased, Parallel Build System
A relatively simple build system can be setup as follows:
- The build is organized into phases. There is a specific order to the phases and commands within a phase can only depend on previous phases.
- A top-level build script runs the full build and orchestrates parallel build tasks.
- Each subdirectory includes a makefile. The makefile is invoked once per phase by the top-level build script.
More information and details are presented below, but the basic system described above is all you will likely need for full-stack web apps. Resultra is a new type of web application for project tracking, and uses this system. Using Resultra’s build system as a case study, this type of build system has proven to be flexible, robust and scalable. Although the system itself is not very sophisticated, it can be used with projects that leverage a full complement of front-end and back-end build and testing tools.
A primary concept for this build system is that builds are organized into phases. Build phases are performed in a specific order. For example, there are phases to compile the code, export the resources, then perform tests.
Another important concept here is that a build command within a given phase can only depend on previous phases; for example, a test can only be run after the code has been compiled.
Top-level Build Script
In general, Python is the glue which ties the entire build system together. All it takes for Python to implement the core functionality of a phased, parallel build system is a simple 100 line build script! To see for yourself, here’s what the code listing currently looks like for Resultra’s top-level build script:
# This script implements a phased build based upon the makefiles in the development tree.
# Within each build phase, make is run on each directory in no particular order. So,
# the build process from each directory is expected to not depend on other directories
# within a a given phase.
# By default a debug build is performed. However to perform a release build, pass the — release
# option on the command line.
from multiprocessing import Poolparser = argparse.ArgumentParser(description=’Main build script.’)
parser.add_argument(‘ — release’,default=False,action=’store_true’,
help=’perform a release build’)
parser.add_argument(‘ — realcleanonly’,default=False,action=’store_true’,
help=’only run the clean and realclean targets across the build’)
parser.add_argument(‘ — windows’,default=False,action=’store_true’,
help=’cross-compile the Windows Electron client.’)
parser.add_argument(‘ — procs’,default=4,type=int,
help=’number of build tasks to run in parallel build on (default = 4)’)
args = parser.parse_args()failedDirs = debugBuild = 1
debugBuild = 0
return “(dir = %s, err = %d) “ % (self.dirName,self.errCode)
def __init__(self, dirName,errCode):
self.dirName = dirName
self.errCode = errCodeclass buildDirSpec:
def __init__(self, dirName,targetName,debugBuild):
self.dirName = dirName
self.targetName = targetName
self.debugBuild = debugBuild
print “Building: dir=”, buildSpec.dirName, “ phase=”, buildSpec.targetName, “ debug=”, buildSpec.debugBuild
bldCmd = “make -C %s — jobs=2 DEBUG=%s %s” % (buildSpec.dirName, buildSpec.debugBuild, buildSpec.targetName)
print “Build cmd: %s “ % (bldCmd)
retCode = os.system(bldCmd)
if retCode != 0:
print “FAIL: failure building dir = %s, target= %s, err = %d” % (buildSpec.dirName,buildSpec.targetName,retCode)
print “Build: Starting phase = “, makeTargetName
makeDirs = 
for root, dirs, files in os.walk(“..”):
for file in files:
if (file == ‘Makefile’) and (not “node_modules” in root):
buildPool = Pool(processes=args.procs)
results = buildPool.map(buildOneDir,makeDirs)
print “Build: Done with phase = “, makeTargetName
for res in results:
if res.errCode != 0:
failedDirs.append(makeTargetName + “:” + res.dirName)
startTime = time.time()if args.realcleanonly:
runMakePhase(“winpkg”)endTime = time.time()print “\n\n — — — — — — — — — — — — — — — — — — — — — — — — — — — — “
print “Build complete: parallel build tasks = %d, elapse time = %d secs “ % (args.procs, endTime-startTime)
print “\nBuild Results:\n”if len(failedDirs) > 0:
print “Build failed on following directories:\n”
print “Build succeeded”
You’re welcome to use the code snippet above as a starting point for your own projects. This type of phased, parallel build system is quite generic; it will not only work for full-stack web applications, but almost any type of software development project.
Makefile per Subdirectory
The second component to the build system is a makefile per subdirectory of the project. Each subdirectory’s makefile only implements targets for the build phases which are applicable to that directory, with the other phases being no-ops. Since dependencies are between phases, there are no recursive makefiles.
Besides the parallel build support provided by the top-level build script, Makefiles also provide a second level of parallel execution using the make tool’s ‘ — jobs’ option. This option allows individual commands (jobs) within the makefile to be executed in parallel.
DEPTH = ../../..include $(DEPTH)/webui/build/autoExportAssets.mk
Once a full build has been completed, development and testing can can occur by running make on the subdirectory where changes have occured.
Project-specific Makefile Commands
Besides the top-level build script and a makefile per subdirectory, the third major component of this system is project-specific makefile commands. These commands are needed to integrate the system with various build and testing tools.
Using Resultra’s build as an example, below is a sampling of build results with different numbers of parallel build tasks enabled in the top-level build script. In addition to the parallel build tasks launched by the top-level build script, the individual makefiles are invoked with the — jobs=2 option for the make command to internally perform parallel execution of jobs/commands inside the makefile:
These builds were performed on a late 2012 Mac Mini with a quad-core i7 processor (8 virtual cores) and a solid-state drive.
Increasing the number of parallel tasks does improve build times, but the improvements are not earth shattering. In particular, going from 1 to 8 parallel build tasks, the build times improved by 33%.
As one more data point, if make’s jobs option is set to 1 and parallel execution is disabled in the top-level build script, the build time increases to 496 seconds. This data point represents the slowest possible build with no parallel execution at all. Versus this slowest possible data point, a build with 8 parallel build tasks and make’s jobs option set to 2, the build is 43% faster.
It’s possible to create a relatively simple build system for full-stack web apps. This system consists of only a single top-level build script, individual makefiles per subdirectory, and makefile commands to integrate with project-specific build and testing tools. Using Resultra as a case study, this simple type of build system has now been proven to be flexible, robust and scalable for relatively large, full-stack web applications.
The benefits of this system not only include simplicity, but also a good separation of concerns. The top-level build script and individual makefiles provide a lightweight scaffolding to run the build. With this scaffolding in place, custom utility scripts and makefile commands are used to integrate the build system with a full complement of front-end and back-end tools.
Nothing is necessarily new or innovative with this type of build system. This system is similar to build systems I’ve encountered on other projects. Nonetheless, for the benefit of other projects, this case study hopefully illustrates the key concepts behind a phased, parallel build system.