Embedding Starlark (Part 1) — Configure Go Programs with Starlark Scripts

Using the Starlark-Go project to embed the Starlark interpreter to configure Go programs at runtime

Starlark is a Python-like scripting language that was originally developed, in Java, to be embedded in the Bazel build tool. Since then, Starlark has evolved into a language spec with runtime interpreters implemented in Go and Rust.

This first post, of a two-part series, shows how to extend the runtime capabilities of your programs with domain-specific scripting functionalities for application configuration using project Starlark-Go.

Coverage of the Starlark language itself is beyond the scope of this post. For a detail reference, see the language spec.

Getting started

To get started, let’s do a quick walk through of embedding and executing a simple Starlark script, that prints “Hello, World from Starlark!”. To do this, the Go code below imports package go.starlark.net/starlark , from project Starlark-Go, and uses it to load and execute Starlark script file “hello.star”:

package main

import (
  "log"
  "go.starlark.net/starlark"
)

func main() {
  _, err := starlark.ExecFile(&starlark.Thread{}, "hello.star", nil, nil)
  if err != nil {
    log.Fatalf("Starlark Exec: %s", err)
  }
}

All examples shown here can be found on GitHub.

If the Go code is executed at this point, it will fail as shown below:

$> go run main.go
2023/01/01 18:25:05 Starlark Exec: open hello.star: no such file or directory
exit status 1

So, let’s create a file name hello.star with the following content:

print("Hello World, from Starlark!")

Now, when program is executed, we get the following:

$> go run main.go
Hello World, from Starlark!

Congratulations🎉 You have successfully embedded the Starlark interpreter in your Go program and use it to execute a script file. In the remainder of this post, we are going to explore how to use Starlark as a way to configure your Go programs at runtime.

Extending Starlark

First, let’s look at the mechanism of extending the Starlark runtime. The Starlark language comes with a large set of useful built-in functions (builtins) that can be used to build functional program.

Using custom functions

Users can, however, define their own script functions, using the def keyword, for customized functionalities as illustrated in the next code snippet:

print_hello()

def print_hello():
    print("Hello, from Starlark!")

Using user-provided builtins

Additionally, the Starlark interpreter runtime can be extended with user-provided builtins. These are Go functions that are registered with the interpreter and can be invoked, at runtime, to handle script function calls.

A Starlark builtin Go function has the following signature :

func (_ *starlark.Thread, _ *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error)

During the execution of a script, the interpreter will transfer control to a registered builtin Go function that is capable of handling the script function call. The Go function will receive the argument values from the Starlark script function call and it can also return a result to the running script.

Using Starlark for runtime configuration

To illustrate how Starlark can be used for runtime configuration, lets create a program, called getfile, that we can use to retrieve text content from a remote HTTP source and save it to a local file as shown below:

package main

var (
 sourceUrl string
 destFile  string
)

func main() {
  sourceUrl = "https://www.gutenberg.org/files/408/408-0.txt"
  destFile = "./soul-black-folks.txt"

  // 1. download resource
  rsp, err := http.Get(sourceUrl)
  if err != nil {
    log.Fatal(err)
  }

  // 2. write resource to destination
  var content []byte
  if content, err = io.ReadAll(rsp.Body); err != nil {
    log.Fatal(err)
  }
  defer rsp.Body.Close()

  if err := os.WriteFile(destFile, content, 0644); err != nil {
    log.Fatal(err)
  }
}

This version of the program (source file) has the source and the destination of the content hard-coded (in variables sourceURL and destFile respectively). When the program is executed, it will automatically pull and save the hard coded document (W. E. B. Du Bois’ The Soul of Black Folks).

Using Starlark as configuration script

What we want is to modify the previous program so that its configuration, for the source and destination arguments, come from a Starlark script as shown below. The script will use function config to configure the source URL and the destination file.

config(
  source_url="https://www.gutenberg.org/files/408/408-0.txt",
  dest_file="./soul-black-folks.txt"
)

Next, lets define a Go function, confingFn, that will be registered as a Starlark builtin to handle the script function call config(…):

func configFn(_ *starlark.Thread, _ *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) {
  if err := starlark.UnpackArgs(
    "config", args, kwargs,
    "source_url", &sourceUrl,
    "dest_file", &destFile,
  ); err != nil {
    return starlark.None, fmt.Errorf("config: %s", err)
  }
  return starlark.None, nil
}

The Go function above uses helper function starlark.UnpakArgs (from project Stalark-Go) to map argument values from the script function call to local Go variables. Script argument source_url is mapped to Go variable sourceUrl and dest_file is mapped to destFile.

Finally, let’s update the main function to register the new Go function as a Starlark builtin function and execute the script file as shown below:

func main() {
  // 0. create a builtin registar as a StringDict (a map)
  registrar := starlark.StringDict{"config": starlark.NewBuiltin("config", configFn)}

  // 1. Execute script file, with the registered builtin
  _, err := starlark.ExecFile(&starlark.Thread{}, "getfile.star", nil, registrar)
  if err != nil {
    log.Fatalf("Starlark Exec: %s", err)
  }

  // 2. download resource
  rsp, err := http.Get(sourceUrl)
  if err != nil {
    log.Fatal(err)
  }

  // 3. write resource to destination
  var content []byte
  if content, err = io.ReadAll(rsp.Body); err != nil {
    log.Fatal(err)
  }
  defer rsp.Body.Close()

  if err := os.WriteFile(destFile, content, 0644); err != nil {
    log.Fatal(err)
  }
}

Now, the program creates a StringDict (a map value) that maps the name of the Starlark script function (config) to the Go builtin function (configFn) and assign it to variable registrar. Next the code executes the Starlark script file getfile.star, passing it the registrar so that when the script encounters the config function it will invoke the Go function.

When the program executes, it will use the script file as a way to configure the program. Using this approach can work with simple values like numbers and strings, but you can also exend your configuration to accept container values such as map, list, etc.

Conclusion

This concludes the first installment of several posts that discuss how to use the Starlark interpreter to extend the capabilities of your Go program using the Starlark-Go project.

This post explores how to get started with embedding the Starlark interpreter. It shows how to extend the Starlark runtime to create your own custom builtin functions to read configuration argument from a Starlark script.

The next post will explore how to use Starlark script as an extension mechanism for your program at run time.

References

• Part 2 — Extend Go Program Functionalities with Starlark Scripts
• Example source code (GitHub)
• Starlark-Go project
• Starlark language spec