Embedding Starlark (Part 2) — Extend Go Program Functionalities with Starlark scripts

Using Starlark-Go to embed the Starlark interpreter to extend the runtime capabilities of Go programs with Starlark script functions

Part one of this series shows how to get started with embedding Starlark and using scripts to configure Go programs at runtime. This post continues the series and explores how to use Starlark scripts as extension points to add functionalities to Go programs at runtime.

All examples shown here can be found on GitHub.

Starlark functions as Go program plugins

The previous post introduced program getfile as an illustrative tool to show how to embed Starlark to configure Go programs. This post continues with the same example and shows how to embed the Starlark interpreter to extend the functionalities of Go programs at runtime, using Starlark script functions, without changing the program itself.

As shown above, the getfile program will use Starlark to extend its functionality using script functions as pluggable extension points. Specifically, the new version of the program will use a script function called proc_line that will allow users to inject custom logic to process a downloaded file line-by-line.

The script file

The new version of the script file is shown below. It now includes the definition of function proc_line:

config(
    source_url="https://www.gutenberg.org/files/408/408-0.txt",
    dest_file="./soul-black-folks.txt"
)

def proc_line(line):
    return line

At runtime, function proc_line will be invoked by the host Go program as it processes each line of a downloading file. The function can transform each text line, as it receives it, and is expected to return its result back to the host program for further processing (in this case, the processed line is saved to a file).

Invoking script functions from Go

In the previous post, we discussed how create and register Starlark builtin functions to configure the Go program at runtime, this discussion will not be repeated here.

Next, lets see how the host program invokes script function proc_line from Go (see item 1 in the comment below). When a Go program uses the Starlark interpreter to execute a script file (with starlark.ExecFile call), it receives a map, assigned to variable script above, of globally declared script values (i.e. variables, functions, etc).

In the snippet below, script["proc_line"] retrieves a reference to the proc_line function from the script.

func main() {
  // 0. Register Starlark builtin (not discussed in this post)
  registrar := starlark.StringDict{"config": starlark.NewBuiltin("config", configFn)}
  script, err := starlark.ExecFile(&starlark.Thread{}, "getfile.star", nil, registrar)
  if err != nil {
    log.Fatalf("Starlark Exec: %s", err)
  }

  // 1. retrieve a text line processor function from script
  var procLineFn starlark.Value
  if procLine := script["proc_line"]; procLine != nil && procLine.Type() == "function" {
    procLineFn = procLine
  }
...
}

Next, let’s focus on item 4 in the source code snippet below. That is where the code loops through the content line-by-line as it is downloaded from the server. With each loop iteration, the Go code uses starlark.Call function (item 5 in comment) to invoke script function proc_line , passing it the text line value.

func main() {
  // 0. Register Starlark builtin
  // 1. retrieve a text line processor function from script

  // 2. download resource from remote server (not discussed here)
  rsp, err := http.Get(sourceUrl)
  ...

  // 3. Create file for output
  file, err := os.Create(destFile)
  ...
  
  scnr := bufio.NewScanner(bufio.NewReader(rsp.Body))

  // 4. Loop through content line-by-line
  for scnr.Scan() {
    if scnr.Err() != nil {
      continue
    }

    // text line
    text := scnr.Text()

    // 5. call Starlark function proc_line with text line
    if procLineFn != nil {
      result, _ := starlark.Call(&starlark.Thread{}, procLineFn, starlark.Tuple{starlark.String(text)}, nil)
      if str, ok := starlark.AsString(result); ok {
        text = str
      }
    }

    // 6. Save result to output file
    file.WriteString(text)

   }
}

At that point, control is delegated to the script function where it can apply user-provided runtime logic from the script. When the proc_line function call completes, it returns a string result value that can be, subsequently, saved in the output file.

Using the program

Now Let’s explore some examples to see how we can use Starlark scripts to extend the capabilities of our new program. First, let’s compile the program into a binary:

go build -o getfile

Next, let’s see how we can extend the runtime capabilities of our program, without recompilation, using the Starlark script file. In the first example run, we are going to print out the content of the retrieved file as it is being downloaded. We will use the following Starlark script to do this:

config(
    source_url="https://www.gutenberg.org/files/408/408-0.txt",
    dest_file="./soul-black-folks.txt"
)

def proc_line(line):
    print(line)
    return line

When we run the example, it simply prints each line of the text file to standard output.

./getfile

The Project Gutenberg eBook of The Souls of Black Folk, by W. E. B. Du Bois

This eBook is for the use of anyone anywhere in the United States and
most other parts of the world at no cost and with almost no restrictions

...

Next, let’s demonstrate how we can change the behavior of our program, without recompiling it, using the Starlark script. To do this, we are going to update the script to transform the text, from regular to upper case, as it is downloaded:

def proc_line(line):
    return line.upper()

Now, when we run the same program, it behaves differently simply by changing the Starlark script file. You can imagine having some interesting transformation logic encoded in the script, to fit different needs, without rewriting or recompiling the host Go program !

Conclusion

This second part of the series on Starlark explores how to embed the Starlark interpreter to extend the runtime functionality of its host Go program. The writeup shows how to use the Starlark interpreter to implement code that can invoke script functions with user-provided logic at runtime without changing or recompiling the host program.

References

• Part 1 — Configure Go Programs with Starlark Scripts
• Example source code (GitHub)
• Starlark-Go project
• Starlark language spec