Instrumentorio: Custom Monitoring for Factorio (or Anything)

Published in

Expected Behavior Blog

7 min readApr 1, 2019

What are Factorio and Instrumental?

Here at Expected Behavior, we love Factorio — by that I mean, some Behaviornauts have played it before, and I am 100% obsessed. It averages out to company-wide love, trust me. Factorio is a factory simulation/automation game where the goal is to produce ever-more complex resources in an ever-more automated manner.

In fact, given the complexity of Factorio and the problem solving it requires, I think the game is an interesting alternative to many technical interviews. Yes, really. If you love Factorio, you can see our open jobs here. But Factorio-as-an-interview should probably be a separate blog post…

In the rare moments when I’m not playing Factorio, I’m working on Instrumental. Instrumental is a monitoring platform designed to ingest absurd amounts of arbitrary data, and then provide tools to help you interpret it. I wanted to write a mod to pull data from the Factorio game engine and get it into Instrumental, just because I can. Let me walk you through the mod.

Writing the Factorio mod

Learning Lua and flying without an Agent

The good news is that Factorio mods are written in Lua, a very simple and stable scripting language. If you’re familiar with Ruby, Python, or JavaScript, you can basically jump right in. On MacOS, installing Lua and its package manager using Homebrew is easy:

brew install lua
brew install luarocks
brew install luasocket

Instrumental does not (currently!) provide a Lua agent, but that’s not a problem. Getting Lua to send data to Instrumental’s collector API is very easy:

nathan@NetworkNotFound instrumentorio/  $ luaLua 5.3.5  Copyright (C) 1994-2018 Lua.org, PUC-Rio> host, port = "collector.instrumentalapp.com", 8000> socket = require("socket")> tcp = socket.tcp()> tcp:setpeername(host, port)1.0> tcp:settimeout(1)1.0> message = "hello version lua/test/1.0\nauthenticate <your_key_here>\nincrement thing.test.lua 1\n"> tcp:send(message)73.0 nil nil 4.4822692871094e-05> tcp:receive()ok nil nil 0.037738084793091

We’re in! Let’s get this code into a Factorio mod!

Writing the mod — easy mode with sockets

The luasocket library installed above provides some C extensions that let you work directly with sockets, which is great for Instrumental. The mod code ends up looking more or less like the REPL example above. Unfortunately, Factorio’s mod environment has a very tight security sandbox which disallows this. Between that and its lockstep synchronization model… sockets aren’t going to fly.

Writing the mod — still-fairly-easy mode with http

Okay, no problem, we’re still cool — we can just whip up a simple http output… But we encounter the same problem. Anything that communicates with the outside world in a way that could result in a meaningful response is disallowed. So, how do we get this data out in a way that Factorio likes? Enter game.write_file().

Writing the mod — somewhat-more-involved mode with InstrumentalD

game.write_file() is a function in the Factorio API that does, well, exactly what you’d expect. We can simply write out data to a file on the local disk and then write a plugin script to consume it with InstrumentalD. This offers the added advantage of keeping all of the authentication and actual transmission outside of the Factorio process.

What does a Factorio mod look like?

Our example does not create any new recipes or entities in the world and is basically invisible to the user. Please note that this is very much proof-of-concept, so copy at your own risk.

You can put your mod into a zip file in the Factorio mods directory or just leave it in a folder — Factorio is happy with either. First, you’ll need some metadata in info.json:

{
  "name": "Instrumentorio",
  "version": "0.1.0",
  "title": "Instrumentorio",
  "author": "Nathan D. Acuff",
  "contact": "http://instrumentalapp.com",
  "homepage": "http://instrumentalapp.com",
  "factorio_version": "0.16",
  "dependencies": ["base >= 0.16"],
  "description": "This experimental mod writes to a local file that Instrumental can understand.  Requires external plugin script."
}

Lastly, the beating heart of the mod is control.lua. This example is pretty basic, but you could expand it to pull basically anything from the game state.

We’ll start by creating a function to save us some headache when actually writing the files:

function instrumental_output(line)
   game.write_file(“factorio.” .. game.tick, line .. "\n", true)
end

By default, the file will be written to a directory called “script-output” in the same place as your mod directory, so that’s fine. This simply writes to a filename tagged with the game tick so that each write will go to a unique file. This is done so that the InstrumentalD plugin script can simply eat the files as they appear and not worry about keeping its place in the log or about the size of the file.

Finally, here’s the magic to read from the game state:

script.on_nth_tick(900, -- every 15 seconds
  function(event)
    for k, force in pairs(game.forces) do
      for item, amount in pairs(force.item_production_statistics.input_counts) do
        instrumental_output("item_production." .. item .. " " .. amount)
      end
      for item, amount in pairs(force.item_production_statistics.output_counts) do
        instrumental_output("item_consumption." .. item .. " " .. amount)
      end
      for item, amount in pairs(force.fluid_production_statistics.input_counts) do
        instrumental_output("fluid_production." .. item .. " " .. amount)
      end
      for item, amount in pairs(force.fluid_production_statistics.output_counts) do
        instrumental_output("fluid_consumption." .. item .. " " .. amount)
      end
    end
  end)

This function will write the amounts of production and consumption every 15 seconds for all items and fluids to a file. Great, we’re done here!

Consuming the Factorio output with InstrumentalD

InstrumentalD Plugin Scripts provide an easy way to pull arbitrary data into your Instrumental project. They can be very complicated with state being passed from execution to execution or as simple as a one-line bit of bash. I’ve chosen to write the script in ruby, but you can use whatever you like:

#!/usr/bin/env rubyFACTORIO_SCRIPT_DIR = "#{Dir.home}/Library/Application\ Support/factorio/script-output/"files_to_scan = Dir["#{FACTORIO_SCRIPT_DIR}*"].sort_by{ |f| File.mtime(f) }[0..-2]files_to_scan.each do |file|
  File.read(file).split("\n").compact.each do |line|
    puts line
  end
  File.delete(file)
end

Pretty simple. Like I said, this could definitely be done in fewer lines (or in bash), but I’m not going for style points today. We get a list of all the files, shoot all but the last one to Instrumental, and then delete them. Why not the last one? It’s a simple way to make sure we aren’t reading the file as it’s being written, with a minor penalty — an extra 15 seconds of latency.

You can install InstrumentalD with homebrew, and running it with scripts is as easy as:

instrumentald -c ~/Desktop/instrumentald.conf --debug -e -s ~/.instrumentald_scripts/

My instrumentald.conf file is just a cut-down version of the example:

project_token = "<your project token here>"
system = ["cpu", "memory"]

Exploring the data with Instrumental

To test this mod out, I built a simple little science base in Factorio:

I hope you’re in the mood for pasta.

Factorio provides an internal mechanism for viewing production stats, but it can be difficult to search, look at historical data, and find patterns. Note: I’m not seriously suggesting that you need Instrumental to play Factorio well. I just like neat graphs. Anyway, here’s what 20 minutes or so of building up this base looks like in game:

So, what does this data look like in Instrumental?

Okay, great, but there’s no way that base is producing 400 Science per minute. It looks like the stats coming out of Factorio are a count of all items produced since whenever. InstrumentalD’s scripting interface provides some ways to handle that, but I’ve written enough code for the day. Instrumental Query Language provides the growth_rate function, which is useful for converting from these sorts of counters into an actions-per-minute view. Looking your way, redis INFO calls.

So, that’s pretty cool. Want to interact with it a bit? Here’s some plate production.

In conclusion

So, that’s how we can use some simple tools to extract data and dump it into Instrumental. Beyond importing a single source, being able to pull data from multiple sources into an Instrumental project so you can slice and dice with the Instrumental Query Language is a great way to make your monitoring life easier. If you don’t like video games, don’t worry, a post about doing the same thing with CloudWatch is coming soon, and you won’t need to write any Lua.

Postscript: Is that all?

There are plenty of other things that we could do in Factorio with a mod like this. For example, we could connect in-game alerts for things like biter attacks, low-power conditions, or research completion to Instrumental Notices. We could also build an in-game entity to pull data from a circuit network into Instrumental. Considering that people have built working computers, televisions, and all sorts of crazy stuff with circuit networks, I think I can put this one down under “seriously, anything you want.”