Docker and Zombies

This story was originally featured on wehkamplabs.com.

We’ve all heard about the incredible chaos monkey from Netflix, killing instances at scale. At Wehkamp we have something similar to attack our microservices clusters: Half-Life2.

Yes, that game from Valve. And using Garry’s mod, one can actually interact with entities inside the game. Spawning items, weapons, non-playing characters and even zombies. Another possibility is talking with HTTP endpoints, and that opens up a wide range of options.

Our Blaze microservices platform is build on Mesos/Marathon. Both expose REST API endpoints over HTTP, which the entire world should do, since it’s 2016.

Using some pieces of Lua coding I was able to connect the game to Marathon and have it fetch a list of running applications, there names, associated count of containers and some meta-data.

for k, v in pairs(util.JSONToTable(body)) do
    for key, app in pairs(v) do
        local n = entitiesSpawned(app['id'])
        if n < app['instances'] then
            for i=n+1,app['instances'],1 do
                blazeSpawn(app['id'], app['labels']['team'], app['labels']['type'])
            end
        end
    end
end

Based on those details we spawn a bunch of characters, randomly equipped with the authentic Half-Life crowbar and put them together on the map grouped by dev-team.

blazeSpawn = function(what, team, type)
        local e = teams["model"]
        PrintMessage(HUD_PRINTTALK, "Spawning for service "..what)
        ent = ents.Create(e)
        ent:SetName(what)
        local x = teams[team]["vector"]["x"]
        local y = teams[team]["vector"]["y"]
        ent:SetPos(Vector(math.random(x-300,x+200),math.random(y-300,y+200),150))
        ent:Spawn()
        ent:Activate()
        ent:DropToFloor()
        // not everyone gets a crowbar
        if math.random(1,10) == 1 then
            ent:Give("ai_weapon_crowbar")
        end
        // all your base are belong to team purple
        if team == "purple" then
            ent:Give("ai_weapon_rpg")
        end
        ent:NavSetWanderGoal(400, 8000)
        ent:SetMovementActivity(ACT_WALK)
        ent:SetSchedule(SCHED_FORCED_GO)
    end

And lets spawn a zombie. And because we’re wrapped inside a big _while(forever)_ loop that iterates on a specific interval, you’ll get a new free zombie every 5 seconds!

blazeSpawnKiller = function(what, team, type)
        local e = "npc_zombie"
        ent = ents.Create(e)
        ent:SetName("none")
        ent:SetPos(Vector(math.random(1,-2000), math.random(1,2000), 200))
        ent:Spawn()
        ent:Activate()
        ent:DropToFloor()
        ent:NavSetWanderGoal(1000, 200)
        ent:SetMovementActivity(ACT_WALK)
        ent:SetSchedule(SCHED_FORCED_GO)
    end
    blazeSpawnKiller()

The objective is simple: defend your containers against a growing horde of zombies, because when an NPC dies, a container dies. Killing an NPC will trigger a call to the Mesos HTTP API, resulting in termination of a task, which is essentially a container. Plus, the dead npc will leave random goodies!

hook.Add("OnNPCKilled", "OnNPCKilled", function(npc, attacker, inflictor)
 local goodies = {
   "item_healthkit",
   "item_ammo_smg1",
   "item_ammo_ar2",
   "item_ammo_crossbow",
   "item_box_buckshot"
 }
 local goods = ents.Create(table.Random(goodies))
 goods:SetPos(npc:LocalToWorld(npc:OBBCenter()))
 goods:Spawn()
 for npcid in string.gmatch(tostring(npc), "%[([%d]+)%]") do
   eid = npcid
 end
 service = tostring(Entity(eid):GetName())
 if service != "none" then
   killContainer(service)
 end
end

Of course Marathon will detect these killed containers and it will deploy them again, making sure service is restored. In-game this results in new NPC’s being spawned for that particular service.

A fun way of interacting with API’s and testing resiliency of our platform. And yes, multiplayer is supported :)

The full source of all this is available at GitHub. Because, where else.