AI & Gaming
Vereya: A Minecraft Mod for Game Virtual Assistants and AI Agents
Creating General-Purpose AI Agents for Minecraft
--
Introduction to Vereya
Minecraft is not only a popular game and an educational platform, but also a good environment for AI R&D.
However, building and training intelligent agents and even studying programming in Minecraft is typically separated from normally playing the game itself. Different Minecraft mods for AI R&D are oriented towards massive training of reinforcement learning models, and may fix the game window size, disable human control, use a concrete Minecraft version for reproducible training and fair comparison of agents, which, however, becomes outdated quite quickly for human players.
Mods and editions of the game such as Minecraft Education, which are oriented towards studying programming, allow for normal playing simultaneously with in-game programming, but their API for artificial agents is quite limited and different from human control.
In our series of blog posts (see part 1, part 2), we started describing our neural-symbolic approach towards general-purpose AI agents in Minecraft. But is there any use of such agents besides advancing towards AGI? In fact, entertainment, educational and research aspects should not necessarily be separated. Neural-symbolic AI agents with explicit knowledge are a perfect fit to game-playing assistants and companions and can also be used for teaching AI.
Vereya Minecraft Mod
TrueAGI incubated within the ecosystem of SingularityNET aims at practical applications of neural-symbolic systems and especially OpenCog Hyperon, which include Minecraft AI assistants and companions. This requires developing a mod for Minecraft AI agents, which:
- Is compatible with the latest versions of the game,
- Supports API functions for AI agents (getting raw video, controlling the playing character emulating human control, etc.),
- Doesn’t interfere with normal game playing, when it is not needed.
With these goals in mind, we have been developing our Minecraft AI mod, Vereya. It was based on Malmö mod, but has considerably move away from it in order to support API of newest versions of Minecraft and to shift from agent-centric approach.
We also migrated from Forge to Fabric modding toolchain for Minecraft. Thus, in order for our mod to work, Fabric Minecraft Launcher is needed, as well as Fabric API should be placed into the mods folder together with Vereya jar file itself, which can be found on curseforge. Sources and the latest builds (including the required fabric API jar file) can be found on github.
Python Wrapper for Vereya
Vereya mod itself provides a low-level API for receiving observations and controlling the playing character. For more convenient higher-level interactions with Minecraft, a Python wrapper together with additional utilities exists.
Vereya inherits (some part of) the mission builder from Malmö, which is used to specify the world for an AI agent, available types of observations and commands, various conditions like allowed mobs, etc. While this might be useful for AI R&D purposes, Vereya is supposed to be used for connecting AI agents to an already running Minecraft world, or to start a new game in the default world. That’s why we skip the mission building part here.
For a quick start, one can connect to a running world of a modded Minecraft using the Python wrapper for Vereya as simply as:
from tagilmo import MCConnector
mc = MCConnector.connect()After connecting, one can send commands, e.g.
mc.sendCommand("jump 1")will make the character to jump permanently, until "jump 0" is sent. Other commands include "attack", "turn", "pitch", and others. A float number can be passed to "turn" and "pitch" to control rotation speed.
Besides commands, observations can be received. For example, mc.getLife() will return 20 for the healthy character, while mc.getAir() will return 300. mc.getAgentPos() will return something like [516.4354, 79.0, -199.0174, 11.550, 115.800], which includes both coordinates and orientation angles (in degrees) of the character. More info can be retrieved using mc.getFullStat function with such arugments as "Food", "WorldTime", "MobsKilled", etc. We also added such keys as "input_type", "isPaused", so an AI agent could check if it has control over the character.
mc.getLineOfSights() will return information about the object in the central cross, e.g. {‘axis’: ‘y’, ‘type’: ‘oak_log’, ‘hitType’: ‘BLOCK’, ‘x’: 509.0, ‘y’: 78.93280442941278, ‘z’: -202.611189065345, ‘distance’: 8.429007640237918, ‘inRange’: False}. If there is no object (e.g. the character is look at the sky) or it is too far, then the result will be {‘hitType’: ‘MISS’}.
mc.getChat() and mc.getInventory() will return the latest message from the game chat and the character inventory correspondingly.
It should be noted that in order to receive new observations, one need to execute mc.observeProc(). The tagilmo library has RobustObserver class with update_in_background method to update observations constantly in a separate thread, but we will not go into detail here.
If one executes mc = MCConnector.connect(video=True)the observations will also include raw video data from Minecraft. For example, the following script will output what the user sees beside control and GUI elements in a separate opencv window until Esc button is pressed:
from tagilmo import MCConnector
import cv2
if __name__ == '__main__':
mc = MCConnector.connect(video=True)
k = 0
while k != 27:
mc.observeProc()
if mc.getImageFrame() is not None:
cv2.imshow("minecraft", mc.getImageFrame().pixels)
k = cv2.waitKey(50)
These inputs and outputs are enough for creating AI agents like those we described previously. Additional information about the game world can be used by some agents and AI assistants/companions. In particular, mc.getRecipeList() and mc.getBlocksDropsList() return information about crafting receipts and items obtained by mining different blocks.
Such additional commands as mc.placeBlock() are added to support some forms of educational tasks for kids programming available in Minecraft Education, but more functionality can still be added.
Join Us
Vereya by itself provides only a low-level API, but it can be used both for creating AI agents and front-end components for educational and game-playing purposes.
The project is open-sourced. Its usage as well as contribution to its codebase and development higher-level libraries and solutions on top of it are welcome.
