Building your own No Limit Texas Hold’em poker bot in Python

Some time ago I came across Libratus, a bot made by Carnegie Mellon for playing heads up (1 vs 1) no limit hold’em. Libratus proved its worth by beating some of the world’s best poker players. I came to wonder just how difficult it would be to create a bot that would do well in larger, say 10 player, tournaments? Well, pretty hard, and for the time being, I’ve got some issues that I’ve yet to find a solution to. And also, I don’t have a super computer at my disposal for training the bots. But that’s a post for another day.

This post will guide you through implementing some simple bots for a 10 player tournament, and give the outline of a more useful bot that can be expanded with better logic.

To start implementing your own bots, you don’t need any machine learning knowledge. You will only need python knowledge. This article is written on Ubuntu, so some of the commands may differ from your setup.

Setup

In a terminal, create and enter a new directory named mypokerbot:
mkdir mypokerbot
cd mypokerbot

Install virtualenv and pipenv (you may need to run as sudo):
pip install virtualenv
pip install --user pipenv

And activate the environment:
pipenv shell

Now with the environment activated, it’s time to install the dependencies. I will be using PyPokerEngine for handling the actual poker game, so add this to the environment:
pipenv install PyPokerEngine

If you are curious about PyPokerEngine, I highly recommend reading the docs.
Verify that PyPokerEngine is correctly installed by creating a file, main.py (touch main.py in the terminal) and add these two lines to it:

import pypokerengine
print("hello world")

Run with:
pipenv run python main.py
If running the code writes “hello world” with no errors, everything is ready.

Note: If you don’t want to create virtual environments, you can just install the dependencies using pip and run the scripts normally.

Creating simple players

To get started, download the players random player, fish player and console player from the PyPokerEngine GitHub repository, and name them randomplayer.py, fishplayer.py and consoleplayer.py respectively inside the mypokerbot directory.
The random player, well, performs random decisions. The fish player always calls, and the console player let you participate using the console.

Once downloaded and put inside the directory, update main.py to now contain:

Run main.py with no errors to verify that the everything gets imported correctly
pipenv run python main.py

Running a game

Now some simple bots are available. So let’s make some of them play each other, which is where PyPokerEngine will be used.
Update main.py to now contain:

This creates a tournament that runs at most 100 rounds, in which each player starts with 1000 chips, and the small blind amount is set to 20. 10 players are registered, five fish players and five random players. Running this will run the tournament for the specified 100 rounds, or less, if a player has won by then.

When running this, you will likely see fish players win most of the time, meaning that performing random actions is a bad choice. Or, at least worse than just calling everything.

Participating through the console

In this section you will see how to participate in the game through the console. If you don’t care, feel free to just skip it.

Recall that you added the console player. To participate, change one of the player registration lines to now be:
config.register_player(name="c1", algorithm=ConsolePlayer())

When running the game, hit the enter key until it’s your turn. When it’s your turn, you can enter either f (fold), c (call/check), or r (raise) and enter the amount to raise. However, note that if the raise option says [-1, -1], raising is not an option.

Participating in the game

Adding a smarter bot

The bots so far have been very simple with absolutely no clue on whether they have a good hand or not. Luckily, PyPokerEngine provides tools for estimating hand strength, which will be the basic of a new bot, the honest player (which is also covered in the docs).
Add a new bot named honestplayer.py containing:

The methods in honestplayer.py are the methods that you can override when creating new bots. The different methods are:

• declare_action: When you are asked for an action, this method is called. You will have to provide your action and an amount. In this bot, we simulate 1000 games to see how many we will win. If the probability of winning the game is greater than 1/num_players (10% for 10 players), we will call. Otherwise we fold. This way, we only play the good hands.
  Note that the fold-action is in place 0 in the valid_actions array, call/check is in place 1 and raise is in place 2.
• receive_game_start_message: Called when a new game is started. This is only called once per game, and the game_info dictionary contains various game info, like the number of players.
• receive_round_start_message: Called preflop when a new round starts. In here, you have the round count, the cards on your hand (holecard) and information about the other players in the game.
• receive_street_start_message: Called when a new “street event” happens, i.e. pre-flop, at flop when community cards are shown, and when turn and river are shown
• receive_game_update_message: Called each time a player performs an action
• receive_round_result_message: Called when a round is over. This is either after showdown or after all players fold. When implementing more advanced bots, this method will be used for updating the bots.

You have now created your first non-trivial bot. If you wish to do so, you can tweak different actions for different scenarios, like going all-in every time you have at least 95% probability of winning the hand or have a pair of aces at the flop. Actually, you should do something like that, since right now it will only play the very best hands and get ruined by blinds. Also, there’s no reason to fold when it costs nothing to check, but the bot does so at the moment. I’ve left these two tasks as exercises for the reader :)

If you wish to try out the bot, import the bot in in main.py by importing it at the top of the file:
from honestplayer import HonestPlayer

And change one of the player registration lines to be:

config.register_player(name="h1", algorithm=HonestPlayer())

Using the HonestPlayer will slow down the game severely, since the honest player performs the many game simulations. If it is too much of a slow-down, consider lowering the NB_SIMULATION value.

Playing through a GUI

The creator of PyPokerEngine has been kind enough to also create a GUI to play through. Currently, the creator’s PyPokerGUI will only work in python 2. Thus, if you are using python 3, this currently doesn’t work. If you want to play through the GUI and are using python 3, I’ve forked the repository and fixed the bugs. Clone this repository and use it if you want to try it out.

Summary

Obviously, none of the bots are complicated and they won’t beat any human players, but the honest bot can be used as basis for creating a decent rule-based bot, and the bots serves as a basis for getting started creating your own bots using PyPokerEngine, for which this post hopefully has been a nice introduction.

If this post have peaked your interest, I have compiled a little list of some suggested reading if you want to create self-learning bots based on modern machine learning approaches. If not, thanks for reading!

The code used in this post is available on GitHub.

Suggested reading

• Reinforcement learning: This is the area in machine learning that deals with reward-based optimizations. This is how dogs are trained; when it finally successfully sits and you give it a treat — the reward (and when it fails, you punish it, in a dog-friendly way). All of the huge game-based machine learning breakthroughs recently (like AlphaGo and Libratus) are based on different reinforcement learning algorithms.
• PyPokerEngine docs: I can only recommend reading through these. I’ve shown some of the most basic functionality, but the library really contains many more features than listed. For instance, it contains an emulator that enables you to really control the game and makes reinforcement learning easier.
• DeepStack: A different reinforcement learning based bot. This is a much more well-documented approach than Libratus. If you are fine with academic level papers, this is a good place to go
• Libratus paper and their endgame solver: The first paper is a short summary of the inner workings of their bot. The second outlines an approach to solving the end-games of imperfect information games (games in which you don’t know all information, like not knowing the opponent’s cards in poker. This is unlike chess and go, where all information is available to everyone)