From Loki to Libratus: A Look at 20+ Years of Poker AI Development

The rise of poker AI has been a strange journey to say the least. Early in 2017 Libratus finished dismantling a human team of heads-up wizards including Jason Les, Dong Kim, Daniel McAulay and Jimmy Chou. It was arguably the most dominant performance by an AI playing poker and a landmark achievement for non-human poker. It’s also been a long time coming as software developers have been tinkering with poker software since the early days of computers.

Poker is a unique challenge for AI developers because it’s an incomplete information game compared to something more static like chess.

So why bother putting in all the work? Computer scientists believe that solving a game like poker could have real-world applications in areas like negotiation, healthcare and more.

We’re going to take look at the long, rather strange, development path that’s taken as all the way to Libratus below.

The Long, Strange Path of Poker AI

1984

Mike Caro Shows Off Rudimentary Poker Software

Poker pro Mike Caro wrote a computer program he called Orac that competed against several pros at the 1984 WSOP.

Caro spent two years developing Orac (that’s Caro backwards, btw) on a glorified Apple II.

Orac was simple by today’s AI standards but it actually managed to beat Doyle Brunson in one match.

Interestingly Orac actually to scan physical cards that had a bar codes. That game itself didn’t take place on the computer.

Orac also took on Bob Stupak at the Stratosphere in a promotional match and got hit with a serious bad beat. In the first match of the best-of-three series Orac moved all-in and Stupak called.

Orac flopped trips but, according to Caro, someone kicked the power cord out and the machine had to be booted up again, which re-set the match. Shady.

1991

University of Alberta Team Begins Work

The University of Alberta Computer Research Group, which would go on to be one of the biggest developers of poker AI, begins their work on poker bots.

The group included a rotating number of faces including game scientist and part-time poker pro Darse Billings in addition to Denis Richard Pap, Jonathan Schaeffer, Duane Szafron, Michael Bradley Johanson, Neil Burch and others. Michael Bowling would join the team later on and become a huge player in the poker AI world.

1996

Deep Blue Conquers Garry Kasparov

IBM’s Deep Blue plays chess world champion Garry Kasparov (and later defeats him).

This sparks a dramatic increase in researchers programming AIs to defeat humans in traditional games like Go or poker.

Poker is markedly different than chess, however, because it’s a game of imperfect information that’s more difficult for computers to master.

Interestingly the Deep Blue project (originally Deep Thought) began at Carnegie Mellon University by Feng-hsiung Hsu.

Carnegie Mellon University would go on to play a huge part in developing poker AIs.

1997

UoA Releases Loki, One of the First Proper Poker A.I.’s

The University of Alberta Computer Research Group, introduces Loki, and uses the legendary rec.gambling/poker forum as a resource in their research.

Game scientist and part-time professional poker pro Darse Billings and two students develop Loki under the guidance of University of Alberta computer scientists Jonathan Schaeffer and Duane Szafron

Loki is designed to play a full table (nine players) poker, which (as of 2017) has yet to be fully solved by computers.

As with most early AIs, Loki was also focused on Limit Hold’em instead of the popular No-Limit variation.

Initially the team was hopeful Loki would someday be advanced enough to compete in the WSOP but Caesars would eventually change the rules to the competition to keep it humans only (it didn’t help that a company tried to buy a monkey into the Main Event in 2006).

Loki was the first in a long line of AIs that would have a huge impact on the poker world.

Loki’s skill level is slightly below an average human poker player.

1999

Loki Turns into Poki, Gets Video Game Contract

The University of Alberta team behind Loki decides to re-christian the bot Poki and shift focus to the two-player game of Texas Hold’em, which has less variables. Poki can play poker at the level of an average poker player.

Many of these early bots (and later ones too) are centered around the concept of a Nash equilibrium, which, put simply, is about making the best possible decision while taking into account the other player’s decision.

Poki would eventually be licensed for the video game Stacked, which also featured Daniel Negreanu’s likeness.

2002

PsOpti/Sparbot Shows Potential for UoA

The University of Alberta unleashes a collection of bots including PsOpti and Sparbot that attempt to solve the heads-up limit poker.

Despite encouraging results none of the bots are better than intermediate strength at two-player Hold’em.

The team uses professional poker pro Gautam “thecount” Rao as an opponent for the AI. Rao had this to say at the time:

“You have a very strong program. Once you add opponent modeling to it, it will kill everyone.”

The U of A spends the next couple years working on opponent modeling or learning how an opponent plays.

2003

Poker AI Developers Transition Away from Chess Model

This was a pivotal year for the development of Poker AI as researchers began to shift their focus away from chess methodology that led to success in AIs like Deep Blue.

Also of note was that Michael Bowling, who did his PHD work at Carnegie Mellon, goes to work at the University of Alberta where he will be the driving force behind their computer poker AI research for the next 10+ years.

2004

Carnegie Mellon, Tuomas Sandholm Begin Poker A.I. Work

Carnegie Mellon University and professor Tuomas Sandholm, the driving force behind the recent Libratus AI, enter the fray by beginning their work on poker AIs.

Over the years Andrew Gilpin, Sam Ganzfried, and Noam Brown also make large contributions to Sandholm’s research stream.

In other news ICCM Poker Bot Challenge hosts No-Limit Hold’em tournament for various bots from around the world. Ace Gruber of the University of Toronto takes the competition.

2005

The World Series of Poker Robots

The poker boom is in full swing and Golden Palace hosts a promotional event at Binions with a $100k first-place prize. There were six different entries.

PokerProbot, designed by a 37-year-old car salesman Hilton Givens from Indiana, emerges victorious. PokerProbot narrowly defeated Poki-X, which was a hastily put together version of the University of Alberta’s famed Poki AI (an update of Loki).

Human pro Phil Laak also beat PokerProbot in a heads-up exhibition match during the competition.

“In three to five years, they’re going to win,” said Kenneth “The Clone” Jones, a poker pro and sometime software engineer told the LA Times at the time.

2006

The Annual Computer Poker Competition Begins

Annual Computer Poker Competition (ACPC or as it was originally known ACM) begins.

Both of the heavyweight teams from the University of Alberta and Carnegie Mellon University go on to compete and win various awards over the course of the competition as well as some lesser-known universities and independents.

2007

Polaris Takes on Phil Laak

The University of Alberta debuts Polaris, which goes on to become one of the most famous poker bots thanks to a heads-up match against Phil Laak (which it lost, although it was close).

Polaris is actually a composite program which consists of a number of bots working together (including the highly-touted Hyperborean08).

The program contains a number of fixed strategies and chooses between them during a match.

Interestingly Polaris is not particularly intensive when it comes to computing power and can be run using consumer-level products like a MacBook Pro.

Polaris only plays two-player heads-up No-Limit Hold’em.

2008

Polaris Bests Six Humans in Informal Test

Polaris competes against six human players during the Gaming Life Expo and posts a record of three wins, two losses and one tie.

The 2008 edition of the bot was upgraded significantly from its predecessor, which competed against Phil Laak in 2007 (but lost).

2009

University of Auckland Introduces Sartre

The University of Auckland begins work on Sartre (Similarity Assessment Reasoning for Texas Hold’em via Recall of Experience).

Sartre would go on to be a major competitor in the AI world and placed well in competitions over the years.

It was one of the rare successful poker AIs that didn’t come from the Carnegie Mellon or University of Alberta teams.

You can still compete against Sartre online.

2011

Poker Sites Full Tilt and PokerStars Crackdown on Bots

For years bots were simply not good enough to compete against humans in online poker but with improved AIs it was only a matter of time before opportunistic programmers took a shot at the potentially lucrative market.

Around 2008 rumors began to circulate about bot activity on several poker sites.

Darse Billings, of the University of Alberta computer team, asserts that most poker bots are very bad and more than 90% are actually losing money.

That doesn’t stop companies like Shanky Technologies openly selling programs that could compete in Hold’em and PLO.

In all came to a head in early 2011 when heavyweight operators PokerStars and Full Tilt made a huge effort to effectively remove bots entirely.

When a player is identified as a bot they are immediately banned and their funds are confiscated.

Bots are no longer a major issue on most poker sites thanks to advanced human-recognition software.

2014

Tom Dwan Takes on Limit Heads-Up Machine, Wins Big

Tom Dwan allegedly wins a huge sum of money by beating a Limit Heads-Up poker machine called Texas Hold ‘Em Heads Up Poker.

The was designed by IGT, a manufacturer of slot machines and video poker machines. Supposedly the machine utilizes a neural net to learn new strategies.

The software for Texas Hold ‘Em Heads Up Poker was designed by Fredrik Dahl of the University of Oslo.

The machine was later endorsed by Phil Hellmuth and Johnny Chan but never caught on in a widespread way, at least compared to traditional slot machines.

Bellagio, in Las Vegas, is still home to one of the Hellmuth machines.

2015

Cepheus Solves Limit Hold’em – University of Alberta

After years of development the University of Alberta finally releases a bot that has essentially solved heads-up Limit Hold’em in the winter of 2015.

Cepheus mastered Limit Hold’em by playing itself for two months. Of course Cepheus had excellent pedigree coming from a long line of famed bots including Loki, Poki, Vexbot, Hyperborean, Polaris and the rest of the U of A lineup.

It’s a landmark for poker AI as Limit Hold’em is the largest imperfect information game to be essentially “solved”.

Cepheus is erroneously referred to as “unbeatable” at the time. In reality Cepheus can lose money on occasion but is unlikely to be beaten over a large sample size.

Despite the fact that Cepheus has a firm grip on Limit Hold’em, the No-Limit version remains unsolved and some scientists believe it may be that way for years to come thanks to the unpredictable nature of the game.

The goal behind Cepheus and other similar AIs is to use it for other applications such as helping governments by improving security strategies or negotiating tactics. Or to help doctors modify treatments for their patients.

You can test Cepheus yourself by going to the UoA website.

Claudico Loses to Humans in Brains vs. AI Challenge

Not to be outdone by the U of A, Tuomas Sandholm and Carnegie Mellon release their own super smart poker A.I. named Claudico in the summer of 2015.

The big difference between Claudico and Cepheus is that Claudico plays No-Limit Hold’em, which is much harder to master.

The Carnegie team issues a public $100,000 challenge to a group of human professional poker pros that included Doug Polk, Jason Les, Bjorn Li and Dong Kim to compete against Claudico in a span of 20,000 hands per player over a 13-day period at Rivers Casino.

In the end the human team prevailed. They finished with a figurative profit of $732,713 over Claudico. The AI was renowned for its odd bet sizing.

Jason Les told PokerListings after the match that he felt it was only a matter of time before A.I.s started to beat humans.

2017

University of Alberta Unleashes DeepStack AI

The U of A team starts 2017 by releasing a paper that claims its new DeepStack AI, is the first AI to beat professional poker players in heads-up No-Limit Hold’em.

DeepStack is a new algorithm that utilizes advanced and the ability to learn from self-play using deep learning similar to the famous AlphaGo AI that beat the famously complex game of go.

DeepStack employs deep neural networks to emulate human intuition and learn on the go.

The study included dozens of participants (although none were as famous as Doug Polk or Dong Kim) and 44,000 hands of poker. There were also cash incentives for the top three performers.

DeepStack is particularly notable because it was able to become a winning poker player with no training from expert poker players.

The study has yet to be peer reviewed, however, and the U of A team is still waiting to discuss it.

Tuomas Sandholm, of Carnegie Mellon, tells Wired Magazine the U of A study doesn’t settle the matter because DeepStack played good players not great ones like Claudico/Libratus.

Libratus Crushes Humans in Brains vs. AI 2

In January of 2017, Libratus finally dealt arguably the most decisive blow in the history of human vs. poker AI competition.

Libratus, a brand-new AI from Carnegie Mellon, didn’t just beat its human opponents… It destroyed them.

By the time the final hand was dealt in the 20-day, 120,000-hand challenge, Libratus was up a staggering $1,766,250.

The human team was comprised of some of the best heads-up NLHE players in the world including Dong Kim, Dan McAulay, Jimmy Chou and Jason Les.

Kim, who had the best record against the AI only losing $85,649, admitted he felt outmatched at times.

“I felt like I was playing against someone who was cheating, like it could see my cards,” he told Wired Magazine. “I’m not accusing it of cheating. It was just that good.”

Although Libratus, which means “balanced” in Latin, was a nominal successor to Claudico, it was actually written from scratch.

Libratus didn’t utilize a fixed built-in strategy and instead relied on a algorithm that computes the strategy.

Players noticed distinct changes in the way that Libratus played each day, which might have had something to do with the fact the AI would analyze its own play and every night and correct mistakes.

Human players can take solace in the fact Libratus was powered by the massive Pittsburgh Super Computer, which is about 7,250 times faster than the average laptop.

In addition Libratus is purely a heads-up AI and adding more than one opponent would be an entirely different (and more difficult) task.

Sandholm is hopeful the technology behind Libratus will have many real-world applications.

Poker Bot Cheat Sheet

Here are some of the most famous poker bots:

University of Alberta

Loki
Poki
PsOpti/Sparbot
Vexbot
Hyperborean
Polaris
Cepheus

Carnegie Mellon University

Tartanian
Claudico
Libratus

University of Auckland

Casper
Sartre

Independent – Fredrik Dahl

Texas Hold’Em Heads-Up Poker