I play a lot of Tetris 99.

I am decent at it. I am not “win the lobby” decent.

So I had a perfectly reasonable thought:

What if my computer played it for me, on the real Switch, pressing the real buttons?

Three days later I had:

• a robot that finishes in the top 10 against 98 real humans
• an Arduino that introduces itself to Nintendo hardware under a false name
• a Tetris coach that draws on top of my own live games
• three confident theories about why things were broken, all of them wrong
• soldered exactly three header pins, badly

This is the story, with drawings.

The original plan

Here is what I actually proposed on day one. I want this on the record.

A camera pointed at the TV. A computer that watches the camera. An Arduino driving a little forest of servo motors, each with a rubber fingertip, physically mashing the buttons of a controller.

It is a beautiful idea. It is also seven mechanical problems wearing a trench coat.

Glare. Moiré patterns from filming a screen. Rolling shutter. Auto-exposure deciding mid-game that yellow is now orange. A tripod nobody is allowed to walk past. And seven servos, each of which needs to press hard enough to register and gently enough not to launch the controller off the desk.

I built this project pair-programming with Claude, Anthropic’s AI, in Claude Code. Its first contribution was to politely describe my plan as “the hardest part of this whole project, harder than the AI,” and then suggest two boring purchases instead.

The saner plan

Eyes: a $15 HDMI capture card. The Switch’s dock outputs HDMI. A capture card turns that into a webcam-shaped device on the PC, except every pixel is exact. No glare, no perspective, no tripod. The board sits at the same pixel coordinates in every frame, forever.

Hands: an Arduino pretending to be a controller. This is my favourite part.

When you plug a USB device in, the host asks it, “what are you?” and the device answers with a few small tables: a vendor ID, a product ID, and a description of the data it will send. For a simple gamepad, that is the entire conversation. No handshake. No encryption. No secret knock.

The Switch accepts certain licensed third-party wired controllers, and one of them, the HORI Pokken Tournament pad, is a completely plain USB gamepad.

So a $10 board with an ATmega32U4 chip says: “Hello. I am vendor 0x0F0D, product 0x0092. I am a HORI controller.” And the Switch says: “Welcome.”

After that it just sends 8 bytes, 125 times a second: two bytes of buttons, a d-pad, four stick axes. A tap of A is one bit that turns on for two frames.

It must be a 32U4 board (Leonardo, Micro, Pro Micro), because that chip has USB hardware the firmware controls. An Arduino Uno cannot do this.

I know this very precisely, because the first board that arrived was an ESP32-C3 somebody had listed as an “Arduino”. It can only ever be a serial port. Back it went.

There was one wrinkle. The Arduino’s only USB port is busy lying to the Switch, so the PC talks to it through a little CP2102 USB-to-serial adapter and three wires. Mine arrived without header pins.

I soldered three pins. It worked first time. I am choosing not to examine how.

The brain is not mine

I did not write a Tetris AI, and you should not either.

Cold Clear, by MinusKelvin, is an open-source versus-Tetris bot written in Rust, and it is genuinely strong. It finds every reachable placement for a piece (including the spins you can only reach through wall kicks), scores the resulting boards with a hand-tuned evaluation, and searches several pieces ahead through the next queue.

It even rewards boards that contain a T-spin slot before a T piece arrives, which is why the bot builds those on purpose. A T-spin double sends as much garbage as a Tetris while using two lines instead of four. That single fact explains most of how strong players play.

My job was everything around it: eyes, hands, and nerves.

Teaching it to see

A 1080p frame is six million pixels, sixty times a second. Python cannot look at all of that. It does not need to.

The board is a 10×20 grid at known coordinates. Sample a tiny patch in each cell, convert it to HSV (hue, saturation, brightness), and ask three questions. Dark? Empty. Bright but grey? Garbage. Vivid? The hue tells you which piece.

The first version took 344 milliseconds per frame. That is three frames per second, which is a slideshow, not a robot. Converting each region once instead of thousands of tiny patches brought it to 2.4 ms. Decoding the video on a background thread took capture from 30 to a true 60 fps.

Then the game started lying to it.

• The “GO!” banner got read as floating blocks. On piece one.
• The Targeting menu is drawn on top of the board, and its glowing pill was read as yellow O pieces.
• Garbage blocks are darker than I assumed. Brightness 111; my threshold was 115. The bot could not see garbage at all, and cheerfully planned placements deep inside it.
• In battles, yellow attack lines fan out from the corner of the board, straight across a cell’s centre. One sample per cell got fooled. Now each cell takes five samples and they vote.
• I noticed battle mode is full of sparks and flashes. Sparks move every frame. Blocks do not. So at each new piece, the bot reads five frames in a row and takes a majority per cell.

My favourite trick is how it knows a piece has spawned. Finding the piece itself is unreliable: it can be half hidden above the visible rows. But the next queue shifts at the exact frame a piece spawns. So the bot watches the queue, not the piece.

The first games

Single-player first. 148 pieces, one mistake, about 1.5 pieces a second. I felt like a genius.

Then I let it into a real Tetris 99 lobby.

94th place. Out of 99.

Then 83rd. Then 89th.

Three theories, all wrong

This was the most educational part, so I will be honest about it.

Theory one: “the game ignores a hard drop right after a rotation.” The logs screamed this. So we ran a controlled experiment against CPU opponents: rotate, wait a precise gap, hard drop, repeat. 27 out of 27 registered, down to a 17 ms gap. The theory was dead. The real culprit was my own safety net: a watchdog that re-sent the hard drop if no new piece appeared quickly. When garbage rises, the next piece is late, so the watchdog fired, and its extra hard drop slammed the next piece straight down.

Theory two: “soft drops fail because gravity gets faster.” Plausible! Wrong. We saved raw frames during soft drops and looked. The pieces were perfectly visible. The bug: after pressing hold, the piece in play is the one that came out of the hold box, which is a different colour. The bot was watching for the colour of the piece that had spawned. It was staring at the hold slot, waiting for it to land.

Theory three: “shorter button taps will make it faster.” 25 ms taps were flawless in single-player. In a 99-player battle, entire moves silently vanished. 34 ms (two frames) is the floor. Single-player timing does not transfer.

The lesson I keep relearning: instrument first, theorise second. Every time we measured instead of guessing, the real cause was something nobody had suspected.

Then I gave it two pieces of advice

I am, after all, the one who plays this game.

1. Set targeting to K.O.s. It aims your garbage at players who are nearly dead, and the badges you collect multiply your attacks. The bot now flicks the right stick up at the start of every match.
2. Stop T-spinning when you are in trouble. T-spin setups are luxuries. Above the halfway line, just clear lines. The bot now switches personality by stack height: spin-hunter below row 10, pure survivor above it, and back again once it has dug down to row 6.

Results:

Note the 79th in there, from the same bot on the same night as a 10th. Tetris 99 is like that. Who decides to target you in the first minute matters more than anything you do.

I also tried a theory of my own: forget T-spins entirely, just keep the board clean. It finished 16th, 79th and 32nd. A clean board that sends no garbage gets buried by the last ten players. My hybrid advice was right; my purist advice was not. I have made peace with being one for two.

It accidentally became my coach

The part I did not plan: since the computer can already see my game and already knows what Cold Clear would do, it can show me.

In coach mode I play with the normal controller. I tap the space bar, and it draws every known piece (current, hold, and the six previews) where the engine would put them, numbered in order, right on top of the live video. t switches it to T-spin setups, a to perfect-clear hunting. Pieces vanish as I place them.

There is also an offline sandbox: a keyboard Tetris with undo, pause, a browser for every possible placement of the current piece, and lessons. One lesson is a T-spin triple that can only be entered through the fifth and last SRS wall kick, and it tells you which kick you used when you land it.

On working with an AI

Almost all of the code was written by Claude, with me steering, testing, soldering, and supplying opinions about Tetris. Once the hardware worked, it could see the screen through the capture card and press buttons through the Arduino, so it started navigating the Switch’s menus by itself: main menu, pick a mode, queue a match, play, read the results, go again.

It was not flawless, which I found oddly reassuring. It twice declared a match over from the logs and switched the bot off mid-game. It once tried to stop the bot with a command that matched its own shell, killed itself, and wondered why nothing happened next. It wrote all of this down in the repo so the next agent does not repeat it.

A word on manners

Most of the testing happened against CPU opponents, which is the right place for it. I ran a few dozen online matches to see how it held up against people. I am not going to grind ranks with it, and if you build one, I would ask the same of you. The fun is in the building.

Things you can try

Everything is on GitHub: omarali-me/tetris99player.

With no hardware at all. Clone it, build Cold Clear, and:

.venv/bin/python tools/sandbox.py --lesson 2     # the T-spin triple that needs kick 5
.venv/bin/python tools/watch.py --garbage-every 6   # watch the bot fight rising garbage

Give the engine a different personality. Cold Clear’s entire character is about 40 numbers. config/weights.json lists them all with notes. Copy it, make singles worth +500, and watch it play like a nervous beginner:

.venv/bin/python tools/watch.py --weights my_weights.json

See how colour classification works in six lines:

import numpy as np
from tetris99.vision.cells import classify_patch
purple = np.full((7, 7, 3), (200, 40, 160), np.uint8)   # BGR!
print(classify_patch(purple))                            # Cell.T

With a $15 capture card: the coach overlay works on your own games with nothing else. This is the best value-per-dollar in the whole project.

.venv/bin/python -m tetris99.loop --source 6 --trainer --mode tspin

With a 32U4 board: one script compiles the firmware with the HORI identity and flashes it. Before going near the Switch, plug it into a Linux PC: it shows up as a joystick under /dev/input/, and you can watch your fake button presses arrive. Then there is an on-screen controller you can click, with rebindable keys.

tools/flash.sh
.venv/bin/python tools/gamepad.py

Steal the stopwatch trick. To measure a game’s input timing, send a press, then count frames in the capture feed until something moves. That is how I learned Tetris 99’s auto-shift delay is 200 ms and its repeat is 33 ms: exactly 12 frames and 2 frames.

Where everything lives

• README: hardware list, setup, quick start
• COMMANDS.md: every command, flag and mode
• ARCHITECTURE.md: what lives in which file, and a table of every threshold, debounce and timing constant
• HISTORY.md: every bug, its real cause, the results, and what is still open
• The field guide: how Cold Clear’s search works, how USB HID works, how the OpenCV pipeline works
• Firmware notes: building, flashing, wiring, recovery
• CLAUDE.md: instructions for the next AI agent who picks this up

It still has not won an online match.

7th is close, though.

And I never did build the servo fingers. I am a little sad about that.

• Tetris
• Arduino
• Computer-Vision
• Nintendo-Switch
• Opencv
• Ai