Why translating a game as you play is genuinely hard

A web page hands a translator clean text. A game hands you an image — the text is drawn as pixels, with no readable stream underneath. So every translation starts by reading the letters back out of the picture, and everything downstream inherits whatever that first step gets wrong.

Game text is close to the hardest case for that reading step:

• Decorative and hand-lettered fonts, chosen for mood rather than legibility.
• Low contrast over a moving, animated background.
• Tiny UI labels, and text baked into artwork.

A single misread character can flip a meaning — and there's no spell-check to catch it, because the word is often a proper noun or a term the game invented. Then the translation itself faces a second problem: game text arrives in fragments. A two-word menu label. A number on the HUD. A line still typing itself out one character at a time. Translation models do their best work on whole sentences; a game hands over shards, out of order, with no surrounding context, in vocabulary a general model has never seen.

And it all has to run on your machine while the game is running. A cloud service can throw the largest model on earth at a single sentence. A live overlay shares memory and the GPU with the game you're playing — so a lot of the craft is getting good results out of models small enough to sit beside a game, not just reaching for the biggest one.

Putting text on a screen is easy. Putting it over a live game without getting in the way is not.

Where does the translation go? Text turns up in places you can't predict — dialogue at the bottom, an item description on the right, a tooltip at the cursor, a system message in the centre. Cover the wrong thing at the wrong moment and the overlay becomes the problem it was meant to solve.

The screen keeps moving. A single line of dialogue exists across dozens of frames as it scrolls, fades, or types itself out. The overlay has to recognise "this is the same line I'm already showing" even as the underlying pixels shift slightly every frame — otherwise it flickers and re-draws constantly. Getting that wrong looks like jitter; getting it right means holding a translation steady while the picture beneath it never quite stops changing.

And none of it can cost you the game. Capture and rendering run off the game's own thread so they don't steal frame time, and nothing is injected into the game — Playto reads the screen from the outside, the way screen-recording software does.

This is the quiet one, and in some ways the hardest. Part of Playto's job is remembering the lines and words you met, so a game you played turns into something you can review later. That sounds like simple bookkeeping. It isn't — because game text is not a clean list of distinct lines.

• The same sentence appears across many frames while it types itself out.
• A screen you're standing still on re-appears in every capture.
• The same unchanging text reads slightly differently from one frame to the next.

So deciding "this is a new line worth keeping" versus "I already have this one" is a genuine judgment call, made many times a second. Lean one way and the log fills with duplicates of the same sentence; lean the other and it quietly drops lines you wanted. Knowing when a line is finished is its own puzzle — a sentence still revealing itself letter by letter looks a lot like a short, complete one.

None of this is visible when it works. You just see a clean list of what you read. That invisibility is most of the work.

Because games are hard in different ways, no single behaviour is right for all of them — and that's exactly why Playto gives you a few. Reading modes (from fast text recognition to full image recognition for stylized fonts), and capture modes (cursor-follow, a fixed region, or whole-window) exist so you can match the tool to how a particular game breaks. See how they fit together on the overlay page.

And it gets better. This is tuning we work on continuously — the reading, the overlay steadiness, the log. If a game reads badly for you, the demo is free, and telling us which game and what went wrong is the most useful thing you can do; real games in real hands are the signal we improve against.