We Spent a Year Finding Out What AI is Bad at
How many arms am i holding up!? 😳
I was designing NLP solutions when NVIDIA was still about making sure the latest Call of Duty could run in 4k and well before our parents were talking about getting recipes from ChatGPT. In 2021 we helped launch"The Bean" for Choose Chicago, a conversational AI that let visitors ask the city's most famous sculpture for directions and dinner recommendations, more than a year before ChatGPT became a household name. Since then, the models have changed. The method didn't: companies say the machine can do the thing, we watch it fail, we write down exactly how, and figure out what job it was actually built for.
Last year on this blog I published five lessons from building Harker's Escape, our LLM-driven Dracula “escape room” game. The short version: dropping an LLM into a game doesn't make it better. Full, working generative 3D gameplay is out of reach. LLMs are brilliant at understanding what a player means and terrible at staying coherent about it. Mixed input methods are a pile of confusion. And Lesson 5 ended on an open question we called Lindstrom's Door: can LLMs actually follow the rules we set for them? Then one final thesis: gen AI for the sake of gen AI doesn’t make a game better. But building systems that allow more freedom, interactivity and infinite possibility is a dream worth chasing.
So, for the past 12 months, we’ve chased that dream.
Those months were a catalog of failures, ours and the technology's, across two games and thousands of manual and automated tests. Which brings me to the thesis I came back to state plainly: AI is genuinely bad at a lot of things. Not temporarily bad. Bad in ways that are structural and repeatable, and, if you can control your frustration…which hasn’t always been easy…bad in ways that are useful, because every failure tells you what the technology’s actual job should be. The teams shipping real products that leverage gen AI aren't the ones who believe hardest. It’s those who believe precisely. They're the ones with the most accurate list of what it can't do.
I'm publishing our list. Every entry is a true story from production, and each one cost us real time, real money, or both before it taught us anything. To give you a taste:
The QA system we built to catch bad AI output. First it wrote its own checklist, hallucinating features our characters never had. Then it failed honest images, over and over, for missing them. The inspector was dreaming, and grading everyone else against the dream.
The image model that was holding a correct reference picture and still drew a three-headed monster, because a corrupt line of text told it to.
The pipeline that lost its music, its quality checks, and its safety nets, and never threw a single error. AI systems don't break loudly. They stop working politely.
The AI storyteller that severed a creature's limb in one scene and had it swinging punches in the next. Continuity is a systems problem, and we have the ledger to prove it.
The AI referee we audit for bias like it's up for Senate confirmation, including whether it can be swayed by nothing but adjectives.
And the one nobody likes to write about: what all of this actually costs to run, per fight, per scene, per penny, and why most AI products couldn't tell you their own number.
None of these are game problems. They're what happens anywhere someone chains this technology together and expects it to hold. Games are just where the failures are vivid, and where we have the receipts.
The stories are already written; they happened. The detailed list posts will land as the work allows.
And, after all this time, Lindstrom's Door gets its answer. It turned out to be the whole company.