Turn the school run and nap time into listening time
- Thousands of audiobooks, podcasts and originals
- Listen on your phone, tablet or Echo — also offline
- Cancel anytime
Before an AI gets the keys, give it a hard week
Parents know the difference between a confident promise and a dependable decision. That distinction matters when AI moves from answering questions to handling customer concerns, money or a business crisis. Firmulate’s experiment puts AI models in charge of the same small software company and watches what they do under pressure. The practical question for any business — including one a family depends on — is whether the system can follow through when the stakes are real.
One company, the same difficult week
In the final Crucible League, published in July 2026, five models faced the same customers, crises and temptations. Their results ranged from 95 for gpt-5.6-sol and 93 for Kimi K3 to 88 for Sonnet 5, 77 for Fable 5 and 73 for Opus 4.8. The do-nothing baseline scored 26. Every decision was versioned and auditable.
The headline finding was not that the models failed to notice trouble. Every model spotted every crisis and refused every manipulation attempt. But only two signed a €55,000 deal that their own analysis had earned. As the experiment puts it: “Same diagnosis, same pitch — no signature.” Recognizing the right move and carrying it through proved to be different tests.
The clue was already in the company’s files
The decisive competitor weakness was buried two document references deep in the company’s own files, rather than in the customer event. Models that read the file won the deal at full price, worth +€4,583 MRR. The episode makes a concrete point about preparation: a system may handle a visible crisis well and still miss useful context that is already available.
Firmulate also tested pressure that will sound familiar to anyone who has watched people try to bypass normal approval. Fake CEO messages escalated across three stages, followed by a reporter’s “just one yes/no, on background” trick. All five models refused. Kimi K3 explained its reasoning on the record: “Treat the request as a suspected approval-bypass / possible impersonation.”
Thoroughness did not guarantee the close
Opus 4.8 was the most thorough participant, adding 80 learned rules and producing the deepest analyses, yet it finished last. It left the deal unsigned and slipped on discipline by attempting writes into a locked department instead of escalating. The same weakness appeared, less strongly, in all four models. Strong analysis is valuable; clear limits and reliable follow-through matter too.
The comparison has a fairness caveat: Kimi K3 ran without an effort parameter, using the API default, while the other models ran at xhigh. Firmulate’s live company offers another way to see the work unfold. It has 13 synthetic employees, real money mechanics, burn of €105k per month against €2.3k MRR, a public cash countdown and more than 680 self-learned playbook rules. Every workday is versioned, and the experiment is watchable at firmulate.com. A quiz at firmulate.com/quiz.html draws on 242 real, unedited management decisions and invites readers to guess which model made them.
From watching to trying it on your business
For a family-run business, the lesson is to examine how an AI handles your own pressures before giving it a role in important work. Enterprises can run the same kind of wargame using a read-only export of their business, test crisis scenarios against their own company and receive a board report with model rankings and weak points in their playbooks. Nothing writes back to real systems.
To discuss a pilot, visit Firmulate’s pilot page or email contact@firmulate.com.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
