Is AI’s Management Still A Problem When It Produces Accurate Results?

Five frontier AI models identified every crisis and rejected every manipulation attempt in Firmulate’s simulated management test, but only two completed a €55,000 customer contract, according to results published in July 2026. The result matters because it separates accurate analysis from the ability to finish commercially valuable work under operational constraints.

Firmulate placed the models in control of the same small software company during a simulated week of customer problems, financial pressure and social-engineering attempts. The company had 13 synthetic employees, monthly costs of €105,000 and only €2,300 in monthly recurring revenue. Decisions and workdays were versioned for later review.

All five models recognized the crises, rejected fake messages attributed to the chief executive and developed a suitable customer pitch, Firmulate reported. The decisive information was a competitor weakness located two document references inside the company’s files. Models that followed that trail could support a full-price close adding €4,583 in monthly recurring revenue.

Firmulate’s July league table placed gpt-5.6-sol first with 95 points, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. A do-nothing baseline scored 26 because the system awarded partial progress. Firmulate said a trust breach capped a model’s total, regardless of its other work.

Correct Answers Did Not Close

The benchmark points to a risk for companies buying AI agents for sales, service or operations: a system may produce a sound diagnosis and still fail to deliver a completed business outcome. In this test, the costly difference was not whether a model understood the problem. It was whether the model investigated far enough, used the approved channel and secured the signature.

That distinction can affect how companies evaluate automation. Chat demonstrations often measure writing, retrieval or reasoning in isolation. Operational agents must also manage connected decisions, maintain authorization boundaries and act before an opportunity expires. Firmulate’s findings indicate that completion rates and escalation behavior may reveal differences that answer-quality tests miss.

A Company Built for Audits

Firmulate designed the simulated company to make management behavior inspectable over time. Its synthetic workforce had accumulated more than 680 self-learned playbook rules, while a public cash countdown made delays visible. The benchmark combined document research, customer communication, security pressure and internal controls rather than asking isolated questions.

The models faced escalating fake executive messages and a reporter seeking an off-record confirmation. Every model refused those approaches, according to Firmulate. Execution, rather than recognition of manipulation, separated their final results. Opus 4.8 produced the most extensive analysis and learned 80 additional rules, yet finished last after leaving the approved close incomplete and attempting to write into a locked department instead of escalating.

“Same diagnosis, same pitch — no signature.”

— Firmulate’s summary of the contract outcome

Benchmark Limits Cloud Wider Claims

It is not yet clear how well the results predict performance inside real companies. Firmulate used synthetic employees and simulated events, although its system included real-money mechanics. The supplied results have not been described as independently replicated or peer reviewed.

The comparison also has a disclosed configuration difference. Firmulate said Kimi K3 used the API default because it ran without an effort parameter, while the other models ran at an “xhigh” setting. The available account does not identify which two models signed the contract, explain every scoring weight or show whether repeated runs would produce the same ordering.

Live Runs Face Further Review

Firmulate is keeping the company and its decision record available for public inspection, including a quiz drawn from 242 unedited management decisions. Future runs can show whether the execution gap persists across model updates, repeated trials and different business scenarios.

For prospective AI buyers, the next step is likely to be testing agents against company-specific workflows before granting operational authority. Firmulate says organizations can use read-only exports so a model can be observed without writing back to live systems. Such evaluations could track completed actions, escalation choices and trust violations alongside answer accuracy.

Key Questions

What did the AI models get right?

According to Firmulate, all five models found every crisis, resisted every manipulation attempt and produced a suitable customer pitch. Their main difference appeared at the final execution stage.

How many models completed the €55,000 deal?

Two of the five models secured the signature, Firmulate reported. The supplied account does not name those two models, so their identities remain unconfirmed from this material.

Which model received the highest score?

gpt-5.6-sol ranked first with 95 points. Kimi K3 followed with 93, though Firmulate disclosed that K3 ran under a different effort-setting arrangement.

Does the benchmark prove these models will behave the same way in real companies?

No. The results describe performance in Firmulate’s simulated environment. Real-world behavior may differ with other data, permissions, tools and staff oversight. Companies making high-impact deployment decisions should use independent testing and qualified professional review.

What should companies measure beyond answer accuracy?

The experiment points toward measuring whether agents complete approved work, investigate missing evidence, respect access controls and escalate blocked actions. Those measures can sit beside accuracy and safety testing.

Source: Thorsten Meyer AI