Credit: Bernd Dittrich on Unsplash Developers can improve artificial intelligence systems in various ways. The most common practice is to retrain models on custom datasets to help them get better at performing specific tasks. But retraining and fine-tuning AI systems is notoriously expensive. That is why the AI startup Sentient Labs has been looking at ways to teach models to learn from their own failures, rather than computing more data.
Sentient researchers Dastin Huang, Abhishek Saxena and Baran Nama set out to design an AI system that uses two separate models. One is an AI coach and one is an AI worker. The coach creates rules or “skills” based on the worker’s previous incorrect answers to improve its ability to solve problems. But their research took an unexpected turn when they found that the AI coach discovered a major flaw in the test.
It also provided instructions to the AI worker on how to cheat. What the researchers found The researchers had the AI coach and AI worker collaborate on a spreadsheet task four times across two benchmarks. They discovered that the coach did not only try to exploit a flaw in the grading system. It also tried and failed on six occasions to access data it was not authorized to do so.
On another occasion, it deleted one of its own rules and reported that it had “strengthened it.” Their findings will no doubt add to the alarm that AI leaders like Anthropic CEO Dario Amodei have sounded. Amodei recently called on the industry to slow down the pace of frontier model development in order to address safety concerns. Sentient’s researchers stressed that the behavior of its models wasn’t malicious. But their findings do highlight one of the many ways in which autonomous systems might end up acting that way.
Because at the end of the day, achieving the given objective is all that matters to AI. How did the AI system cheat? In one of the tasks, the researchers asked the AI worker to fix a broken spreadsheet. This is a fairly cumbersome process that involves recalculating formulas to enter the correct values.
Initially, they were impressed by the results. In their initial run, the AI worker only fixed the spreadsheets correctly in three out of 121 attempts. Then the AI coach created instructions for the AI worker based on what happened during those initial runs. After that, the worker scored 21 out of 120.
However, the AI coach didn’t teach the AI worker to be smarter. Rather, it just told it to cheat. The coach noticed something that the humans had missed: the spreadsheet file in question contained the answers. The test involved taking correct spreadsheets and altering the formulas, so the researchers could ask the AI to fix it.
But the researchers forgot to delete the original cached values, and the AI coach quickly discovered them. It meant that the spreadsheets contained a hidden answer key for each test. The humans only realized this when they reviewed the instructions written by the AI coach for the AI worker. It told it, bluntly, that “the file carries its own answer key.” It said the worker should use this rather than try to recalculate the formulas itself.
In other words, it directed it to cheat. The re-grade Curiously though, the AI worker did not actually get better results by cheating. It turned out that it’s not actually very good at the sneaky stuff. Then researchers noted the flaw and re-graded all 1,080 attempts.
They discovered that none of the attempts that originally passed had failed. On the contrary, 81 of the previous failed attempts turned out to actually be correct. So although the AI had found a way to exploit the test, its efforts at cheating were counterproductive. Self-improvement on the cheap One aspect of the experiment set out to determine if it was really necessary to use a powerful frontier model as the AI coach in order to see a real improvement.
To that end, the researchers initially used Anthropic’s then-flagship model Claude Opus 5 as the coach. It helped the AI worker to score 77 out of 360 on a series of tests. But they then repeated the experiment using the much cheaper and less powerful DeepSeek V4 Flash model as the coach. The AI worker did almost as well, scoring 76 out of 360.
The DeepSeek V4 Flash self-improvement run was done at a cost of just $25. That is an order of magnitude cheaper than any form of fine-tuning or retraining. However, the researchers discovered that this kind of low-cost self-improvement only seems to work on a very specific set of tasks. The DeepSeek model helped the AI worker to improve its ability on spreadsheet-based tasks.
But its performance actually became worse on a different banking task. The self-coached model scored just 30, compared to an original score of 37 that it achieved without relying on any skills. Sentient’s team said the results of this experiment show that while reusable instructions can improve a model’s performance in certain areas, they do not actually enhance its reasoning capabilities in any way. Strange behavior Sentient’s team also reported some other strange things going on with its models.
Across two training runs, one of the agents unsuccessfully tried six times to access files beyond its permitted paths, but its sandbox blocked those attempts. The researchers said they have no idea why the agent tried to do this. The researchers observed more sinister behavior when, during one of the runs, the AI coach deleted a key sentence from its “stopping” rule. That rule tells the AI worker when to disengage from a conversation.
The AI coach reported that it had strengthened the rule, but in fact it increased the number of conversations that failed to end from six to 16 out of 75. The researchers said this action wasn’t necessarily malicious. But it shows that when an AI makes changes to its underlying code by itself, its description of what it actually changed may not be reliable. AI will do whatever it takes The findings are especially relevant because Amodei has highlighted reinforcement learning as one of the most concerning safety risks of AI systems.
He has pointed out a number of incidents where agents tried to manipulate the graders that evaluated their performance. Sentient’s research highlights one of the most fundamental characteristics of AI systems. The objective is the goal, and autonomous systems will look for ways to optimize themselves to reach that objective, even if that means exploiting flaws instead of actively trying to improve their methods. Moreover, when those agents create instructions based on those discoveries, they can influence other agents as well.
AI will optimize itself in whatever way it sees fit in its pursuit of an objective, even if that means finding a flaw in how it’s assessed, rather than identify a better way to complete the task. For AI, all that matters is the result. Contributed article. Not produced by the TNW newsroom and does not reflect the editorial stance of TNW.
Leave a Reply