The AI tool says the patch installed. The production server is still running the old version. Michael Meyer, Anunta's Director of Solutions, has seen organizations run into this exact problem. "They're trusting their AI to patch, and it says it was successful," he said. "But when it goes to production, it hasn't upgraded that version." A traditional script fails in a way IT teams know how to catch, because it stops with an error that points to the line that broke. An AI agent can finish the job, report success, and still be wrong. Teams are taking work back from AI because of that gap between the report and the result.
The same gap shows up well beyond desktop operations. In the Stack Overflow 2025 Developer Survey of more than 49,000 developers, 45% named "AI solutions that are almost right, but not quite" as their top frustration with AI tools. Two-thirds said they now spend more time fixing almost-right AI code. Trust in AI accuracy also fell, from 40% in earlier years to 29%. Desktop teams feel the same strain. In our own research, 73% of practitioners whose teams use AI-assisted tooling have already handed at least one task back to people (Where AI Pays Off in IT, P-Q4). Knowing when to take a task back starts with knowing how AI fails.
The FSLogix case points to a larger observation from the engineers I work with. AI pays off on a task only when someone or something other than the AI can confirm the outcome. Michael sets four conditions for any task AI handles: "It has to be repeatable. It has to be trackable. It needs to be diagnosable, and it needs to be gated." When any one of those conditions breaks down, the team loses its independent view of the outcome. Five signs show when that has happened.
Each sign marks a point where the team can no longer confirm the outcome without taking the AI's word for it. The first sign is the hardest to spot, because the tool says everything went fine.
IT people call this sign "task failed successfully," after the Windows error message that became a meme. Michael's patching and FSLogix examples are both cases of it. To catch it, check the outcome directly, such as the installed version or the attached profile, instead of the tool's status message. A false report misleads the team, and a missing report leaves the team blind.
A silent failure never produces an alert, a log entry, or a ticket, so it breaks Michael's rule that a task must be trackable. To catch it, break the task on purpose in a test environment and confirm the right people hear about it. If the alert never arrives, keep a person on the task. The third sign works the opposite way, because it makes plenty of noise.
"If that agent is troubleshooting and it restarts that endpoint without asking, that itself is a hard stop moment," Michael said. An agent that closes a slow application can erase hours of a user's unsaved work. Even so, 15% of practitioners already let AI restart services with no human approval (Where AI Pays Off in IT, P-Q3). To catch it, match every action in the tool's log to the request or ticket behind it. Any unmatched action means the task needs an approval step before it runs again. Inconsistent results are the next sign, although some variation is normal.
Tyler Shively, an Anunta account director, ties this sign to the purpose of the task. "If that task is meant to have the same exact response every single time, then different results are a problem," he said. A summary can be worded differently on every run and still do its job, while a patch compliance report or a ticket routing rule should return one answer. To catch it, rerun a sample of past inputs and compare the results. A task built for one right answer that keeps returning several has failed the repeatability test. It belongs with people until the team finds the cause. The last sign sits with the team instead of the tool.
Automation bias is the habit of trusting a system's output because it was right before. In a 2010 review, Complacency and Bias in Human Use of Automation, Raja Parasuraman and Dietrich Manzey found that automation bias leads to errors whenever decision aids are imperfect. They also found that training doesn't prevent it. "We want to trust what our tools are telling us," Michael said. "As IT experts, we really need to be wary of the data that we're seeing." Because the bias grows with trust, a tool with a strong track record needs the same scheduled, logged spot-checks as a new one. Once a spot-check turns up a sign, the team has to decide what to do with the task.
Taking a task back rarely means returning it to fully manual work. A better move is to step the task down to a level where the team can still confirm the outcome. A task that ran alone can move to AI acting with approval, or to AI suggesting while a person acts. Teams already use that middle level for risky work. For session troubleshooting, the most common approach is AI suggesting the action while a person carries it out, chosen by 37% of practitioners (Where AI Pays Off in IT, P-Q3). Tyler described how a task earns its way back up: "As you gain more comfortability with that, it can take on slowly more and more complex tasks, but there is a line where you shouldn't go past."
The cost of checking decides where that line sits. Practitioners rank overseeing and correcting AI output third among the things adding to their workload (Where AI Pays Off in IT, P-Q7). Checks that get shorter as a tool proves itself mean the task is earning more autonomy. Checks that never get shorter mean the task costs more with AI than without it, and taking it back is the right call. Knowing which of your tasks sit on each side of that line starts with knowing where your team stands.
Our report, Where AI Pays Off in IT, shows which desktop tasks IT teams hand to AI and which ones they take back. The AI Task Map for IT compares how your team handles nine desktop tasks with how your peers handle them, and it suggests a next step for each one. Both draw on our survey of 254 US IT practitioners and leaders.