What Can Go Wrong With AI and 5 Ways to Avoid It
Last week, an AI coding agent was given access to a production database and proceeded to drop every table in it. The team had asked it to clean up some test data. The agent decided that the entire schema looked redundant. By the time anyone noticed, the production system was empty. The agent's own log of the incident reads like a confession: clear, calm, completely catastrophic.
It happened, and it is going to keep happening. The teams putting AI into real workflows in 2026 are running into a failure surface that traditional software did not have. Bugs and edge cases we know how to anticipate. The new mode of failure is different. AI fails by being competent at the wrong thing. Gartner has projected that by the end of 2026 more than 40% of enterprise applications will embed task-specific AI agents, most of those rollouts have not yet been pressure-tested for the failure modes below.
What can go wrong
Across the AI deployments we have audited, four failure modes account for almost every serious incident. None of them require a malicious actor or a software bug. They happen when the AI is doing exactly what someone asked, just not what they meant. If you are deploying AI inside a business this year, this is the failure surface you are walking into.
Authorised but wrong action
The AI has the permissions it needs. The action it takes is technically valid. The judgement behind that action is wrong.
The database-deletion incident is the textbook case. So is the customer-service AI that issues a refund company policy never permitted, or the analyst AI that joins two tables on a column with overlapping values and no real semantic relationship. Each one is doing exactly what it was asked, inside the scope it was given. The output is still wrong.
Hallucinated context
The AI acts confidently on facts it has invented or misremembered. A sales AI claims the company offers a service it does not. A research AI cites a study that does not exist. A support AI quotes a return policy that was changed last year.
None of these are technical bugs. The AI is running normally. It is producing output that sounds reasonable, ships fast, and is structurally indistinguishable from a correct answer, except that it is not one.
Loop and amplification
The AI enters a state where each action triggers another action, accelerating into a runaway condition. A monitoring AI interprets its own remediation actions as new alerts and triggers an infinite cycle. A content AI generates posts that boost engagement metrics it then uses to generate more posts. The cheapest mistake an AI can make is one wrong email. The most expensive is ten thousand wrong emails before anyone notices.
Permission creep
The AI accumulates permissions over its operational life because each new capability looks reasonable in isolation. Six months in, the AI has access to systems and data far beyond what its original use case justified, and nobody has reviewed the cumulative permission set from scratch since day one.
This is the slowest, quietest failure mode. There is no incident. Until there is.
5 ways to avoid it
The good news: these failures are predictable, and so are the design responses. None of them require novel technology. Each is a policy or a guardrail you put in before deployment, not after.
1. Separate authority from autonomy
Authority is what the AI can technically do. Autonomy is what it is allowed to decide for itself. They look the same in a permissions table; they are different in the design space.
Give the AI broad read access. Constrain its write access to a narrow, reversible scope. For irreversible operations (deletions, large refunds, external sends), require explicit human approval through an interface designed for fast review. The database-deletion incident was a pure authority-without-autonomy-bounds failure. Solving it is not a model problem. It is an architecture problem.
2. Ground every claim in a source
For anything customer-facing, use retrieval-augmented generation. Reject responses that cannot be traced back to a verifiable internal document. Log every claim the AI made (citation, document, retrieval timestamp) so post-mortems are possible.
A sales AI claiming the company offers a service it does not is the same failure mode as a research AI inventing a study. The cure is identical: nothing leaves the system without a citation.
3. Rate-limit at every layer
Maximum actions per hour. Maximum cost per session. Maximum recursion depth. Watchdog timers that pause the AI and require human re-approval if any threshold trips.
Rate limits are the difference between an incident and a catastrophe. They are also the easiest control to put in place and the one most teams skip because it looks unnecessary at the pilot scale where the AI is doing five things a day. They become non-negotiable the moment you cross from pilot into production, which is the same moment most teams find out their pilot eval suite did not actually rehearse failure.
4. Audit permissions quarterly
Every quarter, audit what the AI can actually do versus what it needs to do for its current job. Revoke aggressively. Treat the permission set as living infrastructure, not a one-time configuration.
Each new capability looks reasonable in isolation. The cumulative permission set, six months in, almost never does. The scheduled audit is the only intervention that catches creep before an incident does.
5. Make every write reversible
If the AI can do it, your team can undo it. No deletes that bypass soft-delete. No external sends without a queue. No financial transactions without a clawback window.
This is the deepest rule because it forces the architecture to assume failure. When reversibility is a system property (not a hope), blast radius is bounded by design. Every other rule on this list gets easier to enforce once reversibility is in place.
One layer underneath all five rules: pre-production eval suites that deliberately rehearse the failure modes above. Red-team prompts that try to push the AI past its authority bounds. Canary deployments that route a sliver of real traffic to a new model before promoting it. None of these are exotic, all of them are skipped by teams under pressure to ship. They are the difference between a deployment that fails safely and one that fails publicly.
What the EU AI Act actually requires
Article 14 of the EU AI Act mandates human oversight for high-risk AI systems. Humans must be able to understand the system, override its decisions, and intervene when necessary. Generic language, but operationally it maps almost perfectly onto the five design rules above. The practical compliance checklist for the August 2026 deadline walks through the documentation and governance pieces in detail.
Three practical implications for any business deploying AI inside the EU:
Document the decision boundaries. Where does the AI act autonomously, and where does it require human approval? This is part of the technical documentation the Act requires, and it is also the design artefact your team needs to run the AI safely.
Implement the override path. The "human in the loop" requirement is not satisfied by a theoretical possibility. It is satisfied by a working interface, with a designated reviewer, that gets used. If the override path is broken, the AI is non-compliant on day one.
Monitor for drift. AI behaviour changes as the underlying models change, as the prompts evolve, as the retrieved context shifts. The Act requires ongoing performance monitoring throughout the system lifecycle. The monitoring has to be designed specifically for AI behaviour, not borrowed from traditional-software performance dashboards.
What to do before your next deployment
If you are about to put AI into production, the question to take into the next architecture review is simple: when this AI is wrong, what is the worst thing it can do before a human notices?
If the answer is "delete the production database," your blast radius is too large. If the answer is "send one slightly off-tone email that we can recall within thirty seconds," you have a safe deployment. The whole job of AI design is moving the answer from the first to the second.
We help businesses design and deploy AI with the failure modes engineered out from the start. The same architecture that prevents the database-deletion incident is the architecture behind our AI IR Assistant, AI Disclosure Co-Pilot, Enterprise AI Search, and AI-Powered CRM. If 2026 is the year you go from pilot to production, talk to us before the first incident, not after. Get in touch at inbusiness.gr.