What operating an AI agent actually means

Operating an AI agent means five things, all of them after go-live: monitoring what the agent does on real documents, correcting it when it drifts, escalating every case it is unsure about to a named person, keeping a complete decision log, and being accountable for the result to the organisation that uses it. Building the agent is weeks one and two. Operation is everything after week three, and it never ends.

That definition is the whole difference between an AI agent that runs and one that is quietly switched off after the first quarter. Seven of ten AI failures are caused by data quality and governance, not by the model. Governance is another word for operation: someone watching, someone correcting, someone answerable.

Buying an agent is not the same as owning its outcome

Most organisations that bought an agent in the last two years bought a tool. The tool has an interface, a subscription and documentation. What it does not have is a person whose job is to look at what it did yesterday. After the launch meeting, ownership dissolves: the department assumes IT watches it, IT assumes the vendor watches it, the vendor watches uptime. Nobody watches the decisions.

The symptoms arrive slowly. A new document type appears and is misfiled for a month. A supplier changes its invoice layout. A department is renamed. None of these is a model failure. All of them are operating failures, and all of them end with the same sentence in a meeting: the AI does not work.

The five duties of an operator

An operator is accountable for the agent's output the way a payroll provider is accountable for payslips. In our practice the duties are fixed and written into the operating agreement.

Duty	What it means in a week of operation	What happens without it
Monitor	Every assignment the agent makes is visible with its confidence; a person reviews the distribution daily, not the individual cases.	Drift is discovered by the department, months later, as a complaint.
Correct	Misassignments are fed back as examples; classes are adjusted when the organisation changes.	The same error repeats until someone stops trusting the agent.
Escalate	Cases below the confidence threshold go to a named person automatically, with the reasoning attached.	Uncertain cases are guessed, and nobody knows which ones.
Log	Every decision is written to the decision log with timestamp, input, class, confidence and reasoning.	An auditor, a supervisor or a citizen asks why, and there is no answer.
Answer	One named contact is accountable for the hit rate agreed before go-live and reports on it.	The tool has an owner on paper and none in practice.

Operation starts with a number agreed before the agent exists

An agent can only be operated against a target. Ours is the hit rate: the share of cases the agent assigns correctly, agreed with the Fachbereich (the department that owns the process) before the Pilotbetrieb starts and tested at acceptance in week three. Everything in operation refers back to it. Monitoring asks whether the rate holds. Correction asks what moved it. The monthly report states it.

Without that number, operation becomes opinion. With it, the conversation about whether the agent is working takes one line.

One Vorgang, so that operation stays measurable

We build and operate one Vorgang at a time: one intake, one task, one destination. Everything that arrives daily, answers the same question and lands in the same system. Incoming post assigned to departments. Delivery notes read and forwarded to warehouse or accounting. Invoices captured and handed to the accounting system with a proposed assignment.

The boundary is not modesty. A single-stage Vorgang has one hit rate, one confidence threshold and one target system, so it can be monitored, corrected and reported. A multi-stage workflow with branches and two write targets has none of those as a single number, and operation of it is guesswork dressed as dashboards. A second Vorgang is a second engagement with its own rate.

The agent does	The agent never does
Classifies the document and assigns it to a department or case type	Approves, posts, pays or deletes anything
Extracts the metadata the target system needs	Triggers deadlines or legal effects
Creates the task in exactly one existing target system	Writes back into a second system
Hands cases below the confidence threshold to a person, with its reasoning	Decides on its own when it is unsure

Why the fallback is the most important part of the design

Every agent is wrong sometimes. The question a supervisor, an auditor or a customer asks is not whether it was wrong but what happened next. A confidence threshold turns that question into a rule: below the agreed certainty, the agent stops, hands the case to the named person and logs why. Above it, the agent acts and logs why. There is no third state in which the agent guesses.

The threshold is set with the department, not by us alone, and it is tuned during the four productive weeks of the Pilotbetrieb against real volume. A threshold set too high sends everything to a person and saves nothing; set too low it hides errors in the log. The right value is the one the department can defend. We describe how the handover works in detail in Confidence threshold and the human fallback.

What "answerable" means when an auditor asks

The decision log is the operator's evidence. For every case it records what came in, what the agent decided, how confident it was, which rule or example the decision rests on, and whether a person intervened. It is written in the format the organisation's own auditors can read, and it is kept for the retention period the organisation sets. Operation entirely within the EU, responsibility from Düsseldorf. GDPR-compliant processing with an AVV included. A complete decision log available as evidence for EU AI Act audits. A defined human intervention path, documented and triggerable at any time.

What belongs in that log, field by field, is the subject of What belongs in a decision log.

How the seven weeks turn a purchase into an operation

The Pilotbetrieb is named for what it is: productive operation on real documents from week three, not a demonstration. It has a fixed price, agreed before we start, and a defined end.

Week	What happens
1	Process conversation with the Fachbereich, classes defined, examples reviewed, access clarified
2	Agent built and trained, quality checks set up, test run on real documents
3	Acceptance against the agreed hit rate, go-live, handover document
4 to 7	Productive operation on real volume; we monitor and correct
End of 7	Written evaluation: rate achieved, cases processed, time saved

If the agreed rate is met, operation continues under a monthly operating agreement, cancellable monthly. If it is missed, the Pilotbetrieb ends with no further obligation. Either way the organisation has a number, a log and a written evaluation, which is more than most AI purchases produce in a year.

Five questions to ask any vendor before you sign

1. Who looks at what the agent did yesterday, and what is their name?
2. What is the hit rate you commit to, and how was it agreed?
3. What happens to a case the agent is unsure about, and where is that documented?
4. Show me the decision log for one case, end to end.
5. What exactly is out of scope, in writing?

A vendor who answers all five in specifics is offering operation. A vendor who answers with a roadmap is offering a tool.

Frequently asked questions

Is operating an agent the same as hosting it?

No. Hosting keeps the software running; operation keeps the decisions right. Operation means monitoring the assignments, correcting drift, escalating uncertain cases to a person, maintaining the decision log and being accountable for the agreed hit rate. A hosted agent that nobody monitors is not operated, however good its uptime.

Can our own team operate the agent instead?

In principle, if someone owns the five duties above as part of their job and has the tooling to see confidence and drift. In practice this is the point where most self-service deployments stall: the tool has an owner on paper and no one in the calendar. That is why we sell the operation, not the tool.

What does the agent do when it is unsure?

It stops and hands the case to a named person, with its reasoning attached, and logs the handover. The certainty below which this happens is the confidence threshold, agreed with the department before the Pilotbetrieb and tuned on real volume. The agent never guesses in the uncertain range.

How is the hit rate measured?

On real documents, at acceptance in week three and continuously afterwards. The department reviews a sample of assignments against what it would have decided; the share the agent got right is the rate. It is reported monthly, and a drop triggers correction, not a discussion about whether AI works.

What is not included in the operation?

A second intake channel, a second target system, multi-stage workflows with branches, custom development against a legacy system, and any action beyond classify, extract, route and create a task. Each of those is either a second Vorgang or out of scope by design, so that the first one stays measurable.

The decidable next step

If your organisation has a recurring intake that someone sorts by hand every day, that is a Vorgang, and it is what a Pilotbetrieb is for: one intake, one task, one destination, fixed price agreed before the start, seven weeks with four of them productive operation on real documents. Request a first conversation.