Terms of reference for an AI agent: template, structure and examples of formulations
Terms of reference for an AI agentwhat exactly needs to be automatedmakes a chatbot
An important point: the TOR for an AI agent differs from the usual TOR for software primarily in the acceptance section. For a classic program, you can often check the result according to a strict rule: you press a button and get the expected state. The agent's approach is different: they can answer the same question in different words, and both answers will be correct. "Therefore, it is necessary to accept an AI agent not "by sight" or according to a beautiful demonstration, but through a pre-prepared set of test cases.
The rule of thumb:
How does the TOR for an agent differ from the TOR for a regular program
The main difference is the lack of deterministic inference. Deterministic output means that with the same input, the system returns the same result each time. An AI agent works differently: it can formulate an answer in different ways, choose a different order of arguments, and ask a clarifying question where it made a recommendation the last time. "The system must answer user questions," therefore, is not verifiable. You cannot accept a project based on a line match, as in a simple autotest.
The third difference is that the behavior changes when the model is changed. The agent's model can be updated by a supplier or replaced by a contractor for the sake of price, speed, or quality. Even if the interface remains the same, the agent will start choosing tools differently, rejecting them differently, and interpreting borderline requests differently. Therefore, the TOR needs a point about regression testing when changing the model: after any replacement, the test set is run, and only after that the change is put into operation.
TK structure: eleven sections
A good technical specification for an AI agent is similar to a map of an area: it shows the target, borders, roads, forbidden zones and control points. If you remove at least one layer, the team will start thinking. And where thinking begins, there are different expectations for business, IT, security, and the contractor.
- A business goal and a measurable result.
- "As is" description of the process.
- Agent boundaries.
- Escalation scenarios for humans.
- Sources of knowledge
- Tools and integrations.CRM
- Channels and interfaces.
- Data and security requirements.
- Logging, auditing, and observability.
- Acceptance criteria and test set.
- Exploitation.
Section by section: what to write and how to formulate
Business goal and measurable result
"This section should describe not an "implement AI", but a business change. The goal should be measurable: response time, the proportion of automated solutions, reducing the load on the first line, increasing conversion from an application to a consultation, and reducing the cost of processing a request. Without a number, TK turns into a wish.
Don't do that:
It's necessary:Shorten the first response time
Description of the process "as is"
The baseline is fixed here: how the process works before implementation. You need to specify the participants, the systems, the average processing time, the cost of the operation, frequent errors and bottlenecks. If the initial state is not measured before the project, there will be nothing to compare after the launch. This is how projects appear that seem to "work", but do not give a provable effect.
Working wording:
Agent Boundaries
Borders are one of the most important sections. You can't just write to an agent, "answers customer questions." This phrase does not protect the business from situations where the agent promises a discount, confirms the term of the contract, or gives a legal assessment. We need three zones: he does it himself, he does it with confirmation, and he never does it.
Don't do that:
It's necessary:creates a request in CRM
Escalation scenarios
Escalation is the transfer of dialogue to a person. In the TOR, you need to list the triggers: a direct request to call an employee, two unsuccessful responses in a row, a negative emotional coloring, the transaction amount is above the set threshold, a topic from the stop list, the lack of an answer in the knowledge base. It is indicated separately where the request is sent: the queue in CRM, the responsible manager, the general support channel or the specialist on duty.
It is equally important to describe what the employee sees. A bad scenario is to simply switch the client to the operator and force him to repeat everything all over again. A good scenario is to provide a brief summary of the dialogue, the original question, the data that has already been verified, the reason for the escalation, and the recommended next action.
Sources of knowledge and the owner of the update
The AI agent responds only as well as the sources from which it takes knowledge. The TOR should list documents, databases, website pages, regulations, instructions, FAQ, and commercial materials. For each source, it is useful to specify the format, volume, date of relevance, and owner.
Don't do that:
It's necessary:
Tools and integrations
The agent's tool is a function that it can trigger: create a request in CRM, check the order status, select free time in the calendar, send a notification, receive leftovers from the warehouse. In the TOR, it is convenient to describe the tools in a table: the name of the function, what it does, the input parameters, what it returns, what rights are needed, and whether the action is reversible.
The normal number of tools for one agent at the first stage is from about two to eight. If there are fifteen or more regular items in the TOR, this is a signal that the task is outlined too broadly. For irreversible actions, a separate rule is needed: the agent must not perform them without human confirmation. You can send a draft email automatically, but you cannot sign a contract or change financial terms.
Channels, data, security and operation
The channel affects the agent's behavior. You can show a long response and buttons on the site, it's better to split the message in the messenger, delay and interrupting skills are critical in telephony, links to documents and action history are more important in the internal portal. Therefore, for each channel it is necessary to fix the restrictions: the length of messages, attachments, buttons, authorization, language of communication.
placement in the company's outline
Operation closes the agent's lifecycle after launch: who manages the knowledge base, how often, what to do in case of a failure, which backup scenario is enabled, and how the changes are coordinated. The rule is simple: any significant change — updating the database, adding a tool, changing the model — must go through the test suite.
Acceptance criteria: how to accept something that responds differently each time
The acceptance criteria are the core of the TOR for the AI agent. Without them, the project is usually accepted by demonstration: the contractor shows several successful dialogues, everyone nods, the agent goes to work, and a month later a dispute begins about why he "does not respond the right way" to real clients. For an AI agent, a demonstration is not an acceptance. Acceptance is done through a test kit.
The test set should consist of real cases. At the first stage, at least 30 cases are sufficient; before commissioning, it is better to prepare 100 or more. Synthetic examples are useful for a rough check, but they do not reproduce the real failure modes well: incomplete formulations of clients, mistakes in terms, emotional messages, contradictory introductions, attempts to force the agent to go beyond the boundaries.
Simple onesDifficult onesBorderline
An unambiguous success criterion is needed for each test case. This may be the expected structured result: an application has been created with such fields, such a tool has been selected, and the dialog has been transferred to the appropriate queue. Or a list of response properties: the agent named the current price, did not promise a discount, asked a clarifying question, referred to a source, did not give a legal assessment.
max_steps
An example of acceptance thresholds: the proportion of correctly handled simple cases is at least 95%; the proportion of correct escalations in borderline cases is 100%; the proportion of correct tool selection is at least 90%; the absence of irreversible actions without confirmation is an absolute requirement. The last point should not be averaged: one erroneous irreversible step can cost more than dozens of inaccurate answers in the FAQ.
Why shouldn't you write "99.9% accuracy" in the TOR? Because real agent systems are still noticeably far from such reliability in complex scenarios. In the ToolFailBench study, the best result among 19 models on the fair use of tools benchmark was 86.33% of net calls. On the t2-bench-Verified benchmark of client tasks with policies and actions, the leaders showed about 82% on average, and about 70-74% in the most difficult domain. In SilentProbe, when an external API failed unnoticeably, agents detected the problem in 12% of cases, fixed it in 0%, and reported false success in 41% of cases.
The conclusion for TK is pragmatic: you can't promise magical infallibility. We need to build a system where risk is manageable. To do this, the document contains confirmations for dangerous actions, reversibility of operations, understandable logs, a test set, and regression testing when changing the model.
Metrics: four words that are constantly confused
In projects with chatbots and AI agents, four metrics are often mixed: rejection, retention, decision, and first-time decision. The error seems to be terminological, but in practice it changes the meaning of the report. You can show beautiful retention and still not solve the client's problem.
| Metric | Which means |
|---|---|
| Deflection | The appeal was closed without human intervention. |
| Retention (containment) | The appeal did not escalate, even if the client dropped the dialogue in irritation. |
| The solution | The client's problem is really solved. |
| First-time Solution (FCR) | The problem was solved in the first interaction, without repeated treatment. |
As a rule of thumb, a real rejection is confirmed by the fact that the customer has not returned with the same problem within 5-7 days. "If a person has closed the chat because he is tired of arguing with an agent, this is not a success, although in raw statistics such a dialogue may look like "it did not escalate."
The gap between the stated retention and the actual solution in industry estimates can reach 20-30 percentage points. A tool that shows 70% retention can actually provide about 45% of a real solution. Therefore, it is better to include the decision metric rather than retention in the acceptance criteria. If the contractor reports only on retention, he chooses the indicator that gives the most beautiful number.
For reference: in industry surveys, the median deviation along the first line is often about 41%, the upper quartile is closer to 59%. There is a huge variation in the types of requests: refunds and password resets can be automated above 70%, while complex claims can remain below 25%. These numbers should be used carefully, as an order of magnitude, and not as a universal norm for any business.
Seven mistakes in the TOR, because of which the project is then redone
The first mistake is a goal without a number. "Improve service" cannot be accepted, because each participant in the project understands improvement in their own way. For the director, this means lower costs, for the support manager — fewer queues, for the client — a quick and accurate response, for the contractor — a working scenario.
The second mistake is the lack of a baseline. If you do not measure how it was before the implementation, then it is impossible to prove that it has improved. Third, there is no list of what the agent never does. This is the most frequent source of incidents: the agent begins to confidently respond where he should stop.
"The fourth mistake is the acceptance "by eye." A beautiful demo is not a substitute for a test set. "Fifth— there is the forgotten escalation: the TOR says "switch to operator," but it does not specify which channel, to whom, with what context, or at what time.
The sixth mistake is that the owner of the knowledge base has not been appointed. After a quarter, the agent starts naming last year's prices, because no one is responsible for updating the documents. Seventh, there is no change of model clause. The supplier updated the model, the behavior changed, the quality dropped, but the moment of change was not fixed and the tests were not run.
What not to write in the TOR
The TOR does not need to specify a specific model and its version, unless it is related to strict security or infrastructure requirements. The models change faster than the project goes on. Russian requirements should be better described: quality, acceptable response delay, restrictions on data placement, cost of processing, possibility of regression verification. The choice of the model should be left to the contractor, provided that the metrics are preserved.
It is not necessary to specify the framework and stack unnecessarily. If a company has no infrastructure requirements, such granularity can narrow the solution space and force it to pay for meeting expectations rather than for the result. What matters is not which framework the agent is built on, but whether it passes the test suite, logs actions, adheres to boundaries, and withstands the load.
You also don't need to specify the exact wording of all the agent's responses. For an AI system, this is false accuracy. It is better to describe the tone of communication, prohibitions, response structure, mandatory clarifications and sources. Specific lines will appear after analyzing the actual dialogues. It is also better to describe the number of users as an input for calculating the load, rather than as an independent architectural requirement.
Pattern
The TK template should repeat the eleven sections from this article and contain hints in each field. His task is not to force the customer to become a system analyst, but to collect the minimum sufficient information to assess the budget, deadlines and risks.
"Inside the template, it is necessary to provide fields for the business goal, the description of the process "as is", a list of agent actions, escalation scenarios, knowledge sources, tools, channels, data requirements, logging, acceptance criteria and operation. The most useful part of the template is the table of test cases: user request, case category, expected behavior, required tool, success criterion, risk comment.
A good template disciplines both sides. The customer sees in advance what decisions need to be made before the launch. The contractor does not receive an abstract "make an AI agent", but a document that can be used to evaluate the architecture, integration, knowledge base, and complexity of acceptance.
What to do next
If you need a ready-made document, download the TK template, enter the initial figures and collect 10-15 real customer requests for a future test set. Analyzing a completed template usually quickly makes it clear which class of solution is needed: a simple bot, an agent with a knowledge base, an agent with tools, or a full-fledged scheme with integrations, logging, and regression checks.
Автор
Антон Амосов
