How to evaluate an AI assistant

Define an acceptable answer

For a document assistant, a useful answer might require an accurate extract and a source reference. For an editor, it might be a draft without invented facts that a person can review quickly. Set criteria before choosing a model. Fluent writing and confident tone alone do not establish that the task is being solved.

Build a varied set of examples

Include ordinary requests, incomplete documents, conflicting information and questions the available data cannot answer. Use anonymised material permitted for the chosen service. Keep some examples aside for final evaluation instead of using all of them during tuning. Otherwise the system may improve on familiar wording while still failing on new requests.

Classify errors by their consequences

A punctuation error, a missing key fact and an invented amount need different responses. Decide which errors permit manual correction and which must block the output. Define when the assistant should report missing information. A well-placed statement that evidence is insufficient is more useful than plausible text with no factual basis.

Evaluate the human review step

Specify who checks the result and what they can see: the original document, supporting passages and proposed changes. If AI prepares a publication, approval should be a separate action. ROAVENTA connects text generation to ETL data and fact checking. Evaluating that sequence is more useful than judging an isolated response.

Compare the cost of the complete outcome

Count data preparation, retries, waiting time and human review as well as model requests. A cheaper response can require much more correction. Compare alternatives on the same examples using the same criteria. Record evaluation settings so that changes in quality or cost can be investigated and the comparison repeated.

Agree on stopping criteria

Before the pilot, define the conditions for continuing: acceptable errors, processing time and reviewer effort. If the result fails, understanding why is still useful. Data may be unsuitable, the task too broad or review too difficult. Narrowing the workflow or declining to automate an action can be valid pilot outcomes.