An AI evaluation score without context is just a very confident number.
A vendor tells you their AI system achieved an 87% accuracy score.
Wonderful.
But what was tested? Which model version was used? What counted as a correct answer? Was the system tested using real municipal situations, or a tidy dataset that bears very little resemblance to the beautiful chaos of local government?
That 87% may be impressive.
It may also be very fancy nonsense.
The Canadian Artificial Intelligence Safety Institute recently outlined what AI evaluators should share so that evaluation results can be properly understood, compared and used.
This is technical guidance, but its message matters enormously for local government leaders, procurement teams and anyone being asked to approve an AI tool.
A Score Is Not the Same as Evidence
Think about it this way.
If someone told Council that a bridge was “87% safe,” the next question would not be, “Where do we sign?”
Council would want to know:
- Who conducted the assessment?
- What standards were used?
- What conditions were tested?
- What was not tested?
- How confident are we in the result?
- Who paid for the assessment?
- What happens in the other 13%?
We need to bring that same level of curiosity to AI.
A benchmark score cannot tell us whether a system is appropriate for drafting council reports, summarizing bylaws, reviewing procurement documents, answering resident questions or supporting service delivery.
The evaluation must reflect the way the system will actually be used.
What Should an AI Evaluation Tell Us?
The federal guidance identifies several areas evaluators should clearly report.
1. The Purpose of the Evaluation
What question was the evaluation designed to answer?
A test of an AI system’s general writing ability is not the same as testing whether it can accurately summarize a 200-page municipal agenda package.
An evaluation may measure capability, safety controls, alignment or performance in a specific environment. Unless the purpose is clear, we cannot know whether the result applies to our intended use.
2. The Context in Which the AI Was Tested
Who was expected to use the system?
Was it tested with ordinary users, technical experts, malicious users or people repeatedly trying to bypass its safeguards?
The context matters because AI performance can change depending on the task, the user, the instructions and the information provided.
A model that performs well in a controlled demonstration may behave very differently when ten departments use it for ten different purposes.
3. The Methodology
Evaluators should explain:
- What was measured
- Which metrics were used
- Which datasets were included
- How answers were scored
- Whether another AI system judged the results
- Whether humans reviewed the outputs
- Why the chosen method was appropriate
This is where it gets interesting.
An evaluation may use automated scoring, human reviewers or even another large language model to judge whether the first model provided a good answer.
If AI is grading AI, we should probably ask whether a human checked the homework.
4. The Exact System and Settings Tested
AI tools change quickly.
Evaluators should identify the model version, configuration, prompts and relevant settings used during testing. A result from one model version may not apply after the provider updates the system.
This creates a very practical procurement question:
Are we buying access to the same system that was evaluated, under the same conditions?
If the answer is unclear, the benchmark may not tell us very much about what municipal staff will actually receive.
5. Uncertainty and Limitations
A score is rarely as precise as it appears.
The federal guidance recommends reporting confidence intervals and other information that helps readers understand uncertainty. For example, one model may score 83% and another 87%, but overlapping confidence intervals could mean there is no statistically meaningful difference between them.
Four percentage points can look very impressive in a sales presentation.
It may not mean the second system is actually better.
Evaluators should also explain where the results may not apply, whether the benchmark is becoming outdated and what errors may have been introduced by the testing method.
6. Potential Conflicts of Interest
Who conducted the evaluation, and what is their relationship with the company that developed or sells the system?
A vendor’s internal evaluation can still provide useful information, but it is not the same as independent testing.
Local governments should know whether the evaluator is completely independent, funded by the developer or otherwise connected to the system being assessed.
That information helps decision-makers determine whether additional testing or verification is required.
Transparency Does Not Mean Publishing Everything
There is an important balance here.
The federal guidance recognizes that some information should not be publicly released.
Cybersecurity testing methods, jailbreak instructions and datasets containing dangerous information could become a step-by-step guide for misuse. Publishing an entire test dataset can also contaminate future evaluations if AI models are later trained on those same questions and answers.
The recommended approach is layered disclosure.
Different information may be shared with:
- The public
- Governments and trusted oversight bodies
- Independent evaluators
- System developers and owners
The goal is to provide enough information for meaningful accountability without creating new safety risks or weakening future evaluations.
That is not secrecy.
It is responsible transparency.
What This Means for Local Government Procurement
Before accepting an AI benchmark, assurance statement or vendor performance claim, municipalities should ask:
- What exactly was tested?
- Does the test reflect our intended municipal use?
- Which model version and settings were evaluated?
- What data, prompts and languages were included?
- How were the answers scored?
- Did humans audit the results?
- What uncertainty or limitations were identified?
- Who conducted and funded the evaluation?
- Can the test be independently replicated?
- What important information has been withheld, and why?
These questions should form part of AI procurement, pilot planning, privacy reviews and ongoing vendor management.
AI evaluation is not simply a technical exercise for IT.
It affects procurement, governance, privacy, service delivery and public trust.
The Human Still Owns the Decision
AI can produce the benchmark.
A vendor can present the score.
An evaluator can explain the methodology.
But the organization still has to decide whether that evidence is relevant, reliable and sufficient for the proposed use.
AI is the tool.
You are the human.
The decision is still yours.
Before approving the next AI pilot, ask one critical question:
Are we evaluating the system, or are we simply trusting the marketing?
Need Help Asking the Right AI Questions?
If your municipality is exploring, purchasing or already using AI, I can help your staff and Council understand the technology, evaluate the risks and build practical guardrails around its use.
Learn more about my practical AI Workshop for local government and book me to help your team use AI with confidence, curiosity and human accountability.
