As we move from AI that simply answers questions to AI that can actually take action, one governance question is becoming much more important:
What counts as an AI incident?
That sounds simple until you start thinking about what AI agents can now do.
They can access systems, use credentials, move information, trigger workflows and interact with other tools.
So an incident cannot only mean:
“The system crashed.”
It may also mean the system did something it was never supposed to do.
Canada is already asking the question
Canada’s federal AI transparency consultation is looking at serious-incident reporting and how organizations should track the activity and interactions of AI agents.
That is important because there is a balance to find.
If every strange AI output becomes a reportable incident, organizations could drown in noise.
But if the threshold is too high, genuinely important behaviour may never receive the human review it deserves.
And now we have some real-world examples to learn from.
OpenAI has publicly discussed evaluation cases involving AI agents using credentials outside their intended boundaries, exposing systems to the public internet, and attempting to move information outside expected environments.
Those examples make this conversation much less theoretical.
What might an AI incident look like in local government?
Think about an AI agent working inside a municipality.
Maybe it is reviewing bylaws.
Maybe it is helping with records management.
Maybe it is extracting Council resolutions and updating a tracking system.
Maybe it is connected to a workflow that can send information or update a database.
Now imagine it:
- accesses information it was not supposed to access;
- uses credentials it was never authorized to use;
- sends information somewhere outside the approved environment;
- continues working after the task should have ended;
- takes an action outside its approved scope;
- or behaves in a way that cannot easily be reconstructed afterwards.
At that point, we are no longer talking about a strange AI answer.
We are talking about a possible governance event.
This is where “Who is the Human?” matters
For me, this is the part local governments need to define before agents become deeply embedded in everyday work.
Who is the Human?
Who receives the alert?
Who reviews what the agent did?
Who decides whether the incident is serious?
Who has authority to stop the agent?
Who determines whether privacy, security, records management, legal or executive staff need to be involved?
And who decides what happens next?
If those answers are unclear, the organization may have an AI governance policy, but it may not yet have an AI incident response process.
Governance needs an escalation path
The same way organizations have processes for cybersecurity incidents, privacy breaches and operational failures, agentic AI may need its own escalation framework.
That does not necessarily mean building an entirely separate bureaucracy.
It could start with something very practical:
What happened?
What did the agent access?
What action did it take?
Was the action authorized?
Was information exposed or moved?
Can we reconstruct what happened?
Who needs to know?
Should the agent be stopped while we investigate?
Simple questions.
But they create something incredibly important:
human control after something unexpected happens.
Because that is the bit we should not forget.
AI may take the action.
It may even take thousands of actions.
But it cannot own the consequence.
The human still owns the response.
And as AI agents become more capable, defining exactly who that human is may become one of the most important parts of public-sector AI governance.
Who is the Human?
That should not just be a catchy question.
It should be something your governance framework can actually answer.
AI is the tool.
You are the human.
The responsibility is still yours.