SAMP Engineering Notes #07
What happens when an AI agent crashes, becomes unreachable, or leaves an operation's outcome uncertain? Explore SAMP failure management, recovery, trust, enforcement, and lessons from AgMS Alpha.
What Happens When an AI Agent Fails?
A failed Agent is not necessarily a failed operation. A missing response is not proof that nothing happened. And recovering connectivity does not automatically restore trust.
These distinctions become increasingly important as AI agents move from isolated experiments into operational environments.
An agent may be executing a task, querying an external service, interacting with another agent, or performing an authorized action when something goes wrong.
The process may crash. The network may fail. The runtime may become unresponsive. Or the operation may complete successfully while its response never reaches the management system.
From the outside, these situations can look remarkably similar.
Yet their operational consequences can be very different.
This is where agent management becomes an engineering discipline rather than simply another dashboard.
1. Failure Is Not a Single State
Consider an environment containing multiple AI agents operating across different frameworks, hosts, and execution environments.
When one agent stops responding, several explanations are possible:
- Agent failure: The runtime or application has crashed or stopped functioning correctly.
- Connectivity failure: The Agent may still be operational, but communication with the Manager has been interrupted.
- Dependency failure: The Agent is reachable but cannot complete its work because an external service is unavailable.
- Management failure: The management infrastructure itself has lost visibility or experienced an operational problem.
- Execution uncertainty: An operation was initiated, but its actual outcome cannot be established.
These situations should not be collapsed into one generic status labeled failed.
A management system needs to distinguish what it has observed from what it has inferred.
A timeout is an observation.
A crashed process is a possible explanation.
A confirmed execution failure is a different fact altogether.
Observability is not the same as certainty.
2. Detect, Degrade, Recover
The first responsibility of an agent management system is to recognize that something has changed.
SAMP, the Simple Agent Management Protocol, provides an experimental management-plane framework through which heterogeneous agents can expose operational information and participate in structured management exchanges.
A Manager may use authenticated communication, reachability information, monitoring data, and operational events to assess an Agent's current condition.
When communication is interrupted, the system may need to classify the Agent as stale or offline.
But an offline Agent is not necessarily untrusted.
Likewise, a reachable Agent is not necessarily authorized to execute management operations.
SAMP explicitly separates liveness from trust.
This distinction matters during recovery.
An Agent that becomes reachable again has demonstrated renewed communication, not necessarily restored operational authority.
Depending on its trust state and applicable policy, additional validation may be required before new management operations are permitted.
The objective is not to restore every Agent to active operation as quickly as possible.
It is to restore safe, observable, and authorized operation.
3. When the Response Never Arrives
Consider a more difficult scenario.
A Manager sends an authorized EXEC request to an Agent.
The Agent receives the request and begins executing it.
Then the connection disappears.
Did the operation succeed?
Did it fail?
Is it still running?
Or did the Agent never receive the request?
Without additional information, the Manager cannot safely choose between these possibilities.
Automatically repeating the operation may create a second, unwanted effect.
Automatically declaring failure may be equally misleading.
SAMP addresses this problem through operation identity, request correlation, replay protection, recorded execution outcomes, and evidence-based reconciliation.
A consequential operation is associated with its original request identity and digest. Where the relevant authoritative operation record remains available, a matching retry can recover recorded state without initiating the effect again.
If execution genuinely cannot be established, the outcome remains indeterminate.
Resolving that uncertainty requires valid evidence attributable to the original operation, not merely a later observation that the desired state happens to exist.
And if that evidence is unavailable, the uncertainty must remain visible.
An unknown outcome is not permission to execute again.
SAMP does not promise universal exactly-once execution across crashes, lost records, or external systems. Reliable implementations must respect those limits.
4. Quarantine Is a Decision. Isolation Is a Fact.
Another important failure scenario involves an Agent whose behavior raises a security or operational concern.
The management system may determine that the Agent should be quarantined.
But recording that decision does not prove that the Agent has actually been isolated.
SAMP distinguishes the trust decision from its enforcement status.
For quarantine enforcement, the specification identifies four possible values:
| Enforcement status | Meaning |
|---|---|
requested | Isolation was requested but has not been confirmed. |
applied | An authorized enforcement actor confirmed isolation. |
failed | The requested isolation failed. |
indeterminate | The enforcement outcome remains unknown. |
This prevents an especially dangerous management illusion:
A dashboard displaying a successful quarantine while the Agent continues operating without the expected restrictions.
A Manager must not represent requested or indeterminate enforcement as confirmed isolation.
Intent is not enforcement. A command is not evidence of its effect.
5. From Detection to Evidence
A practical operational model can be expressed as:
Detect → Degrade → Recover → Correlate → Human Authority → Act Once → Evidence
This sequence is an illustrative operational workflow, not a mandatory SAMP state machine.
Detect: Identify an abnormal condition through monitoring, authenticated exchanges, or operational events.
Degrade: Represent reduced availability or capability without inventing certainty about the underlying cause.
Recover: Restore communication or service where possible, without silently restoring authorization or bypassing security policy.
Correlate: Connect observations, requests, responses, and execution records to reconstruct the incident.
Human Authority: Obtain human authorization where the operation, organizational policy, or potential consequences require it.
Act Once: Avoid unintended duplicate effects by preserving operation identity, validating authorization, and respecting recorded or indeterminate outcomes.
Evidence: Preserve the information needed to explain which decisions were made, which actions were attempted, what was confirmed, and what remains unresolved.
Some stages may overlap or repeat. Recovery can precede the complete reconstruction of an incident, and new evidence can change what investigators know.
The important principle is that uncertainty must not be silently converted into success.
6. Lessons from AgMS Alpha
AgMS, the experimental Agent Management System built around SAMP concepts, provides a practical environment for exploring these questions.
Its Alpha demonstrations include controlled operational disturbances such as an ingestion API timeout, missed Agent communication, and a temporarily unreachable Agent.
These scenarios help exercise monitoring, event reporting, degradation, and recovery behavior across heterogeneous components.
An Agent may disappear temporarily and return.
An external dependency may time out without causing the Agent runtime to crash.
An operational event may be detected before the complete cause is known.
These are useful tests because real operational environments rarely fail in perfectly predictable ways.
However, AgMS Alpha is an experimental implementation.
Its demonstrations do not establish production certification, exhaustive fault coverage, or complete conformance with every requirement of the evolving SAMP Internet-Draft.
The purpose of the Alpha is to test management concepts, expose engineering problems, and improve the implementation through observable behavior.
A successful demonstration is evidence of the particular behavior tested, not proof that every failure mode has been solved.
7. The Real Question Is What We Can Prove
As AI systems become more autonomous, management infrastructure must distinguish among three things:
- What the system was instructed to do.
- What the system reported doing.
- What can actually be verified.
This applies to availability, execution, authorization, quarantine, recovery, and incident investigation.
AI agents will fail.
Networks will fail.
Management systems will sometimes lose information.
Even a successfully executed operation may produce an uncertain management outcome when its evidence is lost.
The goal of SAMP is not to make those realities disappear.
It is to provide a structured foundation for managing agents without concealing them.
Because in operational engineering, the most dangerous answer is not always failed.
Sometimes, it is an unjustified succeeded.
A trustworthy agent management system must know when it does not know.
About SAMP: The Simple Agent Management Protocol is an experimental, vendor-neutral management-plane proposal for heterogeneous AI agents. Its Internet-Draft and implementation resources are available at samp-protocol.org and the IETF Datatracker.
