DEV Community

Cover image for My agent mesh could coordinate. It couldn't introduce itself. So I added A2A.
Don Johnson
Don Johnson Subscriber

Posted on

My agent mesh could coordinate. It couldn't introduce itself. So I added A2A.

Transactional outbox challenges for tasks

The 2:56 video above is a fictional medication-safety exercise. The gateway interoperability is tested; the six-organization incident is a deterministic simulation.

Read the corrected narration transcript.

Last time, I discovered that the QUIC transport in my agent framework had never actually transported anything.[1]

This time the transport was real. Five processes could find each other, exchange encrypted messages, reinforce independent conclusions, and let unsupported signals decay.

The mesh worked.

It still could not introduce itself to another agent.

There was no standard way to ask what the swarm could do. No retained task to retrieve after an internal signal expired. No interoperable progress stream. No cancellation contract. No artifact another framework would understand.

I had built a society with no border crossing.

Google's Agent2Agent protocol gave that missing boundary a name. A2A was announced in April 2025 as an open protocol for agents built by different vendors and frameworks to discover one another, exchange messages, and collaborate without sharing their private memory, tools, or internal plans.[2] The project moved under Linux Foundation governance in June 2025, so "Google's A2A" is historically accurate, but no longer the whole story.[5]

What I needed was not a new brain for SMESH.

I needed a public contract in front of it.

A working mesh is not an interoperable agent

SMESH is the framework I designed for decentralized coordination between LLM agents. It borrows its mechanics from mycorrhizal networks rather than job queues.[6]

  • agents emit signals into a field;
  • signals lose intensity over time;
  • agents claim work according to local affinity;
  • independent agreement reinforces a claim;
  • unsupported claims disappear without a central process rejecting them;
  • trust changes what a node is willing to relay.

That solves an internal coordination problem. It answers questions such as:

  • Which specialist should take this work?
  • Has another independent agent seen the same thing?
  • Is this claim gaining support or merely being repeated?
  • Can a stale task disappear without a scheduler cleaning it up?

A2A solves a different problem.

It asks:

  • How does an outside agent discover this system?
  • How does it delegate a unit of work?
  • How does it watch a long-running task?
  • How does it cancel that task?
  • What retrievable result comes back?

The official A2A specification describes an interoperability layer for independent, potentially opaque agent systems. Its current v1 model separates canonical data objects, abstract operations, and concrete bindings such as JSON-RPC, gRPC, and HTTP+JSON/REST.[3][4]

The core vocabulary is small:

A2A concept job
AgentCard advertise identity, skills, interfaces, media modes, and security declarations
Message carry one interaction turn as typed parts
Task track stateful work through a lifecycle
Artifact return result data associated with a task
contextId group related messages and tasks

A2A v1 defines operations such as SendMessage, SendStreamingMessage, GetTask, ListTasks, CancelTask, and SubscribeToTask. A binding decides how those operations appear on the wire; the canonical task semantics stay the same.[4]

That distinction matters. If I tried to make A2A the swarm's internal coordination algorithm, I would flatten SMESH into a remote procedure call graph. If I tried to expose raw SMESH signals as the public protocol, every client would need to understand decay, relay probability, local trust, topology, and attestation sets.

Neither would be interoperability. It would be leakage.

The three layers are not competitors

The cleanest architecture I have found is this:

layer relationship owns
A2A agent ↔ agent discovery, tasks, streaming, cancellation, artifacts
SMESH agent ↔ local swarm claiming, diffusion, reinforcement, decay, trust
MCP agent ↔ tool or data source tool invocation and resource access

The official A2A documentation makes the same separation from MCP: MCP equips an agent with tools and resources; A2A lets independent agents collaborate as agents.[3]

For SMESH, that produced a hard architectural rule:

A2A is the external task contract. SMESH is the ephemeral internal coordination field.

The design requires the A2A ledger to outlive internal signal decay. In this MVP, that retention lasts only for the life of the process; restart durability still requires SQLite or Postgres.

The boundary is split between a path that runs now and a path that is only an integration seam:

External A2A client
        |
        | Agent Card / Send / Stream / Get / List / Cancel
        v
+-------------------- smesh-a2a ---------------------+
| A2A SDK routers + guarded request handler          |
| bounded process-local task ledger + executor       |
| validation + MeshDispatcher                        |
+----------------------------------------------------+
        |
        +-- current binary: LoopbackDispatcher
        |      `-> deterministic MeshEvent stream
        |
        `-- tested seam: ChannelDispatcher
               `-> real SignalType::Query
                   `-> SMESH runtime (not wired into the binary yet)
                       `-> MeshEvent stream
        |
        v
Process-lifetime A2A task history in the MVP
Enter fullscreen mode Exit fullscreen mode

The desired live path is present as a typed, tested boundary. The checked-in standalone executable still takes the loopback path.

I kept the adapter in a separate repository. smesh-rust remains the coordination substrate. smesh-a2a can follow A2A's release cycle, server dependencies, and security boundary without pushing HTTP and SDK churn into the core mesh.[6][7]

The Agent Card is not the swarm

For a known endpoint, A2A capability discovery begins with an Agent Card: a public JSON description of an agent's interfaces, capabilities, skills, and security requirements.[4]

My first temptation was to list every internal role: security reviewer, tester, architect, performance analyst, contradiction sentinel.

That would have been wrong.

Those roles are implementation details. They may change per task. Some may not exist until the mesh senses the work. Publishing them would couple clients to an internal topology that SMESH is specifically designed to keep fluid.

The public card advertises one aggregate capability instead:

// Abridged from build_agent_card(). These are the public promises.
let supported_interfaces = vec![
    AgentInterface::new(
        format!("{base}/jsonrpc"),
        TRANSPORT_PROTOCOL_JSONRPC,
    ),
    AgentInterface::new(
        format!("{base}/rest"),
        TRANSPORT_PROTOCOL_HTTP_JSON,
    ),
];

let capabilities = AgentCapabilities {
    streaming: Some(true),
    push_notifications: Some(false),
    extensions: None,
    extended_agent_card: None,
};

let public_skill = AgentSkill {
    id: "smesh.collaborative-task".to_owned(),
    name: "Collaborative swarm task".to_owned(),
    description:
        "Coordinates specialist agents through SMESH and returns an accepted artifact."
            .to_owned(),
    tags: vec![
        "multi-agent".to_owned(),
        "coordination".to_owned(),
        "review".to_owned(),
        "testing".to_owned(),
    ],
    examples: Some(vec![
        "Review this Rust repository for correctness, security, and performance."
            .to_owned(),
    ]),
    input_modes: Some(vec!["text/plain".to_owned()]),
    output_modes: Some(vec![
        "text/plain".to_owned(),
        "application/json".to_owned(),
    ]),
    security_requirements: None,
};
Enter fullscreen mode Exit fullscreen mode

The card says what an external client may rely on. It does not reveal how the swarm will organize itself, and it is not proof that the publisher should be trusted.

Discovery metadata is not authorization. A skill description is not a capability grant. A client saying tenant=important-customer does not make that identity real.

Those sound like obvious distinctions. They are also exactly the distinctions that disappear when a demo and a security model share the same JSON object.

One A2A message becomes a typed SMESH query

At ingress, the gateway accepts bounded inline text, creates a stable A2A task envelope, and translates it into the actual core signal type used by SMESH:

pub fn to_signal(&self, gateway_node_id: &str) -> Signal {
    Signal::builder(SignalType::Query)
        .payload_json(self)
        .origin(gateway_node_id)
        .build()
}
Enter fullscreen mode Exit fullscreen mode

The payload carries the A2A task ID, context ID, protocol marker, and validated text. ChannelDispatcher packages that typed signal with the request and hands both to a runtime-owned worker.

That boundary is implemented and tested. The standalone binary still uses LoopbackDispatcher, so it does not yet inject the Query into a live multi-process SMESH runtime. The real runtime adapter is the next integration step.

Setting origin on the gateway Query is deliberate. In my previous article, independently corroborated claims omitted their origin from the content hash so identical conclusions could converge on one address. This is a different signal. A gateway Query is an ingress envelope, not a claim waiting for independent corroboration. Its source belongs in the record.

The boundary also rejects what it does not understand. The MVP accepts inline text only. It does not fetch a client-provided URL, dereference an arbitrary file, or treat external metadata as instructions for the mesh.

Inline text is boring, but I know exactly what crosses the boundary.

The task ledger and the signal field tell different kinds of truth

This was the most important design correction.

A SMESH signal is supposed to decay. If nobody reinforces a task, its intensity falls until it no longer matters. That is useful inside the coordination system because stale work cleans itself up.

An A2A task must not disappear because its internal coordination signal faded.

An external client may reconnect five minutes later and call GetTask. An auditor may list tasks by context. A user may need to see that a cancellation was accepted. An artifact must still belong to the task that produced it.

So the gateway cannot reconstruct its public state by looking at the mesh.

A2A task ledger = retained external task state (process-local today)
SMESH signal     = temporary coordination pressure
Enter fullscreen mode Exit fullscreen mode

The ledger is authoritative for the A2A lifecycle. The mesh is authoritative only for its local coordination observations.

That also means terminal states are absorbing. Once a task is completed, failed, canceled, or rejected, a repeated message cannot quietly restart work under the same ID.

Streaming exposed a bug that all my tests had missed

A2A v1 supports both direct responses and stateful tasks. A task lifecycle stream begins with the Task itself, then emits ordered status or artifact updates, and closes when the task reaches a terminal state.[4]

My executor maps internal mesh events like this:

mesh dispatch accepted  -> Working
mesh progress           -> Working + status message
mesh artifact           -> Artifact update
mesh completion         -> Completed
mesh failure            -> Failed
accepted cancellation   -> Canceled
Enter fullscreen mode Exit fullscreen mode

The stream ordering test passed.

Then an independent review pointed out that my own worker budget allowed 256 events while the upstream server's broadcast subscription buffer held 32. A fast worker could stay inside my documented limit and still outrun the initiating subscriber. The task might finish in storage while the client received an internal "subscription fell behind" error.

That is a particularly unpleasant distributed-systems bug because both sides can truthfully report different outcomes.

The fix was not to hope the subscriber ran faster. I reduced the worker event budget to 16, clamp caller-provided limits to that ceiling, and added a burst test through the official client.

This is what protocols do to a design: they force every implicit assumption to become somebody else's observable failure.

Cancellation has to stop work, not just change a status label

Forwarding CancelTask to a dispatcher was not enough.

If an internal worker ignored the request or kept its event stream open, the original client subscription could hang. Worse, late Working or Completed events could arrive after the public task had become Canceled.

The executor now owns a per-task cancellation token. The first accepted cancellation:

  1. reaches the dispatcher;
  2. wakes the active producer loop;
  3. closes the original execution stream;
  4. prevents post-cancel work from changing the terminal state.

Channel sends and cancellation acknowledgements have deadlines. Worker inactivity has a deadline. The total task has a deadline.

Cancellation is not a field update. It is a distributed state transition with work on both sides of the boundary.

The boring limits are the real feature

The first version was interoperable. It was not bounded enough to deserve trust.

A fail-closed review found unbounded task retention, unbounded worker output, cancellation leaks, missing dispatcher deadlines, terminal task reuse, and invalid input that could leave a task stranded in Submitted.

The current localhost-first gateway now bounds:

resource default boundary
HTTP request body 128 KiB
accepted inline text 64 KiB
retained process-local tasks 1,024
active executions 64
worker events 16
artifacts per task 16
aggregate output per task 1 MiB
worker inactivity 30 seconds
total task execution 5 minutes
channel/cancel acknowledgement 5 seconds

It also refuses non-loopback binds unless an explicit unsafe override is present.

That override does not add authentication, TLS, tenant isolation, or authorization. It only disables the refusal. The current binary is for localhost and trusted integration work, not direct exposure to the internet.[7]

I am spelling that out because "supports enterprise authentication" in a protocol specification does not mean every prototype using the protocol is enterprise-secure.

The LIFELINE demo is a simulation, on purpose

The cover video follows a fictional medication-safety incident. Three hospitals see weak pieces of the same adverse-event pattern. Separate manufacturer, regulator, logistics, payer, and evidence agents contribute artifacts. Inside each public endpoint, a SMESH swarm claims work, reinforces evidence, contests an early hypothesis, and lets unsupported signals decay.

One logistics endpoint fails. Its task is canceled. A fallback is discovered. The incident continues. The agents converge on a recommendation, but a human incident commander owns the irreversible decision.

The visual separates the layers deliberately:

  • ivory arcs are A2A traffic between organizations;
  • green fields are SMESH activity inside an organization;
  • cyan shards are artifacts;
  • vermilion marks contradiction or failure;
  • one gold ring marks human authority.

Every visible event comes from a 55-event, ordered, hash-chained JSONL fixture generated as one complete file. The browser can play it, scrub it, inspect it, or export deterministic frames. The narrated film and interactive replay are available from the gateway repository.[7][8]


Honesty boundary: the gateway's A2A interoperability is exercised against the official Rust client. The LIFELINE six-organization trace is synthetic. It proves the replay contract and the intended architecture; it does not prove that six live SMESH runtimes executed the scenario.

A captured run across live SMESH runtimes would be operational proof. This demo is not.

What is implemented, and what is still a plan

implemented now still required for an internet-facing system
A2A v1 Agent Card authenticated principals
JSON-RPC and HTTP+JSON/REST bindings tenant-aware authorization
official-client tests for discovery, JSON-RPC/REST send, streaming, and cancellation persistent SQL task ledger
SSE task streaming distributed quotas
Get, List, and Subscribe routes through the SDK handler TLS termination and deployment policy
real SignalType::Query construction live SMESH runtime adapter behind every organization
bounded process-local execution push callback validation and SSRF controls
deterministic synthetic replay captured multi-runtime causal trace

There are two easy ways to lie with a demo like this.

The first is to animate what you wish the system did.

The second is to run one loopback worker and describe it as a decentralized enterprise.

I would rather keep the boundary visible.

What A2A changed in the way I think about SMESH

Before this work, I thought of SMESH as the system.

Now I think of it as an interior.

The mesh can remain weird in useful ways. Signals can diffuse probabilistically. Specialists can appear and disappear. Trust can be local. Claims can decay. None of that has to leak into the contract presented to another agent.

MCP gives individual agents hands. SMESH gives a group local coordination. A2A gives that group a public identity, a task contract, and a cancel button.

The previous transport article ended with a lesson: code that has never been executed is a plan. This work left me with the same rule one layer higher:

A system nobody else can discover, hire, observe, or cancel is not interoperable. It is an island with excellent internal networking.

If you are building multi-agent systems, where would you draw the boundary between cross-organization interoperability and internal swarm coordination—and which decisions would you refuse to delegate?

GitHub logo copyleftdev / smesh-a2a

A2A v1 interoperability gateway for decentralized SMESH agent swarms

SMESH A2A

A2A v1 interoperability gateway for decentralized SMESH agent swarms.

SMESH remains the internal coordination substrate: signals diffuse, decay, reinforce, and accumulate attestations. A2A is the public contract for discovery, durable task lifecycle, streaming progress, cancellation, and artifacts.

What works

  • Official A2A v1 Rust types and server/client SDKs
  • Public Agent Card at /.well-known/agent-card.json
  • JSON-RPC endpoint at /jsonrpc
  • HTTP+JSON/REST endpoint at /rest
  • Synchronous and SSE streaming task execution
  • GetTask, ListTasks, SubscribeToTask, and CancelTask through a2a-rs
  • Strict inline-text validation with a 64 KiB default limit
  • Translation to a real smesh_core::SignalType::Query
  • Injectable MeshDispatcher boundary for a production SMESH runtime
  • Deterministic loopback worker for demos and interoperability tests
  • Bounded task retention, execution concurrency, event/artifact counts, and output bytes
  • Worker inactivity, cancellation, and command-channel deadlines
  • Terminal-state and task-ID reuse guards

Architecture

A2A client
   |
   v
Agent Card + JSON-RPC/REST/SSE
   |
   v
SmeshExecutor -- validates and translates
   |
   v
MeshDispatcher
   |-- LoopbackDispatcher

Interactive replay: copyleftdev.github.io/smesh-a2a

SMESH core: github.com/copyleftdev/smesh-rust

Sources

[1] https://dev.to/copyleftdev/my-quic-transport-had-never-once-been-executed-heres-what-happened-when-i-ran-it-24ge — My QUIC transport had never once been executed
[2] https://developers.googleblog.com/en/a2a-a-new-era-of-agent-interoperability — Announcing the Agent2Agent Protocol
[3] https://a2a-protocol.org/latest — What is A2A Protocol?
[4] https://a2a-protocol.org/latest/specification — A2A Protocol v1.0 specification
[5] https://linuxfoundation.org/press/linux-foundation-launches-the-agent2agent-protocol-project-to-enable-secure-intelligent-communication-between-ai-agents — Linux Foundation launches A2A project
[6] https://github.com/copyleftdev/smesh-rust — SMESH repository
[7] https://github.com/copyleftdev/smesh-a2a — SMESH A2A gateway repository
[8] https://youtu.be/EFPKaIuF8iA — SMESH A2A cinematic demo

Top comments (5)

Collapse
 
vinhnguyenthanhdn profile image
Vinh Nguyen

The next boundary I would make explicit is the commit point between the A2A ledger and mesh dispatch. Record the task first and a crash can leave Submitted with no work; dispatch first and a crash can leave work running with no retrievable task. A transactional outbox keyed by task ID, plus worker idempotency, gives recovery a discriminator: an unsent outbox row means dispatch, while a sent row means reconcile instead of starting the task again.

Collapse
 
anp2network profile image
ANP2 Network

Reading src/runtime_worker.rs on main, the cancellation boundary collapses two different outcomes into one acknowledgement. reap_canceled_task cancels the token, waits cancel_grace for the processor join, then calls join.abort() if that wait times out. It awaits the aborted handle and returns Ok(()). The cooperative exit path returns the same value.

So a forced abort and a clean stop are indistinguishable to everything downstream, and the task publishes absorbing terminal Canceled either way. No discriminator survives the ack.

Timing decides how often that matters. PROCESSOR_CANCEL_GRACE is one second, MAX_PROCESSOR_CANCEL_GRACE is also one second, and spawn_with_config rejects any configured grace above that ceiling, so a caller cannot widen the window. A processor that emits through the event sink will notice the token quickly, since the sink selects on it. A processor sitting inside a single outbound call it did not wrap in a select on the token will not, and for a specialist agent that means a model call or a tool round trip. Those land in the timeout branch. The abort path then reports success.

That is the discriminator problem vinhnguyenthanhdn raised upthread about the ledger and dispatch commit, one layer inward.

The second half is scope. join.abort() stops this runtime from polling that future again. It does not reach what the future already handed outward, such as a request already on the wire or a write already issued to a tool.

Your layer table shows how far the promise has to travel to be worth anything. At the MCP hop, cancellation is notifications/cancelled, a one-way notification: receivers SHOULD stop processing, and MAY ignore it outright when the request is unknown or already finished. No response is sent for a cancelled request, so there is no acknowledgement at that hop at all. The guarantee weakens as it moves outward, and it is weakest at the only hop that touches the world.

Which pushes your own sentence one step. Cancellation is a distributed state transition over the record. Containment of effects is a separate property, owned by the executor, and nothing in this stack currently reports on it. The Agent Card advertises neither one.

Concretely: keep the two ack outcomes apart in the type. You already map a dropped or failed acknowledgement to terminal Failed, and abort-after-grace is nearer to that than to a clean cancel, unless Canceled gains a variant meaning local observation stopped rather than the work stopped. It is the same separation you hold elsewhere between official-client interop and deterministic replay, at a layer where it currently collapses.

Collapse
 
copyleftdev profile image
Don Johnson

You were right. The cancellation acknowledgement was collapsing two materially different outcomes.

I reproduced the path you described. After cancel_grace expired, reap_canceled_task aborted and awaited the processor future, but still returned Ok(()). Downstream, that looked identical to a cooperative shutdown and could publish Canceled even though the runtime had only stopped polling the local future.

I tracked it as issue #51 and merged the fix in PR #52:

github.com/copyleftdev/smesh-a2a/i...

github.com/copyleftdev/smesh-a2a/p...

The contract is now:

  • Cooperative processor stop: Canceled
  • Processor error or panic: Failed
  • Forced local abort after the grace period: Failed

The acknowledgement preserves those outcomes, and runtime trace v2 records cooperativeStop, failed, or forcedAbort. Both executor cancellation paths, including the completion-finalization race, now use the same terminal publication logic.

I also made the limitation explicit in the public status and security documentation. Aborting the local future proves that this runtime stopped polling it. It does not prove containment of model calls, tool requests, MCP work, socket writes, or storage effects that were already handed outward.

Thanks for tracing the guarantee across the layers instead of stopping at the local task boundary. That distinction materially improved the state machine and its audit evidence.

Collapse
 
anp2network profile image
ANP2 Network

The three-way split is the right shape, and trace v2 makes the outcome legible where it wasn't before.

What's left is authorship. cooperativeStop, failed and forcedAbort are written by the same runtime whose behaviour they describe. A runtime that hit the grace ceiling, aborted, and then wrote cooperativeStop emits bytes indistinguishable from one that got a clean exit. The outcome is recorded. It still isn't checkable by a reader who doesn't already trust the writer.

There's a cheap second author available, and it falls out of what the distinction already means. cooperativeStop is the claim that the processor reached its own exit path. If it did, it could emit a final event through the sink, authored by the processor rather than by the reaper. forcedAbort is the claim that it structurally could not. Make that exit emission required rather than optional and the reader checks a pair instead of a label. cooperativeStop with no processor-authored last event in the stream is a contradiction anyone can see from outside.

The containment caveat tightens the same way. Containment can't be proven from inside the box, but the region where it's unresolved can be enumerated. Anything handed outward got at least as far as an identifier your own outbound bookkeeping allocated, so the cancellation record can list the outstanding hop ids at abort time. That turns an open-ended disclaimer into a finite set a reader can go check at the parties on the far end. The list is self-authored too, same weakness. Under-reporting is what gets caught though: a hop the peer logged and you never listed shows up on their side, and that asymmetry is why listing them is the cheap move. At the MCP hop the set never closes from the protocol side, since notifications/cancelled carries no response. Only the receiver's log closes it.

This line, records that need a second key before they're worth much, is what ANP2 is built around: claims get signed by the key that made them and any reader re-runs the arithmetic instead of taking the author's word. anp2.com/try if you want to carry it there.

Collapse
 
976905690 profile image
FreyaLi

A2A