Post Syndicated from Ivo Kammerath original https://aws.amazon.com/blogs/architecture/scaling-patterns-for-self-organizing-multi-agent-clusters-with-kiro/
Most multi-agent systems today follow the same shape: a supervisor agent breaks a task down, hands the pieces to subagents, and stitches the results back together. This is how Kiro CLI delegates to subagents, and what the Strands Agents SDK gives you primitives for with graphs and agents-as-tools. It is a good default. One process holds the plan, so behavior stays predictable and every result passes through a single gate.
That one process is also the limit. Every assignment and every result flows through the supervisor, so its context window caps how much work the system can hold at once. If it dies, the run dies with it. And because a single planner fixes the decomposition upfront, you get one take on the problem, multiplied by N workers.
Plenty of distributed systems still coordinate centrally, and should. But the alternative has been around for decades: let participants converge through shared state instead. We wanted to know what happens when you apply that move to artificial intelligence (AI) agents, so we built kiro-flock, an open-source reference implementation. It runs clusters of Kiro CLI agents on Amazon Elastic Compute Cloud (Amazon EC2) with nothing between them but an Amazon Simple Storage Service (Amazon S3) bucket. No orchestrator, no message bus. Agents coordinate by reading each other’s append-only logs. This post explains the pattern and gives you enough to deploy the sample and watch a cluster converge yourself.
When to use this pattern
Architecture has to match the task. In the 2025 study “Towards a Science of Scaling Agent Systems” of 260 agent-system configurations they found exactly that: task performance ran from +80.8 percent on decomposable financial
reasoning to -70.0 percent on sequential planning, against a single-agent baseline.
Neither the supervisor nor this pattern wins everywhere.
A self-organizing cluster fits work that splits into many quasi-independent contributions toward one goal: reviewing a large code base, migrating hundreds of modules against a known target, generating tests or design alternatives at scale. It also suits brainstorming where you want real variety instead of one planner’s take. Parallelism matters more than ordering. Agents can join, fail, and leave without ceremony.
A supervisor fits the opposite profile. The task tree is known upfront, steps depend on each other, or you need a verification gate before results ship. The same study found that architectures without centralized verification propagate more errors. That is a real cost of removing the arbiter, though you can still gate the finished result the way the migration example ends in a full test pass. What a cluster will not give you is a gate between every step, and if you need that, the supervisor earns its bottleneck.
| Workload profile | Better fit |
| Many independent contributions, one goal | Cluster |
| Decomposition should emerge from the work | Cluster |
| Diversity of approaches is an asset | Cluster |
| Long-running, agents come and go | Cluster |
| Known task tree, strict ordering | Supervisor |
| Central verification gate required | Supervisor |
| Interactive, latency-sensitive | Supervisor |
The pattern
The core decision: coordination lives in shared state. No component plans, assigns, or aggregates for the rest. Three parts make it work:
- Agents. Independent processes that read and write a shared store and never connect to each other. One agent failing stops only its own log.
- A shared environment. A single store holds a direction file, one append-only log per agent, and a working area for artifacts.
- A direction. A markdown file that states the goal and leaves the path to the agents.
Each agent runs a loop. It starts a fresh session, reads the direction and the logs of a bounded set of peers, decides on one contribution that moves the goal forward, writes its artifacts, and appends one line to its own log:
That line is the entire coordination message. No broker delivers it, no acknowledgment comes back. The next agent that reads it decides for itself what to do about it. Remove an agent and its neighbors read one fewer log. Add one mid-run and it joins the division of labor already underway.
The bounded peer set is deliberate. Agents sit in a logical ring and each reads a fixed number of neighbors on either side, set by a radius parameter. Give every agent full visibility and the cluster collapses onto whatever the first agent wrote, because each later agent reads that as consensus. Limited visibility lets signals spread gradually, and agents working from different context get room to develop alternatives.
In kiro-flock, each agent is a headless Kiro CLI session on its own Amazon EC2 instance, and the shared environment is an Amazon S3 bucket. Which tools an agent may use without review, and what each iteration reads and writes, are design decisions you make once per cluster. We think of them as harness engineering and loop engineering, and the drift failure mode in the following section shows why the fresh session per iteration matters.
What a run looks like
Figure 1. Reference architecture for a kiro-flock cluster on AWS.
Agents run as headless Kiro CLI sessions on EC2 instances, each reading and writing the S3 bucket that holds the direction, one log per agent, and the shared artifacts. An Amazon API Gateway and AWS Lambda control plane behind Amazon Cognito starts, stops, and steers clusters from the dashboard. Agents publish metrics to Amazon CloudWatch, and Amazon Bedrock backs the post-run analysis.
Take a concrete run: sixteen agents in a ring, directed to hold a distributed discussion on AI agent clustering and converge on a shared synthesis. The operator writes one direction file and starts the cluster. Nothing else is assigned.
The following lines are from that run (result fields shortened for print). In the first iteration the agents fanned out with no assignment: failure modes, coordination topologies, distributed-systems parallels, and several overlapping stigmergy pieces, all written in parallel within one minute of start. The later lines show an agent correcting course after reading its neighbors, the synthesis forming, and the cluster winding itself down:
The cluster converged on a shared synthesis covering the angles in the direction, and by iteration 7 all sixteen agents had declared themselves idle. Nobody assigned the topics, arbitrated the synthesis, or told the cluster it was done. Even done is only a signal read from the logs, since an agent goes idle when its neighbors are idle and the output is stable.
Figure 2. A single cluster in the kiro-flock dashboard. Six agents run at radius 1, each on its own EC2 instance. Every agent card shows its neighbors and its latest log line, the “did / result / next intent” message its neighbors read. The right panel shows the shared environment in S3 and the direction the cluster works toward.
Three ways to answer “whose work do I read?”
Every iteration starts with that question, and the answer defines the coordination algorithm. kiro-flock ships three, swappable at runtime.
Amorphous (ring). Each agent reads a fixed window of neighbors set by radius R. An agent at radius 2 reads four neighbors whether the cluster holds 8 agents or 800, so per-agent work stays constant as the cluster grows. The ceiling is your EC2 vCPU quota, not the algorithm. The largest system we have run so far totaled 184 agents across 11 cooperating clusters, creating a programming language. Rings beyond the low hundreds are extrapolation from that constant per-agent cost, not tested territory. The price is speed: a signal moves one hop per iteration. That slowness is also what lets dissenting agents mature alternatives before the neighborhood locks in. Use it for parallel work, or as the opening phase before consensus.
Mesh (full visibility). Every agent reads every other agent’s latest entry. Alignment is fast and context grows linearly with the cluster, so mesh stays comfortable to about 30 agents and workable to about 50. Diversity collapses, because agents reacting to the same first signal agree instead of exploring. Use it when a small group must converge quickly.
Swarm (recency). Each agent reads the K most recently active peers, so the cluster reorganizes around where the action is. Good for ideation, runs well past 100 agents. If K stays small while N grows, most agents read the same few peers and pile onto one subtask. Raise K or switch to amorphous.
A productive sequence uses all three: open amorphous to explore, switch to swarm as a direction forms, finish in mesh to align on the output.
How long does convergence take? In a ring, one iteration carries a signal 2R positions, so full propagation takes ceil(N / 2R) iterations, and consensus roughly two to three times that, because agents observe, react, and confirm. The wall-clock column assumes an iteration interval of 30 seconds per agent loop, the default interval in the reference implementation. The interval is configurable per cluster.
| Agents (N) | Radius (R) | Propagation ceil(N/2R) | Consensus (2-3x) | Wall clock to propagate |
| 8 | 1 | 4 iterations | 8-12 iterations | about 2 minutes |
| 100 | 2 | 25 iterations | 50-75 iterations | about 12 minutes |
| 1,000 | 4 | 125 iterations | 250-375 iterations | about 62 minutes |
| 1,000 | 20 | 25 iterations | 50-75 iterations | about 12 minutes |
Radius trades per-agent context for convergence speed, as the last two rows show. For parallel map-style work, propagation barely matters. Agents only need to avoid duplicating each other. It costs you when the task needs consensus, so match radius and cluster size to the context you are in. The cost model follows the same logic: no always-on orchestrator and no broker. You pay for Kiro credits and the EC2 instances while they run, plus S3 storage and requests. You also pay for the AWS Lambda, Amazon API Gateway, and Amazon Bedrock usage the control plane and post-run analysis incur. See AWS Pricing.
None of this is new theory. Identical unreliable parts producing coherent global behavior through local reads is amorphous computing. Coordinating through traces left in a shared medium instead of messages is stigmergy, described by Grassé for termites in 1959 and formalized for artificial systems by Theraulaz and Bonabeau. And a set of append-only logs is a grow-only conflict-free replicated data type (CRDT) spreading gossip-style: replicas converge without locks, which is all the consistency this workload needs.
Where it breaks
Self-organizing clusters fail in ways orchestrated systems do not. With no supervisor to arbitrate, a bad signal can spread before anyone corrects it. Four failure modes recur, and each maps to a design choice rather than a safeguard bolted on afterward.
| Failure mode | Where it comes from | Design choice that addresses it |
| Groupthink | Mesh visibility collapses the cluster onto the first signal | Open amorphous to build diversity, switch to mesh only to align |
| Drift | Persistent session history builds behavioral momentum | Fresh session per iteration. State lives only in shared logs |
| Hot spots | Swarm with K too small for N starves subtasks | Raise K, or switch to amorphous |
| Carry-over | Stale files from a previous run read as current context | Archive environment/ and store/ to history/ on every start |
Drift deserves one more sentence, because it is the least obvious. An agent that keeps its session history carries a narrow reading of the direction forward even after its neighbors move on. Starting every iteration with no conversational memory sounds wasteful. It is actually the control that keeps a thousand independent loops steerable. The only state an agent carries is what it reads back from the shared logs.
Composing clusters
The same decision works one level up: clusters coordinate by reading each other’s shared environment, the way agents read each other’s logs. We run a structure we call WeltenBuilder: a feature cluster implements against an agreed interface, a shared-infrastructure cluster owns common services, a QA cluster reads across the others and reports inconsistencies as artifacts. A coordinator cluster writes conflict-resolution notes the others pick up on their next iteration. A resolution note is a trace, not a command. Remove the coordinator and you remove a signal, not a dependency.
Because the shared environment is the coordination plane, all clusters launch at the same time with no dependency graph to sequence. Contract bottlenecks dissolve the same way: a small mesh cluster converges on interface definitions in a few iterations while other clusters build against its latest stable output. This is where the pattern points: standing clusters, each producing one class of artifact, composed into a factory whose unit of work is a direction file and a topology.
Figure 3. Multiple specialized clusters in the WeltenBuilder dashboard, each with its own algorithm and agent count, coordinating only through the shared S3 environment on the right.
Try it
The kiro-flock reference implementation is open source under Apache 2.0. It is a sample to study and adapt, not a production system.
One setup script provisions the stack with the AWS Cloud Development Kit (AWS CDK): Amazon S3 for the shared environment, Amazon EC2 for the agents, and AWS Lambda with Amazon API Gateway as a control plane behind a dashboard. The dashboard starts and stops clusters, changes the algorithm, and updates the direction mid-run. Amazon Cognito handles access, and Amazon Bedrock backs a post-run analysis that summarizes how the cluster converged.
You need an AWS account with the AWS CDK bootstrapped. Install kiro-cli and create a Kiro API key for headless mode (requires a Kiro subscription). Then:
Direct a cluster in plain language: “Start a flock of 8 agents to review the files in my project and suggest improvements.” The default runs 8 agents at radius 1 and converged in 5 to 7 iterations in our runs. Before wider use, scope each agent’s EC2 AWS Identity and Access Management (IAM) role, restrict security-group egress to the endpoints agents should call, and add AWS Budgets alerts.
Conclusion
The supervisor pattern remains the right default for bounded task trees, whether you build it with Strands, Kiro CLI subagents, or any of the coding agents that delegate this way. When the work decomposes into many independent contributions and diversity matters more than a central gate, moving coordination into shared state helps remove the throughput ceiling and the single point of failure in one move. The convergence math and the failure modes both follow from that decision, and the distributed-systems results they rest on have been known for decades. Deploy the sample, read the logs as a cluster converges, and decide where your own multi-agent workloads belong.
