Engineering the Agentic Development Loop

Engineering the Agentic Development Loop

How we build software with agents at Collective AI

When we started shipping real work with coding agents, writing code stopped being the constraint. The agent is fast, and the output is usually decent. What we watched happen instead was drift: two modules that should share a helper carrying near-identical blocks, a service printing instead of logging, a boundary returning a raw dict, a test suite green in a way nobody quite trusted.

Our first answer was the obvious one — write the conventions down. A CONVENTIONS.md, a page in the wiki, a longer system prompt. That helped, and it is still where we start. But it turned out to be a different kind of artifact from the one we ended up building, and the difference is mechanical rather than editorial. A list of conventions is context an agent may or may not weigh on a given task. A pre-commit hook that fires when the agent tries to commit — and comes back with an error naming the deterministic rule it broke — is not weighed against anything else in the context window. The only way past it is to comply.

That distinction is the shape of our stack. It follows from one claim we now organise around: when the cost of writing code drops, the bottleneck moves to two things that are still hard — defining what to build and verifying that what got built is acceptable. So most of our effort goes into making the standard executable rather than into making it well written.

This post is what we run today: the loop, the definition chain, the rules layer, the AFK runtime, and then the concrete sequence we follow on a repo, command by command.

1. The loop has three stages, and each has a different owner

Definition is human and does not get delegated. This is the input the system cannot invent for us. What is being built, and how we will know it is done.

The agent loop runs on its own, against gates, until it goes green. Implement, verify, retry. No human in that circle.

Delivery goes back through a human, but over a bounded artifact: a pull request.

The boundary between the first stage and the second is where we spend the most attention. If the definition is fuzzy, the agent loop is a very efficient machine for producing a thousand coherent lines built on a wrong premise. So definition is the stage we instrumented hardest — and, counterintuitively, we instrumented it by making the agent ask us questions.

2. Definition: interrogation first, then vertical slices

The interrogation comes first

The failure mode we keep returning to was never bad code. It was misalignment: you thought the other side understood you, you saw the result, and you discovered it did not. Agents don't fix that. They accelerate it, because an agent doesn't push back on an underspecified request — it fills the gaps silently and confidently.

So we invert the direction of the conversation. Instead of asking the agent to produce, we put it to work interrogating us: concrete questions about the change until every branch of the design tree is resolved or explicitly deferred. We use Matt Pocock's skills for this, and it is the highest-leverage part of our stack. An hour of questions has consistently cost us less than a day of implementation in the wrong direction.

There are three front doors, and which one you pick depends on what is in front of you rather than on preference:

• /grill-me — no repo, no files. An idea, a decision, a change you can hold in one session. This is our default.
• /grill-with-docs — the same interview, but against the codebase and stateful: it reads the repo and keeps what it learns in CONTEXT.md and ADRs.
• /wayfinder — for an effort larger than a single agent session. It charts the work as a map of decision tickets on the tracker and runs grilling sessions inside it, so the map grows as questions surface and shrinks as they get answered.

The chain from there is deliberately boring, and each link has one job:

• /to-spec — turns the conversation into a spec on the tracker. It does not interview again; it synthesises what was already discussed. That restraint is the point: a spec that re-interviews you reintroduces assumptions the interrogation already ruled out.
• /to-tickets — splits the spec into tracer-bullet tickets, each declaring which other one blocks it. That declaration is what later makes parallelism possible without two agents colliding.

ADRs and CONTEXT.md are outputs of the interrogation, not homework

The part that made this stick for us is that grilling against the repo writes. Decisions come out as ADRs; vocabulary comes out as CONTEXT.md entries. Neither is a documentation chore scheduled for later — they are the residue of a session that already happened.

CONTEXT.md is the cheaper of the two and pays off every session. Its job is to give a recurring idea one name, so nobody has to describe it again from scratch. Take a course made of sections, and a lesson that only counts as real once it has been written to the filesystem. Without a name, every session re-describes that in a sentence and a half, and each re-description drifts a little: becomes real, gets created, lands on disk. With one, it's materialization — and the word means the same thing in the ticket, in the function name and in the review.

The payoff isn't brevity. It's that names stay consistent across modules, functions and files, the agent navigates the repo better, and it stops burning context rebuilding jargon on every cold start.

The ADRs do the other half: they keep the next slice from relitigating a decision the previous one settled. When an agent is the one reading them, a decision that lives only in a closed chat window is a decision that will be made again, differently.

We slice vertically, and this is the reason

When we split a spec into tickets, each one is a thin path through every layer it touches — a tracer bullet with an observable behaviour at the end — rather than a layer built out across the whole feature. No "add the data model", no "build the endpoints", no "wire the UI".

We didn't adopt this for the usual agile reasons. We adopted it because vertical slices are the only shape that fits the rest of the machine:

• A horizontal slice has no acceptance criterion you can execute. "Add the persistence layer" can't be asserted end to end, so the gates go green over code nothing exercises. A vertical slice ends in behaviour, which means the agent loop has something concrete to close against and the verify step is doing real work rather than confirming that the code compiles.
• A horizontal slice can't be reviewed on its own. Nothing is demoable until the last layer lands, so review arrives late and all at once. Agents lower the cost of producing a diff; they don't raise our capacity to read one. Each slice needs to be independently mergeable.
• Vertical slices collide less. Two agents both working "the model layer" fight over the same files. Two agents each owning a path through the system mostly don't — and where they do, the blocking edge is declared on the ticket, so the collision is scheduled instead of discovered.
• Wayfinder's decision tickets are the same idea one level up. A question whose resolution is a decision is a vertical unit too: it closes.

One rule of thumb we say out loud a lot: if you cannot write the acceptance criterion, you do not have a ticket — you have an intent. An agent can implement a criterion. It cannot implement an intent; it can only guess at one.

3. The rules layer: a standard that executes

We ship our house rules through a small bootstrap CLI (Botica) that installs them into a target repo. Each rule is a CES — Collective Engineering Standard, documented in docs/engineering-standards.md — and the property that matters is that a CES is not a paragraph. It is one of: an ast-grep pattern, a prek hook, a pyproject setting, or an AGENTS.md directive plus a detail file.

Two identifiers, two jobs

Every rule carries both:

Written together where there's room: CES-46 · log-no-print.

The number never goes inside the slug. Slugs are what tooling and suppressions key on, so they have to stay stable and semantic even when the issue moves. And the number comes from the tracker issue, which means gaps are meaningful — a declined proposal burns its number, PEP-style. Nothing gets a code by existing only in the code.

Three tiers of context, on purpose

A design decision that looks like a folder convention:

• Always-on index — a marker-delimited ## Engineering Standards section in AGENTS.md. One line per rule: terse directive, enforcer marker, pointer.
• On-demand detail — .agents/rules/<slug>.md. Full rationale, examples, migration, escape hatches.
• Canonical code — .agents/snippets/. Drop-ins for new repos, comparison templates for existing ones.

We keep the index in AGENTS.md rather than packaging it as a skill because for horizontal standards — the ones that apply to all the code, all the time — passive always-on context does the job that on-demand retrieval can't. A skill gets invoked once the agent decides it's relevant. A style rule has to be present before the agent decides anything.

The marker tells you who stops you

Every CES declares its enforcer, and the marker is not decorative:

That last row matters. Not everything fits in a pattern, and when we forced one anyway we got rules that were technically enforced and practically useless. [judgment] rules are honest about needing a reader — they just make sure the reader has the argument in front of them.

Violation messages are self-contained

A small thing with a large effect. The message states its own rule, the fix, and its code — nothing else:

message: "CES-46 (log-no-print): libraries log, they don't print — use the house get_logger."

What it must never do is name a sibling rule, or prescribe another rule's suppression key. The reason is context poisoning: scattering policy across neighbouring messages lets one rule restate — or quietly supersede — another rule's decision, and the agent takes the last thing it read as authoritative. The grouped rationale lives exactly once, in the detail file.

Never overwrite the repo

The installer is clean-adds-only. It never edits or overwrites an existing file. Whatever differs is reported as [defer] and left for a human or an agent to merge. There is no uninstall, because installing requires a git repo — git checkout and git clean are the undo.

That refusal is what makes it safe to run on a live repo: the CLI does the deterministic part, and everything that needs judgment — new-vs-existing detection, additive JSONC merges, per-item conflict resolution — is handed to an agent driving the install guide. Which is exactly how we run it, and the subject of section 6.

4. Skills carry the judgment that rules can't

Rules cover what fits in a pattern. Everything else — how to diagnose a hard bug, how to design a module with a lot of behaviour behind a small interface, how to spot a test that passes without protecting anything — travels as skills: how-to modules the agent loads when the task fits.

The axis that matters is who can invoke them. User-invoked skills are reachable only when you type them, and their job is to orchestrate. Model-invoked skills can be reached for by the agent on its own, and they hold the reusable discipline. A user-invoked skill may invoke model-invoked ones, never another user-invoked one — which keeps orchestration from nesting into something nobody can follow.

Which skills a repo uses is declared in skills-lock.json, a tracked manifest; .agents/skills/ is derived from it and stays gitignored. Think pyproject.toml / uv.lock: the manifest is the source of truth, the directory is the .venv.

Our review path is the clearest example of skills doing real work. /code-review reviews the diff since a fixed point along two axes, run as parallel sub-agents so they don't contaminate each other: standards (does it follow this repo's rules, plus a smell baseline?) and spec (does it faithfully implement the originating issue?). When we ran them as a single prompt we got a blurry compromise of both. Run separately, each comes back with things the other missed.

One hygiene rule we state explicitly: we review the diff and the PR, not the agent's conversation. The transcript explains how it got there. The diff is what merges. When a review starts arguing with the agent's reasoning instead of the code, it has stopped being a review.

5. Making it run without you

Definition and review are human. The middle should be AFK, and for us that turned out to be an infrastructure problem rather than a prompting one.

We built Orquesta (orq-lite) for it: a local, durable workflow runtime for coordinating coding agents, in Go. Two properties do the heavy lifting.

Behaviour is data. Flows, subflows, schemas, policies and prompts ship together as digest-verified packs. Installation validates the manifest, every declared SHA-256, unlisted files and symlinks before an atomic install under .orquestalite/packs/<name>/<version>. The compiler resolves versioned references and pins resource and pack digests into an immutable IR. The pack format and the runtime architecture are documented in the repo.

State lives outside the agent. Runs, materialised step instances, attempts, outputs, approvals and outbox events go to .orquestalite/workflows.db. A resume uses the pinned compiled IR and pack digest stored with the run, not whatever happens to be installed later.

That second one is the whole ballgame. A run that dies at minute 40 and can't be resumed hands back nothing. And a resume that picks up a definition edited in the meantime is worse than no resume at all, because you'll believe you ran the thing you designed.

Providers are configured through a team.json that separates agents from roles. A flow invokes a role; the role resolves to an ordered list of agents plus escalation agents, with rate-limit classification and bounded fallback between providers:

{
  "agents": {
    "sonnet":    {"provider": "claude-code", "model": "claude-sonnet-5"},
    "gpt-terra": {"provider": "codex", "model": "gpt-5.6-terra", "effort": "medium"},
    "opus":      {"provider": "claude-code", "model": "claude-opus-5"},
    "gpt-sol":   {"provider": "codex", "model": "gpt-5.6-sol", "effort": "high"}
  },
  "roles": {
    "coder": {
      "agents": ["sonnet", "gpt-terra"],
      "prompt": ".orquestalite/packs/development/5/prompts/coder.md",
      "result_path": ".orquestalite/results/coder.json",
      "timeout_seconds": 1800
    },
    "adversary": {
      "agents": ["opus", "gpt-sol"],
      "prompt": ".orquestalite/packs/development/5/prompts/adversary.md",
      "result_path": ".orquestalite/results/adversary.json",
      "timeout_seconds": 900
    }
  },
  "rate_limit_backoff": {"initial_seconds": 30, "factor": 2, "max_seconds": 1800},
  "lint_argv": ["go", "vet", "./..."],
  "test_argv": ["go", "test", "./..."]
}

Three things are worth reading off that file.

The first is that roles get the model their job actually needs: coder runs on a fast model, because writing the code is the part the gates will check anyway, while adversary runs on the strongest reasoning available, because its mandate is to find what the ticket failed to say. Model choice becomes a per-role decision instead of a global one, and it changes independently of the flow.

The second is that each role's fallback crosses providers on purpose. A rate limit or a degraded endpoint on one vendor should slow a run down, not end it — which is the same reason the state lives in SQLite rather than in the agent's session.

The third is the last two lines. The governed pack obtains project-specific gates only through lint_argv and test_argv; it hardcodes no Go, Python or Node commands. That keeps us from forking the engine per language, and it is also the seam where the rules layer plugs in: those two lines point at the checks Botica installed. One standard, four consumers — the agent reads it, the hook runs it, the runtime gates on it, CI confirms it.

The governed delivery loop separates delivery from post-delivery governance, and the governance phase is mandatory in both batch and ticket modes: plan a bounded objective, implement it, run deterministic gates and initial QA/repair, then always run integrated QA, adversarial analysis and code criticism, integrate findings within a policy budget, and run final gates. qa, adversary and critic have deliberately different contracts — QA validates behaviour, the adversary evaluates the declared product objective and its invariants rather than the literal ticket text, the critic reviews maintainability and convention fit. Three readings with separate mandates find different things; one combined reading, in our experience, finds the first thing and stops.

Where Orca fits

For the interactive side — several agents in parallel, each in its own worktree, with something to watch them from — we adopted Orca rather than building it. Durable and deterministic is our problem; a good desktop surface is not.

The split turned out clean. Orca owns the fleet: N agents, one worktree each, a surface to watch them from, and a trigger layer with cron and RRULE schedules. Orquesta owns the graph inside a worktree: compiled IR, pinned packs, checkpoints, approvals, resume. The seam between them is just a command — an Orca automation is a trigger plus an agent, and that agent's job can be to run an orq-lite flow. That is how our scheduled PR review works without us having written a scheduler, and it's the arrangement described at the end of the next section.

6. How we actually use it

Everything above is easier to see as one sequence. This is what we do, in order.

Step 1 — Botica into the repo

Open an agent at the root of the repo and paste the instruction from the Botica README:

Set up this repo by following the instructions here: https://raw.githubusercontent.com/collectiveai-team/scaffolding/main/guide.md Don't summarize it — follow every step.

An agent rather than the CLI, because the bootstrap needs judgment the CLI can't automate: new-vs-existing detection, additive JSONC and AGENTS.md merges, Python detection, per-item conflict resolution. The CLI does the deterministic clean-adds; the agent drives it and handles the merges. On a new or empty repo there's nothing to merge, so the fast path is the CLI directly:

uvx --from git+https://github.com/collectiveai-team/botica scaffolding install
scaffolding check          # nonzero exit if the bootstrap is incomplete

What lands: gitignore, per-agent config, the AGENTS.md standards index, prek hooks, ast-grep rules, CI workflows, Varlock secrets, and the skills declared in skills-lock.json. In other words, the whole rules layer from section 3 and the whole skills layer from section 4 — the gates the agent loop will run against, and the always-on context it reads before writing.

Step 2 — Orquesta into the repo

Same move, second agent session. Paste the instruction from the orquesta-lite README:

Set up Orquesta Lite in this repository by following every instruction in: https://raw.githubusercontent.com/collectiveai-team/orquesta-lite/main/guide.md Do not summarize the guide—perform the steps, verify each gate, and stop to report any blocker that cannot be resolved safely.

The guide covers installing orq-lite if it isn't on PATH, initialising the built-in development pack, configuring team.json, proving the lint and test gates actually run, choosing a development and review strategy, launching idempotently, and monitoring or resuming the durable run.

That third item is the one to watch, because it's where the two halves meet. team.json's lint_argv and test_argv should point at the checks Botica installed in step 1. When they do, the runtime isn't enforcing a private opinion about quality — it's enforcing the repo's standard, the same one the pre-commit hook and CI enforce.

init leaves behind team.json, .orquestalite/packs/development/5 (the built-in digest-verified pack), and .orquestalite/workflows.db on the first run. Then:

orq-lite doctor   # pack, roles, prompts, providers, credentials, gates

Worth not skipping. A run that dies at minute 40 on a missing credential is a run you could have failed in two seconds.

We also install the runtime's own skill, using-orq-lite, which is the working reference for the flow lifecycle:

npx skills add collectiveai-team/orquesta-lite --skill using-orq-lite

Step 3 — Per feature: grill, spec, tickets

For each feature, one definition session before any code:

1. /grill-me — the default. /grill-with-docs when there's a codebase to align against and we want the ADRs and CONTEXT.md entries written; /wayfinder when the effort is bigger than one session.
2. /to-spec — synthesise the conversation into a spec on the tracker. No second interview.
3. /to-tickets — vertical slices, each with an acceptance criterion and its blocking edges declared.

That last step is the handoff. /to-tickets can write the set as text in a local file, and that file is what the runtime takes as input.

Step 4 — Hand it to the default development flow

The pack that ships with orq-lite init covers the whole middle stage. The familiar commands are aliases into it:

For a feature, that's two commands and then nothing:

orq-lite plan features.md
orq-lite factory features.md --pr

The reason this is AFK and not a gamble is that none of it depends on the agent being right, or on the agent reporting honestly:

• The gates are the repo's own, via lint_argv / test_argv.
• A green step is evidence; a claim isn't. "It's working already" is not a state the runtime recognises. The gate result is.
• Governance always runs. QA, adversary and critic execute after delivery, not instead of it, with findings integrated inside a policy budget.
• A dead run is recoverable. orq-lite flow resume <run-id> continues the definition that started the run, because the IR and the pack digest are pinned to it.
• A rate limit slows it down, it doesn't end it. Each role falls back across providers.

While it runs, orq-lite status --watch and orq-lite flow events <run-id> are what we actually look at; orq-lite cost rolls up what it spent.

Step 5 — PR review on a schedule

The review flow is in the same pack (orq-lite review --pr N), so it doesn't need to be triggered by hand. We configure an Orca automation on an hourly cron trigger, with a precheck so it only spends model time when there's something to look at, and the automation's job is to run the orq-lite review flow against the open PR.

What that buys is ordering. By the time a human opens the PR, the deterministic pass and the standards-versus-spec review have already run and left their findings on it. The human is reading a diff that has been checked, not checking a diff. The third stage stays human — we just don't want it to be the first thing that looks at the code.

Changing a flow, or writing a new one

Flows are data, so changing one is an editing task, and the using-orq-lite skill installed in step 2 is the reference for it: choosing a flow, writing team.json, monitoring and recovering runs, and authoring flows. We point an agent at it, then verify before spending model time:

orq-lite flow validate development/task-list@1
orq-lite flow inspect development/task-list@1
orq-lite pack install ./my-pack

Pack and flow versions are independent, and both can be pinned (development@5/task-list@1) when a run has to be reproducible.

7. What this doesn't solve yet

Our rule set is mature for Python, not yet for TypeScript. ruff, pyrefly, ast-grep rules, deptry and complexipy give us real deterministic coverage on the Python side, and the polyglot hooks (duplication detection, secret scanning, Conventional Commits, Dockerfile lint, dependency scanning) apply to any repo. The gap isn't the tooling. Biome now ships several hundred rules, type-aware and cross-file project rules, and custom syntactic patterns through GritQL plugins — which is close to the role ast-grep plays in our Python set. What's missing is the part that's ours: the CES set that decides which of those rules we turn on, which house patterns we add, and what each violation message says. Until we write it, TS is covered by the polyglot hooks and by skills rather than by rules of its own, and that's the biggest gap between the story and the artifact.

Our skills manifest is a manifest, not a lock. It carries a ref only when the source came pinned, tags are mutable, and the hash it writes is never verified on restore. It reproduces which skills, from where, at what ref — not bytes. We call it a manifest deliberately, because calling it a lock invites trust in a guarantee it doesn't give.

Local hooks can be skipped. --no-verify exists. That's why anything that genuinely matters — commit policy, dependency vulnerabilities — also lives in CI. A pre-commit hook is a fast feedback loop, not a control.

The short version

If you take one thing from this: write your standard as something that executes. Not a document, not a system prompt — a rule with an id, an enforcer, and a message that names itself.

The rest is what we found follows from there. Deterministic first, judgment second, because judgment is the expensive resource. Slice vertically, because a slice that ends in behaviour is a slice a gate can check. Put the decisions back in the repo as ADRs and vocabulary, because a decision that lives in a chat window will be made again, differently. The standard travels with the repo, not with the person or the prompt. Never overwrite decisions a repo already made. Trust the gate, not the agent's report that it's working.

None of this makes the agent smarter. It makes the agent's output checkable, which for us has mattered more.

Botica and Orquesta are open source. AI Bundles, our data-science counterpart, is private during an internal development window. The engineering workflow skills we lean on come from mattpocock/skills.