RCL 2.1.1: AI code review has to remember when it was wrong

One of our pull requests went through 24 Review Council evidence rounds because several AI reviewers kept rediscovering the same false positive.

We inspected the claim and dismissed it. In the next round, two models raised it again. RCL treated that corroboration as new evidence and reopened the finding. We dismissed it again. Then it came back again.

The models were doing what models do. The loop was our bug.

Today we published RCL 2.1.1. The core change is simple: a dismissal is now terminal on its evidence. If a finding returns against the same code, in the same location, with the same substance, more models repeating it do not make it new information.

This sounds like a small state-machine fix. It changes whether multi-model code review is a useful delivery gate or an infinite argument with a goldfish.

What RCL is trying to do

We built Review Council because asking one language model to review another language model's code is useful, but not remotely trustworthy enough to gate production changes.

RCL runs different models in different review roles, deduplicates their findings, measures agreement and writes a structured report. Our convergence workflow then follows a pull request across multiple exact Git heads. Findings receive stable identities and explicit outcomes: fixed or dismissed.

“Exact head” matters. A review of commit A does not approve commit B. If you fix a finding, you changed the code, so the new commit needs a new review. The requested fix may be incomplete or may create another defect.

The intended loop is:

review exact commit
      ↓
triage each gating finding
      ↓
confirmed → reproduce → fix → new commit → fresh review
unsupported → dismiss with evidence
      ↓
converge only when the machine can account for every finding

The difficult word in that loop is not “review.” It is “converge.”

The plausible mistake in 2.0

RCL 2.0 introduced cross-round finding identities. Titles are useless for this because models almost always describe the same problem differently. RCL instead anchors identity to file, category and location, with overlap matching as lines move.

That gave the review process memory. It could distinguish a new finding from a repeat, suppress a dismissed finding and avoid asking an agent to re-triage the same prose forever.

We still allowed one way for a dismissed finding to reopen: fresh corroboration. If at least two models raised it again, RCL classified it as regating and put it back in front of the engineer.

The reasoning seemed sensible. Independent agreement should count as evidence.

The problem is that AI reviewers are not independent witnesses. They see the same diff, share many training priors and often follow similar review prompts. A popular false positive can attract fresh agreement every round without a single fact changing.

In other words: consensus can increase confidence in a claim. It cannot, by itself, invalidate the evidence that disproved the claim.

RCL 2.1.0 replaced an overly tight three-round default because real reviews needed more room. That exposed the flaw immediately. Once the cap stopped ending the process, repeated corroboration could create an unbounded cycle:

dismiss
  ↓
two models repeat the claim
  ↓
regate
  ↓
re-triage
  ↓
fresh round
  ↓
repeat forever

One internal pull request made it to 24 rounds on one claim. That is not thorough review. That is broken termination logic.

Dismissals are now terminal on their evidence

RCL 2.1.1 changes the rule.

Once an engineer dismisses a finding with evidence, that finding stays suppressed when it returns. It does not matter whether one model repeats it or ten do. Repetition is visible in the ledger, but it does not reopen the gate.

This does not give dismissals unlimited scope.

• If a model raises a claim about different code, location-anchored identity makes it a new finding.
• If a previously non-critical claim escalates to critical, it regates once and needs fresh triage.
• If it is dismissed again at critical severity, that critical dismissal is terminal too.
• If the code changes to fix any finding, the changed head still requires a fresh review.

The important phrase is “on its evidence.” We are not teaching RCL that a class of bug can never exist. We are recording that a specific claim about a specific piece of code was investigated and rejected.

That is how human review works when it works well. “Three people left the same comment” is a reason to look carefully. It is not a reason to ignore the answer after somebody did.

Dismissal-only rounds now converge immediately

The second change removes a pointless confirmation round.

Suppose a review produces three gating findings. The engineer inspects all three, proves that each is unsupported and dismisses them. No code changes.

What could another review of the exact same commit establish?

Nothing. The patch is unchanged. The evidence is unchanged. Another round can only restate claims that already have dispositions or produce genuinely new findings, which the current round had every opportunity to do.

RCL 2.1.1 therefore makes the resolution explicit:

• converged-dismissal-only: every gating identity was dismissed and nothing was fixed; the unchanged head converges immediately.
• fixes-pending-fresh-round: at least one finding was fixed; the code changed and needs review at the new head.
• unresolved: one or more gating identities still lack a recorded outcome.

The CLI decides this from persisted run state. It records the severity at which a verdict was made and retains the current round's classified identities. The driving agent no longer has to reconstruct the stop condition from prose.

That last part is more important than it looks.

Agents need machine-readable stop conditions

A human reviewer can look at a thread and say, “we have dealt with this; move on.” An agent needs a rule it can execute.

If the rule says “continue until there are no comments,” an agent can review forever. Models can always produce another comment.

If the rule says “continue until all important comments are fixed,” the agent will fix plausible nonsense and make the code worse.

If the rule says “stop after three rounds,” the agent will call a capped process successful even when a real defect remains open.

The stop condition has to account for evidence:

1. Did enough reviewers return to make the round conclusive?
2. Which findings actually gate?
3. Is each gating finding new, repeated, suppressed or regating?
4. Was it fixed or dismissed, and why?
5. Did any fix change the Git head?

Only the machine has enough durable state to answer those questions consistently across sessions and agents.

This is a recurring lesson in agentic engineering. Natural language is good for expressing intent. It is bad at enforcing invariants. The moment a workflow matters, the boundaries need to become data, commands and state transitions.

There are good alternatives

We did not build RCL because nobody else was working on AI code review. The category is crowded, and several alternatives are better than RCL at parts of the job.

• GitHub Copilot code review is the lowest-friction option for a GitHub team. It behaves like a native reviewer, can re-review new pushes and offers suggestions where developers already work. GitHub's own documentation also says a re-review may repeat comments that were previously resolved or downvoted - which is almost exactly the lifecycle problem RCL 2.1.1 addresses.
• Graphite Agent automatically reviews pull requests with codebase context, suggested fixes and a dashboard for acceptance and downvote rates. It has a much better human-facing feedback loop, especially for teams already using Graphite's stacked-PR workflow.
• Greptile goes beyond reading the diff. Its TREX layer runs pull-request branches in a sandbox and attaches logs, screenshots, traces or videos to findings. That is stronger runtime evidence than a council of models reasoning statically about code.
• Amazon Q Developer combines generative review with rule-based detectors for SAST, secrets, infrastructure-as-code, software composition and deployment risks. RCL does not try to replace a mature deterministic security scanner.
• Bito's review CLI is closer to RCL operationally. It reviews local changes from the terminal, supports several source-control systems and exposes output intended for coding agents and automation. Its documented workflow even tells the coding agent to validate a finding against the actual code before fixing it (good advice).
• PR-Agent, Qodo's community-maintained open-source review project, is the closest alternative on deployment control. It can run locally, in GitHub Actions or self-hosted, supports several Git providers and can use different model providers.

CodeRabbit is the closest hosted comparison

CodeRabbit is probably what most teams actually want from AI code review. Install the app, open a pull request and get useful comments in the place where developers already work. No local ledger. No council command. No new operational vocabulary.

The workflow is genuinely good. CodeRabbit can review automatically, run an incremental review over new changes or start a full review from scratch. Developers can pause and resume it through pull-request comments. They can also tell it to resolve the comments it created.

That makes it a better human interface than RCL. If I am reviewing a pull request between meetings, I would rather type @coderabbitai review than inspect a local run directory and reason about convergence state.

But the two tools place the source of truth in different places.

CodeRabbit's documented incremental review considers its comments since the most recent full review and focuses on new changes. A full review deliberately disregards those prior comments and reviews the whole pull request again. That is sensible for a comment-oriented workflow. It is less useful when an autonomous agent needs to prove why an old finding must not gate again.

In RCL, resolving a finding is not just tidying a thread. The agent records whether the finding was fixed or dismissed, why that decision was made, and which Git head the evidence belongs to. A repeated finding can be matched to that history. A changed head creates a new review obligation. The machine, rather than the operator's feeling, decides whether the round converged.

I did not find stable finding identities, exact-head evidence, fixed-versus-dismissed verdicts, terminal suppression across rounds or a machine convergence result in CodeRabbit's public documentation. That does not mean CodeRabbit could not implement those mechanics internally. It means they are not the contract a developer is asked to operate.

If humans decide when review is done, CodeRabbit's model is excellent. Our agents need to carry that decision across sessions, context resets and linked worktrees without quietly reopening a disproven claim. For us, the ledger is worth the worse interface.

CodeRabbit asks: what should we comment on this pull request? RCL asks: has this exact head completed adversarial review, and can we prove why?

The same pattern holds across the rest of the market. The official material I reviewed does not document any of these tools combining all of RCL's specific mechanics: several independent generative reviewers, exact-head evidence, stable finding identities across rounds, explicit fixed and dismissed verdicts, terminal suppression and a machine-decided convergence result.

That does not prove those systems cannot implement similar behavior internally. It means their public contract is mostly the review and its comments. RCL's public contract is the state of the review process.

Why RCL works for us

If our goal were to give human developers useful comments inside GitHub, I would probably not have built another tool. Copilot or CodeRabbit would get us there faster. If the primary goal were runtime validation or security scanning, Greptile or Amazon Q would bring evidence RCL does not produce on its own.

Our problem is different.

An agent writes the change, pushes it, receives review hypotheses, checks them against source and tests, records fixed or dismissed outcomes, changes the code, launches another exact-head review and eventually merges. That loop can cross terminal sessions, cleared context and linked worktrees. A GitHub thread is a useful interface for a human. It is a weak source of truth for that machine process.

RCL gives us four things the workflow depends on:

1. Independent reviewers, not one reviewer persona. Different models run in different roles. Their failures, timeouts and agreement remain visible instead of being collapsed into one authoritative-sounding answer.
2. The Git head is part of the evidence. A finding and its verdict belong to the commit that was reviewed. A fix creates a new head and therefore a new review obligation.
3. Findings have a lifecycle. New, repeated, suppressed and regating are machine states. Fixed and dismissed are recorded decisions. The process survives the agent that started it.
4. Completion is executable. Attempt budgets, reviewer-health thresholds and convergence results are commands and persisted state, not instructions we hope the next agent interprets the same way.

There are costs. RCL takes more setup than installing a GitHub app. The terminal experience is not as polished as CodeRabbit's comments or Graphite's dashboard. Running several models costs more than running one. Static multi-model review still needs tests, CI, security scanners and, for risky changes, runtime evidence.

But we can inspect and change the whole mechanism. When a production review loop reached 24 rounds, we did not have to file a vendor ticket and work around it. We could find the bad state transition, change the contract, add tests and publish 2.1.1.

We did not need another bot that could leave comments. We needed a referee and a bookkeeper for agents reviewing agents.

We are making the review system skeptical of itself

RCL already treats review findings as hypotheses rather than instructions. A confirmed finding gets a reproducer and a fix. An unsupported finding gets a reasoned dismissal. Severity language and model confidence do not count as proof.

Version 2.1.1 applies that skepticism to consensus itself.

Agreement is valuable. It is one of the reasons to run a council in the first place. But agreement is not truth, especially when the reviewers are probabilistic systems looking at the same artifact.

A useful review system has to remember both kinds of evidence:

• Models agreed that a defect might exist.
• An engineer checked the runtime path and established that it did not.

Throwing away the second fact every time the first one repeats is not caution. It is amnesia.

RCL 2.1.1 shipped with 487 passing tests covering suppression, critical escalation, terminal critical re-dismissal, partial triage, old run states and all three resolution outcomes. Existing pre-2.1.1 state loads unchanged.

Install or upgrade it with:

npm install -g review-council@2.1.1
rcl --version

The full implementation and release notes are in PR #36.

The best code-review tool is not the one that produces the most findings. It is the one that helps you establish which findings are true, remembers the answer, and eventually lets you ship.