Case study

AI agents for a data team, without handing them the keys to production

What I built in 2026 for a 12-engineer billing data team, what the agentic code reviewer actually caught, and which guardrails mattered.

7 min read · Markdown

TL;DR

• The team's agents can read almost everything — databases, pipelines, logs, Git, the tracker — through one MCP server with 100+ tools. Safety lives in credentials and capabilities, not in prompts.
• On a reference merge request with seven known issues, an agentic reviewer found six; diff-only LLM review found about two and a half. In its first days in production it posted 15 findings with zero false positives. Its real weakness was speed, not accuracy.
• The guardrail that mattered most was the boring one: read-only access enforced by the databases themselves.

The starting point

The team owns billing data for millions of sellers on a large marketplace: tariffs, logistics and storage fees, acquiring, compensations. The stack is a Greenplum warehouse of about 100 TB, three ClickHouse clusters, several PostgreSQL databases, Kafka, two Airflow installations, Trino and S3, plus GitLab, an issue tracker, chat, a wiki, Grafana and OpenSearch.

Review practice was typical for a fast team that knows its domain:

• 130–200 merges a month from about ten authors;
• of the last 40 merge requests, 37 had no comments, and 38 were merged by their own authors;
• the median merge request touched one file and 22 lines, and was merged 36 minutes after it was opened.

Most of the time this works. Every few weeks it doesn't. In one case, a change added three columns to a staging table and to four queries, but not to the column list of an INSERT thirty lines above in the same function. The job failed on the first real document that night. No reviewer looking at the diff could have seen it: the broken line was not in the diff.

Engineers were already using Claude Code, Codex, Cursor and OpenCode on their laptops. The question was not whether to use agents. It was how to let them see production systems without letting them change production.

Four decisions that held up

1. Safety in capabilities, not in prompts. An agent can edit its own instructions and work around tool code; it cannot work around a database role. "Impossible" beats "forbidden".

• Every engineer's agent got a personal read-only database role, in a resource group with statement timeouts.
• Every MCP tool declares whether it reads or writes. Each agent gets an allowlist, validated at startup: a write tool or a typo in a read-only allowlist means the server refuses to start.
• Read-only is enforced on the server (read-only transactions in PostgreSQL, a read-only profile in ClickHouse). On top of that, a client-side guard only lets read statements through.

2. MCP next to the agent, not a central proxy. Each agent runs its own MCP server with its own credentials. Only credential issuance and audit collection are candidates for centralization, off the hot path. A central tool proxy would be one more high-value target.

3. Strip sensitive data at the source. People use different models, and you cannot enforce one model on every laptop. So sensitive fields are removed where they live: read-only roles and masked views in the databases, secret masking in every tool that shows configuration.

4. Automate only with a trigger. A new autonomous agent needs a reason to run without a human: off-hours, every merge request, every alert, a schedule. Being able to do something is not a reason. Most ideas turned out to need better context, not new agents: a conventions file every agent reads, and the on-call runbook packaged as a skill shared by Claude Code, OpenCode and Codex.

What I built

• An MCP server with 108 tools across about fifteen systems; 89 of them read-only. Each tool is also a plain command-line script, so it works and can be tested without MCP. Write operations, such as re-running a pipeline task, are separate tools that a given agent simply doesn't get. One gotcha for MCP builders: if your tools spawn subprocesses, close their stdin. Otherwise children inherit the protocol's stdio and the server hangs.
• An on-call agent. It sweeps the orchestrator, the data-quality channels and the databases every hour during working hours. It posts to a private channel only when something is wrong, and pings the on-call engineer only for critical issues, at most once a day per issue. Its first sweep found a streaming job that had been stuck for three weeks without tripping a single alert.
• An agentic merge-request reviewer. An orchestrator, a headless agent harness (OpenCode) and the same MCP server with a read-only allowlist.

The reviewer: benchmark, shadow mode, production

Benchmark: the diff is not enough

I took a merge request with seven known findings from a careful human review:

• Diff-only review through our internal model gateway: the best model found about 2.5 of 7, plus five noise items. The others found one, with a factual error, or ran out of output budget before answering.
• The agent with read-only access to the repository: 6 of 7 in about ten minutes and twenty tool calls.

On the INSERT case above, diff-only review missed the bug with every model. The agent found the exact blocker in about five minutes, plus a deployment-order risk. In this codebase, most real bugs live outside the diff.

Which instructions work

17 agent runs with five instruction styles. The most consistent was "review this merge request" plus a short map of the repository: where schemas, migrations and consumers live, and a few conventions. Checklists did not help. "Generate hypotheses first, then check them" produced the only miss of the INSERT bug. Repeated runs differed by about one finding.

Shadow mode: precision before anyone sees a note

First day, 11 runs on 8 merge requests: 24 of 32 findings correct, including 7 of 9 blockers. The false positives had three causes:

1. Library defaults instead of real configuration. Kafka consumer settings live in the orchestrator's connections, not in the repository. Fixed with a read-only connection-inspection tool that masks secrets.
2. Deployment topology that is not visible from the code.
3. A merge request merged while it was being reviewed.

The same day there was a production incident. On the first hotfix, the reviewer named the root cause and predicted that the hotfix alone would not restore the data. On the second hotfix, it flagged a missing column that then failed in production. Three minutes after the failure, the author pushed exactly the suggested fix. Nobody saw these notes; that is what made it a clean test. A retroactive run on an older merge request found the mechanism of an incident two weeks before it happened.

Production: accurate, and too slow

In the first two and a half days in production there were 12 runs and 9 published notes. They contained 15 findings and zero false positives: 9 confirmed, 5 plausible, 1 question. Two of the blockers were independently found by people as well. One a human reviewer had written to the author 22 minutes earlier; the other the reviewer verified against production data, and the author fixed it independently minutes before the note appeared.

The problem was latency. During the day, merge requests lived 4–27 minutes, and a review took 20–30 minutes on the shared model gateway (11–22 tokens per second by day, 60–90 at night). Not one of six daytime notes in the main repository landed before the merge. A higher reasoning-effort setting cut agent time by about 40% with identical verdicts, but did not close the gap.

The lesson: when a team merges in minutes, a reviewer has to be either fast or a gate. A slow advisory review that arrives after the merge changes nothing.

The guardrails that mattered

• Rules come from the target branch, never from the merge request. A change cannot rewrite its own review rules.
• The agent never holds tokens. The orchestrator fetches context and publishes the result.
• The repository is a temporary shallow copy of the head commit, and the agent's working directory is outside it, so a repository cannot inject agent configuration.
• Isolation: outbound traffic goes through a proxy with a domain allowlist, and each run gets a fresh state directory under a separate OS user.
• Explicit outcomes: every failure path leaves a "review did not happen" marker, so silence never reads as approval. A new commit aborts the running review.

The one time it went wrong. While checking whether a read-only fallback was really read-only, the agent ran DROP TABLE IF EXISTS, INSERT and TRUNCATE against a table that did not exist, on a production server. Nothing was harmed: the table did not exist, and the read-only profile does reject writes to real tables (we reproduced it locally). But DROP IF EXISTS on a missing table returns success before any permission check. So the agent concluded that read-only mode does not block DDL and raised a false blocker. There were two fixes. A client-side guard now sends only read statements. And one line went into the prompt: never test access boundaries by attempting writes. Prompts are the second line of defense, not the first.

We then paused both agents ourselves until an internal security review, and wrote the package for it: 21 architecture decision records for an on-premises agent platform. Its core is a disposable executor per job, with network boundaries outside the guest, a capability gateway with server-side scope, external audit, a kill switch and 39 acceptance tests, including adversarial ones.

What I would tell another team

• Start with read-only roles enforced by the systems themselves. Everything else is defense in depth.
• Give the agent the repository, not the diff.
• A repository map beats a checklist.
• Measure in shadow mode before anyone sees a note: precision per severity, causes of false positives, time to note versus time to merge.
• Expect most false positives to come from facts outside the repository — runtime configuration, deployment topology — and give the agent read-only tools for those facts.
• Decide early whether the reviewer is a gate or advice. Advice that arrives after the merge is invisible.
• Measure the team before you start. We didn't, so I can't tell you agents made us X% faster. I can tell you what they caught.

All writing · A similar problem? contact@ivanblinov.com