Tests Pass, Structure Fails: Keeping Long-Lived Systems Maintainable in the Agent Era

Written with AI assistance. Research, structure, and every claim are reviewed by me.

A few weeks into building BriefCheck, I hit a wall. Certain features could no longer be implemented without breaking something else that had worked before. And I had really tried to keep quality high and not to vibe-code my way through it. I reviewed the changes by hand, and the project has a harness around the agent: architecture documentation, a list of architectural invariants in AGENTS.md, and a QA gate that runs the formatter, the analyzer, and the test suite. Everything was green. The structure gave out anyway.

That made me wonder how research and industry deal with this gap: code that works, and a structure that does not hold. This article is what I found, and what I took from it for my own work.

I want to draw a line between vibe coding and agentic engineering here. Vibe coding, in Martin Fowler’s sense, means building software by prompting “without looking at any of the code that the LLM generates”, and he considers it suited to disposable software.¹ In agentic engineering the agents write the code, but humans supervise, review, and stay responsible for it. That is the case I care about: the tests are green and a human has looked at the diff.

Most of what follows comes from studies, so I should be upfront about their limits. Nearly all of them examine open-source projects, greenfield tasks, or benchmarks, and applying them to long-lived enterprise systems is my own extrapolation. Several are preprints, and I say so when they are. Statements with a footnote report a finding. Everything else is my own judgement.

Why structure matters more the longer a system lives

Structure is irrelevant for a prototype that is thrown away in a month. Long-lived software is different. Software Engineering at Google puts it as “software engineering is programming integrated over time”, notes that the lifespan of code varies by a factor of roughly 100,000, and asks teams to distinguish between code that “happens to work” and code that “is maintainable”.³

What makes maintenance expensive is understanding, not typing. In a field study with 78 professionals over 3,148 hours of work, developers spent about 58 % of their time on program comprehension.⁴ Enterprise budgets point the same way. The US Government Accountability Office reported that about 80 % of federal IT spending goes to operating and maintaining existing systems, with the ten most critical legacy systems between 8 and 51 years old.⁵ That figure includes hardware and licences and covers agencies rather than companies, so I use it as an order of magnitude and not as a measure of software maintenance cost.

This is where agent-written code becomes a risk. Storey distinguishes cognitive debt, the erosion of shared understanding in a team, from intent debt, the absence of externalised rationale.⁶ The paper is conceptual and has no data, but it names the mechanism I care about: if comprehension is the dominant cost and agents produce code faster than a team can come to understand it, the bill arrives later and in the most expensive place.

Agents erode structure while the tests stay green

A study of ten projects built with Cursor, averaging about 17,000 lines of code, shows the gap in numbers. The projects reached 91 % functional correctness, yet static analysis still reported 1,305 design issues (CodeScene) or 3,193 (SonarQube), mostly duplication, complexity, and oversized methods.² Ten greenfield projects, measured by tools, without peer review: a hint, not proof. The studies below look at how the gap arises and whether it lasts.

Erosion shows up over iterations, not in a single result

SlopCodeBench (a preprint) has agents extend their own solutions over several checkpoints instead of solving one task. Structural erosion rose in 77 % of the trajectories and verbosity in 75.5 %. Prompts that asked for quality lowered the starting values, “without affecting degradation rates”.⁷ In other words, asking for quality in the prompt improves the first result, but the code still degrades at the same pace as the iterations pile up.

Agents work on the excerpt, not on the structure

Duplication is where this becomes concrete. RepoReuse (a preprint, three days old at the time of writing, not replicated) found that agents stop exploring the repository as a session continues. By turn five, 50.8 % of the task chains contained duplicated logic, “all while pass rates barely move”.⁸ In real pull requests, the picture is similar: across 7,851 agent PRs, researchers found 28,425 code clones that, once introduced, persisted and were merged.⁹ Reviewers did not compensate. They reacted to agent PRs more positively or neutrally and overlooked the redundancy.¹⁰

This follows from what the agent sees. Copying what is visible is cheaper than finding what already exists, and a test suite will never complain about it. Birgitta Boeckeler tested maintainability sensors on a TypeScript dashboard and described the result in Maintainability sensors for coding agents: agents “are quite happy to copy and paste” and rarely refactor when they repeat code for the third or fourth time without an explicit nudge.¹¹ That is one greenfield project without data, but it matches the studies.

The debt does not stay in the agent session

The strongest causal evidence comes from a peer-reviewed difference-in-differences study of open-source projects that adopted Cursor. Compared with a matched control group, the projects gained a significant but transient speed-up and a sustained increase in static-analysis warnings and code complexity, which in turn slowed development again.¹² A correlational preprint across 182 repositories adds that agentic code needs more corrective maintenance, and that each additional 10 percentage points of PRs merged without review goes along with roughly 6 % more maintenance load.¹³

Tests do not see it, and benchmarks do not reward it

A test suite encodes behaviour. In a benchmark that adds a structural oracle on top of the functional tests, 13.3 % of the agent results passed all functional tests and still failed the structural checks.¹⁴ SWE-Gate (a preprint) measures the same gap against the rules that reviewers enforce in real pull requests: among 644 repairs that passed the functional tests, 221 violated at least one review constraint.⁴⁰ The incentives point in the same direction. Common benchmarks measure test success, and the tests themselves are weaker than assumed: in one analysis, 31.08 % of the patches counted as resolved passed because of weak tests,¹⁵ and another found 345 patches wrongly credited as passing.¹⁶ Whether model training optimises for this signal is a hypothesis. I found no evidence for it. The same goes for the intuition that long sessions degrade the agent’s grip on structure: models use information in the middle of long contexts less reliably,¹⁷ but I found no study that measures this effect on coding tasks.

The gap is also invisible to some of the metrics that teams reach for first. In SWE-CI, 15 of 20 models scored better than the human reference code on the Pylint score, and all 20 scored worse on the Maintainability Index. The authors describe the result as code that is “superficially clean but intrinsically complex”.¹⁸

Not every study finds a problem

A registered-report RCT with 151 participants, 95 % of them professionals, let people extend code that had been written with or without AI. It found no significant differences in time or quality and concluded that any effect is small and uncertain.¹⁹ Agent PRs also introduce fewer breaking changes than human PRs (3.45 % against 7.40 %).²⁰ Both look at a narrower slice than the erosion studies: one Java feature in a short task, and breaking changes at merge time. The erosion studies follow longer iterative work, but on open-source and greenfield projects, and some of them rely on SonarQube, whose false positives the authors of one benchmark name as a validity risk.²¹

What I could not find is a longitudinal study of agent-driven architectural drift in real enterprise systems. For those, I treat the risk as plausible but unproven.

Enterprise conditions make it worse

Two conditions amplify the problem in enterprise systems. The first is delivery stability. Google’s DORA survey for 2025, which is correlational, reports that AI adoption “does continue to have a negative relationship with software delivery stability”.²² The second is that teams lose the understanding of code they are responsible for. In an Anthropic RCT with 52 mostly junior developers, participants who worked with AI assistance scored 50 % on a comprehension quiz against 67 % for the hand-coding group, with the largest gap in debugging.²³ It is a small study, run by an AI vendor, with a chat assistant instead of an agent, so I would not lean on the exact numbers. But it points at the mechanism behind the cognitive debt Storey describes.

Measuring maintainability without trusting one number

ISO/IEC 25010:2023 defines maintainability as the degree of effectiveness and efficiency with which a product can be modified to improve, correct, or adapt it, and breaks it into modularity, reusability, analysability, modifiability, and testability.²⁴ It is a design property. Nobody can measure it directly. Every number is a proxy.

Tools compute these proxies in three ways:

• A formula over code measures. The classic Maintainability Index combines Halstead volume (a size measure based on how many operators and operands a program uses), cyclomatic complexity (the number of independent paths through the code), and lines of code. Each term lowers the score as it grows, and Visual Studio rescales the result to 0 to 100.³⁶
• Counting rule violations. Pylint’s score starts at 10 and subtracts the weighted violations per statement, with an error counting five times as much as a warning.³⁷ SonarQube assigns each rule an estimated fix time, adds the minutes up to a technical debt, and relates that debt to the estimated cost of developing the code, which gives a letter rating.³⁸
• A weighted catalogue of code smells. CodeScene’s Code Health scans for more than 25 factors such as duplicated logic, deeply nested conditionals, and large or low-cohesion modules, scores each file, and averages the scores weighted by file size.³⁹

All of them measure what can be counted in the source at one point in time: size, branching, nesting, duplication, smells. None of them observes whether a change has become harder. Beyond code structure there are two more levels of evidence, and each shows something different:

Internal metrics are weak as a score and useful as a trend. Agent-written code is formatted well and named plausibly, which is exactly why the Pylint score and the Maintainability Index disagree in SWE-CI. In a study of 403 agent commits labelled as readability improvements, the Maintainability Index dropped in 56.1 % of them.²⁵ The studies in the previous section avoided the problem by tracking how metrics change over iterations. The diagnosis there rests on change behaviour, with internal metrics as corroboration. I would weight the three levels the same way.

The process level is closest to the actual pain. Boeckeler’s heuristic for the first sign of cracks is simple: the number of files changed for a small adjustment increases, or changes start breaking things that used to work.¹¹ Both match what I saw in BriefCheck. In her project, a change to a date range ended up touching more than 40 files. This is a practitioner observation without data, but it is cheap to track and hard to game.

Tests, the established sensor, have two blind spots. Coverage records that a line ran, not that its effect was verified: in one of her files, 100 % statement coverage coexisted with 13 surviving mutants, and she summarises that “coverage tells us that a line was executed, but not that its impact was verified”.¹¹ And agents often change code that no existing test executes. In a coverage analysis of agent PRs, 64.8 % of the Python PRs contained no changed line that an existing test runs, and only 49.6 % of all PRs touched tests at all.²⁶ Mutation testing closes part of the first gap at a high resource cost. I found no study of test quality in agent-written suites, so that question stays open.

I would combine the three levels so that each covers the blind spot of another. Internal metrics warn early and cheaply, process metrics show whether the warning matters, and the human experiment calibrates both. This combination is my own and has not been validated as a package.

Three layers that make structure enforceable

A qualitative study of five practitioners describes oversight of AI-assisted work as three layers: preventive guardrails (architectural intent made machine-readable), executable guardrails (linting, tests, CI), and human oversight that shifts from line-by-line review towards architecture and long-term maintainability.²⁷ Boeckeler’s Harness engineering for coding agent users arrives at a similar structure from practice: guides steer before the agent acts, sensors observe afterwards, and each can be computational or LLM-based. With sensors alone, the agent keeps repeating the same mistakes. With guides alone, it encodes rules but never finds out whether they worked.²⁸

The layers form a chain in which each one answers the gap that the previous one leaves open. There is a caveat for enterprises: Boeckeler notes that “the harness is most needed where it is hardest to build”. Heavily indebted legacy systems need guardrails most and are the hardest to retrofit, while greenfield projects can build them in from the start.²⁸

Intent has to be readable by the agent

Context files such as AGENTS.md are the obvious first step, and their evidence is mixed. An ETH preprint found that context files generally do not raise task success and add more than 20 % inference cost, although the agents follow the instructions, and the files help most for non-standard conventions.²⁹ A study by a single author reports that agents fail at “feature design, pattern selection, exact wiring”, decisions that are hard to capture in a prose convention.³⁰ In another single-author preprint, code-proximate formats such as OpenAPI or TypeScript contracts closed most of the capability gap for weaker models, while for strong models the format barely mattered.³¹ All three measure task success, not structure.

So the files should hold what the agent cannot infer, as close to the code as possible, and they have to be maintained. A consistency checker found outdated code references in 23.0 % of 356 repositories’ context files.³² A context file that has drifted from the code teaches the agent the wrong structure with full authority.

Each module states what it promises and what it may depend on

I described the repository layout for this in an earlier article: architecture documentation as Markdown next to the code, with a root AGENTS.md for universal rules and nested ones for individual modules. BriefCheck follows that layout, and each feature has its own AGENTS.md that says what the feature is responsible for, what other code can rely on, and what it depends on. The quota feature, for example, states that UI code must ask it for the user’s tier and never work the tier out from the payment provider.

An agent that sees only one file cannot know such promises, which is why I think a structure definition has to capture them, per module and next to the code. I know of no study that tests this format.

A rule in the prompt is a request, a rule as a test is a merge condition

Readable is not the same as followed. Converting the rules in AGENTS.md into AST checks raised constraint compliance from 67.0 % (prompt only) to 88.3 % in a single-author preprint.³³ Boeckeler saw a similar pattern in practice: an agent pointed to its sensors by a hint in AGENTS.md called them unreliably, git hooks enforced them more reliably, and dependency-cruiser layer rules, which the agent violated a few times, were corrected after feedback.¹¹ Agents also keep dependency boundaries poorly on their own: in the structural benchmark mentioned earlier, dependency control reached 4.3 % and responsibility decomposition 15.2 %, though each rests on only two cases and the authors call it a diagnostic trend.¹⁴

The layer rule of BriefCheck is listed in AGENTS.md as “Enforced by convention and code review”. In a Java project, ArchUnit turns such a rule into a test that fails the build:

@ArchTest
static final ArchRule domain_stays_independent =
    noClasses().that().resideInAPackage("..domain..")
        .should().dependOnClassesThat()
        .resideInAnyPackage("..data..", "..presentation..");

Other ecosystems have equivalents, for example dependency-cruiser for TypeScript and import_lint for Dart, and Building Evolutionary Architectures calls such checks fitness functions.³⁴ A rule like this only covers what dependencies and package structure can express. Duplication or misplaced responsibility need other sensors.

Of the eight rules in the root AGENTS.md of BriefCheck, only two concern structure, and neither is backed by a test. The other six are conventions about widgets, libraries, and logging.

Enforcement works like a ladder. It starts with a sentence in a context file and continues through review, a lint rule, and a test that fails the build, up to something the compiler enforces, and the higher a rule sits, the harder it is to ignore. The structural rules are the first ones to climb.

Two practical details follow from Boeckeler’s report. She observed that the agent often raised the cyclomatic-complexity threshold instead of refactoring. If an agent can edit the sensor’s configuration, a threshold is a negotiation. I would therefore gate on “not worse than main” instead of an absolute score, and keep sensor configuration under code-owner review. I found no study that tests fitness functions with agents.

Review moves from lines to structure

The correlation between the share of PRs without review and maintenance load is the best evidence I found for the effect of review, but it is a correlation: teams that skip review may also be under more time pressure or have fewer guardrails.¹³ The finding that reviewers overlook redundancy in agent PRs is the argument for giving them structural input.¹⁰ A preprint on weaker models reviewing stronger coding agents (Groundability) points the same way. Reviewer size did not predict review quality, while official execution evidence improved the detection of five of six reviewers, and reviewers without such evidence over-rejected correct patches.⁴¹ The study is about functional correctness, not structure. The lesson I take from it: a reviewer needs a decisive check more than a bigger model, and for structure that check is a sensor like the ones above. Boeckeler uses the impact radius of the changed files as a triage signal, and an LLM-based modularity review that she ran occasionally found duplicated routes and a parameter passed through every layer. She stresses that this is expensive and probabilistic and not something to run on every commit.¹¹

The 2025 State of AI-assisted Software Development report (DORA) describes the same shift. The friction does not vanish. It moves “from manual grind to deciding and verifying, possibly in the form of prompt iteration, result vetting, and assessing code that looks remarkably similar to correct code”.²²

Healthy code makes every layer cheaper

Maintainable code also pays off for the agent. In 5,000 Python files, five mid-size open-weight models broke significantly fewer refactorings in healthy code (CodeScene’s Code Health of at least 9). The effect was not significant for Sonnet 4.5 called directly, nor for Claude Code running on Sonnet 4.5, which broke only about 5 % of the test suites either way.³⁵ The data comes from competition code, the authors work for the vendor of the metric, and the benefit shrinks as models get stronger. It supports the direction but does not make a case for it.

Structure has to come from the system around the model

Writing code is getting cheaper, and understanding it stays the expensive part. That is the argument for spending effort on whatever keeps understanding cheap: architectural intent that agents can read, rules that fail a build instead of asking politely, sensors on three levels instead of one number, and review aimed at structure.

BriefCheck had a green QA gate, a contract per module, and a reason for every rule, and still the layer rule rests on convention and review. A rule that nobody can fail a build with stays a request.

The evidence is young, comes mainly from open source and greenfield work, and the study that follows agent-written code through years of enterprise maintenance has not been done. Teams will have to form their own judgement, and the three levels give them a way to do it with data.

I will start with the two structural rules and the conventions that are cheapest to check.

Sources