Causal AI for Agentic Software Engineering: Evaluating LLM as a Judge

LLM-as-a-Judge in Software Engineering

Large language models are already producing a growing share of the software artifacts we rely on: specifications, test plans, code, documentation. As we reflected a year ago in our retrospective on AI-orchestrated SDLC, the central question is no longer whether AI can produce an artifact, but how we validate the artifacts that AI produces.

Software product quality is commonly organized around the ISO/IEC 25010 standard, which breaks it down into characteristics such as functional suitability, performance efficiency, security, and maintainability. Some of these can be assessed using rule-based methods—for example, functional suitability through test suites or security through scanners that detect known vulnerabilities. However, even for maintainability, static analysis provides only proxies, such as complexity or coupling, that correlate with quality rather than measuring it directly. Other aspects require an understanding of meaning and context.

For such aspects, human expert review remains the reference, but it is costly and does not scale to the volume of artifacts that AI now produces. LLM as a Judge offers a scalable alternative by providing the semantic and contextual understanding that rule-based metrics lack. The approach is already in use: agentic coding frameworks can make agents evaluate their own outputs against explicit criteria, while dedicated review agents assess artifacts such as code or user stories (see our blog article on skillsets).

Empirical Challenges

Knowing which factors make a judge more reliable requires separating their individual contributions. The cleanest way is a controlled experiment: vary one factor at a time while holding everything else constant. But for a substantial part of the relevant factors, no such control is possible: the properties of the code under evaluation, the project it comes from, and the human experts‘ judgments are not free variables. These can only be observed, and they vary together, so that no comparison isolates the contribution of a single factor.

Even a single factor is hard to measure. The complexity of code is usually reduced to simple counters, such as the number of lines or functions, but that is not the same as how hard the code feels to a human. And the human judgment that is compared against is not a fixed target: different experts do not always agree, and datasets in which several experts judge the same code are rare.

Some influences are also invisible: who wrote the code, how long it has existed, or what it is used for can shape both the code and the experts‘ opinions without leaving a trace in the data. No comparison can remove an influence that is not visible. For such factors, only observational data remain.

Causal Inference for Software Engineering

Causal inference provides a framework for drawing causal conclusions from observational data, in exactly the situations where a controlled experiment is not possible. Its central move is to make the causal assumptions explicit before the data is analyzed, and to represent them as a causal graph (a directed acyclic graph): a link between two variables stands for a possible direct causal effect, and the absence of a link stands for the assumption that no such effect exists.

From this graph it can be derived how the data must be analyzed in order to separate causal effects from mere correlations. The method behind this is the do-calculus, and it yields a second, often more valuable answer: which effects cannot be estimated from the data at hand at all. The advantage of the approach is twofold. The causal assumptions stay explicit and can be checked against the data, and the analysis method is derived from the assumptions rather than chosen after the fact.

Causal inference is a well-established tool in statistics and the social sciences, but only a relatively recent addition to software engineering. Early applications appeared in the 2010s [2,3], and only in the last two years has the work begun to consolidate into a field, Causal Software Engineering, with several tutorials [5,6], and a vision and roadmap [4] and a dedicated special issue [7].

LLM-as-a-Judge for software maintainability

In our recent work, we applied causal inference techniques to investigate agreement between assessments made by LLM judges and human experts. [1]. We used a public dataset [8] as our reference, in which several human experts independently rated 304 Java classes from five projects (ArgoUML, AOI, Diary Management, JSweet and JUnit 4) on five aspects: readability, understandability, complexity, modularity and overall maintainability.

We evaluated seven judge models from the Qwen 3.5 family, ranging from 0.8 billion to approximately 400 billion parameters and spanning both dense and mixture-of-experts architectures. Each model was tasked with answering the same five questions as the human experts. Their answers were then compared with those of the experts.

Findings

Complexity is the enemy of agreement. The most practically important factor is the complexity of the code. Controlling for project and code size, code complexity drastically reduces agreement with human experts. On simple code the judge and the human largely see the same thing; on complex code the judge starts to actively disagree, not just waver.

Bigger is not (monotonically) better. Holding architecture and quantization constant, model size does not follow the „bigger is better“ intuition: the 9B model outperforms the 4B model, yet the 35B model outperforms the 122B model. Parameter count alone does not explain agreement quality.

Question formulation matters. How a quality criterion is phrased is a real lever: readability and understandability questions yield more aligned judgments than modularity and overall-maintainability questions. The framing of a question is a design parameter on par with the model choice.

Temperature has no effect. Sampling temperature (0.6 versus 1.0) has no significant effect on any agreement measure.

What this means for practice

Four takeaways for teams that already use LLMs to review, score, or self-check generated artifacts:

• Do not assume the biggest model is the best judge. Model-size effects on judge quality are non-monotonic. When choosing or upgrading a judge, candidates should be benchmarked on the organization’s own artifacts and criteria, under constant architecture and quantization.
• Route complex artifacts to humans, or break them up. Judge reliability degrades sharply with the complexity of the code under evaluation, and degrades in a directional way (the judge starts to disagree, not just waver). A pragmatic design is to use the judge for first-pass screening of simple artifacts and escalate the rest.
• Treat question formulation as part of the system. How a quality question is phrased changes agreement with human experts. Rubrics are not interchangeable; they are a design parameter on par with the model choice.
• Design evaluations like causal experiments. Before running an expensive benchmark, it pays to check which effects the design can actually identify. This step can save months of misinterpreted correlations and it shows exactly which new experiments are needed (e.g., posing every question in both its positive and negative form).

Next steps

The evaluation in [1] is a first step on a specific task and a specific kind of judge; the detailed limitations are documented in the paper. Two directions stand out for what comes next.

Breadth. The same causal approach can be applied to other quality aspects and to other artifact types beyond the maintainability of Java classes, for example functional suitability, security, and the specifications, test plans, and documentation that agentic pipelines produce.

Realism. The current evaluation is a controlled study on a curated dataset, and the goal is to apply the method in real settings, in particular inside an agentic AI coding assistant, where an LLM-as-a-Judge runs continuously in the loop of actual development. Such settings are precisely where a controlled experiment is impossible: the artifacts, the code, and the expert judgments are observed, not controlled. That is exactly where causal inference applies. Even without the ability to vary a factor at will, it remains possible to estimate and reason about causal effects, provided the causal assumptions are explicit and can be checked against the data.

The promise of LLM-as-a-Judge is a scalable quality gate for the flood of AI-generated artifacts. Causal reasoning can make such judges not just usable, but understandable, so that when a judge diverges from human experts, the reason is known and a remedy can be applied.