Measuring Software Quality with AI

Quasar is an approach developed by Fraunhofer IESE for the automated measurement of software quality. Unlike traditional tools for static code analysis, Quasar captures quality aspects throughout the entire software lifecycle: from early requirements and design decisions through architectural documents to the final code. This page describes the methodological basis behind it.

Quality assessment using generative AI is not a new concept. The key question is how reliable and transparent the results are. Quasar relies on a methodology derived from software quality research and combines it with the controlled use of language models.

The Goal-Question-Metric (GQM) approach is a well-established method for systematically defining quality goals in software development. The basic principle is simple: First, a specific quality goal is defined, such as “The architecture of a system should be maintainable and extensible.” From this, questions are derived that clarify how this goal can be identified: Are dependencies between components minimized? Are responsibilities clearly separated? Finally, measurable indicators (metrics) are defined for each of these questions, enabling a concrete evaluation.

We have derived specific quality objectives from recognized and widely used software quality models (e.g., UX and architecture standards, heuristics, ISO 25010, etc.) and, using the GQM approach, translated them into concrete, automatically measurable metrics.

This approach ensures that Quasar does not simply look for “quality” in general, but always evaluates performance against a precisely defined goal. This makes the individual results from each measurement point justifiable and transparent to development teams and product managers.

Quasar has a modular design. Each module specializes in a specific quality aspect and implements the GQM approach for its respective area. Generative AI is used specifically for individual, clearly defined measurement points, rather than as a general evaluator with a single, comprehensive prompt.

There are currently two modules:

The UX Module evaluates the user-friendliness of interfaces and interaction designs based on established usability heuristics. It can be used directly within design tools such as Figma and provides feedback immediately after individual design decisions are made, without having to wait for time-consuming manual user tests.

The Architecture Module analyzes software architectures for maintainability, extensibility, and structural quality. It evaluates architectural documents and related documentation, identifying weaknesses that would be difficult to correct in later development phases. The Architecture Module can be integrated directly into tools such as draw.io.

Both modules access the same infrastructure: data sources such as repositories, document storage systems, and design tools are integrated once and provide the modules with the necessary information in a structured format.

Quasar views quality not merely as the result of an assessment, but also as a starting point for constructive improvement. Through the continuous analysis of various artifacts, quality issues are identified early on—precisely when they can be resolved with relatively little effort. This is particularly true for conceptual or architectural weaknesses, which are difficult to correct in later phases.

The automated, AI-powered analysis provides transparency into a product’s strengths and weaknesses at various stages of development. This makes Quasar a valuable resource for developers, architects, and product managers who need to make informed decisions. The assessment not only delivers a result but also provides a transparent basis for targeted improvements throughout the entire development process.

In this way, Quasar helps ensure that quality isn’t just checked at the end, but is built up and secured step by step. By interacting with Quasar, a user interface can be built piece by piece, or different architectural options can be “discussed.” We’re talking about AI pair engineering here—the AI becomes a personal assistant and works in a similar way to pair programming.

One aspect of AI-powered quality measurement that is often underestimated is the question of what information is provided to a language model for a specific evaluation task. Too little context leads to superficial results; too much or contradictory context degrades the quality of the output.

Quasar addresses this issue through systematic context management, which compiles the exact relevant subset of available information for each measurement task.

In practice, comprehensively measuring numerous quality aspects is a complex undertaking that requires a high level of expertise in systematic implementation. Often, it is unclear what works well and why—or why it doesn’t. This is where we excel, using proven scientific methods and a systematic approach to determine how such questions can be answered with empirical evidence. To achieve efficiency gains, it is essential that the quality of AI-generated answers and results is accurate. To this end, we also use plugins we developed ourselves for Figma and draw.io to test added value directly within widely used, practice-relevant tools.

All modules are designed to work exclusively with internal company data. The underlying language models can be run locally or in trusted, hosted data centers. This requirement is not a compromise made after the fact, but was part of the architectural decisions for Quasar from the very beginning.