Why EU AI Act Compliance Evidence Keeps Failing

Workshop study reveals EU AI Act compliance faces three tensions—metric translation, human oversight, and evidence timeliness—and compliance should be a continuous process.
By involving 32 researchers, policymakers, and practitioners in participatory workshops covering healthcare, hiring, and public transportation, the study examines the practical difficulties of translating the EU AI Act into technical requirements. Thematic analysis identified three recurring tensions: poorly chosen fairness metrics can mask substantive discrimination; human oversight becomes a formality under time pressure and insufficient information; and compliance evidence becomes invalid as population structure and deployment conditions change. The study concludes that technical, organizational, and evidentiary challenges are interdependent, compliance should not be treated as one-time certification but requires context-sensitive, continuously maintained mechanisms, and calls for policy design to leave room for contextual judgment while requiring transparent documentation.
A participatory workshop study involving 32 researchers, policymakers, and practitioners reveals the real challenges of translating the EU AI Act from legal text into technical requirements: compliance is not a one-time technical assessment but an ongoing process that requires continuous maintenance.

Research Background: When Regulation Meets Engineering Practice
The EU AI Act sets technical and organizational requirements for high-risk AI systems, but in practice, translating these requirements into measurable system properties demands extensive contextual judgment—what to measure and what evidence to use in support of compliance claims have no standard answers.
The study employed a participatory workshop method, covering three real-world scenarios: healthcare, hiring, and public transportation, and identified recurring tensions through thematic analysis. The researchers' core conclusion is that technical, organizational, and evidentiary challenges are interdependent, and that establishing and maintaining compliance requires attention not only to technical assessments but also to how AI systems are used, governed, and monitored over time.
The EU AI Act (EU AI Act) officially entered into force in 2024 and is the world's first comprehensive legislation to regulate AI systems through risk classification. It categorizes AI systems into four levels: unacceptable risk, high risk, limited risk, and minimal risk. High-risk systems (such as AI used for medical diagnosis, hiring screening, and critical infrastructure management) must meet a series of compliance obligations, including risk management systems, data governance, technical documentation, transparency, and human oversight. The Act is implemented in phases: prohibitive provisions have applied since early 2025, while the core obligations for high-risk systems have a longer transition period. This "risk-based classification + phased implementation" architecture means that compliance is not a single-point acceptance check but an ongoing obligation spanning the entire system lifecycle—this is also the institutional premise for understanding why this study emphasizes "continuous compliance."
Three Key Tensions: Gaps in Translation, Responsibility, and Evidence
Metric Translation: Fairness Measures Can Mask Discrimination
Translating regulatory obligations into measurable system properties is the first link in the compliance chain and the most prone to distortion. Workshop participants pointed out that in hiring scenarios, fairness metrics themselves can actually mask discrimination—when measurement methods are poorly chosen, numbers that appear to meet standards can obscure substantive unfairness.
This points to a deeper issue: regulatory provisions describe value goals (such as fairness), while technical metrics can only capture certain aspects of those values. Choosing which metric to use and how to set thresholds is itself a decision laden with value judgments, yet this decision-making process is often invisible in compliance documentation.
Fairness measurement is itself a highly contested technical field. Common metrics such as statistical parity (equal pass rates across groups), equal opportunity (equal true positive rates across groups), and predictive parity (equal precision across groups) are mathematically incompatible—satisfying one often means violating another. This means no single metric can fully capture "fairness." In scenarios such as hiring, if group pass rates alone are used as the basis for compliance, a model may "game the metrics" by slightly adjusting decision boundaries while substantive distributive unfairness persists. This is known as "fairness washing." Therefore, metric selection is essentially a question of value prioritization, requiring justification for why a particular notion of fairness is prioritized in a given scenario, rather than equating metric compliance with completion of compliance.
Human Oversight: Effectively Nominal Under Time Pressure
The Act emphasizes human oversight, but the workshops revealed gaps in practice. In public transportation scenarios, time pressure and limited information undermine the effectiveness of human oversight—overseers must make judgments in extremely short timeframes but lack sufficient information to support them.
This touches on a blind spot in compliance assessment: institutional design assumes that overseers have the capability, information, and time to make rational judgments, but real-world operating conditions often fail to satisfy these premises. Merely checking whether "a human oversight step has been set up" is insufficient; one must also ask whether oversight is actually effective under real conditions.
"Human oversight" in AI governance literature is usually distinguished into three modes: human-in-the-loop (the overseer must confirm each system output), human-on-the-loop (the overseer monitors system operation and intervenes when anomalies occur), and human-in-command (the overseer can halt the system at any time). The EU AI Act requires high-risk systems to have effective human oversight capabilities but does not specify particular modes. The public transportation scenario problems revealed by the study are a typical dilemma of the "human-on-the-loop" mode: overseers nominally retain intervention rights, but when the pace of decision-making exceeds human reaction speed, or when the information presented by the interface is insufficient to support judgment, the right to intervene becomes a formality. Effective oversight is therefore not only a matter of institutional arrangements but also involves engineering constraints related to human-computer interaction design, information presentation, and operational time budgets.
Evidence Validity: Environmental Changes Can Invalidate Old Evidence
The issue of the shelf life of compliance evidence was repeatedly raised in the workshops. In healthcare scenarios, changes in patient population structure can invalidate previously established system performance evidence—a system performing well under the original data distribution does not mean it remains reliable under a new population composition.
This means compliance evidence is not a static asset but a dynamic conclusion with an expiration date. When a system's usage conditions, deployment environment, or service population change, the validity of existing evidence must be re-examined.
In machine learning engineering practice, the concepts directly corresponding to "evidence invalidation" are data drift and concept drift. Data drift refers to changes in the distribution of input data (such as changes in patient population composition), while concept drift refers to changes in the true mapping relationship between inputs and outputs (such as updates to clinical guidelines rendering existing diagnostic criteria no longer applicable). Both can cause performance assessments based on historical data to lose extrapolation validity. For this reason, the EU AI Act requires high-risk systems to establish post-market monitoring and risk management systems, the core logic of which is precisely to recognize that compliance evidence has a time limit: deployers must continuously collect operational data, set drift monitoring thresholds, and revalidate system performance when trigger conditions are met, rather than treating a one-time test report as permanent proof of compliance.
From One-Time Certification to Continuous Compliance
An important conclusion of the study is that technical, organizational, and evidentiary challenges are intertwined and cannot be addressed separately. The choice of a fairness metric (a technical decision) affects how an organization allocates oversight responsibilities and also determines what evidence is sufficient to support compliance claims.
This points to a context-sensitive and continuous approach to compliance. Compliance should not be understood as delivering a document or passing a single assessment, but should include ongoing attention to how systems are used, governance structures, and monitoring mechanisms. For developers and deployers, this means establishing mechanisms capable of tracking changes in system operating conditions and reassessing the validity of evidence when conditions change. For policymakers, this means leaving room in rule design for contextual judgment while requiring transparent documentation of the basis for those judgments.
The value of this research lies in pulling the compliance issue from paper provisions back into real work scenarios, presenting specific gaps in the three links of translation, oversight, and evidence. These gaps cannot be automatically bridged by more detailed technical standards; they require coordinated responses at the technical, organizational, and governance levels.
Note: This article is based on a single-source participatory workshop study. The specific scenarios and conclusions involved all come from the study's thematic analysis and have not yet undergone multi-source cross-validation.
One-line Summary
A workshop study reveals that EU AI Act compliance faces three tensions—metric translation, human oversight, and evidence timeliness—and that compliance should be a continuous process.
Paragraph Summary
By involving 32 researchers, policymakers, and practitioners in participatory workshops covering three real-world scenarios—healthcare, hiring, and public transportation—the study examines the practical difficulties of translating the EU AI Act from legal text into technical requirements. Thematic analysis identified three recurring tensions: first, metric translation problems, where poorly chosen fairness measures can mask substantive discrimination; second, human oversight becoming a formality under time pressure and insufficient information; and third, compliance evidence becoming invalid due to changes in population structure, deployment environment, and other conditions. The study concludes that technical, organizational, and evidentiary challenges are interdependent, that compliance should not be viewed as one-time certification but requires context-sensitive, continuously maintained mechanisms, and calls for policy design to leave room for contextual judgment while requiring transparent documentation of the basis for judgments.
Related articles

New AI Governance Initiative: Mandatory Independent Audits and Pre-Deployment Safety Proof
Macron convened government leaders, AI company executives, and researchers to discuss advanced AI risks. Statements call for a rigorous governance framework requiring mandatory independent audits and pre-deployment safety proof.

Roblox Funds Controversial Anorexia Treatment
Roblox funds flow to a controversial creator who promotes ketogenic diets over conventional treatment for anorexia and bulimia, raising medical and regulatory concerns.

AI Safety by Design: European Scientists and Policymakers in Dialogue
European scientists and policymakers discuss the urgency of safe-by-design AI in Berlin and Zurich. This article explains the concept, practical challenges, and implications for global AI governance.