Skill Extraction Under the SFIA Framework: Is Agentic RAG Actually Better?

From Flat Labels to Responsibility Levels: The New Challenge of Skill Extraction

Automated skill extraction is a foundational step in workforce planning, yet most systems treat skills as flat labels — ignoring the critical dimension of responsibility levels in skill practice. The same skill looks very different in the hands of a junior practitioner versus a senior architect, and that difference is precisely the most valuable signal in recruiting, job matching, and talent assessment.

The Skills Framework for the Information Age (SFIA) was designed specifically to capture this dimension. It defines 147 professional skills mapped across seven responsibility levels. Despite SFIA's widespread adoption in European ICT job profiling, no LLM-based automated extraction approach targeting SFIA had been reported — until now.

This new arXiv paper formalizes the task for the first time as a structured prediction problem: extracting (skill, level) pairs from free text, with evaluation organized around three core questions.

SFIA's seven responsibility levels (Level 1–7) progress in steps from "following instructions" to "setting industry strategy," each with explicit requirements around autonomy, complexity, and impact. Take "data analysis" as an example: a Level 2 practitioner produces standardized reports under supervision, a Level 5 practitioner independently designs analytical frameworks and guides a team, while Level 7 involves defining organizational data strategy. This granular scale makes SFIA not just a skill inventory, but a measuring stick for professional maturity. It is precisely this characteristic that makes SFIA widely used in European ICT job certification, compensation band design, and talent development planning — and also what makes automated extraction far harder than ordinary skill annotation. The model must not only identify what skill is being used, but also infer at what depth from context.

Three Core Questions and Five Strategies

The researchers posed three highly targeted questions:

• How accurately can text be mapped to SFIA's closed vocabulary?
• Which strategies can reliably predict responsibility levels alongside skill predictions?
• Do more complex agentic designs genuinely outperform simpler retrieval and prompting approaches?

To answer these, the authors compared five strategies spanning the full spectrum from traditional to cutting-edge:

1. Lexical Baseline: Simple keyword matching.
2. Dense Retrieval + LLM Reranking: Vector retrieval to surface candidates, followed by LLM reranking.
3. Zero-shot, Schema-constrained LLM: Direct LLM generation under constrained output structure.
4. Single-Agent Agentic RAG: A single agent autonomously performing retrieval-augmented generation.
5. Three-Agent Collaborative Team: A "Retriever–Matcher–Validator" setup with role-based division of labor.

All strategies were evaluated against a SFIA 9 corpus built by the authors using a fully automated agentic pipeline, released as open source alongside the paper. The gold standard for evaluation is expert-annotated European ICT job profiles.

Agentic RAG — the agentic variant of retrieval-augmented generation — extends traditional RAG by giving the model the ability to actively decide when and how to retrieve. Standard RAG follows a fixed pipeline: query → retrieve → generate. Agentic RAG allows the model to decide mid-process whether another retrieval is needed, how to rewrite the query, or when to stop iterating. The single-agent version handles this loop autonomously; the multi-agent version distributes responsibilities like "retrieval," "matching," and "validation" across different roles that communicate to improve reliability. The theoretical rationale is that specialization reduces single-point error rates — but the cost is communication overhead and compounding latency. This study's experimental results provide a quantitative reference for exactly that trade-off: in classification tasks with a closed candidate space and well-defined rules, the negotiation cost of role division may exceed its error-correction benefit.

Key Findings: Retrieval Casts Wide, Generation Aims Precisely, Levels Require Explicit Decisions

The experimental results are illuminating — and in some ways counterintuitive.

Retrieval and generation each have their strengths. Retrieval-based matching identifies the most skills (high recall), while generative strategies are significantly more precise (high precision). This confirms the classic division: retrieval casts a wide net, generation aims accurately.

Responsibility levels must be predicted as an explicit decision. This is one of the paper's most important insights: only strategies that treat level determination as a deliberate decision step can reliably predict the correct responsibility level. Methods that rely on similarity-based selection to indirectly infer levels show error rates more than twice as high. In other words, responsibility levels cannot "naturally emerge" from semantic similarity — the model must reason about them specifically.

Multi-agent is not a silver bullet. The three-agent collaborative team doubled system latency with no improvement in accuracy. This result directly challenges the popular assumption that "more agent roles = better performance" — at least for closed-taxonomy matching tasks, stacking agent roles does not automatically yield gains.

In engineering terms, "level determination as an explicit decision step" means the model's chain-of-thought or prompt structure must include a dedicated stage requiring the model to first confirm skill assignment, then separately reason about and output the level — rather than letting the level appear as a byproduct of vector similarity ranking. This aligns with the classic principle of "task decoupling" in multi-label classification: when two prediction targets have heterogeneous decision bases (skill identification depends on semantic similarity; level determination depends on the depth of responsibility described in context), forcing them into a single joint prediction tends to implicitly sacrifice one target. This finding has direct relevance for all NLP tasks involving "capability level assessment" in HR tech, including resume parsing, job requirement structuring, and performance description analysis.

Implications for AI Engineering Practice

The value of this research extends beyond the SFIA framework specifically — it offers lessons for LLM application engineering more broadly.

Faced with structured, closed-vocabulary extraction tasks, engineers often reach for the most complex architecture to "play it safe." But this paper reminds us: architectural complexity should match task payoff. When the candidate space is closed and finite (such as 147 fixed skills), multi-agent negotiation and verification mechanisms may only add cost without addressing the true bottleneck of accuracy.

What actually determines success is task design itself — whether level determination is explicitly modeled as a decision matters more than whether you use a single agent or multiple agents. This aligns closely with the engineering intuition of "first clarify what you're predicting, then decide how to predict it."

Additionally, the researchers open-sourced the agentic pipeline for automatically constructing SFIA corpora, providing the first reproducible, level-aware skill extraction baseline for future research. For teams working on HR tech, talent profiling, or job matching, this is a ready-made starting point.

Summary

This work systematically answers, for the first time, how to accurately map free text to SFIA's closed vocabulary while capturing responsibility levels. Its three conclusions — retrieval recalls broadly while generation is precise, levels require explicit modeling, and multi-agent architectures aren't necessarily superior — are both concrete contributions to the SFIA domain and a measured recalibration amid today's "agentic hype": using the right level of complexity for the right task matters far more than blindly chasing architectural sophistication.