semanticfa: An R Package for Response-Free Semantic Factor Analysis

Psychometric scales typically rely on large amounts of respondent data to validate their structure, but language models offer a new path: converting scale items into semantic embeddings and directly performing factor analysis on the semantic similarities among items. An R package called semanticfa implements this approach and provides supporting features such as factor interpretation, redundant item flagging, short-form construction, and new item screening.

The Core Logic of Semantic Factor Analysis

Items from reflective psychometric scales can be viewed as "proposals" about the structure of a construct. A language model converts each item into an embedding vector—a string of numbers where items with closer semantic meaning have more similar vectors. semanticfa directly performs factor analysis on the similarity matrix of these item embeddings, and the entire process requires no respondent answer data whatsoever.

This forms a complementary relationship with traditional factor analysis based on response data. The paper explicitly states that semantic factor analysis "complements rather than replaces" response-based factor analysis. Its value lies in the ability to test whether items cluster according to the expected structure before data collection, or to quickly assess scale quality when administration is not feasible.

Embedding Vectors and Semantic Similarity

An embedding vector is a dense vector representation obtained after a language model maps text into a high-dimensional real-valued space. The model is trained on large-scale corpora so that semantically similar texts are close to each other in vector space, typically measured by cosine similarity, which captures the directional alignment of two vectors (ranging from -1 to 1). Semantic factor analysis exploits this geometric property: each item is treated as a point, pairwise cosine similarities between items are computed to obtain an item × item similarity matrix, and factor analysis is then applied to that matrix. This is mathematically isomorphic to the traditional practice of computing a Pearson correlation matrix among items based on respondent answers and then performing factor analysis; the only difference is the source of the matrix—one comes from semantic space, the other from response data. Therefore, any factor extraction and rotation methods applicable to correlation matrices (such as principal axis factoring and promax rotation) can be used directly, and semanticfa's interface design follows this logic.

A Complete Empirical Example

The authors demonstrated with the 50-item International Personality Item Pool Big-Five Factor Markers, using Qwen3-Embedding-8B as the embedding model, and compared the results with factor analysis based on 874,434 respondents.

Factor retention criteria wavered between five and six factors, and following a预设 tie-breaking rule, five were ultimately retained. Three of these factors mainly corresponded to a single dimension, while the other two mixed most extraversion items with five agreeableness items. The congruence coefficient (1 indicating perfect agreement) between each factor and its best-matching human-data factor ranged from .92 for openness to .52 for agreeableness.

This result shows that the semantic structure overlaps considerably with the classic Big Five dimensions, but the presence of mixed factors also suggests that semantic similarity does not equal psychometric homogeneity.

Congruence Coefficient

The congruence coefficient (often denoted Tucker's φ) is a psychometric index for comparing two sets of factor loading patterns. To compute it, the loading vectors of the same factor in two solutions (e.g., each item's loading on that factor) are standardized and their inner product is taken; the value theoretically ranges from -1 to 1, with 1 indicating that the two loading directions are perfectly aligned, 0 indicating no relation, and negative values indicating opposite directions. It is more rigorous than simply comparing factor variance contributions or visually inspecting loading tables, because it simultaneously considers the relative magnitude and sign of each item's loading. This article uses it to quantify the agreement between semantic factors and human-data factors based on 874,434 respondents, ranging from .92 for openness to .52 for agreeableness, visually showing which dimensions are well captured at the semantic level and which exhibit structural misalignment. In practice, φ above .90 is generally considered very congruent, .80–.90 acceptable, and below .80 requires cautious interpretation.

Agreement and Divergence Between Semantics and Response Behavior

An intuitive hypothesis is that items with more similar meanings also have more similar responses. The data show that this trend does exist, with a correlation of .46 between semantic similarity and raw response similarity.

However, reverse-scored items break this pattern. Oppositely keyed items within the same dimension are highly similar semantically but have opposite responses; their semantic similarity (.58) is almost on par with cross-dimension item pairs (.57). The authors attempted to flip the sign of embeddings for reverse-scored items, but this did not improve consistency with raw responses and instead reduced consistency with reverse-scored responses.

This finding has direct practical implications: if a scale contains reverse items, structural interpretation via semantic factor analysis requires extra caution.

Reverse-Scored Items

Reverse-scored items are items whose wording direction is opposite to the construct they belong to, for example adding an item like "I usually keep quiet" to a scale measuring extraversion. Respondents' raw scores on such items must be reverse-transformed (e.g., subtracting the raw score from 6 on a 5-point Likert scale) before being combined with other positively keyed items; otherwise, method effects are introduced and internal consistency is reduced. The original purpose of reverse items is to suppress acquiescence bias and careless responding, but in semantic space, reverse items are highly similar to positively keyed items of the same dimension (both revolve around the same construct) while being negatively correlated in responses. Semantic factor analysis only sees the former, so it groups reverse items and positive items into the same semantic factor, causing that factor to fail to directly correspond to the factor structure in response data where "positive items load positively and reverse items load negatively." This is precisely the root cause of the finding that sign-flipping embeddings did not improve consistency.

Short-Form Construction and Semantic-Behavioral Coherence

Under the five-factor specification, the 25-item short form generated by semanticfa (five items per dimension) outperformed all 200 random five-item combinations, indicating that semantic information can help select more representative items. However, parallel analysis retained only three factors, suggesting that the short-form structure still requires validation with response data.

The authors propose the concept of "semantic-behavioral coherence," namely the degree of agreement between semantic structure and response structure given a particular embedding model, which can serve as a measurable property of a scale. This provides a new diagnostic dimension for scale development: the extent to which semantic structure deviates from response structure may reflect misalignment between item wording and the psychological construct.

Applicable Scenarios and Limitations

semanticfa is suitable for rapid item screening in the early stages of scale development, constructing candidate short forms, or assessing whether translated versions preserve the semantic structure. Its limitations are equally clear: the choice of embedding model affects results, the handling of reverse-scored items is not yet ideal, and factor retention decisions may depend on additional rules.

For psychometric researchers and practitioners who need to quickly evaluate scales, this tool offers a low-cost preliminary check, but it cannot replace confirmatory analysis based on real responses.

One-line Summary

semanticfa uses language model embeddings to perform response-free semantic factor analysis of scale items, complementing traditional methods.

Paragraph Summary

semanticfa is an R package that converts psychometric scale items into language model embedding vectors and directly performs factor analysis on the semantic similarity matrix among items, all without respondent answer data. Using a 50-item Big Five personality scale as an example, the authors compared Qwen3-Embedding-8B embeddings with factor analysis of 874,434 respondents' data; congruence coefficients between semantic factors and human-data factors ranged from .92 to .52, with mixed factors present. The correlation between semantic similarity and response similarity was .46, but reverse-scored items were highly similar semantically while responses were opposite, and sign-flipping embeddings did not improve consistency. Under the five-factor specification, semantic information can generate a 25-item short form superior to random combinations. The authors propose the concept of "semantic-behavioral coherence," emphasizing that the method complements rather than replaces traditional validation and is suitable for early-stage screening in scale development and translation evaluation.