Retrieval Over Scale: Improving Log Level Prediction with Socio-Technical Context

The Challenge of Log Level Prediction: Why Random Examples Fall Short

Developers insert log statements to record code execution, and these logs are essential for detecting and troubleshooting system anomalies. The log level (e.g., DEBUG, INFO, WARN, ERROR) determines whether a message is recorded, and choosing a level is a trade-off: too low a level can mask important information in production, while too high a level produces noisy logs and incurs extra I/O and storage overhead.

Recent research has attempted to use large language models (LLMs) with in-context learning (ICL) to predict log levels, but typically samples examples randomly in the prompt. This approach overlooks the fact that logging practices have socio-technical attributes, with conventions varying across components and development teams. The paper points out that off-the-shelf LLMs using random examples achieve only a median AUC of 0.56 to 0.76, leaving substantial room for improvement without fine-tuning.

In-Context Learning (ICL) refers to enabling a large language model to perform a new task by providing a few examples in the prompt without updating the model parameters. The effectiveness of ICL depends heavily on the quality and relevance of the examples: the closer the examples are to the target task, the easier it is for the model to capture the correct patterns. In log level prediction, examples are pairs of "code snippet—log statement—level." While random sampling is simple, it may introduce examples irrelevant to the target code, causing the model to learn noisy patterns. Therefore, prompt design for ICL is essentially an example selection problem, which motivates the retrieval-based approach in this paper.

AUC (Area Under the Curve) typically refers to the area under the ROC curve, a common metric for ranking quality in binary classification tasks. AUC ranges from 0 to 1, where 0.5 corresponds to random guessing, and values closer to 1 indicate a stronger ability to distinguish positive from negative samples. In log level prediction, AUC can be interpreted as the model's tendency to assign a higher probability to the correct level. The paper reports median AUC rather than mean because of large variance across projects, and the median better reflects typical performance.

Retrieval Strategy: Finding Examples from Two Dimensions—Functionality and Ownership

The paper proposes a retrieval-based LLM log level prediction method, with the core idea of prioritizing contextual relevance over random selection or basic similarity. The method retrieves relevant log examples along two socio-technical dimensions:

• Functionality dimension: code that performs similar operations. The same type of operation (e.g., database query, API call) typically follows similar log level conventions.
• Ownership dimension: code maintained by the same group of developers. Logging habits within a team tend to be consistent, and even if code functionality differs, the ownership signal can provide valuable reference.

These two dimensions together form a "Multiplex Socio-Technical Context," providing the LLM with examples that are closer to real-world development scenarios.

Socio-Technical Attributes originate from software engineering research, emphasizing that software systems are not only technical artifacts but are also embedded in organizational structures, team collaboration, and development conventions. Logging practices are a typical example: one team may agree that all external call failures should be logged as WARN, while another team prefers ERROR; the same functionality may also have different conventions in different components. These conventions are not written in code syntax but genuinely influence log level choices. Therefore, code text similarity alone is insufficient; it is also necessary to capture socio-technical signals such as "who wrote it" and "what it does."

Ablation Study is a common method in machine learning to verify the contribution of each component: removing or replacing a module one at a time and observing performance changes. In this paper, ablation experiments use only the functionality signal, only the ownership signal, and compare with random retrieval. If using a single signal alone still outperforms the random baseline, the signal itself is effective; if combining the two outperforms the stronger single signal across all projects, the two dimensions are complementary rather than redundant.

Experimental Results: Significant AUC Improvement, Outperforming Supervised Predictors

The study evaluated four LLMs on ten popular Java projects. Results show that this retrieval method improves the median AUC by 0.07 compared to the random baseline, and combining the two signals achieves a median AUC of 0.83 to 0.97. Ablation experiments show that each signal alone outperforms random retrieval, and combining the two outperforms the stronger single signal across all ten projects.

More interestingly, the method also significantly outperforms state-of-the-art supervised predictors (such as DeepLV), with a median AUC higher by 0.12 to 0.19. This means that without fine-tuning the model, better results can be achieved solely through clever contextual retrieval.

Supervised Predictors refer to specialized models trained on labeled data, such as DeepLV. These methods typically require a large amount of manually labeled log level data and are trained for specific projects or datasets, thus performing strongly within their training distribution but with limited cross-project generalization. The retrieval-augmented LLM method in this paper does not perform any fine-tuning and surpasses DeepLV solely with examples in the prompt, indicating that in scenarios with high data annotation costs and large differences in team conventions, contextual retrieval may be more cost-effective and adaptable than traditional supervised training.

Implications: Socio-Technical Signals Matter More Than Model Scale

The paper's findings highlight the importance of socio-technical signals such as code functionality and ownership for LLM log level prediction. In an era of increasingly powerful LLMs, how to provide high-quality contextual examples to the model may be more effective than simply scaling up model size. For developers, this means they can leverage team-internal logging conventions and code patterns of similar functionality to more efficiently generate reasonable log levels, reducing information loss or noise in production environments.

This idea may also generalize to other code-related tasks, such as comment generation and exception handling recommendation. As long as the task itself is influenced by team conventions and functional semantics, retrieving socio-technical context is worth trying.