asmsense: 228K x86-64 Function Dataset Tackles Reverse Engineering Annotation Challenges

Title

asmsense: 228K x86-64 Function Dataset Tackles Reverse Engineering Annotation Challenges

Meta Description

asmsense releases a 228K x86-64 function dataset with source-level labels and trains a Qwen3-8B+LoRA model for stripped binary function naming. The project openly discusses data gaps and evaluation challenges while seeking community feedback.

Article Content

In reverse engineering and binary analysis, automatically naming and describing functions from stripped binaries has long been a challenge. Recently, an open-source project called asmsense released a new dataset containing 228,177 x86-64 functions and trained a small model to tackle this task.

According to the project author on Reddit, asmsense didn't simply crawl existing binaries—rather, it compiled 341 permissively licensed C projects from scratch. This means each function retains its real name and source code context, providing high-quality "source-aware" labels for supervised learning.

Dataset Composition and Core Features

The asmsense dataset currently contains 228,177 functions, spanning multiple optimization levels from -O0 to -Os. This cross-optimization-level design exposes the model to code forms from different compilation stages, better reflecting the diversity of binaries in the real world.

Each function comes with descriptions in English and Turkish, with Chinese descriptions planned. The author emphasizes that descriptions were generated by a "teacher LLM" that could see the C source code, and only sampled by another LLM for checking—which somewhat limits the absolute accuracy of the labels but provides a feasible compromise for large-scale data.

The code and data are open-sourced on GitHub under a permissive license, making it easy for researchers to reproduce and extend.

Model Approach: Qwen3-8B + LoRA

The project uses Qwen3-8B as the base model and fine-tunes it with LoRA to perform function naming and description generation. This combination strikes a balance between computational cost and performance, making it suitable for experimentation in resource-constrained environments.

The author candidly admits that the model currently targets clang/Mach-O compilation results on macOS and does not yet cover GCC/ELF builds. This means the dataset has a clear platform skew, which may affect the model's generalization ability in Linux environments.

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique. Its core idea is to inject low-rank decomposition matrices alongside the weight matrices of each layer in a pretrained model, training only these newly added small parameters while freezing the original model weights. Compared to full-parameter fine-tuning, LoRA significantly reduces memory usage and computational cost while maintaining performance close to full fine-tuning. Qwen3-8B is an open-source large language model from the Qwen series, with 8 billion parameters and strong code and reasoning capabilities. Applying LoRA to Qwen3-8B enables adaptation for assembly code understanding tasks on a single or a few consumer-grade GPUs, which explains why the project can iterate quickly in resource-constrained environments.

Data Gaps: A Clear Map of "What's Missing"

Perhaps the most commendable aspect of asmsense is its candid listing of the current dataset's shortcomings. The author uses a single page on the project site to walk through what's missing step by step:

• Other architectures: ARM64, RISC-V, MIPS, etc. are not yet included; currently x86-64 only.
• GCC/ELF builds: All samples currently come from clang/Mach-O, lacking the GCC compilation outputs common in the Linux ecosystem.
• Chinese descriptions: Multilingual support is still in planning.
• Human-verified labels: Current labels have only been sampled by LLMs, lacking comprehensive review by human experts.
• Agent trajectories: No records of interactive analysis processes are included.
• Public benchmark evaluations: No evaluations have been run on benchmarks such as BinMetric, REBench, BinSum, and DeBinVul.

This transparency is highly valuable for subsequent researchers—it clearly identifies directions for improvement and avoids overpromising.

Vision and Challenges of a Reinforcement Learning Environment

The author is considering turning the entire data processing pipeline into a reinforcement learning (RL) environment: compile, strip symbols, have the model name functions, then score based on token-level F1 against the real names.

However, the author is also keenly aware of the flaws in this reward design: the model could game the reward function by generating generic names like "get_data" to obtain partial credit. In the post, he asks the community for help: "If you've trained on similarly fuzzy objectives, how did you shape the reward? Did RL beat plain fine-tuning?"

This touches on a core difficulty of the binary function naming task: the semantic ambiguity of function names. The same functionality may have multiple reasonable naming conventions, and token-level F1 cannot fully capture semantic correctness.

Token-level F1 is a commonly used evaluation metric for sequence generation. It computes precision and recall at the token level between the predicted sequence and the ground truth sequence, then takes the harmonic mean. In the function naming context, it measures the degree of overlap at the subword level between model-generated function names and real function names. However, this metric cannot distinguish semantically equivalent but differently spelled names (e.g., get_data vs. fetch_data), and it tends to reward outputs containing high-frequency generic affixes. This is precisely the technical root of the author's concern about reward gaming: the model may learn to generate plausible-sounding but low-information generic names, achieving high scores without truly understanding function behavior.

Evaluation Challenges: What If There's No Human Panel?

Another key question the author raises: how do you evaluate the quality of generated descriptions without a panel of human experts?

Currently, descriptions are generated by a teacher LLM and only spot-checked by another LLM. While this automated evaluation approach is scalable, it carries the risk of circular reasoning—an LLM evaluating an LLM's output may fail to detect systematic errors.

For an offline model, which data gap is most critical? The author himself is seeking community input. From a practical application standpoint, the lack of GCC/ELF support may be the most fatal, since most production binaries are in ELF format.

Circular reasoning risk is a classic problem in automated evaluation: when the evaluator and the evaluated share the same origin (e.g., belong to the same LLM family), the model may systematically favor outputs generated by itself or similar models, resulting in inflated evaluation scores that fail to reflect true quality. In the asmsense scenario, the teacher LLM generates descriptions and another LLM spot-checks them—both may share similar training data distributions and biases, making it difficult to detect factual errors or omissions in the descriptions. Common mitigation strategies include introducing human evaluation subsets, using LLMs from different vendors as judges, and designing objective metrics based on verifiable properties of the source code (such as whether API names from function calls are correctly mentioned).

Multi-Source Cross-Validation and Community Feedback

A note: the information above mainly comes from the project author's unilateral statements on Reddit, and no independent third-party verification has been seen yet. The actual quality of the dataset, the true performance of the model, and the effectiveness of the evaluation methods all need to be tested by the community through use.

The author also candidly states that he used LLMs to assist with writing and grammar checking during the construction process, so minor errors may exist, and welcomes corrections. This openness helps build trust, but also reminds users to remain cautious about data quality.

Significance for the Binary Analysis Field

Despite its many shortcomings, the emergence of asmsense cannot be ignored. It provides a reproducible, large-scale dataset with source-level labels, filling a gap in publicly available resources for binary function understanding.

More importantly, the author explicitly lists data gaps and evaluation challenges, pointing the way for future research. Whether it's extending architecture support, introducing human verification, or designing more robust reward functions, there are clear entry points.

For researchers working in reverse engineering, malware analysis, or binary code similarity detection, asmsense is a starting point worth watching and contributing to. The project's open-source nature also means the community can collectively drive its evolution and gradually address current shortcomings.

With more researchers participating, we may see the birth of a more comprehensive and reliable benchmark for binary function understanding.

One-line Summary

asmsense releases a 228K x86-64 function dataset and fine-tunes a Qwen3-8B+LoRA model, transparently listing data gaps and evaluation challenges.

Paragraph Summary

asmsense is an open-source project for automatic naming and description of functions in stripped binaries. It released a dataset of 228,177 x86-64 functions from 341 permissively licensed C projects, covering optimization levels from -O0 to -Os, with bilingual English/Turkish descriptions generated by a teacher LLM. The project uses Qwen3-8B as the base model with LoRA fine-tuning for naming and description tasks, but current samples are limited to clang/Mach-O compilation outputs. The author explicitly lists data gaps (missing ARM64/RISC-V/MIPS architectures, GCC/ELF builds, Chinese descriptions, human-verified labels, agent trajectories, and public benchmark evaluations) and is exploring turning the pipeline into an RL environment, while worrying that token-level F1 rewards can be gamed by generic function names. On evaluation, there is no human expert panel, and reliance on LLM automated checks carries circular reasoning risk. The project stands out for its transparency and reproducibility, providing a scalable starting point for the binary function understanding field.