AI-Atlas: File-by-File Assessment of Real Skill in Deep Learning Projects

Deep learning project descriptions on resumes often sound identical: a from-scratch Transformer implementation and a notebook that merely calls .fit() look nearly indistinguishable to recruiters. A developer on Reddit has released a tool called AI-Atlas that attempts to solve this information asymmetry through code analysis—it reads public repositories file by file, scores projects across four dimensions (depth, originality, completeness, and reproducibility), and cites the specific files that support each score.

It tackles the old problem that "resumes can't distinguish real ability"

When evaluating a deep learning project, recruiters typically only see the README, a few code snippets, and a verbal description. A repository that hand-writes attention mechanisms from scratch and one that imports a pretrained model and runs the pipeline can both be described with equally impressive sentences on a resume. AI-Atlas's starting point is to shift the object of evaluation from "what the project claims to do" to "what the code actually contains": the tool directly reads the file structure of public repositories, judges the content of each file, aggregates those judgments into scores across four dimensions, and annotates which files contributed to the score.

This "file citation" design is key. A bare numeric score easily becomes a black box, but when a score maps to specific files, developers can self-check which parts are seen as highlights and which as filler, and recruiters can follow the citations to verify.

The difference between implementing a Transformer from scratch and calling a high-level API usually shows up at the code level as whether handwritten tensor operations appear—constructing attention matrices, multi-head splitting, positional encoding, masking logic—and whether the training loop, loss function, and optimizer scheduling are implemented by hand. Projects that directly call model.fit() encapsulate these details inside the framework: less code, high readability, but they don't reflect how well the author understands the model's internal mechanisms. The reason tools like AI-Atlas read file by file is precisely that this difference is rarely written in the README, yet abounds in specific module files—for instance, a handwritten attention.py versus a dataset.py that only loads data carry completely different weight as capability signals.

What the four scoring dimensions mean

According to the author, AI-Atlas's scoring revolves around depth, originality, completeness, and reproducibility, but the specific weights and judgment criteria for each dimension are not public. Judging from the tool's positioning, these dimensions likely correspond to different questions:

• Depth: Does the project stop at calling high-level APIs, or does it involve low-level implementations such as model architecture, training pipelines, and data processing?
• Originality: Is the code a common tutorial reproduction, or does it contain independently designed components?
• Completeness: Does the repository include runnable training, evaluation, and inference pipelines, rather than only fragmented notebooks?
• Reproducibility: Are engineering details such as dependency declarations, configuration, and random seeds in place?

Note that these dimension names come from the author's description; how they are quantified and what thresholds are set is not elaborated in the original text, so the precise meaning of the scores should not be over-inferred.

Reproducibility in machine learning engineering refers to whether a piece of code can produce the same or statistically equivalent results on another machine at another point in time. It typically depends on several concrete practices: fixing random seeds (for Python, NumPy, and the framework separately), explicitly declaring dependency versions (requirements.txt, environment.yml, lock files), centralizing hyperparameters and paths in configuration files rather than hardcoding them, providing data acquisition or preprocessing scripts, and recording the runtime environment. For recruiters, high reproducibility means the author has the habit of delivering experiments for others to run, which says more about engineering competence than model metrics themselves. In file-by-file analysis, configuration directories, dependency manifests, and entry scripts are often the main sources of evidence for this dimension.

The engineering value of file-by-file analysis

Pushing the evaluation granularity down to the file level is where this tool is more targeted than generic code quality scanners. Deep learning repositories vary greatly in structure: some concentrate core logic in a few modules, while others are a pile of many experiment scripts. File-by-file scoring can distinguish "a carefully implemented model file" from "a dozen parameter-tuning config files," which directly helps judge where the author invested their engineering effort.

In terms of usage cost, the tool is free, requires no registration, and only reads public repositories, which means recruiters, open-source maintainers, or learners can try it directly without configuring credentials or authorization flows. In the post, the author also explicitly poses "what else would you like it to check" as an open question to the community, indicating that the current version's functional boundaries are still expanding.

Limitations and open questions

The core difficulty of such tools lies in the transparency of their judgment criteria. All four dimensions are fairly subjective concepts, and different teams may have very different understandings of "originality" and "depth." If the scoring logic is not public, developers find it hard to improve in a targeted way, and recruiters find it hard to judge the credibility behind a score. The original text does not say whether a scoring explanation page or interpretability report exists, which is something to confirm before actual adoption.

Another practical issue is repository visibility. The tool only reads public repositories, while a large number of job seekers' project code is not public, which limits its reach as a resume aid. In addition, whether non-capability factors such as code style, comment language, and directory conventions affect scoring is not mentioned in the original text.

For developers, the greater significance of such tools may lie not in the score itself but in the evaluation mindset it suggests: breaking project quality down to a locatable file level rather than stopping at the README's self-description. As for whether AI-Atlas's scores are accurate, more people need to verify with real repositories.

The distinction between static code analysis and black-box scoring is worth clarifying. Static analysis means inferring project characteristics without running the code, solely by parsing source structure, import relationships, function call chains, and file organization; it is low-cost and batch-processable, but it cannot verify whether the code actually runs, nor whether the implementation is correct. LLM-driven repository understanding usually falls into this category: the model reads file contents and makes judgments, which is essentially semantic inference based on code text. Therefore such scores reflect "how deep the code looks," not "whether the experiment succeeded." Understanding this is necessary to reasonably interpret the scores AI-Atlas gives—it is better suited as a screening and self-check clue than as a certification of ability.