Data Autopsy: Using Open-Source AI Tools to Investigate Dataset Risks Before Training

Most machine learning workflows start with a familiar routine: load the dataset, look at a few rows, check the shape, handle missing values, and then start training the model. This process is fine in itself, but it leaves behind some less obvious hazards—duplicate records can distort distributions, extreme values deserve closer inspection, categorical features may represent the same value in multiple ways, and the target variable may be severely imbalanced. The most dangerous case is target leakage: the model performs brilliantly in evaluation but fails miserably on real data because what it learned is not the underlying pattern but information it should never have had access to at prediction time.

These problems are often discovered through manual inspection, scattered notebook cells, or debugging sessions after the model has been trained. Data Autopsy tries to bring this investigative work together in one place. It is an open-source, AI-assisted forensic investigation tool for tabular datasets. Its goal is not to treat a dataset as a collection of rows and columns, but to investigate its structure, identify suspicious patterns, examine the evidence, and understand what might have gone wrong.

What it is, and why deterministic analysis comes first

Data Autopsy currently focuses on CSV datasets. The workflow is straightforward: upload a CSV, analyze structure and content, run deterministic forensic checks, review findings by severity, inspect affected columns, measurements, and example rows, understand the meaning and limitations of the findings, export a report, and optionally use a local language model to explain the findings in plain language.

One of the most important decisions in the project architecture is the separation of detection and explanation. Statistical and rule-based checks written in Python are responsible for identifying potential problems, returning structured findings that include measurements, thresholds, affected columns, and supporting evidence. The language model is only an optional explanation layer: it can translate technical findings into easy-to-understand language, but it does not decide whether a detector has found a problem.

This separation brings several benefits. Reproducibility: the same dataset, configuration, and detector version should produce reproducible results. Transparency: every finding can be traced back to a measurement or an explicit rule, rather than an unexplainable statement generated by a model. Testability: detection logic can be validated with synthetic datasets that have known properties. Privacy: core investigation does not require an external AI API. Graceful degradation: the application should remain useful when a local model is unavailable.

AI can make technical reports easier to understand, but it should not be responsible for inventing the underlying evidence.

Ten categories of checks in the forensic engine

The forensic engine is the core of the project, and the current implementation covers ten categories of investigation.

Missing values and duplicate records

Missing data is the first thing that should be checked in any dataset, but the missing percentage is only part of the story. A column with a small amount of missing data may be easy to handle; a column that is half missing requires more careful decisions; a completely empty column may be unusable for its intended purpose. The tool classifies missing-value findings using configurable thresholds, with completely or heavily missing columns receiving higher severity. Reports can include affected columns, missing counts, missing percentages, and the threshold that triggered the finding. Missing values are not automatically errors—in some datasets, the absence of a value itself carries meaning.

Duplicate data affects counts, distorts distributions, and can also cause evaluation problems when related records appear in both the training and test sets. The engine checks for exact duplicate rows and potential duplicate candidates based on selected non-identifier features. These are two different situations: exact duplicates mean that the relevant row values match, while potential duplicates mean that records are similar under a specific comparison rule. Potential duplicates require more caution—two people may share the same age and city, two transactions may have the same amount, and repeated measurements in time-series data may be expected. Therefore, findings should be treated as investigation candidates, not as evidence that must be deleted.

Median absolute deviation (MAD) and the modified Z-score are more robust methods for outlier detection than standard deviation. The traditional Z-score measures how far a value deviates from the center using the mean and standard deviation, but the mean and standard deviation are themselves sensitive to extreme values—a few outliers can raise the mean and inflate the standard deviation, thereby "hiding" those extreme values. MAD takes the median of the absolute differences between each observation and the median, and it is almost insensitive to outliers. The modified Z-score is defined as 0.6745 × (x − median) / MAD, where 0.6745 is a constant that aligns the statistic with the standard deviation scale under a normal distribution. A threshold of 3.5 is commonly used, corresponding to a tail probability of about 3 in 10,000 under a normal distribution. Because it does not rely on the mean, the MAD method is more reliable than IQR and ordinary Z-scores on heavy-tailed, contaminated data.

Constant columns, identifier columns, and numeric outliers

Constant columns contain only one distinct value and usually offer no useful variation for predictive models. Near-constant columns contain a dominant value in almost every row, with little variation elsewhere. The tool flags constant columns as well as columns where a single value accounts for at least 99% of observations. These features may be redundant, incorrectly filled, or simply part of the dataset design. A nearly constant operational flag may still be meaningful in a specific application, so the tool reports findings rather than silently removing features.

Columns such as record IDs, unique codes, and generated identifiers may be mistaken for meaningful predictive features. The current detector flags columns with very high uniqueness under a defined minimum sample condition. Unique identifiers may be suitable for joining tables or tracking records, but if they merely memorize individual observations, they are not suitable as predictive inputs. The goal is to help users ask: does this feature represent meaningful information, or does it merely identify the row?

Outlier detection is not just about finding values far from the mean. Different datasets have different distributions; a high-value transaction may be suspicious in one context and completely ordinary in another. Naturally heavy-tailed variables may contain extreme values without being errors. Data Autopsy uses two statistical methods: the interquartile range (IQR) method with a configurable multiplier of 1.5, and a modified Z-score based on median absolute deviation (MAD) with a threshold of 3.5. Using multiple methods allows anomalously distant values to be viewed from different perspectives. Findings can include the number and percentage of flagged observations, relevant bounds or thresholds, summary measurements, and examples of extreme values. These methods are screening techniques; they cannot determine that a value is wrong, fraudulent, or safe to delete.

Categorical feature issues

High-cardinality categorical columns can cause problems for some preprocessing and modeling methods. A column containing thousands of distinct labels may represent free-text input, product codes, user identifiers, or unnormalized categories. The engine flags categorical features using configurable unique-value count and uniqueness ratio thresholds. High cardinality is not necessarily bad—a product catalog may legitimately contain many product codes, and a geographic field may have many valid categories. The purpose of this finding is to prompt inspection of how the feature should be represented or encoded.

Data collected from multiple forms, systems, or manual entry often contains variants that represent the same apparent category, such as Mumbai and mumbai, Active and active, or Completed and COMPLETED. The current check looks for conflicting string representations after trimming surrounding whitespace and normalizing case. This inconsistency increases the number of apparent categories and complicates grouping or encoding. However, normalization should not be applied blindly—some datasets distinguish case by design, and different strings are not always interchangeable. The tool identifies inconsistencies and lets the user decide how to handle them.

High cardinality refers to a large number of distinct values in a categorical column, such as a column with tens of thousands of unique user IDs or product codes. For linear models and most tree models, directly one-hot encoding a high-cardinality column produces an extremely wide and extremely sparse feature matrix, which both consumes memory and leaves each category with very few samples to estimate parameters, making overfitting easy. Common approaches include target encoding, frequency encoding, hashing, or merging rare categories into "other." High cardinality itself is not an error: fields such as postal codes and SKUs naturally have many legitimate values. What really deserves caution is mistaking them for ordinary categorical features or encoding them directly when the category distribution differs between training and test sets.

Class imbalance, correlation, and target leakage

In classification tasks, the target distribution strongly affects training and evaluation. A dataset with a large majority class and a small minority class may produce a model that looks accurate but fails to identify the most important cases. The tool examines the distribution of the selected target and flags severe imbalance using configurable ratios. The current rules include two thresholds of different severity: 4:1 and 20:1. These values are screening thresholds, not a universal definition of an imbalanced dataset. Class imbalance is common in legitimate applications such as fraud detection, medical screening, and anomaly detection, and the correct evaluation strategy depends on the problem, the cost of different errors, and the number of available samples.

Strong correlations between numeric features may indicate redundancy or relationships worth investigating. The engine identifies highly correlated feature pairs using a Pearson correlation threshold. To keep analysis manageable on larger datasets, the current implementation limits pairwise analysis to a subset of high-variance numeric columns and limits the number of rows considered. This is a practical trade-off between usefulness and computational cost. Correlation does not prove causation, and high correlation is not automatically a data quality problem—some features are expected to vary together because they measure related attributes. The report helps users locate strong relationships and decide whether they are expected, redundant, or potentially problematic.

Target leakage deserves particular attention because it can make model evaluation misleading. Leakage occurs when a feature contains information about the target that should not legitimately be available at prediction time, or when information about the target or evaluation process enters training in an invalid way. The current checks look for several suspicious signals, including features identical to the target, very strong numeric relationships, unusually strong categorical class purity, and, where applicable, high ROC-AUC scores. These signals can identify features worth careful examination, but they do not by themselves prove leakage. A feature may legitimately be a strong predictor, a numeric relationship may be expected in a scientific dataset, and a categorical variable may almost determine the target because the underlying process really works that way.

A more complete leakage investigation should also consider how the data was collected, when each feature was available, how preprocessing was performed, and whether the evaluation setup reflects a realistic prediction scenario. The long-term goal is to make this kind of investigation more systematic, rather than treating suspicious correlations as conclusive evidence.

ROC-AUC (area under the receiver operating characteristic curve) measures a classifier's ability to distinguish positive from negative classes across all possible thresholds, with a value of 0.5 equivalent to random guessing and 1.0 representing perfect discrimination. An important property is that it is insensitive to class distribution—when the ratio of positive to negative samples changes, ROC-AUC remains basically stable as long as ranking ability is unchanged, so it is often used for model comparison in imbalanced settings. But in target leakage checks, an unusually high ROC-AUC (for example, a single feature reaching 0.95 or above) is precisely a red flag: it may mean that the feature is not available at prediction time, or that target information has already been mixed in during data collection or preprocessing. It should be emphasized that a high AUC only shows that the feature is highly separable from the target; it cannot directly prove leakage, and judgment still needs to consider when the feature is generated and whether it is available.

Evidence matters more than severity

A dashboard that only shows red warnings without explaining why is not very useful. Data Autopsy uses structured findings to carry the context needed for investigation. Depending on the check type, a finding can include a stable identifier and check name, severity classification, affected columns, relevant row indices, measurements and detection thresholds, an explanation of why the issue matters, suggestions for further investigation, and known limitations of the detection method.

This structure separates results from presentation. The dashboard can use it to filter and organize findings, the report exporter can use it to generate documentation, tests can validate measurements and thresholds, and the explanation layer can use the evidence to generate more readable summaries. Severity and confidence should remain distinct concepts—a potentially critical issue is not necessarily a confirmed problem, and the strength of a detector depends on the evidence it can actually observe.

The role of the local language model

The project aims to explore practical AI integration without making paid cloud APIs a requirement. Data Autopsy is designed to use Ollama with Qwen2.5 1.5B as the default local explanation model, and smaller models can also be used as alternatives. The model's role is to explain existing findings in plain language, for example by describing why a high proportion of missing values matters, what the extreme-value detector measured, or what follow-up questions developers should consider. The underlying measurements still come from the deterministic engine.

This design has several advantages: data analysis does not depend on a remote language model service; the project remains usable without a model; model output can be checked against structured evidence; users can experiment with local inference on supported hardware; and an external AI provider is not a required part of the core workflow. Small models also have limitations: they may produce vague explanations, misunderstand context, or fail to follow the requested output structure. Local inference performance depends on the machine, model, and dataset size. Therefore, model responses should be validated, kept within reasonable time limits, and supported by deterministic fallbacks. Local inference is a design choice and does not guarantee that every part of the application is automatically private—the application must still inspect external requests, telemetry, logs, and other data-handling behavior.

Ollama is an open-source tool for running and managing local large models. It packages model weight downloads, quantized format loading, and inference services into simple commands. Developers use ollama pull to obtain models and call them through a local HTTP interface, without needing a cloud service account. Qwen2.5 1.5B refers to an instruction-tuned model with about 1.5 billion parameters in the Qwen series. After quantization, it can run in the memory of an ordinary laptop and is suitable for lightweight text tasks such as summarization and rewriting. The inference speed and memory usage of such small models are far lower than those of large models with tens of billions of parameters, but language fluency and instruction-following ability are correspondingly limited, making errors in structured output or long contexts more likely. For this reason, Data Autopsy places it only in the explanation layer and retains deterministic fallbacks, avoiding the impact of model quality fluctuations on core analysis conclusions.

Tech stack, interface, and testing

The project uses a deliberately familiar Python stack: Python for core analysis logic, Pandas and NumPy for tabular data and numerical operations, SciPy and scikit-learn for statistical tools and modeling metrics, Streamlit for building the interactive interface, Plotly for visual exploration of distributions and relationships, Ollama and Qwen2.5 1.5B for optional local language model explanations, and Pytest for testing detectors, data processing, reports, and related behavior. The goal is to make it easy for contributors already using Python and data science tools to get started, without requiring a complex distributed architecture.

The forensic engine can produce many findings, but displaying everything at once causes information overload. The interface is designed around the investigation workflow, with main areas including: overview (dataset summary and the most important findings), dataset profile (column types, missingness, distributions, and other structural information), forensic findings (a prioritized view of detected issues), leakage investigation (a focused view of target-related risks), evidence inspector (measurements, thresholds, affected rows, and limitations), AI investigator (optional explanations based on existing findings), and report export (saving results for later review). The visual direction is restrained and information-centered; severity should be clear, but the entire interface should not look like an alarm screen. Charts should answer specific questions, and visualizations should not exist merely to make the application look more sophisticated.

Deterministic checks are only useful when they behave predictably on both ordinary and abnormal inputs. Tests need to cover more than a single clean CSV. Synthetic datasets are particularly useful because they let us introduce known conditions and verify whether the engine detects them. Examples include: completely empty columns, exact duplicate rows, constant features, features containing extreme numeric values, categories represented with inconsistent case, severely imbalanced targets, features that duplicate the target, datasets containing legitimate strong predictors, boolean and datetime columns, small datasets and wide tables, and columns containing missing values or unexpected types. Tests should also verify that analysis does not silently modify the original dataset and that edge cases do not crash the application. Target leakage checks require especially careful testing—the detector should not flag every legitimate strong predictor as leakage simply because it correlates with the target. Report generation should also be tested independently, so that findings can be exported consistently even when the language model is unavailable. The goal is not to claim that the test suite proves the application is flawless, but to make detector behavior explicit and to catch regressions as the project evolves.

What it cannot prove, and how to run it locally

A good forensic tool should be honest about its boundaries. Missing values are not necessarily errors, duplicates are not always redundant, outliers are not necessarily invalid, highly correlated features are not necessarily leakage, and high-cardinality features are not necessarily poorly designed. Likewise, a dataset can pass all implemented checks and still contain problems the engine cannot detect. Some problems require domain knowledge, source-system context, temporal reasoning, or an understanding of how the data is used in production. Statistical patterns can highlight suspicious behavior, but they cannot always explain its cause. Therefore, Data Autopsy is an investigation assistant, not an automated data cleaning system. It should help developers find where attention is warranted while keeping consequential decisions visible and reviewable.

The project is designed to run locally using Python and a virtual environment. A typical setup includes cloning the repository, creating a virtual environment, installing dependencies, running tests, and then launching the application with Streamlit. The optional explanation layer requires Ollama and a supported local model; the pull command for the current default model is ollama pull qwen2.5:1.5b. When the model is not installed or running, the application should still support its deterministic analysis and reporting workflows.

Next steps and the value of open source

The goal of the next phase is to make findings more useful and investigations more trustworthy. Directions include: stronger leakage analysis that investigates suspicious features using data availability, target encoding, and evaluation methods; better evidence inspection that makes the path from warning to exact rows, measurements, and thresholds easier; reproducible reports that contain enough dataset and check metadata for later review while avoiding unnecessary exposure of sensitive row-level data; useful demo datasets that demonstrate specific problems without relying on hard-coded findings or misleading screenshots; more robust tests covering multiclass targets, small samples, anomalous types, and legitimate but suspicious-looking relationships; and contributor-friendly detectors that maintain a consistent detector interface so new checks can be added without tight coupling to the UI.

Data Autopsy combines statistical checks, evidence-oriented reporting, a local AI explanation layer, and an interface for investigating tabular datasets. It is also a useful way to explore an important question in AI engineering: how can language models be used to make tools easier to understand without making them an unverified source of truth? The project aims to keep the deterministic parts inspectable, the limitations documented, and the tool useful without paid services. It is still evolving, and the goal is not to promise that every dataset problem can be discovered automatically, but to make investigating these problems more systematic, transparent, and accessible. Sometimes, before we can train a better model, we need to understand the data at hand.

One-line Summary

Data Autopsy uses deterministic checks plus local AI explanations to systematically investigate tabular dataset risks.

Paragraph Summary

Data Autopsy is an open-source, AI-assisted forensic investigation tool for tabular datasets, focused on CSV, with the goal of systematically investigating data hazards before training models. It clearly separates detection from explanation: Python statistical and rule-based checks produce structured findings with measurements, thresholds, and evidence, while a local language model (Qwen2.5 1.5B on Ollama) serves only as an optional explanation layer and does not decide detection results. The forensic engine covers ten categories of checks, including missing values, duplicate records, constant columns, identifier columns, outliers, high-cardinality categoricals, inconsistent category representations, class imbalance, highly correlated features, and target leakage. The tool emphasizes that evidence matters more than severity and is honest about its boundaries: statistical patterns can only highlight suspicious behavior, not prove that data is wrong, nor replace domain knowledge and human review.