Data Autopsy

Data Autopsy is an open-source, AI-assisted forensic investigation tool for tabular datasets that focuses on CSV datasets; its workflow includes uploading a CSV, analyzing structure and content, running deterministic forensic checks, reviewing results, exporting reports, and optionally using a local language model to explain findings.

Data Autopsy's core architectural decision is to separate detection from explanation: statistical and rule-based checks written in Python are responsible for identifying potential issues and returning structured findings, while the language model serves only as an optional explanation layer and does not decide whether a detector finds an issue.

Data Autopsy's forensic engine currently implements coverage of ten categories of investigation, including missing values, duplicate records, constant columns, identifier columns, numeric outliers, high-cardinality categorical features, inconsistent categorical values, class imbalance, feature correlation, and target leakage.

Data Autopsy uses median absolute deviation (MAD) and a modified Z-score for outlier detection, where the modified Z-score is defined as 0.6745 × (x − median) / MAD, typically with 3.5 as the threshold, corresponding to a tail probability of about three in ten thousand under a normal distribution.

Data Autopsy uses two statistical methods to detect numeric outliers: the interquartile range (IQR) method (using a configurable multiplier of 1.5) and the MAD-based modified Z-score (threshold 3.5).

Data Autopsy classifies missing-value findings using configurable thresholds, with fully or heavily missing columns given higher severity, and reports can include the affected column, the missing count, the missing percentage, and the threshold that triggered the finding.

Data Autopsy flags constant columns as well as columns where a single value accounts for at least 99% of observations.

Data Autopsy's class imbalance check flags severe imbalance using a configurable ratio, and the current rules include two thresholds of different severity, 4:1 and 20:1.

Data Autopsy uses a Pearson correlation coefficient threshold to identify highly correlated feature pairs, and restricts pairwise analysis to a subset of high-variance numeric columns while also limiting the number of rows considered, to strike a compromise between usefulness and computational cost.

Data Autopsy's target leakage check looks for several suspicious signals, including features identical to the target, very strong numeric relationships, unusually strong categorical class purity, and, where applicable, high ROC-AUC scores, but these signals alone cannot prove leakage.

After trimming surrounding whitespace and normalizing case, Data Autopsy looks for conflicting string representations in categorical features (such as 'Mumbai' and 'mumbai'), but the tool only identifies the inconsistency, and the user decides how to handle it.