How to Do EDA with Real Depth: A Practical Guide from Mechanical Operations to Insight-Driven Analysis

A Common Frustration: What Should EDA Actually Look For?

Recently, in Reddit's machine learning community, a learner raised a question that many beginners encounter: "I know the basics of EDA — handling missing values, outliers, checking distributions and correlations. But every time I get a new dataset, I don't know what to look at. I don't want to just run a describe(), plot a few charts, and say 'okay, EDA is done.'"

It's worth noting that while Pandas' describe() function is convenient, it only provides summary statistics for numerical columns — count, mean, standard deviation, min, quartiles, and max — capturing central tendency and dispersion. It cannot reveal the shape of distributions (such as bimodal distributions), temporal trends in data, characteristics of categorical variables, or complex relationships between variables. Over-reliance on a single function's output can even lead to Simpson's Paradox-style misjudgments, where summary statistics mask drastically different patterns across subgroups.

This question strikes at the core pain point of Exploratory Data Analysis (EDA): Many people treat EDA as a fixed checklist of operations, rather than a skill of asking questions and thinking critically. Real EDA isn't about mechanically executing code — it's about examining data with business questions in mind and gradually building intuition about the data.

This article systematically covers how to advance from "knowing the operations" to "knowing how to think," helping you know what questions to ask when facing any unfamiliar dataset.

The Essence of EDA: From Code Execution to Question-Driven Analysis

The Academic Origins of EDA

To understand the essence of EDA, it's worth revisiting its origins. The concept of Exploratory Data Analysis was formally introduced by American statistician John Tukey in his 1977 book of the same name. Tukey argued that statistical analysis should not merely verify preset hypotheses but should let the data "speak for itself." He invented visualization tools like the box plot, which is still widely used today, and championed an open-ended, discovery-oriented philosophy of data exploration. Understanding this academic origin helps us recognize that EDA was never meant to be a mechanized process from the very beginning — it is an analytical philosophy that emphasizes curiosity and flexibility.

Why Checklist-Style EDA Gets You Stuck

A common mistake among beginners is treating EDA as a set of mandatory actions: first check missing values, then look at distributions, then plot a correlation heatmap. There's nothing wrong with this workflow per se, but it's "tool-centered" rather than "question-centered." When you only focus on "which function should I run," it's easy to finish all the standard operations and still have no idea what the data is telling you.

Truly experienced analysts, when opening a dataset, aren't thinking "what charts should I plot" — they're thinking "what is this data about? What am I ultimately trying to predict or understand? What hypotheses do I need to validate to achieve that goal?"

Building a Goal-Oriented Analytical Framework

Before getting your hands dirty, ask yourself three levels of questions:

• Business level: What problem is this dataset meant to solve? What's the target variable? What does success look like?
• Data level: What does each column represent? How was the data collected? What biases might exist?
• Modeling level: Which features might be relevant to the target? Does the data quality support the modeling I want to do?

Only with clear objectives does every chart and statistic become meaningful. For example, when examining distributions, if you know the goal is a classification task, you'll pay special attention to differences in feature distributions across classes, rather than looking at individual variables in isolation.

A Reusable EDA Thinking Framework

Step 1: Understand the Data's "Origin Story"

When you get the data, don't rush to plot charts. Spend time reading the data dictionary and field descriptions, and understand the business meaning of each column. Figure out the data source, collection method, and time range. Many data traps are hidden in the data generation process, not in the data itself.

The two most common traps are survivorship bias and data leakage. Survivorship Bias means the dataset only includes samples that "survived" or "succeeded" while omitting failed or churned samples, leading to systematic bias in analytical conclusions. For example, when analyzing "common traits of successful companies," if you only collect data from companies still in operation, you can't learn about companies with the same traits that failed, thus overestimating the impact of certain features. Data Leakage refers to training data inadvertently containing information that wouldn't be available at actual prediction time — such as using features containing future information to predict current events, or having some encoded form of the target variable exploited by the model as a feature. Neither of these problems will typically be caught in standard data quality checks; only a deep understanding of the data generation mechanism can identify them.

Step 2: Progress Layer by Layer from Univariate to Multivariate

Univariate analysis: Examine each variable individually for distribution, value range, and missing data. For numerical variables, check skewness and outliers; for categorical variables, check frequencies and rare categories. The purpose of this step is to discover data quality issues and the basic characteristics of each variable.

Regarding skewness and outliers, it's worth understanding them in depth. Skewness measures the degree of asymmetry in a data distribution: zero skewness indicates perfect symmetry (like a normal distribution), positive skewness indicates a long right tail (like income distributions), and negative skewness indicates a long left tail. Highly skewed variables typically need log transformation or Box-Cox transformation to be effectively utilized by linear models. Common outlier detection methods include: the IQR-based method — flagging points below Q1-1.5×IQR or above Q3+1.5×IQR as outliers; the Z-score method — flagging data points more than 2-3 standard deviations from the mean; and more advanced algorithms like Isolation Forest and DBSCAN. It's worth noting that outliers aren't necessarily errors — they might be the most valuable business insights.

Bivariate analysis: This is the most commonly overlooked yet most valuable step. Focus on the relationship between each feature and the target variable — do the distributions of numerical features differ across target classes? Do different values of categorical features correspond to different target means? This step directly impacts subsequent feature engineering.

Multivariate analysis: Examine interactions and collinearity between features. A correlation heatmap is just the starting point — you also need to think about "why are these two variables correlated" and "are there confounding factors."

Regarding collinearity, this is an issue with profound impact on model quality. Collinearity or multicollinearity refers to a high degree of linear correlation between two or more independent variables. When collinearity is severe, coefficient estimates in models like linear regression become extremely unstable — small data changes can cause coefficient signs to flip, making feature importance interpretation unreliable. Common methods for detecting collinearity include: calculating the Variance Inflation Factor (VIF — generally, VIF exceeding 5-10 indicates serious collinearity), checking for feature pairs with absolute correlation coefficients above 0.8 in the correlation matrix, and observing the condition number of the feature matrix. Solutions include removing redundant features, using Principal Component Analysis (PCA) for dimensionality reduction, or applying regularization methods (such as Ridge Regression or Lasso Regression) to mitigate the effects of collinearity.

As for confounding factors, this touches on a core issue of causal inference. A confounding variable is a third-party variable that simultaneously affects both the independent and dependent variables, causing us to incorrectly conclude a causal relationship between two variables. The classic example: ice cream sales and drowning incidents are positively correlated, but the real driver is "temperature" as a confounding variable. Identifying potential confounders during EDA is crucial because it directly affects the correctness of feature engineering and model interpretation. Simpson's Paradox is a classic manifestation of confounding — trends in the overall data may completely reverse when the data is grouped by a certain variable. In recent years, Judea Pearl's causal inference framework and DAGs (Directed Acyclic Graphs) have provided a systematic methodology for identifying and handling confounders.

Step 3: Validate with Hypothesis-Driven Exploration

Excellent EDA is hypothesis-driven. When you observe a phenomenon, actively formulate hypotheses and validate them. For example: "I hypothesize that long-term users have higher retention rates" — then group by user registration date and compare retention rates. This cycle of "formulate hypothesis → validate → generate new questions" is what truly makes you stronger.

How to Deliberately Practice EDA Skills

Build Intuition Through Real Projects

Returning to the Reddit user's core question: how do you practice? The answer is keep doing projects, but do them with reflection. Don't just complete them for the sake of completion. After finishing each dataset, ask yourself: What non-obvious insights did I discover? Did I miss any important relationships? If I could start over, how would I adjust my analysis sequence?

Learn from Outstanding Kaggle EDA Notebooks

Kaggle is the best learning resource. Kaggle is the world's largest data science competition and community platform, acquired by Google in 2017, with over 15 million registered users and tens of thousands of public datasets. Its core value lies not only in the competitions themselves but also in the community-contributed Notebooks (formerly called Kernels) — these public code notebooks document complete analytical thinking processes, from EDA to feature engineering to model tuning. Kaggle's ranking system spans four dimensions: Competitions, Datasets, Notebooks, and Discussion, with the highest title being Grandmaster.

Choose popular competitions and carefully read highly upvoted EDA notebooks. You'll discover how top analysts ask questions and dissect data step by step. Pay attention to their thinking process, not just their code. Observe what they do after seeing a particular chart and why they do it. It's especially recommended to focus on EDA notebooks that appear in the early stages of competitions (within one to two weeks of data release) — these notebooks typically spend extensive space interpreting data features and uncovering hidden patterns, which is extremely helpful for developing analytical thinking.

Train Your Instincts with Diverse Datasets

Deliberately expose yourself to data from different domains and structures: time series, text, image metadata, and structured tables. Each data type has its unique EDA focal points.

Specifically, time series EDA requires special attention to trend, seasonality, and autocorrelation, with common tools including ACF/PACF plots and STL decomposition. Text data EDA focuses on word frequency statistics, TF-IDF distributions, text length distributions, and preliminary topic modeling exploration — while word clouds are common, they have low information density, so it's better to combine them with n-gram analysis and co-occurrence matrices. Structured tabular data EDA is the most classic, with emphasis on data quality assessment, relationship mining between variables, and discovering feature engineering clues. Image metadata EDA needs to address image size distribution, brightness histograms, color channel statistics, and visual differences between classes. Each data type has its unique "traps" and "golden signals" — broad exposure is what builds comprehensive analytical intuition.

Fill in Your Domain Knowledge

Getting stuck during EDA is often not a technical problem but a domain knowledge gap. If you don't understand the business logic of finance, healthcare, or e-commerce, it's hard to judge which features are important or which outliers are reasonable. Spending time understanding the basic concepts of the domain your data belongs to will instantly add depth to your analysis.

Conclusion: EDA Is a Data Intuition That Can Be Cultivated

The essence of EDA is not a set of operations but the ability to ask questions and an intuition for data. Shifting from "what function should I run" to "what is this data telling me, and what else do I want to know" is the key leap from beginner to proficient analyst.

This ability cannot be fast-tracked through a single course. It comes from: setting clear objectives, asking systematic questions, extensive practice, deep reflection, and continuously building domain knowledge. When you stop worrying about "is my EDA done yet" and start immersing yourself in "what stories in this data haven't been discovered yet," you've truly begun to master it.