How to Write the Research Design for a Machine Learning Paper? A Detailed Guide Using Player Churn Prediction as an Example

Problem Background: Common Methodological Confusion in ML Papers

In academic research related to machine learning, a seemingly basic yet often overlooked question is: How do you accurately define the overall "research design" of your study? Traditional social science research methodology frameworks (experimental research, observational studies, case studies, etc.) don't map perfectly onto ML research centered on model comparison, leaving many thesis-writing students confused.

This confusion has deep-rooted causes. The social science methodology system matured over decades of development in the twentieth century—from Fisher's foundational randomized experimental design, to Campbell and Stanley's quasi-experimental framework, to Creswell's mixed methods paradigm—each research design has clear applicability conditions and identification criteria. However, machine learning, as a field that has grown rapidly between engineering practice and academic research, has not yet formed a similar level of methodological consensus. "Experiments" in ML papers typically refer to computational experiments, which are fundamentally different from social science experiments involving human subjects. This terminological ambiguity further exacerbates the classification difficulty.

Recently, a student writing a bachelor's thesis in machine learning sought help on Reddit, with a highly representative research topic: In a mobile gaming context, comparing three machine learning models across two dimensions—"predictive performance" and "interpretability"—and exploring whether a trade-off exists between them and under what application scenarios the trade-off is more pronounced.

The mobile gaming context provides this research with a highly practical grounding. Data science applications in the mobile gaming industry are already quite mature—from user retention prediction and payment propensity modeling to real-time recommendation systems, ML models permeate every aspect of operations. However, with the implementation of data privacy regulations like GDPR and increasing user demands for algorithmic decision transparency, pursuing prediction accuracy alone is no longer sufficient—operations teams need to understand "why" a model makes a certain prediction to make responsible business decisions. This is precisely the industrial origin of the "performance vs. interpretability" trade-off question.

This question touches on a gray area in ML research methodology that deserves in-depth analysis.

Methodological Breakdown: A Typical Mixed-Methods Sample

Based on the original post description, this study's methodological design is actually quite complete. We can break it down into several layers.

Data Processing Layer: Accurate Positioning of CRISP-DM

The author clearly stated that the data preparation phase followed the CRISP-DM (Cross-Industry Standard Process for Data Mining) model, covering data understanding, cleaning, feature engineering, and other steps. A critical clarification is that CRISP-DM was only used as a process guide for the data preparation phase, not as the research methodology for the entire paper.

CRISP-DM (Cross-Industry Standard Process for Data Mining) was born in 1996 from an EU-funded project, jointly developed by companies including SPSS, Teradata, Daimler-Chrysler, and OHRA. It divides data mining projects into six phases: Business Understanding, Data Understanding, Data Preparation, Modeling, Evaluation, and Deployment, with iterative backtracking allowed between phases. Despite being nearly thirty years old, CRISP-DM remains the most widely adopted data science process framework in industry—multiple industry surveys show its adoption rate has consistently exceeded competing frameworks like SEMMA and KDD Process.

This is a very correct understanding. CRISP-DM is essentially an engineering process model—it describes the execution steps of a data mining project, not an academic research design. Mistaking it for a research methodology is a common error among beginners. CRISP-DM answers "what to do and in what order," rather than the methodological questions of "why do it this way and how to demonstrate validity" in academic research—this is the essential distinction between the two.

Performance Evaluation Layer: Quantitative Research Methods

For model evaluation, the author employed standard quantitative approaches:

Training three models under identical conditions (differing only in hyperparameters) to ensure comparability;
Using multiple metrics including Recall, F1-score, ROC-AUC, and PR-AUC to evaluate performance;
Applying the Wilcoxon signed-rank test for statistical significance testing.

The combination of these four metrics itself reflects the researcher's deep thinking about evaluation dimensions. Recall measures the model's ability to capture positive samples, which is particularly critical in scenarios like player churn prediction where "the cost of missed detections is high." F1-score is the harmonic mean of precision and recall, providing a single balanced metric. ROC-AUC (Area Under the Receiver Operating Characteristic Curve) measures the model's overall discriminative ability across all classification thresholds, but can be overly optimistic under severe class imbalance—which is extremely common in mobile gaming user behavior data, since paying users or churning users are typically the minority. For this reason, PR-AUC (Area Under the Precision-Recall Curve) is introduced as a complementary metric, as it is more sensitive to minority class performance. This multi-metric strategy avoids evaluation bias that a single metric might introduce.

The choice of the Wilcoxon signed-rank test is also noteworthy. This is a non-parametric statistical method proposed by Frank Wilcoxon in 1945, used to compare differences between paired samples without requiring the data to follow a normal distribution. In 2006, Janez Demšar published the highly influential paper Statistical Comparisons of Classifiers over Multiple Data Sets, systematically demonstrating why traditional paired t-tests have serious problems when comparing classifiers (such as violating independence assumptions and sensitivity to outliers), and recommending the Wilcoxon signed-rank test (for comparing two classifiers) or the Friedman test with Nemenyi post-hoc test (for comparing multiple classifiers). This paper has been cited over ten thousand times and established the methodological standard for statistical comparisons in the ML field.

This layer falls entirely within the quantitative research paradigm, and the introduction of statistical testing reflects research rigor—something not commonly seen in undergraduate theses. Choosing the Wilcoxon test rather than simply reporting mean differences indicates that the researcher recognizes that performance differences across folds (cross-validation folds) or across experiments may be affected by randomness, requiring hypothesis testing to determine whether differences are statistically meaningful.

Interpretability Evaluation Layer: Qualitative Research Methods

The interpretability evaluation took two paths:

Intrinsic interpretability: Judgment based on transparency criteria derived from the literature;
Post-hoc interpretability: Analysis using SHAP.

The distinction between these two types of interpretability originates from a foundational theoretical framework in the Explainable AI (XAI) field. In 2017, Finale Doshi-Velez and Been Kim systematically organized three levels of interpretability evaluation in their pioneering paper Towards A Rigorous Science of Interpretable Machine Learning: application-level evaluation (judged by domain experts in real tasks), human-level evaluation (judged by laypeople in simplified tasks), and function-level evaluation (no human participation required, using formalized proxy metrics). Intrinsic interpretability refers to the transparency of the model structure itself—for example, the decision paths of shallow decision trees can be directly read and understood by humans, and linear regression coefficients can be directly interpreted as marginal effects of features. Post-hoc interpretability applies external explanation techniques (such as SHAP, LIME, etc.) to opaque models (such as deep neural networks, ensemble methods). The former is an inherent property of the model, while the latter is an approximate explanation provided by external analysis tools—this distinction has direct implications for research design.

SHAP (SHapley Additive exPlanations) was proposed by Scott Lundberg and Su-In Lee in 2017, with its theoretical foundation rooted in the Shapley value concept from game theory—proposed by Nobel laureate Lloyd Shapley in 1953 for fairly allocating contributions among participants in cooperative games. SHAP treats each feature as a "player" and quantifies that feature's impact on individual predictions by calculating its marginal contribution across all possible feature combinations. SHAP's core advantage is that it is the only attribution method that simultaneously satisfies three theoretical properties: Local Accuracy, Missingness, and Consistency.

Interestingly, the author did not compute additional quantitative metrics for SHAP, but instead qualitatively analyzed global SHAP summary plots (e.g., consistency of feature effects, distribution range of SHAP values) and discussed their implications. Summary plots display global feature importance and directional influence—the more concentrated the SHAP value distribution and the more consistent the direction, the easier it is for human experts to form intuitive understanding of the model's decision logic. This makes the interpretability evaluation predominantly qualitative—the researcher is essentially making expert judgments about visual patterns rather than computing a numerical interpretability score.

Core Classification: What Research Design Is This Exactly?

Returning to the author's primary concern: Should this research be classified as an empirical evaluation study, a comparative study, an experimental study, or something else?

The Most Fitting Classification: A Mixed-Methods Comparative Empirical Study

Considering the methodological characteristics, the most accurate classification for this research is a "mixed-methods comparative empirical study" with a distinct quasi-experimental flavor. Here's why:

Why is it "experimental"? The author trains models under controlled conditions (differing only in hyperparameters), which constitutes a controlled manipulation of variables—the "treatments" being compared are different model algorithms. While this isn't a strict experiment with random assignment in the social science sense, in the ML context, controlling training conditions and standardizing evaluation protocols already possesses the core characteristics of experimental research.

The quasi-experiment concept was proposed by Donald Campbell and Julian Stanley in their 1963 classic work, referring to experimental designs where the researcher can manipulate independent variables but cannot fully control all confounding variables or cannot perform random assignment. In the context of ML model comparison, researchers can control train-test data splits, feature sets, and evaluation metrics, but cannot control the data generation process itself or algorithms' internal randomization (such as neural network weight initialization, random forest feature subsampling). For this reason, classifying ML model comparison as quasi-experimental rather than true experimental is the more accurate positioning.

Why is it "comparative"? The core objective of the research is to horizontally compare three models—this naturally falls under comparative study.

Why is it "empirical"? The research is based on real datasets, draws conclusions through actual execution and measurement rather than pure theoretical derivation, and therefore constitutes empirical research.

Why is it "mixed-methods"? The performance dimension uses quantitative metrics plus statistical testing, the interpretability dimension uses qualitative analysis (SHAP plot interpretation plus transparency criteria), and the final trade-off analysis integrates results from both sides through qualitative discussion. This is a typical mixed-methods design.

Mixed Methods Research, as an independent methodological paradigm, gained widespread academic recognition from the late 20th to early 21st century, with John Creswell and Vicki Plano Clark's series of works establishing its theoretical framework. Its core premise is that quantitative methods excel at answering "how much, to what extent" questions, qualitative methods excel at answering "why, how" questions, and combining both provides a more complete research picture than either method alone. In traditional classification, mixed methods have several integration modes: convergent design (quantitative and qualitative conducted simultaneously then integrated), explanatory sequential design (quantitative first, then qualitative explanation), and exploratory sequential design (qualitative first, then quantitative validation). This research case more closely resembles a convergent design—quantitative evaluation of performance and qualitative analysis of interpretability proceed in parallel, ultimately integrated in the trade-off discussion. It's worth noting that explicit application of mixed methods in computer science and AI remains relatively uncommon; most ML papers default to a purely quantitative paradigm. This undergraduate student's conscious integration of two methodological traditions reflects the penetration of interdisciplinary methodological awareness.

Important Reminder: Distinguishing the Hierarchical Relationship Between "Methods" and "Design"

A misconception the author may fall into is trying to select the single correct answer from labels like "experimental/comparative/evaluation." In fact, these concepts are not mutually exclusive—a study can simultaneously be comparative, experimental, and empirical. Research design description should be layered and combinatorial: the top level is "mixed-methods comparative empirical study," beneath which are two sub-paths: "controlled model training experiment" and "SHAP-based qualitative interpretability analysis."

This layered thinking is well-supported in methodology literature. Saunders et al.'s "research onion" model in Research Methods for Business Students decomposes research design into multiple concentric layers: philosophical stance, methodological orientation, strategy choice, time horizon, and technical methods. Choices at each layer do not constrain options at other layers but provide different dimensional positioning for the overall design. For ML research, the outermost philosophical stance is typically pragmatism, the strategy layer is experimental or quasi-experimental, and the methodological orientation can be quantitative, qualitative, or mixed—these dimensions are orthogonal and should not be compressed into a single label.

Recommended Methodology Literature for ML Research Design

The author also hoped to find methodology literature specifically for ML research design (rather than traditional social science). Here are several directions worth exploring:

Empirical Software Engineering and ML Evaluation: Look into methodological discussions on "empirical evaluation in machine learning"—this literature specifically addresses how to design reproducible, comparable ML experiments. The methodological accumulation in the Empirical Software Engineering field is particularly noteworthy—over the past two decades, this field has systematically addressed the question of "how to apply rigorous empirical research methods in computational experiments," and its experience is highly transferable to ML research.
Explainable AI (XAI) Evaluation Frameworks: Doshi-Velez and Kim's classic exposition on interpretability evaluation classification provides theoretical support for distinguishing intrinsic from post-hoc interpretability. Additionally, Christoph Molnar's open-source book Interpretable Machine Learning provides a systematic review of various interpretability methods and has practical value for constructing evaluation frameworks.
Benchmarking and Statistical Testing: Demšar's methodology for statistical testing when "comparing classifiers over multiple datasets" provides the theoretical basis for applying the Wilcoxon test in ML. This paper not only provides specific testing method recommendations but more importantly argues why the ML field needs to shift from "looking at mean magnitudes" to "performing statistical inference"—a paradigm shift crucial for improving the scientific rigor of ML research.

Conclusion: The Long-Term Value of Developing ML Research Methodology Awareness

The value of this case lies not only in answering a classification question, but in revealing a widespread phenomenon: An increasing number of ML researchers are beginning to value rigorous methodological articulation.

As AI research matures, merely "running models and reporting metrics" is no longer sufficient. Clearly defining research design, articulating the division of labor between quantitative and qualitative methods, and accurately positioning tools (like CRISP-DM) within the research—this methodological awareness is the key differentiator between "engineering practice" and "academic research." In recent years, reproducibility checklists introduced at top ML conferences (such as NeurIPS, ICML) and requirements for detailed reporting of experimental design in papers are concrete manifestations of this methodological standardization trend. The ML community is gradually building its own system of methodological norms—one that absorbs the rigor requirements of traditional empirical research while adapting to the unique characteristics of computational experiments.

For ML learners at the undergraduate and even graduate level, cultivating this awareness may be no less important than mastering any specific algorithm. A researcher who can clearly answer "what type is my research, why did I choose these methods, and under what conditions do my conclusions hold" will produce academic output with significantly higher credibility and impact than peers who can only demonstrate technical implementation.