The Current State and Future of AI Drug Discovery: Data Bottlenecks, Clinical Challenges, and the Path Forward

Can AI Reshape Drug Development?

Drug development has long been one of the most expensive and time-consuming challenges in the biopharmaceutical industry. Industry data shows that bringing a new drug from the laboratory to market takes an average of 10 to 15 years, costs billions of dollars, and faces a failure rate exceeding 90% during clinical stages. It is against this backdrop that artificial intelligence has been placed on a pedestal—seen as a key technology that could potentially break the curse of "high investment, high failure rates, and long timelines."

Behind these numbers lie deep structural reasons. A 2020 study by the Tufts Center for the Study of Drug Development (Tufts CSDD) estimated that the average R&D cost per new drug is approximately $2.6 billion, with a large portion being "failure costs"—resources consumed by candidate drugs that are eliminated at various stages. This phenomenon is known in the industry as "Eroom's Law" (Moore's Law spelled backwards), referring to the fact that since the 1950s, the number of new drugs approved per billion dollars of R&D spending has roughly halved every 9 years, forming a stark contrast with Moore's Law in the semiconductor industry. The causes of this trend include: ever-increasing regulatory standards, the "low-hanging fruit" having already been picked, increasingly complex clinical trial designs, and continuously escalating safety requirements.

A recent article that sparked widespread discussion in the tech community systematically examined AI's positioning in drug discovery, its current stage, and future development paths. This topic reflects the biopharmaceutical industry's complex attitude toward AI implementation: full of expectations yet maintaining caution.

Core Application Scenarios of AI in Drug Discovery

Many people have misconceptions about "AI drug development," imagining it as a black box that automatically generates new drugs from start to finish. In reality, AI's value is distributed across multiple stages of the R&D pipeline, with each stage having different levels of maturity and contribution.

Target Identification and Validation

The first step in drug development is finding biological targets (typically proteins) associated with diseases. A biological target refers to a biomacromolecule closely related to disease onset and progression—most commonly proteins (such as kinases, receptors, ion channels, etc.), but also nucleic acids (such as mRNA, microRNA). AI analyzes massive genomics, proteomics, and literature data to help researchers discover previously overlooked potential targets and predict their associations with diseases. This stage dramatically shortens the time traditionally spent on manual literature screening and experimental validation.

Notably, "target validation" is the process of confirming that modulating a specific target can actually produce therapeutic effects—a step whose failure is one of the biggest risk sources in drug development. Statistics show that approximately 50% of Phase II clinical failures can be attributed to incorrect target selection—where the drug successfully acts on the target, but modulating that target does not effectively treat the disease. AI's core value in this stage lies in integrating multi-omics data (genomics, transcriptomics, proteomics, metabolomics, etc.) and revealing disease mechanisms through causal inference and network analysis, thereby reducing the error rate in target selection.

Molecular Generation and Virtual Screening

This is one of the most closely watched applications in AI drug discovery. Generative models can "design" candidate molecules that meet specific requirements within vast chemical space, while virtual screening technologies can rapidly identify promising lead compounds from libraries of billions of compounds. Compared to traditional high-throughput experimental screening, AI screening speeds have improved by several orders of magnitude.

"Chemical space" refers to the totality of all theoretically possible drug-like molecules. It is estimated that the number of potential drug-like small molecules reaches 10^60, far exceeding the total number of atoms in the universe (approximately 10^80). Traditional medicinal chemistry can only screen within known compound libraries, while generative AI models (such as Variational Autoencoders/VAE, Generative Adversarial Networks/GAN, diffusion models, and Transformer-based molecular generation models) can directionally "create" new molecules within this nearly infinite space. After learning the structure-activity relationships of existing drug molecules, these models can generate entirely new molecular structures that satisfy specific constraints (such as binding affinity to targets, solubility, metabolic stability, etc.). Virtual screening then uses computational methods like molecular docking and Free Energy Perturbation (FEP) to prioritize candidate molecules.

Protein Structure Prediction

DeepMind's AlphaFold represents a landmark breakthrough in this field. It elevated the accuracy of protein 3D structure prediction to levels approaching experimental determination, opening new possibilities for structure-based drug design. AlphaFold has now made hundreds of millions of protein structure predictions publicly available, becoming an indispensable tool for researchers worldwide.

From a technical perspective, AlphaFold's core innovation lies in introducing an attention mechanism architecture called "Evoformer," which simultaneously processes multiple sequence alignment (MSA) information of protein sequences and spatial relationships between amino acid residue pairs. In the CASP14 (Critical Assessment of protein Structure Prediction) competition in 2020, AlphaFold2's prediction accuracy reached a GDT score of approximately 92.4 (out of 100), approaching the precision of experimental methods like X-ray crystallography and cryo-electron microscopy for the first time. The AlphaFold database released in 2022 contains over 200 million protein structure predictions, covering nearly all known protein sequences. The subsequent AlphaFold3 further expanded to complex structure prediction involving proteins with DNA, RNA, small molecule ligands, and more. However, it's important to note that AlphaFold predicts static structures, while drug design often requires understanding dynamic conformational changes in proteins—a limitation of current technology.

Three Core Challenges in AI Drug Discovery

Despite rapid inflows of capital and technology, the industry remains cautious about the actual effectiveness of AI in drug development. The following are the key bottlenecks currently faced.

Data Quality: The Biggest Constraint

The scarcity, noise, and inconsistency of biomedical data are the core obstacles limiting AI model performance. Unlike natural language processing, which has access to massive amounts of high-quality text, high-quality experimental data is often expensive to obtain, limited in quantity, and difficult to reproduce across different laboratories. The "garbage in, garbage out" principle is particularly pronounced in drug discovery.

Compared to internet text data, biomedical data faces unique challenges. First is the "small data" problem: a typical drug-target interaction dataset may contain only hundreds to thousands of data points, far less than the trillion-level tokens needed to train large language models. Second is the experimental noise problem: different laboratories use different assay methods (such as fluorescence detection, mass spectrometry, etc.), and activity measurements for the same compound can vary by several fold. Additionally, there's the data silo problem: for competitive and intellectual property reasons, pharmaceutical companies typically don't publish their internal screening data. While public databases like ChEMBL aggregate large amounts of bioactivity data, data quality varies considerably. Privacy-preserving technologies such as Federated Learning are being explored to address cross-institutional data sharing challenges.

No AI-Native Drug Has Been Approved

This is an unavoidable reality: as of now, no drug fully designed by AI has successfully completed all clinical trials and received regulatory approval. While several leading AI drug companies have advanced candidates into clinical stages, they have also encountered clinical failures or slow progress. This reminds us that AI primarily optimizes efficiency in the early stages of R&D, while clinical trial outcomes still depend on complex biological realities.

From an industry progress perspective, several AI drug companies' pipelines have achieved milestone advances. Insilico Medicine's anti-fibrotic candidate ISM001-055 was the first drug to enter Phase II clinical trials with both the target discovered and the molecule designed by AI; Recursion Pharmaceuticals has advanced multiple pipeline programs using high-content phenotypic screening combined with AI analysis; and Exscientia has brought multiple candidates developed with partners into clinical stages. However, in 2023, BenevolentAI's atopic dermatitis candidate failed to meet its primary endpoint in Phase II trials, becoming one of the most notable setbacks in the AI drug discovery space. These cases demonstrate that AI can indeed accelerate early drug discovery (compressing the time from target to candidate drug from 4-5 years to 1-2 years), but once entering clinical stages, AI-discovered drugs still face the same biological uncertainties as traditionally discovered drugs.

The Gap Between Computational Models and Biological Systems

The real difficulty in drug discovery isn't computing power—it's the extreme complexity of biological systems. A molecule that performs excellently in computational models may fail in the human body due to toxicity, metabolic issues, or off-target effects. AI can accelerate the process of "generating hypotheses," but it cannot replace the wet lab experiments and clinical research necessary for "validating hypotheses."

This involves the complementary relationship between "wet lab experiments" and "dry lab/in silico" work. Wet experiments refer to physical experiments conducted in laboratories using real biological samples and chemical reagents, such as cell culture, animal model testing, and biochemical assays; dry experiments refer to computer model-based analysis and prediction. The core challenge of drug discovery lies in the fact that biological systems exhibit emergence—molecular-level predictions often cannot fully capture complex behaviors at the cellular, tissue, or whole-organism level. For example, a drug's ADMET properties (Absorption, Distribution, Metabolism, Excretion, Toxicity) in the body involve the coordinated action of multiple organ systems, and current computational models can only make partial predictions. This is why even when AI predicts highly optimized molecules, they still need to be progressively validated through in vitro experiments, animal experiments, and human clinical trials.

Pragmatic Development Paths for AI Drug Discovery

Facing the challenges described above, the industry is exploring several more pragmatic directions.

From "Replacing Researchers" to "Augmenting Researchers"

A more realistic positioning is to build AI as a powerful assistant for researchers, rather than a fully autonomous replacement. AI excels at processing massive data, discovering hidden patterns, and accelerating design iterations, while human experts handle scientific judgment, experimental validation, and strategic direction. This human-AI collaboration model is considered the most viable implementation path in the short term.

Establishing a "Prediction—Experiment—Feedback" Loop

The key to the future lies in creating a complete loop of "AI prediction—automated experimentation—data feedback—model iteration." Through laboratory automation and robotics, AI predictions can be quickly validated, and validation data can then feed back into models, creating a virtuous cycle of continuous learning. This mechanism has the potential to fundamentally alleviate the data scarcity problem.

One key technology for building this loop is lab automation and self-driving labs. This concept integrates robotics, IoT sensors, and AI decision-making systems into experimental workflows. Typical automation platforms include liquid handling robots, automated cell culture systems, and high-throughput detection platforms. In 2023, Carnegie Mellon University's "Coscientist" system and the University of Toronto's "Ada" autonomous chemistry lab demonstrated the complete loop of AI-planned experiments, robotic execution, and automated result analysis and feedback. This "closed-loop learning" model has the potential to shorten each experimental iteration cycle from the traditional weeks to days or even hours.

Building Stronger Foundation Models and Data Infrastructure

As domain-specific large models for biopharmaceuticals continue to emerge, and as the industry increasingly emphasizes data standardization and cross-institutional sharing mechanisms, there remains significant room for improving AI's capability ceiling in drug discovery. Building high-quality, reusable data infrastructure may be more critical than simply scaling up computing power.

Inspired by the success of Large Language Models (LLMs), the biopharmaceutical field is seeing the emergence of domain-specific foundation models. In the protein domain, Meta AI's ESM-2 and ESMFold leverage the "language" characteristics of protein sequences for pre-training, capturing evolutionary information and structural features; in the molecular domain, Microsoft's MoLFormer and multiple open-source projects attempt to train general molecular representation models using SMILES strings or molecular graphs as input; in the genomics domain, Google DeepMind's Enformer and Nucleotide Transformer can predict the functions of gene regulatory elements. The core advantage of these foundation models lies in the "pre-train—fine-tune" paradigm: first learning general representations on large-scale unlabeled data, then fine-tuning for specific tasks (such as drug activity prediction, toxicity prediction, etc.), thereby alleviating the problem of scarce labeled data.

A Rational View of the AI Drug Discovery Marathon

AI is tangibly changing the way drugs are discovered—particularly in early-stage R&D for target identification, molecular design, and structure prediction, where efficiency gains can no longer be ignored. But this is a marathon requiring patience, not an overnight disruption.

The real breakthrough may not come in the form of "a single AI drug appearing out of nowhere," but rather as a systemic improvement in efficiency across the entire R&D pipeline. For practitioners and investors following the AI drug discovery space, maintaining technological optimism while preserving scientific rigor is the most appropriate stance.