Claude Code Powers Text Mining: How Humanities Researchers Can Master Web Scraping and Data Analysis

Text data mining has long been one of the higher-barrier areas in quantitative research — especially for researchers from humanities backgrounds, where Python's complex syntax and web scraping techniques can feel daunting. As AI Agent tools mature, that landscape is changing. In an academic livestream on Bilibili, Professor Chen from Zhongnan University of Economics and Law gave a systematic walkthrough of how to use Claude Code (CC) to complete an entire text mining research workflow — from data collection and cleaning to feature extraction and in-depth analysis.

From Conversational AI to AI Agents: Two Leaps Toward Equal Access in Research

Professor Chen frames the evolution of AI-assisted research in two stages. In the early phase, researchers using conversational AI tools like DeepSeek^Ad, Kimi, or Wenxin Yiyan had to copy AI-generated code into Jupyter Notebook or PyCharm to run it, then paste error messages back into the chat for fixes. While this workflow broke the barrier of having to write code by hand, it still had real pain points.

"Code is like math — if you don't get it, you just don't get it" — that's the honest reality for many humanities researchers. Professor Chen argues that code is fundamentally just another language (machine language), and it can absolutely be generated with AI assistance. In previous courses, he had students skip installing Python entirely and use conversational AI to generate hundreds of lines of scraping code — completing in seconds what a human programmer might take half an hour to write.

That said, this workflow often hits a wall at the scraping stage. Web scraping is highly site-specific — every website is built differently and has its own anti-scraping mechanisms, meaning you essentially need a custom script for each target. Professor Chen acknowledges that if conversational AI can't produce a working solution after three or four iterations, it tends to get stuck in a loop. At that point, you need to manually search developer communities for direction, or the AI will just keep spinning its wheels.

Claude Code's Core Breakthrough: Autonomous Trial-and-Error and Automation

What truly transforms efficiency is an AI Agent like Claude Code. Professor Chen illustrates the difference with a personal example: he asked CC to scrape a website with anti-scraping protections. CC automatically tried eight different methods, succeeding on the eighth — with no human intervention required. When the first approach failed, it automatically switched to the second, then the third, without the researcher needing to copy and paste a single error message.

"With the old conversational AI, a lot of students would give up by the third or fourth failed attempt," Professor Chen emphasized. CC's autonomy means it actively explores solutions until the task is done. In his demo, green indicators meant a step succeeded, white indicated a progress report, and red indicated a script error. The key insight: even when red errors appear, there's no need to panic — CC will resolve them on its own.

The skill requirements have also dropped significantly. Learning Python used to mean hundreds of hours of coursework — what people call going from "beginner to dropout." With CC, researchers only need to understand the logic of code rather than write it themselves — and if you're not concerned about managing risk, you barely need to read the code at all. CC delivers results directly: data files, charts, and a summary report.

Cost is worth noting too. Professor Chen runs CC with the domestic model DeepSeek rather than Anthropic's official API (to avoid account ban risks). Scraping over 800 Douban film reviews cost approximately 1 RMB — compared to roughly 0.1 RMB per entry if you outsource the job on Taobao, this approach wins on both efficiency and cost.

The Complete Text Mining Workflow: From Collection to Three Core Analysis Methods

Professor Chen used Douban reviews of the controversial film Niulai as a demo case to walk through the complete text mining process:

Data Collection: Don't limit yourself to scraping. Alongside automated web collection, exporting from databases or reading packaged PDF files are all legitimate ways to obtain text. CC can accept a URL and generate a scraper, or read local files for analysis.

Data Preprocessing (Cleaning): This is the step that determines the accuracy of everything that follows. Tasks include handling garbled text, standardizing between traditional and simplified Chinese, cleaning mixed Chinese-English content, and removing punctuation and irrelevant numbers. Chinese text also requires tokenization using the jieba library, since Chinese lacks the natural word boundaries that spaces provide in English. Stopword configuration is also essential — for Niulai reviews, words like "movie" and "Niulai" appear constantly but carry no analytical value and should be filtered out, letting meaningful terms like "director," "story," and "mother" come through.

Feature Extraction: Converting text into multi-dimensional word vectors so computers can process it. This includes word frequency statistics and word cloud generation — analogous to descriptive statistics in quantitative research, offering an initial look at the shape of the data.

For core analysis, Professor Chen recommends that beginners master three methods, which he compares to building blocks that can be freely combined:

• Cluster Analysis: Identifies what camps or categories the text falls into. For example, when analyzing elderly care policies, hundreds of policy documents can be grouped to reveal the dimensions the government is addressing.
• Topic Analysis (LDA Topic Model): Identifies the core themes expressed in a body of text. LDA (Latent Dirichlet Allocation) is the mainstream model for uncovering deeper meaning. In his demo, Professor Chen extracted roughly eight topics from Niulai reviews, covering plot, characters, family relationships, and production quality critiques.
• Sentiment Analysis: Determines the emotional orientation of text. Early methods could only distinguish positive, negative, and neutral (e.g., a score near 1 indicates positivity, near 0 indicates negativity). Modern approaches can further break down emotional types — measuring proportions of joy, anger, sadness, and more.

Professor Chen cautions that relying on a single method makes for a thin paper. Combining two or three "blocks," or pairing them with methods like QCA or fsQCA, makes research far more competitive. He shared one paper that used qualitative thematic analysis to extract a theoretical model from a small interview sample, then used LDA to validate the model against a large web-based sample.

---

Jieba is currently the most widely used open-source tokenization library in Chinese NLP, supporting three segmentation modes: precise, full, and search engine. The challenge of Chinese tokenization lies in word boundary ambiguity — "研究生命科学" could be segmented as "研究生／命科学" (graduate students / life science) or "研究／生命科学" (research / life science), with context determining the correct reading. A stopword list is a predefined collection of non-meaningful words — including function words like "的," "了," and "是," plus domain-specific high-frequency irrelevant terms — that are filtered out so meaningful content words receive appropriate weight in frequency statistics and vectorization. Word vectorization methods like TF-IDF or bag-of-words convert cleaned text into numerical matrices, serving as the shared foundation for all downstream machine learning analysis — clustering, topic models, and sentiment classifiers all take this matrix as input.

---

Conceptual Clarification: LDA Topic Modeling Is Not Qualitative Thematic Analysis

One important methodological distinction raised during the livestream: LDA topic analysis and qualitative thematic analysis are not the same thing.

The host outlined three pathways for text analysis: the interpretivist path (focused on meaning, motivation, and context behind text, as in interpretive phenomenology or grounded theory); the linguistic/structuralist path (focused on how text is constructed, as in narrative analysis); and the positivist path (focused on the structure of the text itself, including word frequency, content analysis, clustering, and topic modeling). Professor Chen's course falls squarely within the quantitative paradigm of the third pathway.

Strictly speaking, LDA should be called a "topic model" or topic modeling — its outputs are topics. Qualitative thematic analysis produces themes through iterative coding. These are fundamentally different things. LDA is a form of "unsupervised learning" in machine learning — there are no preset criteria, and neither the researcher nor the computer knows in advance how the results will cluster. This means no theoretical hypotheses need to be, or can be, specified beforehand.

---

LDA (Latent Dirichlet Allocation) was proposed by Blei et al. in 2003. Its core assumption is that each document is composed of multiple topics in varying proportions, and each topic is characterized by a probability distribution over words. For example, a film review might be 70% "plot" topic and 30% "acting" topic, and the model simultaneously outputs the most representative high-frequency words for each topic. "Unsupervised" means researchers don't need to label texts or define categories in advance — the model infers structure entirely from word co-occurrence patterns, which is the fundamental difference from supervised sentiment classifiers. In practice, researchers must manually specify the number of topics (the K value) and then assign interpretable semantic labels to each topic after results are generated. This labeling step still requires human judgment and serves as the bridge between the model's objectivity and the researcher's interpretation.

---

Topic Selection Over Tools: Should Controversial Issues Be Analyzed for Consensus?

The most valuable discussion in the livestream wasn't about technology at all — it was a debate around whether researchers should pursue "consensus analysis" of controversial online topics.

The host argued strongly against rushing into such topics and drew careful distinctions: questions of ultimate values (like abortion or capital punishment), zero-sum interest conflicts (like siting a waste incinerator), and minority rights issues involving power dynamics — none of these are situations where seeking consensus is appropriate. Drawing on Rawls's concept of "overlapping consensus" from A Theory of Justice, he argued that genuine consensus is most needed when disagreement stems from information asymmetry and incomplete understanding — not from clashes over terminal values.

This discussion points to a core insight: text analysis, quantitative methods, and qualitative methods are all just tools — like pots and pans in a kitchen. What matters is whether you can cook a dish that moves your readers. Professor Chen agreed: now that AI has lowered the technical barriers, the center of gravity for academic ability should shift from "can you write code" to "can you ask valuable questions."

Practical Tips for Using CC

Based on the livestream, several operational notes are worth recording:

• CC will automatically locate and install Python, but it deletes any temporarily installed programs after the task is complete. It's advisable to install Python manually in advance.
• CC sometimes fails to detect an existing Python installation on the local machine and falls back to Java or other options — but Java is far less suited for text analysis than Python. In such cases, explicitly prompt CC to use the local Python installation.
• Sentiment analysis has multiple available models with different trade-offs. CC sometimes takes shortcuts and picks the simplest method, producing results too rough for publication. Researchers need to understand the principles well enough to guide CC toward better choices.
• LDA performs poorly on short texts (such as Bilibili bullet comments). For those use cases, consider BTM or other topic models specifically designed for short text. Douban, Bilibili comments, and Zhihu Q&A are relatively scraper-friendly platforms worth considering when evaluating data availability during topic selection.

On scraping ethics, Professor Chen draws a clear line: web scraping is fundamentally the automated bulk collection of publicly visible information that any ordinary user can see (such as public reviews) — that's a legitimate application. Brute-forcing passwords, accessing VIP content or private user data, or scraping financial data crosses ethical and legal boundaries and must never be attempted.

---

BTM (Biterm Topic Model) is designed specifically for short texts, addressing the root cause of LDA's poor performance in that context: LDA relies on within-document word co-occurrence statistics, but a single bullet comment or microblog post typically contains only a dozen words, making document-level co-occurrence signals extremely sparse and topic estimates unstable. BTM instead counts word pair (biterm) co-occurrences across the entire corpus, bypassing the short-document problem and typically producing more coherent topics for bullet comments, microblogs, and short reviews. When choosing a topic model, average text length is a key criterion: Douban long-form reviews (usually over 100 characters) are well-suited for LDA, while Bilibili bullet comments (usually under 20 characters) are better served by BTM or methods that incorporate pre-trained language models.