SELENE Open-Source Project: An Interactive Notebook Library for Systematically Learning AI and Machine Learning from Scratch

The Growth Story of an AI Learning Resource

In an era of rapidly evolving AI technology, how to systematically master the core principles of machine learning, deep learning, and large language models from scratch remains a major challenge for beginners. There are plenty of tutorials available, but they tend to be either highly fragmented or jump straight to the practical level of calling APIs, lacking in-depth analysis of underlying mechanisms. Recently, an open-source learning project called SELENE gained attention on the Reddit community — it just surpassed 200 stars on GitHub, making it a public learning resource worth serious attention.

SELENE was originally created by a lecturer at the National University of Singapore (NUS), starting as interactive lecture notes for courses. As the content was continuously refined and expanded, it gradually evolved into an open knowledge base for all AI learners. The project author specifically thanked the community for their support in the post, revealing that GitHub stars are currently the primary channel for receiving feedback. This kind of open-source resource — driven by teaching practice and continuously iterated — often gets closer to the essence of knowledge than commercialized courses.

What AI Learning Content Does SELENE Cover

SELENE is a public repository built on Jupyter Notebook, covering topics including artificial intelligence, machine learning/deep learning, natural language processing (NLP), data mining, and data science. Its greatest feature is interactivity — every knowledge point is presented as a runnable Notebook, allowing learners not only to understand the principles but also to run code, modify parameters, and observe results firsthand.

Jupyter Notebook is an open-source interactive computing environment that originally emerged from the IPython project and later developed independently into a general-purpose tool supporting Python, R, Julia, and other languages. Its core design philosophy integrates code, text explanations, mathematical formulas (rendered via LaTeX), and visual outputs into a single document, forming what's called a "computational narrative." This format is particularly suited for teaching scenarios — instructors can intersperse theoretical derivations with code implementations, while learners can execute code cell by cell and observe changes in intermediate variables in real time. Google Colab is essentially a cloud-hosted Jupyter environment, eliminating the tedious steps of configuring a local Python environment and installing CUDA drivers.

The project's current positioning is very clear: focused on fundamentals, aimed at beginners. However, "beginners" here comes with a prerequisite — basic mathematical foundations including linear algebra, calculus, and probability theory. This positioning distinguishes SELENE from quick-start tutorials that only teach "how to use" things, placing greater emphasis on understanding underlying mathematics and algorithmic mechanisms.

Traditional Machine Learning Models

In the traditional models section, SELENE systematically covers the classical algorithm spectrum of machine learning:

• Regression models: Linear Regression, Logistic Regression
• Probabilistic models: Multinomial Naive Bayes
• Tree models: Decision Trees/CART, Random Forest
• Ensemble methods: AdaBoost, Gradient Boosting Machine (GBM), and the widely used industrial tools XGBoost, LightGBM, CatBoost

This section is crucial for building a solid machine learning foundation, especially ensemble learning methods, which remain the primary tools for structured data competitions and industrial applications. The core idea of Ensemble Learning is to build a strong learner by combining multiple weak learners. In data science competitions like Kaggle, models from the GBDT family have long dominated winning solutions. XGBoost was proposed by Tianqi Chen at the University of Washington in 2014, significantly improving performance through regularized objective functions and efficient tree construction algorithms; LightGBM is an improved version released by Microsoft in 2017, using histogram-based decision algorithms for significantly faster training on large-scale data; CatBoost was developed by Yandex, specifically optimizing the handling of categorical features. Even in today's deep learning era, these models remain the tools of choice in scenarios requiring structured data processing, such as financial risk control, recommendation system ranking, and medical diagnosis.

Neural Networks and Deep Learning

The deep learning section starts from the most fundamental Multi-Layer Perceptron (MLP) and dives deep into the derivation of Backpropagation — the core mechanism. The backpropagation algorithm is essentially the systematic application of the Chain Rule on computational graphs, computing gradients of the loss function with respect to each parameter layer by layer from output to input, making gradient descent optimization possible. Although modern deep learning frameworks (such as PyTorch's Autograd and TensorFlow's GradientTape) have fully automated this process, understanding its mathematical details is crucial for diagnosing training problems (such as vanishing gradients and exploding gradients), designing new network architectures, and understanding why certain initialization strategies work.

Notably, the project includes a pure NumPy implementation of MLP training — this "build from scratch" approach requires developers to manually write every step of forward propagation, loss computation, gradient calculation, and parameter updates, forcing learners to confront practical issues like dimension matching in matrix operations and numerical stability handling, truly understanding what happens behind the framework rather than merely calling PyTorch or TensorFlow's black-box interfaces.

Additionally, it dissects key components of modern neural networks: linear layers, Residual Connections, Layer Normalization, Dropout, and the currently popular Mixture-of-Experts (MoE) model. The MoE concept can be traced back to 1991, but has regained attention in recent years due to its successful application in large language models. Its core architecture contains multiple parallel "expert" sub-networks and a "gating network," activating only a small number of experts for each input (e.g., selecting 2 out of 8), allowing the model to have enormous total parameters while using only a fraction during each inference. Google's Switch Transformer and Mistral's Mixtral 8x7B both adopt this architecture — this "sparse activation" design philosophy is significant for understanding how current large models scale capabilities under limited compute.

Transformer Architecture and Large Language Models in Detail

SELENE's most timely section is its systematic breakdown of the Transformer architecture and Large Language Models (LLMs). This is the technical core of the current AI wave.

In the Transformer chapter, the project covers in detail:

• Attention mechanism: The soul of the Transformer
• Complete Transformer architecture
• Positional Encodings: Explored in depth across three parts
• Masking mechanism

The Transformer architecture abandons the sequential processing approach of RNNs and CNNs, using pure attention mechanisms to process all input tokens in parallel, bringing enormous computational efficiency advantages. However, the self-attention operation itself is permutation invariant — shuffling the input order doesn't change the output — but language is inherently ordered; "the cat chases the dog" and "the dog chases the cat" have completely different meanings. Positional encoding was introduced precisely to solve this fundamental problem. The original Transformer paper (Vaswani et al., 2017) used sinusoidal functions to generate absolute positional encodings; BERT adopted learnable absolute positional embeddings; while current mainstream large language models (such as LLaMA, GPT-NeoX) generally use Rotary Position Embedding (RoPE), which encodes relative position information by applying rotational transformations to query and key vectors in attention computation, offering better length extrapolation capabilities. SELENE's three-part deep dive into positional encoding reflects the complexity and importance of this topic.

In the LLM section, the content is even more practice-oriented and cutting-edge:

• Fundamental principles of language models
• Retrieval-Augmented Generation (RAG): A key technique for addressing knowledge timeliness and hallucination problems in large models
• Fine-tuning
• Training an LLM from scratch
• Efficiency optimization strategies and data preparation

Retrieval-Augmented Generation (RAG) is a framework proposed by Meta AI in 2020, designed to address two core limitations of large language models: the knowledge cutoff date problem and the hallucination problem. The RAG workflow typically involves three steps: first, documents from an external knowledge base are chunked and converted into vectors via embedding models, stored in a vector database; when a user asks a question, the system similarly converts the question into a vector and retrieves the most relevant document chunks through similarity search; finally, the retrieved context and original question are fed together into the large language model to generate an answer. This architecture allows models to "consult" the latest or private knowledge sources without retraining, and has become the standard architecture pattern for enterprise-level LLM applications, widely used in intelligent customer service, enterprise knowledge base Q&A, legal document analysis, and other scenarios.

This complete path from attention mechanisms to training an LLM from scratch covers exactly the critical leap from understanding to practice. For learners who truly want to understand "why ChatGPT-like models work," this is extremely valuable systematic material.

Optimizers and NLP Fundamentals

Beyond the models themselves, SELENE also dedicates a separate Optimizers chapter, covering mainstream optimization algorithms including gradient descent with momentum, RMSProp, AdaGrad, and Adam. Understanding the differences between these optimizers is crucial for hyperparameter tuning and model training.

The development history of neural network optimizers reflects researchers' progressively deeper understanding of training dynamics. The most basic Stochastic Gradient Descent (SGD) uses only gradient information to update parameters at each step, making it prone to getting stuck at saddle points and local optima. Momentum borrows from physics concepts, introducing exponential moving averages of historical gradients to accelerate convergence and suppress oscillation. AdaGrad (2011) first introduced the idea of adaptive learning rates, maintaining independent learning rates for each parameter, but the accumulated sum of squared gradients causes learning rates to decay too quickly. RMSProp, proposed by Hinton in a lecture, solved this problem by using exponential moving averages instead of simple accumulation. Adam (2014) combines the advantages of momentum and RMSProp, simultaneously maintaining first-moment and second-moment estimates of gradients with bias correction, becoming the most widely used default optimizer in deep learning. In recent years, AdamW (the decoupled weight decay version) has become the standard choice for training large language models.

In NLP fundamentals, the content includes:

• Tokenization: Covering modern mainstream methods like Byte Pair Encoding (BPE) and WordPiece
• Text normalization, Lemmatization, and Stemming
• Embeddings: From overview to the classic Word2Vec

Tokenization is the first step in how large language models process text, and a key factor affecting model performance. Modern LLMs use subword segmentation strategies. Byte Pair Encoding (BPE) was originally a data compression algorithm; OpenAI's GPT series introduced it to the NLP domain: starting from the character level, it repeatedly merges the most frequently occurring adjacent character pairs in the corpus, ultimately building a fixed-size vocabulary. WordPiece is a similar method developed by Google for BERT, differing in that it uses likelihood values rather than frequency as the merging criterion. The advantage of these methods lies in balancing vocabulary size and coverage — common words exist as whole tokens, while rare words are split into meaningful subword fragments, avoiding the out-of-vocabulary (OOV) problem faced by traditional word-level models. Tokenizer design directly affects a model's multilingual capability, mathematical reasoning ability, and token efficiency.

Although these fundamentals are "traditional," they are required knowledge for understanding how large models process text input.

Usage and Future Plans

SELENE has also put effort into the user experience. The project provides an overview page that connects all topics, with each knowledge point offering three access methods simultaneously:

1. Directly viewing the HTML-rendered version (readable without any environment setup)
2. Accessing the GitHub repository for source files
3. One-click opening in Google Colab (eliminating the hassle of local environment configuration)

The author also revealed that the team is building a web navigation interface to help users browse topics and recommend learning paths. This means SELENE may evolve in the future from a mere collection of notes into a structured learning platform.

Regarding content planning, the author is currently writing material on time series analysis and classical statistical models (such as AR, ARMA, ARIMA), planned for use in next semester's data mining course. This further confirms SELENE's unique model of being "teaching-driven and continuously growing."

Why SELENE Is Worth Bookmarking

In today's landscape of highly homogenized AI educational resources, SELENE's value lies in three aspects: systematic coverage (from traditional ML to LLMs in one coherent path), interactivity (Jupyter + Colab makes theory executable), and emphasis on principles (not shying away from mathematical derivations and implementations from scratch).

It doesn't aim to replace in-depth textbooks or paid courses, but rather provides a clear and free path for learners with a certain mathematical foundation who truly want to understand AI's underlying mechanisms. For AI learners, such an open-source, free, and structurally complete resource repository is undoubtedly a worthwhile starting point to bookmark in the self-learning journey.

Key Takeaways