Machine learning,
from the roots up.
A ground-up guide to the whole terrain — algorithms, evaluation, deep nets, and the LLM era — organised so that by the last section you can hold your own with data scientists.
AI, ML, deep learning, generative AI — the nesting dolls
These four words get used interchangeably and it drives data scientists slightly mad. They're actually nested: each one lives inside the one before it. Get this straight and half the confusion disappears.
Four words, one hierarchy
Artificial intelligence is the whole ambition: machines doing things that seem to require intelligence. Everything else is a subset. Click each band in the diagram to see where it sits and what belongs to it.
"All deep learning is machine learning, but not all machine learning is deep learning — a random forest is ML with no neural network in sight." That one sentence signals you understand the map.
The loop hiding inside every model
Before the algorithms themselves, it pays to know the engine underneath them. Almost every model — from linear regression to GPT — learns through the same four-beat loop. This is the single most useful mental model you can carry.
1 · Predict
The model makes a guess using its current internal numbers (its parameters, also called weights).
2 · Measure error
A loss function scores how wrong the guess was — e.g. squared error for numbers, cross-entropy for categories.
3 · Find the slope
Backpropagation works out which direction to nudge each weight to reduce the loss (the gradient).
4 · Nudge & repeat
Gradient descent takes a small step downhill. Do this millions of times and the model "learns".
Parameters are learned by the model during training (the weights). Hyperparameters are set by you before training — learning rate, number of trees, network depth. You tune hyperparameters; the model tunes parameters. Confusing the two is a rookie tell.
The knob that controls step size in beat 4 is the learning rate — too big and training overshoots and diverges; too small and it crawls. That single hyperparameter causes more failed training runs than almost anything else.
Supervised, unsupervised, reinforcement
This is the backbone of the whole field, and the right place to start. The difference is entirely about what feedback the model gets. Every algorithm a data scientist would expect you to know is here — especially the ensemble methods, which do most of the real-world heavy lifting on tabular data.
Labelled data. Every training example comes with the right answer. The model learns the mapping from inputs → answer, then predicts answers for data it's never seen.
Linear RegressionRegression+
Fits a straight-line relationship between input features and a continuous number, then reads predictions off the line.
Logistic RegressionClassification+
Despite the name, it's a classifier. It squashes a linear score through a logistic (sigmoid) curve to output a probability between 0 and 1.
It's often described as "just binary classification." True, but incomplete: with a softmax it extends naturally to multiclass (multinomial logistic regression). Say "binary by default, multiclass with softmax" and you'll sound precise.
Decision TreesBoth+
A flowchart of if/else splits learned from data. Fully interpretable, handles numbers and categories, needs little preprocessing — but a single deep tree overfits badly.
Support Vector Machine (SVM)Both+
Finds the boundary (hyperplane) that leaves the widest possible margin between classes.
The linear case is only half the story. The reason SVMs are famous is the kernel trick — it bends the boundary to separate classes that aren't linearly separable, without ever computing the high-dimensional coordinates. SVMs also do regression (SVR).
k-Nearest Neighbours (k-NN)Both · added+
No training at all — it just stores the data. To classify a new point, it looks at the k closest known points and takes a vote.
Naïve BayesClassification · added+
Applies Bayes' theorem with a bold "naïve" assumption that features are independent. Wrong assumption, surprisingly good results — and blazingly fast.
Random ForestBoth · added · important+
An ensemble: train hundreds of decision trees, each on a random slice of data and features, then average their votes. The randomness cancels out individual trees' overfitting.
Ensembles rarely make the beginner lists, yet they win a huge share of real-world tabular problems. If a data scientist mentions "the forest" or "bagging", this is it.
Gradient Boosting (XGBoost / LightGBM / CatBoost)Both · added · important+
Also an ensemble of trees, but built sequentially: each new tree focuses on fixing the errors the previous ones made. Where a forest builds trees in parallel and averages, boosting builds them one after another and corrects.
XGBoost and its cousins are the workhorse that wins most Kaggle competitions on structured data and quietly power much of fintech. If you remember one "advanced" name from this page, make it this one.
No labels. The model is handed raw data and asked to find structure on its own — natural groupings, or a simpler way to represent the data.
K-Means ClusteringClustering+
Splits data into K groups by repeatedly assigning points to the nearest centre and moving each centre to the mean of its points, minimising within-cluster spread.
You must choose K in advance, and it assumes roughly round, similar-sized clusters. The "elbow method" and silhouette score help pick K — dropping either term shows depth.
Hierarchical ClusteringClustering+
Builds a tree of nested clusters (a dendrogram) by repeatedly merging the closest groups. No need to pre-set K — you cut the tree at whatever height you like.
DBSCANClustering · added+
Density-based clustering: it finds dense regions and, crucially, labels sparse points as noise/outliers rather than forcing them into a cluster. It also finds arbitrarily-shaped clusters that K-means can't.
Principal Component Analysis (PCA)Dimensionality reduction+
Finds new axes (principal components) along the directions of greatest variance, then keeps only the top few — compressing many features into a handful while losing as little information as possible. It's a linear method.
t-SNE & UMAPVisualisation · sharpened+
t-SNE is a non-linear technique that squeezes high-dimensional data into 2-D so you can see the clusters, preserving local neighbourhoods.
First: in a t-SNE plot the distances between clusters are not meaningful and cluster sizes are misleading — it's for eyeballing, not measuring. Second: UMAP is the modern alternative — faster, and it preserves more of the global structure. Data scientists reach for UMAP now.
Learning by trial and reward. An agent acts in an environment, gets rewards or penalties, and learns a strategy (a policy) that maximises reward over time. No labelled answers — just consequences.
Q-LearningValue-based+
Learns a table (or network) estimating the future reward of taking each action in each state — the "Q-value". Act greedily on the highest Q-value and you have a policy. It's off-policy: it can learn the best strategy while exploring with a different one.
Policy Gradient MethodsPolicy-based+
Skip the value table and optimise the policy directly, adjusting it toward actions that led to higher reward. Handles continuous actions (steering angle, throttle) that Q-learning struggles with. Modern variants: PPO, A2C.
RL is no longer a niche corner. RLHF — reinforcement learning from human feedback — is how models like Claude and ChatGPT are aligned to be helpful. Policy-gradient methods (PPO) are the engine. Knowing RL feeds straight into understanding today's assistants.
What separates a model that works from one that looks like it does
These concepts aren't algorithms, which is exactly why they're easy to skip — but they're what data scientists actually argue about. A model can score beautifully in the lab and fail completely in production for exactly these reasons.
Overfitting vs underfitting — the central tension
An overfit model memorises the training data, including its noise, and flops on new data. An underfit model is too simple to capture the real pattern and flops on everything. Every modelling decision is a negotiation between these two.
Underfitting is high bias (too rigid). Overfitting is high variance (too sensitive to the exact training sample). You can't drive both to zero — the art is finding the sweet spot. This phrase comes up constantly; now it means something to you.
The three-way split — and why "test set" is sacred
You split your data into training (the model learns here), validation (you tune hyperparameters here), and test (touched once, at the very end, to estimate real-world performance). Keeping the test set untouched until the end is what makes your final number honest.
K-fold cross-validation splits the training data into K parts, trains K times each holding out a different part, and averages the scores — giving a far more reliable estimate than a single split, especially on smaller datasets.
Regularisation — deliberately handicapping the model
Techniques that penalise complexity to fight overfitting: L1 (Lasso) can zero out useless features entirely; L2 (Ridge) shrinks weights smoothly; dropout randomly switches off neurons during training; early stopping halts training before it starts memorising. Naming any of these signals real understanding.
When information from the future (or from the test set) sneaks into training, your model looks brilliant and then collapses in production. Example: scaling your features using the whole dataset before splitting. Leakage is the number-one cause of "it worked in the notebook but not in prod." Mention it and you'll sound seasoned.
Metrics — and the trap of accuracy
The metric definitions are the easy part. The single most important caveat in all of model evaluation is the one most people miss — so let's make you feel it with a live tool.
Live confusion matrix
Adjust the four counts and watch the metrics shift. Then hit the "rare disease" preset.
Load the "predict all healthy" preset. A model that never flags anyone still scores 99% accuracy when only 1% are sick — while catching zero real cases (recall = 0). This is why on imbalanced problems — fraud, disease, churn — data scientists distrust accuracy and lean on precision, recall, F1, and ROC-AUC instead.
Precision vs recall — pick your poison
Precision answers "when the model says yes, how often is it right?" — you optimise it when false alarms are costly (flagging a legit transaction as fraud annoys customers). Recall answers "of all the real cases, how many did we catch?" — you optimise it when misses are dangerous (missing a cancer). You usually can't max both; F1 balances them.
ROC-AUC: a single number (0.5 = random, 1.0 = perfect) summarising how well the model ranks positives above negatives across every threshold — the standard headline metric for classifiers. Confusion matrix: the 2×2 table you just played with, the source of all these numbers.
Regression metrics — beyond MSE
MAE (mean absolute error) is the average miss in plain units, robust to outliers. RMSE is MSE's square root, back in the original units but punishing big errors harder. R² tells you what fraction of the variance the model explains — 1.0 is perfect, 0 is no better than guessing the mean.
The unglamorous work that wins
There's a saying: models are commodities, features are the moat. Four techniques form the core — with one precision point worth making.
Scaling
Putting features on a comparable range so one big-numbered feature (income) doesn't drown a small one (age). Vital for distance- and gradient-based models (k-NN, SVM, neural nets); irrelevant to trees.
"Mean 0, standard deviation 1" is specifically standardisation (z-score) — one kind of scaling. The other common one is normalisation (min–max), which squeezes values into a 0–1 range. "Scaling" is the umbrella; know both children.
Encoding categorical variables
Turning categories into numbers. One-hot makes a 0/1 column per category (safe, but explodes width on high-cardinality fields). Ordinal maps to ranked integers (only when order is real). Target/embedding encoding handles high-cardinality cases like postcodes.
Handling missing data
Imputation fills gaps (mean/median, or a model-based guess); sometimes the fact that a value is missing is itself signal worth a flag column. Dropping rows is a last resort — it can quietly bias the data.
Generating new features
Creating features that expose the pattern more directly — moving averages, ratios, date parts (day-of-week, is-holiday), interactions between fields. This is where domain knowledge becomes accuracy, and where a strategist who understands the business often out-contributes a pure modeller.
From a single neuron to self-attention
Deep learning climbs a natural ladder, each rung solving a limitation of the one before. Here it is rung by rung — with the one transformer claim that gets garbled most often set straight.
Neural Network (the MLP)
Layers of interconnected "neurons", each connection weighted. Data flows through, activation functions (like ReLU) add the non-linearity that lets it model curves, and backprop tunes the weights.
Convolutional Neural Network (CNN)
Slides small filters across an image to detect local patterns — edges, then textures, then whole objects as layers deepen. Vastly fewer weights than connecting every pixel to every neuron.
Recurrent Neural Network (RNN)
Has a memory that carries information from earlier steps forward, so order matters — suited to language and time series. Its weakness: it forgets across long gaps (the vanishing gradient problem).
LSTM & GRU
Add gates that decide what to keep, forget, and output — fixing the long-memory problem. LSTM uses three gates (input, forget, output); GRU simplifies to two (reset, update).
Transformers
Ditches sequential processing entirely. Self-attention lets every word look at every other word at once and weigh how relevant each is — using queries, keys and values, scaled dot-products and a softmax. Multi-head attention does this several times in parallel to capture different relationships, and positional encoding re-injects word order (since nothing is processed in sequence).
It's tempting to assume BERT and GPT both come from the full encoder-decoder. More precisely: the original 2017 transformer was encoder–decoder (built for translation), but BERT is encoder-only (great at understanding — search, classification) and GPT is decoder-only (great at generating text). Getting this distinction right is a genuine credibility marker with ML people.
Parallelism (whole sequences at once, so they train fast on huge data), long-range memory (attention reaches across the entire input), and scalability (bigger models + more data kept getting better — the insight behind GPT-scale systems).
Where most intros stop, the conversation starts
Many overviews end at transformers. But that's exactly where today's most-discussed ideas begin — and where your strategy work lives. These are the terms that dominate current data-science and product conversations.
Foundation models & the pretrain → fine-tune shiftParadigm+
Instead of training a fresh model per task, you pretrain one enormous model on vast general data, then fine-tune or prompt it for specific jobs. This is transfer learning at scale — the reason a handful of labs can serve thousands of downstream applications.
EmbeddingsRepresentation+
Turning words, images, or users into vectors of numbers where closeness = similarity. This is the k-NN intuition from earlier, industrialised. Embeddings underpin semantic search, recommendations, and RAG.
RAG — Retrieval-Augmented GenerationLLM pattern+
Before an LLM answers, retrieve relevant documents (via embeddings) and feed them in as context. It grounds answers in your data and cuts hallucination — without retraining the model. The default enterprise pattern for "chat with our knowledge base."
Fine-tuning vs prompting vs RAGDecision+
Prompting: change the instructions, no training — fastest, cheapest. RAG: inject fresh/proprietary knowledge at query time. Fine-tuning: adjust the model's weights on your examples to change its behaviour or style. Knowing which lever to pull for which problem is a genuinely strategic skill.
RLHF & alignmentTraining+
Reinforcement Learning from Human Feedback: humans rank model outputs, a reward model learns those preferences, and policy-gradient RL (PPO) nudges the model toward them. It's how raw text-predictors became helpful, honest assistants — the direct bridge from Section 03's RL.
Diffusion modelsGenerative+
The engine behind AI image generation (Stable Diffusion, DALL·E, Midjourney). Train a model to remove noise step by step; run it in reverse from pure noise and a coherent image emerges. A different generative family from transformers — worth knowing they're not the same thing.
Agentic AISystems+
LLMs that don't just answer but plan, use tools, call APIs, and act in loops toward a goal — checking their own work along the way. The frontier of applied AI, and squarely your domain. The through-line: an agent pursuing a goal by taking actions is the same RL framing from Section 03, now wrapped around a language model.
Explainability and responsible AI matter more the more powerful models get — SHAP and LIME explain individual predictions, and in regulated fields like wealth management, "why did the model decide this?" is often a hard requirement, not a nice-to-have. Bringing this up unprompted marks you as someone who thinks past the demo.
The quick-reference layer
A tight glossary, then a handful of phrases that make conversations flow. You don't need to build models to sound like you understand the people who do.
- Feature
- An input column the model learns from.
- Label / target
- The answer you're trying to predict.
- Parameter (weight)
- A number the model learns during training.
- Hyperparameter
- A setting you choose before training.
- Loss function
- The score of how wrong a prediction is.
- Gradient descent
- The step-by-step method that minimises loss.
- Epoch
- One full pass through the training data.
- Overfitting
- Memorising noise; great on train, poor on new data.
- Regularisation
- Penalising complexity to curb overfitting.
- Ensemble
- Combining many models to beat any single one.
- Baseline
- The simple model everything else must beat.
- Inference
- Using a trained model to make predictions.
- Embedding
- A vector where nearness means similarity.
- Attention
- Weighing which parts of the input matter most.
- Fine-tuning
- Further-training a base model on your data.
- Hallucination
- A confident but false LLM output.
Prove it to yourself
Eight questions across everything above. Pick an answer to lock it and see why. No pressure — this is just your mirror.