One language carries data from raw leaf litter on the forest floor to sunlight at the top of the tree. Today we learn Python from first principles, use it to train an ML model, build a data pipeline around that model, and scale the whole thing with Spark.
Python won data work for one reason: it reads like the pseudocode you'd scribble on a whiteboard. No semicolons, no type declarations to start, and indentation is the syntax — the visual structure of the code is its logic.
A variable in Python is a name tag tied to an object. You never declare types; Python infers them at runtime (it's dynamically typed). Press Run on every cell — outputs are real.
# Variables: assign with = , type is inferred client_name = "Thaddeus" # str portfolio_value = 1_250_000.50 # float (underscores for readability) num_accounts = 3 # int is_accredited = True # bool print(type(portfolio_value)) print(f"{client_name} holds S${portfolio_value:,.2f} across {num_accounts} accounts")
That f"..." is an f-string — Python's way of weaving variables into text. The :,.2f part is a format spec: thousands separators, 2 decimal places. You will use f-strings every single day.
Two workhorses: if / elif / else for branching, and for loops for iteration. Notice there are no braces — the indented block is the body.
aum_sgd = [180_000, 950_000, 4_200_000, 60_000_000] for aum in aum_sgd: if aum >= 30_000_000: segment = "UHNW" elif aum >= 1_500_000: segment = "HNW" elif aum >= 300_000: segment = "Affluent" else: segment = "Mass" print(f"S${aum:>12,} → {segment}")
This slider drives the exact if / elif ladder above. Move it and watch the branch that fires.
Functions are defined with def. Type hints (aum: float) are optional but professional teams use them — they document intent and let tools catch bugs.
def classify_segment(aum: float) -> str: """Map assets under management to a wealth segment.""" if aum >= 30_000_000: return "UHNW" if aum >= 1_500_000: return "HNW" if aum >= 300_000: return "Affluent" return "Mass" # Functions are values too — pass them around like data segments = [classify_segment(x) for x in (50_000, 2_000_000, 45_000_000)] print(segments)
Four built-in containers cover 95% of daily work. Choosing the right one is half of writing good Python.
| Structure | Syntax | Ordered? | Mutable? | Reach for it when… |
|---|---|---|---|---|
| list | [1, 2, 3] | Yes | Yes | An ordered sequence — trades, rows, files |
| tuple | (lat, lon) | Yes | No | A fixed record that must not change |
| dict | {"key": val} | Yes* | Yes | Lookups by key — a client profile, JSON, config |
| set | {1, 2, 3} | No | Yes | Uniqueness & membership — dedupe, "have we seen this ID?" |
* dicts preserve insertion order since Python 3.7 — handy, but don't build logic on it.
# A dict is how structured data usually arrives (think: JSON from an API) client = { "id": "C-1042", "name": "Thaddeus", "holdings": ["VWRA", "IWDA", "SGD_CASH", "IWDA"], # oops, a dupe } unique_holdings = set(client["holdings"]) # dedupe in one move print(f"{client['name']} holds {len(unique_holdings)} distinct assets: {sorted(unique_holdings)}")
A list comprehension builds a new list by transforming and filtering another, in one readable line. It's the idiom that most clearly separates "writing Python" from "writing Java in Python". It's also conceptually a map + filter — remember that phrase, because Spark is built on the same two verbs.
balances_sgd = [120_000, -500, 2_400_000, 0, 88_000] # The loop way (4 lines)… usd = [] for b in balances_sgd: if b > 0: usd.append(round(b / 1.34, 2)) # …and the comprehension way (1 line): [transform for item in source if filter] usd2 = [round(b / 1.34, 2) for b in balances_sgd if b > 0] print(usd == usd2, usd2)
from dataclasses import dataclass # A dataclass: a class that's mostly data — Python writes the boilerplate @dataclass class Trade: symbol: str qty: int price: float def notional(self) -> float: return self.qty * self.price t = Trade("VWRA", 150, 142.30) print(t, "→ notional:", t.notional()) # Errors: catch what you expect, let the rest crash loudly try: bad = float("n/a") except ValueError as e: print("Handled:", e)
Data science in Python rests on a stack of libraries, each one layer up the tree: NumPy (fast arrays) → pandas (tables) → scikit-learn (classical ML) → PyTorch / TensorFlow (deep learning). You'll spend most of your life in the middle two.
The core object is the DataFrame: a table with named columns, like a SQL table or Excel sheet you command with code. The universal data science loop is load → inspect → clean → transform → model.
import pandas as pd df = pd.read_csv("clients.csv") # load print(df.head(3)) # inspect: first rows df = df.dropna(subset=["age", "aum"]) # clean: drop missing df["engaged"] = df["logins_90d"] >= 6 # transform: new feature # The single most useful pattern in analytics: groupby → aggregate print(df.groupby("segment")["aum"].agg(["count", "mean"]).round(0))
groupby("segment")["aum"].mean() in pandas, GROUP BY segment in SQL, and df.groupBy("segment").avg("aum") in PySpark are the same idea in three dialects. Learn the idea once; the syntax follows.Every scikit-learn model — regression, random forest, gradient boosting — follows the same four-step ritual. This is a real example: predicting a client's propensity to accept an advisory nudge.
from sklearn.model_selection import train_test_split from sklearn.ensemble import RandomForestClassifier from sklearn.metrics import roc_auc_score X = df[["age", "aum", "logins_90d", "tenure_yrs"]] # features y = df["accepted_nudge"] # label (0/1) # 1. Split — hold out data the model never sees, to test honestly X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42) model = RandomForestClassifier(n_estimators=300) # 2. Choose model.fit(X_tr, y_tr) # 3. Fit (learn) probs = model.predict_proba(X_te)[:, 1] # 4. Predict print(f"AUC on unseen clients: {roc_auc_score(y_te, probs):.3f}")
Click (or tap) to drop data points — say, engagement score vs AUM. The line refits instantly using least squares, the exact math inside LinearRegression().fit(X, y). Watch R² judge your data's messiness.
Why hold data back? Because a model graded on its own homework always looks brilliant. Slide to change the split and see how the honest score behaves.
scikit-learn covers tabular problems — churn, propensity, credit scoring. When the data is unstructured (text, images, speech), you climb to PyTorch or TensorFlow, where models are stacks of layers trained by gradient descent on GPUs. The Python you write barely changes — it's still objects, functions, and loops — only the library and the compute bill grow.
A model is only as good as the water flowing to it. Data engineering is the river system: moving data from source systems into shapes the business — and the model — can drink from. The universal pattern is ETL/ELT: Extract, Transform, Load.
Modern lakehouse platforms (Databricks, Delta Lake, Iceberg) organise data into three refinement layers. Think of it as light moving up through forest strata. Click each stratum to watch one real record climb.
Data lands exactly as the source system produced it — duplicates, nulls, typos and all. Append-only, never edited. Why keep the mess? Because it's your audit trail and replay button: if downstream logic was wrong, you rebuild everything from here.
Deduplicated, typed, validated, joined with reference data. One row per real-world entity. This is the layer data scientists trust for feature engineering — clean enough to use, granular enough to be flexible.
Aggregated, business-named marts: AUM by segment by month, nudge acceptance by channel. This is what dashboards, regulators, and executives touch. Sunlight — refined, expensive, and worth the climb.
Before frameworks, understand that a pipeline is just functions composed in order. Here's bronze → silver → gold in ~25 lines. This exact shape scales up: swap pandas for PySpark and the structure survives.
import pandas as pd def extract() -> pd.DataFrame: # BRONZE: read raw return pd.read_json("s3://lake/bronze/events/2026-07-18.jsonl", lines=True) def transform(raw: pd.DataFrame) -> pd.DataFrame: # SILVER: clean & conform df = raw.drop_duplicates(subset=["cust_id", "ts"]) df["cust_id"] = df["cust_id"].str.strip().str.upper() df["aum_sgd"] = df["aum"].str.replace(",", "").astype(float) return df[df["aum_sgd"] >= 0] # data quality gate def load(clean: pd.DataFrame) -> pd.DataFrame: # GOLD: aggregate & publish gold = clean.groupby("segment", as_index=False).agg( clients=("cust_id", "nunique"), total_aum=("aum_sgd", "sum")) gold.to_parquet("s3://lake/gold/aum_by_segment.parquet") return gold print(load(transform(extract()))) # the whole river
Press each stage in order. Watch the record count fall as quality gates bite, and note the file format changing — JSON at the floor, Parquet in the canopy.
In production, an orchestrator (Airflow, Dagster, Prefect — all Python) runs your pipeline on a schedule, retries failures, and alerts on breaks. You describe the pipeline as a DAG — a directed acyclic graph of tasks. Hold that word: Spark uses the very same concept one level down.
from airflow.decorators import dag, task from datetime import datetime @dag(schedule="0 6 * * *", start_date=datetime(2026, 1, 1), catchup=False) def wealth_daily(): @task def bronze(): ... # land raw events @task def silver(): ... # clean & conform @task def gold(): ... # aggregate marts @task def score_clients(): ... # run the Part-3 model on fresh silver data bronze() >> silver() >> [gold(), score_clients()] # dependencies wealth_daily()
pandas holds your data in one machine's memory. At 10 GB it groans; at 500 GB it dies. Apache Spark's answer: chop the data into partitions, spread them across a cluster of machines, and run your logic on all of them at once. PySpark is the Python steering wheel for that cluster.
The one machine running your Python script. It plans the work, builds the DAG, and collects results. The conductor — it holds the score, not the instruments.
Worker processes on cluster machines that hold partitions in memory and execute tasks on them in parallel. The orchestra.
The chunks your dataset is split into. 500 GB might become 4,000 partitions of ~128 MB. Parallelism = how many partitions can be worked at once.
When an operation like groupBy needs rows with the same key on the same machine, data moves across the network. The expensive monsoon — necessary, but you minimise it.
A 96-partition job. Scale the cluster and run it — each executor works its partitions in parallel (gold = processing, mint = done). Watch wall-clock time change. This is what "horizontal scaling" means.
In pandas, every line executes immediately. In Spark, transformations (filter, select, groupBy) only describe work — they build a DAG and nothing runs. Only an action (count, collect, write) triggers execution. Why? It lets Spark see the whole plan and optimise it — pushing filters early, pruning unused columns, minimising shuffles — before touching a single byte.
Click transformations in any order — notice the cluster does nothing. Then click an action.
Here is the Part-4 pipeline rewritten for a cluster. Squint and it's the same code — that's deliberate. The DataFrame API is a lingua franca.
from pyspark.sql import SparkSession, functions as F spark = SparkSession.builder.appName("wealth_gold").getOrCreate() events = spark.read.json("s3://lake/bronze/events/") # 500 GB? fine. silver = (events .dropDuplicates(["cust_id", "ts"]) .withColumn("cust_id", F.upper(F.trim("cust_id"))) .withColumn("aum_sgd", F.regexp_replace("aum", ",", "").cast("double")) .filter(F.col("aum_sgd") >= 0)) # all lazy so far… gold = silver.groupBy("segment").agg( F.countDistinct("cust_id").alias("clients"), F.sum("aum_sgd").alias("total_aum")) gold.write.mode("overwrite").parquet("s3://lake/gold/aum_by_segment") # ACTION → now it runs
df.filter(...).groupBy(...).agg(...) in PySpark and it returns instantly on 2 TB of data. Why?.count(), .show() or .write and the DAG actually executes — that's when you wait.Compute is the CPU/GPU + memory that executes your code — as opposed to storage, where data rests. The defining move of the modern stack is separating the two: data lives cheaply in object storage (S3, ADLS, GCS); compute spins up, drinks from it, and shuts down.
Buy a bigger machine. Simple, but there's a ceiling — and one very expensive tree can still fall over.
Add more machines — the Spark way. Near-limitless, resilient, but you pay a coordination (shuffle) tax. You watched this trade-off in the cluster playground.
| Layer | What lives there | Python tool | Who works here |
|---|---|---|---|
| Bronze | Raw, immutable events & extracts | spark.read.json / kafka | Data engineers |
| Silver | Clean, typed, deduped entities | PySpark / pandas transforms | Data engineers + scientists |
| Gold | Aggregated business marts & features | groupBy → parquet / SQL | Analysts, ML models, dashboards |
| Model | Trained artefacts scoring fresh data | scikit-learn / PyTorch | Data scientists + ML engineers |
| Orchestration | Schedules, retries, lineage | Airflow / Dagster DAGs | Data engineers |
| Idea | Python | pandas | PySpark |
|---|---|---|---|
| Filter | [x for x in xs if p(x)] | df[df.aum > 0] | df.filter(F.col("aum") > 0) |
| Transform | [f(x) for x in xs] | df["usd"] = df.sgd / 1.34 | df.withColumn("usd", ...) |
| Aggregate | sum(xs) | df.groupby("seg").sum() | df.groupBy("seg").sum() |
| Execute | immediately | immediately | only on an action (.count()) |
Install Python + VS Code. Rewrite one Excel task as a script using lists, dicts, functions, f-strings.
pandas on a real CSV from your world: load, clean, groupby, chart with matplotlib.
Train a scikit-learn classifier end-to-end. Split honestly, measure AUC, explain overfitting to a colleague.
Build a bronze→silver→gold pipeline in pandas, schedule it with Airflow, then port one stage to PySpark on Databricks Community Edition (free).