Technical Track
Courses for people already building with LLMs who need the system to survive production. Coding required.
Written without tying anything to a tool or model version: what is taught here still holds when the libraries change. Where something is not known with certainty, the course says so instead of filling the gap.
Evaluating LLM Systems
How to know whether your prompt change improved anything or broke something else: datasets, LLM-as-judge and its biases, and regression suites.
🎓 3 levels📚 12 lessons
1. Why You Need Evals
- •The problem: you do not know whether your change improved anything
- •Your evaluation set: where it comes from and how to build it
- •What can actually be measured, and what cannot
- •The small-numbers mistake
2. LLM as a Judge
- •LLM as a judge: how it works and why it is so widely used
- •The judge's biases, and what to do about each
- •Calibrating the judge against humans
- •Rubrics that produce agreement
3. Evals in Your Workflow
- •The regression suite: making sure a fix does not break what worked
- •Offline and online: two different questions
- •Traces: without observability there is no diagnosis
- •Your minimum eval system, running this week
RAG That Works: Retrieval, Not Magic
Why your document chatbot answers badly: chunking, the real limits of embeddings, hybrid search, reranking, and how to measure each step.
🎓 3 levels📚 12 lessons
1. Why Retrieval Fails
- •RAG is search before it is generation
- •Chunking: the decision with the most impact and the least thought
- •Embeddings: what they capture and what they do not
- •Measuring retrieval on its own
2. Retrieving Better
- •Hybrid search: why BM25 is still alive
- •Reranking: the step almost nobody adds
- •Metadata and filtering: the cheapest improvement
- •The user's question is not the search query
3. RAG in Production
- •Lost in the middle: context order matters
- •Citing, and knowing how to say you do not know
- •The index changes: maintenance and data that expires
- •When RAG is not the answer
Securing LLM Applications
Direct and indirect prompt injection, over-privileged tools and data exfiltration: why there is no fix, and how to shrink the blast radius.
🎓 3 levels📚 12 lessons
1. The Underlying Problem
- •There is no separation between instructions and data
- •Direct and indirect injection: the second is the dangerous one
- •Blast radius: the only variable you control
- •OWASP for LLM as a map, not a certificate
2. Attack Surfaces
- •Tools: where injection becomes an incident
- •Poisoned RAG: when the attacker writes your context
- •Data and system-prompt leakage
- •Cost and availability as an attack vector
3. Defence in Depth
- •Least privilege: the defence that actually works
- •Model output is untrusted input
- •Human in the loop, where it actually matters
- •Testing your own system before someone else does