Large Language Models (LLMs) have transformed how we build software systems, search engines, developer tools, conversational applications, and AI-powered products.
But understanding an LLM requires much more than knowing that it is a neural network trained on text.
You need to understand:
- What an LLM actually learns
- Why next-token prediction works
- How Transformers enable contextual understanding
- Why GPT uses a decoder-only architecture
- How data, model size, and compute interact
- How training differs from inference
- How model architecture affects memory and latency
- How to design an end-to-end LLM platform
- What trade-offs matter when scaling an LLM
This Part builds that foundation.
1. The Most Important Mental Model
If an interviewer asks:
What is a Large Language Model?
A weak answer would be:
An LLM is a neural network trained on lots of text.
A stronger definition is:
An LLM is a parameterized probabilistic model that learns the distribution of token sequences from large-scale text and uses that learned distribution to predict the next token given the preceding context.
At a high level, the process looks like this:
Text
↓
Tokenizer
↓
Token IDs
↓
Embeddings
↓
Transformer
↓
Contextual Representations
↓
Vocabulary Logits
↓
Probability Distribution
↓
Next Token
↓
Append Token to Context
↓
Repeat
Mathematically, the probability of a sequence can be represented as:
P(x₁, x₂, ..., xₙ)
=
∏ P(xₜ | x₁, ..., xₜ₋₁)
The key idea is that the model does not directly learn:
“How to answer questions.”
At its fundamental level, it learns:
“Given this context, what token is likely to come next?”
Modern LLMs can process and generate language in ways that appear coherent and contextual. However, this should not be interpreted as human-like consciousness or comprehension.
Pretraining vs Instruction Following
This distinction becomes extremely important later:
Pretraining
↓
Next-Token Prediction
↓
General Language Capability
↓
Instruction Tuning
↓
Instruction Following
↓
Application-Specific Behavior
Therefore:
Pretraining ≠ Instruction Following
2. Why Did LLMs Become So Powerful?
Modern LLM capabilities are closely connected to advances in deep learning and the ability to train models on enormous quantities of text.
For an expert-level discussion, think about three major scaling dimensions:
LLM Capability
│
┌──────────────┼──────────────┐
↓ ↓ ↓
Model Scale Data Scale Compute Scale
│ │ │
Parameters Tokens GPU/TPU Hours
Layers Quality Distributed Training
Hidden Size Diversity Optimized Kernels
A weak answer is:
More parameters make the model smarter.
A better answer is:
Model capability depends on the interaction between model capacity, training-data quantity and quality, optimization, and compute. Increasing parameters without sufficient high-quality data or compute does not automatically produce proportional gains.
What Happens If You Double the Model Size?
Do not immediately say:
Performance doubles.
Instead, discuss:
- Parameter count
- Training tokens
- Compute budget
- GPU memory
- Optimizer state
- Communication overhead
- Training time
- Inference latency
- Inference memory
- Serving cost
- Scaling behavior
This is where an LLM discussion starts becoming a systems discussion.
3. Why Is Next-Token Prediction Enough to Learn Useful Behavior?
This is one of the most important questions in Part 1.
Suppose the training corpus contains:
Paris is the capital of France.
The model learns relationships such as:
Paris
→ is
Paris is
→ the
Paris is the
→ capital
Paris is the capital
→ of
But across billions of examples, the model also encounters patterns such as:
Question:
What is the capital of France?
Answer:
Paris
It may also encounter:
Translate:
Hello → Bonjour
or:
def fibonacci(n):
...
or:
Article → Summary
The model is not necessarily given a separate training objective for every possible task.
Instead, these patterns exist within the training distribution.
The model learns statistical relationships between tokens and contexts. When a prompt activates a particular pattern, generation can behave like task execution even though the underlying objective remains next-token prediction.
The book describes capabilities such as translation emerging from this process.
Expert Interview Answer
The next-token objective is deceptively general because the training corpus contains many latent task structures. The model learns statistical relationships between tokens and contexts. When a prompt activates a particular pattern, generation can behave like task execution even though the underlying objective remains next-token prediction.
4. LLM vs Traditional Machine Learning
Another common interview question is:
How is an LLM different from traditional machine learning?
Traditional ML often follows a pipeline like:
Problem
↓
Feature Engineering
↓
Task-Specific Model
↓
Prediction
For example:
Email
↓
TF-IDF Features
↓
Classifier
↓
Spam / Not Spam
Traditional machine-learning systems frequently depend on humans identifying and engineering useful features.
An LLM follows a very different paradigm:
Raw Text
↓
Tokenizer
↓
Large Neural Network
↓
Learned Representations
↓
Next-Token Prediction
One foundation model can support many downstream tasks.
That is a major architectural shift.
5. LLM Applications
Part 1 discusses applications including:
- Text generation
- Translation
- Sentiment analysis
- Question answering
- Contextual text generation
For system-design interviews, however, it is more useful to classify applications by workload.
A. Generation
Prompt
↓
LLM
↓
Generated Text
Examples:
- Chat
- Coding
- Writing
- Summarization
B. Classification
Text
↓
LLM
↓
Class
Examples:
- Spam detection
- Sentiment analysis
- Intent classification
C. Transformation
Input Text
↓
LLM
↓
Transformed Text
Examples:
- Translation
- Rewriting
- Summarization
D. Reasoning and Tool Use
A broader system-design extension is:
User
↓
LLM
↓
Tool / Retrieval
↓
LLM
↓
Answer
This goes beyond the basic scope of Part 1 and should therefore be treated as an interview extension rather than a direct claim of the Part itself.
6. The LLM Lifecycle
One of the most important system-design lessons from Part 1 is that building an LLM is not simply:
Train a neural network.
Think about the complete lifecycle:
DATA
↓
Data Collection
↓
Data Cleaning
↓
Tokenization
↓
Dataset Creation
↓
PRETRAINING
↓
Base LLM
↓
Fine-Tuning
↓
Instruction Following
↓
Evaluation
↓
Model Registry
↓
Deployment
↓
Inference
↓
Monitoring / Feedback
This is where LLM theory transitions into system design.
Designing an LLM-Building Platform
If an interviewer asks:
Design a system for building an LLM.
Your architecture should contain at least:
Data Pipeline
│
┌──────────┼──────────┐
↓ ↓ ↓
Ingestion Cleaning Deduplication
│
Filtering
│
Tokenization
↓
Training Pipeline
│
┌──────┴──────┐
↓ ↓
Checkpoints Evaluation
│ │
└──────┬──────┘
↓
Registry
↓
Serving
│
┌────────┬────────┼────────┬────────┐
↓ ↓ ↓ ↓ ↓
Routing Batching Inference Caching Monitoring
This architecture demonstrates that an LLM is not merely a model artifact.
It is a complete data, training, evaluation, deployment, and serving system.
7. Transformer Architecture
The Transformer architecture is probably the single most important technical topic in Part 1.
The original Transformer architecture contains:
Encoder + Decoder
The encoder converts input text into contextual numerical representations, while the decoder generates output text.
Self-attention allows the model to capture relationships between tokens.
Consider:
“I went to the bank to deposit money.”
The representation of:
bank
should depend strongly on:
deposit money
rather than only on nearby words.
That is the core motivation behind attention.
8. Encoder vs Decoder vs Encoder-Decoder
This is a very common system-design interview question.
| **Architecture** | **Main Purpose** |
| ---------------- | ----------------------------------- |
| Encoder-only | Understand / represent input |
| Decoder-only | Generate continuation |
| Encoder-decoder | Transform one sequence into another |
Encoder-Only
Input
↓
Encoder
↓
Representation
↓
Classifier
Good for:
- Classification
- Embeddings
- Representation learning
Encoder-Decoder
Input
↓
Encoder
↓
Context
↓
Decoder
↓
Output
Good for:
- Translation
- Sequence-to-sequence generation
Decoder-Only
Prompt
↓
Decoder
↓
Next Token
↓
Next Token
↓
Next Token
This is the GPT-style architecture emphasized in the book.
9. Why Decoder-Only GPT?
The original Transformer was designed with encoder and decoder blocks, particularly for sequence-to-sequence tasks such as translation.
GPT simplifies this into:
Decoder-Only
+
Causal Self-Attention
+
Next-Token Prediction
The decoder-only architecture provides a simple universal interface:
Context → Continuation
Many different tasks can be represented as text generation.
For example:
Translate:
English: Hello
French:
Or:
Question:
What is 2 + 2?
Answer:
Or:
Summarize:
<document>
Summary:
This makes the architecture remarkably general.
10. GPT Architecture — Know This Diagram
For an interview, remember the following architecture:
Token IDs
│
▼
Token Embeddings
+
Positional Embeddings
│
▼
┌───────────────────────┐
│ Transformer Block 1 │
├───────────────────────┤
│ Causal Self-Attention │
│ Feed Forward │
│ Residual + Norm │
└───────────────────────┘
│
▼
Transformer Block 2
│
▼
...
│
▼
Transformer Block N
│
▼
Final LayerNorm
│
▼
Output Projection
│
▼
Vocabulary Logits
│
▼
Softmax
│
▼
Next Token
GPT models are built from repeated Transformer blocks.
The core architecture remains the same while model size can change through:
- Embedding dimension
- Number of layers
- Number of attention heads
11. Tensor Shapes — Expert Coding Interview Level
Suppose:
B = Batch Size
T = Sequence Length
D = Embedding Dimension
V = Vocabulary Size
H = Number of Attention Heads
Dh = D / H
The tensor flow is:
Token IDs
[B, T]
│
▼
Embedding
[B, T, D]
│
▼
Transformer × N
[B, T, D]
│
▼
Linear Projection
[B, T, V]
The final vocabulary logits have shape:
[B, T, V]
Why [B, T, V] Instead of [B, V]?
Because during training, the model predicts the next token at every sequence position simultaneously.
Therefore, we need a vocabulary distribution for every one of the T positions.
During autoregressive generation, we typically use the logits corresponding to the final position.
12. Model Scaling
The book introduces GPT-2 configurations ranging from approximately 124M to 1.558B parameters.
The important design insight is that model size is influenced by:
Embedding Dimension
+
Number of Layers
+
Attention Heads
+
Feed-Forward Dimensions
+
Vocabulary
Increasing model size affects both training and inference.
Training
More Parameters
↓
More Compute
↓
More GPU Memory
↓
More Training Time
Inference
More Parameters
↓
More Weight Memory
↓
More Compute / Token
↓
Potentially Higher Latency
Distributed Systems
Large Model
↓
Single GPU Insufficient
↓
Model Parallelism
↓
Communication Overhead
Therefore, increasing model size is not simply a capability decision.
It is also an infrastructure decision.
13. Parameter Count — Coding and Design Interview
Suppose an interviewer asks:
How would you estimate the size of a GPT model?
Reason component by component.
Embedding Parameters
Approximately:
V × D
where:
V= vocabulary sizeD= embedding dimension
Attention Parameters
The major projections are:
Q
K
V
Output
For a basic dense formulation, these contribute approximately:
4D²
per Transformer block.
Feed-Forward Parameters
If:
Dff = 4D
then the feed-forward component contributes approximately:
2D × Dff
which becomes:
8D²
Therefore, a rough block-level estimate is:
≈ 12D²
plus:
- Normalization parameters
- Biases
- Other architecture-specific parameters
The exact architecture can differ, so always state your assumptions during an interview.
14. Why Large Datasets Matter
Part 1 emphasizes training LLMs on vast quantities of text.
Large-scale training allows models to capture deeper contextual information and subtleties of human language.
However:
Data quantity is not enough. Data quality is equally important.
Think about the data pipeline as:
Raw Web
↓
Filtering
↓
Deduplication
↓
Quality Scoring
↓
Language Identification
↓
PII / Safety Filtering
↓
Contamination Checks
↓
Training Corpus
Consider this interview question:
Would you rather have 10T low-quality tokens or 2T high-quality tokens?
Do not answer automatically.
A mature answer is:
I would evaluate data quality, diversity, duplication, domain coverage, contamination, and the target model’s scaling regime. Token count alone is not a sufficient measure of training value.
That demonstrates architectural maturity.
15. Data Quality Becomes a System-Design Problem
Imagine you have:
10 PB Web Corpus
and need to process it for LLM training.
A High-level design might look like:
Object Storage
↓
Distributed ETL
↓
Normalization
↓
Language Detection
↓
Quality Filtering
↓
Deduplication
↓
PII / Safety Filtering
↓
Tokenization
↓
Sharded Dataset
↓
Distributed Training
At this point, interviewers can ask:
- How do you deduplicate?
- How do you prevent data leakage?
- How do you shard the dataset?
- How do you resume after a worker failure?
- How do you prevent training-data contamination of evaluation sets?
These are natural system-design extensions of Part 1.
16. Training vs Inference
This distinction is essential.
Training
Input Tokens
↓
Transformer
↓
Logits
↓
Loss
↓
Backpropagation
↓
Gradient
↓
Weight Update
Inference
Prompt
↓
Transformer
↓
Logits
↓
Select Next Token
↓
Append Token
↓
Repeat
GPT generation repeatedly predicts a subsequent token and appends it to the input context.
The Critical Difference
Training:
Parallel Across Positions
Inference:
Autoregressive / Sequential
This difference drives much of modern LLM infrastructure.
17. Why Inference Is a Systems Problem
At first glance, serving an LLM seems simple:
User → LLM
Production inference is much more complicated:
Request
↓
Tokenizer
↓
Queue
↓
Scheduler
↓
GPU
↓
Prefill
↓
Decode
↓
Streaming
Now you need to solve:
- Batching
- GPU utilization
- Memory
- KV cache
- Latency
- Throughput
- Concurrency
- Streaming
- Cancellation
- Timeouts
- Model routing
- Autoscaling
These topics go beyond the narrow implementation covered in Part 1, but they are exactly where Part 1 becomes relevant to expert LLM system-design interviews.
18. A High-Level System-Design Question
Consider the interview question:
Design a platform for serving an LLM to 100,000 concurrent users.
Do not immediately start drawing boxes.
First, clarify the workload.
Step 1 — Understand the Workload
Ask:
- What is the model size?
- What is the context length?
- What is the requests-per-second requirement?
- What is the average input-token count?
- What is the average output-token count?
- What is the peak traffic?
- What is the latency SLO?
- Is streaming required?
- What availability is required?
- Is tenant isolation required?
Step 2 — Design the Architecture
A high-level architecture could look like:
API Gateway
│
▼
Auth / Rate Limit
│
▼
Load Balancer
│
▼
Request Router
│
▼
Scheduler
│
┌──────────┴──────────┐
▼ ▼
GPU Pool A GPU Pool B
Model A Model B
│ │
└──────────┬──────────┘
▼
Streaming
│
▼
Client
Then discuss:
- Model Registry
- Tokenizer Registry
- Prompt Registry
- Metrics
- Tracing
- Logging
- Evaluation
- Canary Deployment
- Autoscaling
- Cost Controls
That is the difference between:
“I know Transformers.”
and:
“I can design an LLM platform.”
19. Expert Coding Interview Questions from Part 1
Q1. Explain an LLM in one minute.
Strong Answer
A decoder-only LLM such as GPT is an autoregressive Transformer that models the conditional probability of the next token given previous tokens. Text is tokenized into IDs, mapped to embeddings, processed through repeated Transformer blocks using causal self-attention and feed-forward transformations, and projected into vocabulary logits. During training, next-token cross-entropy drives parameter updates; during inference, the model generates tokens autoregressively.
Q2. What is the shape of the GPT output?
[B, T, V]
where:
B= batch sizeT= sequence lengthV= vocabulary size
Q3. Why doesn’t GPT output a single token during training?
Because training predicts every next-token position simultaneously.
For example:
Input:
I love machine learning
Targets:
love machine learning <next>
Conceptually:
I
→ love
I love
→ machine
I love machine
→ learning
Therefore, one forward pass generates multiple training predictions.
20. Why Does GPT Need Positional Information?
Self-attention operates on token representations and does not inherently encode sequence order.
Without positional information:
A B C
and:
C B A
would not have sufficient explicit positional distinction.
GPT models use positional information by adding positional embeddings to token embeddings.
This allows the model to distinguish tokens based on their positions in the sequence.
21. Why Can GPT Perform Translation If It Wasn’t Specifically Trained for Translation?
This is one of the best Part 1 interview questions.
A strong answer is:
Because the model learns statistical relationships across large multilingual corpora. If the training data contains translation-like patterns and aligned linguistic structures, next-token prediction can implicitly learn those mappings. This capability can emerge even though translation is not the fundamental training objective.
The important concept is that the model’s training distribution contains many different linguistic patterns.
22. What Is the Difference Between GPT and the Original Transformer?
Original TransformerGPTEncoder + DecoderDecoder-onlyDesigned for sequence-to-sequence tasksDesigned around autoregressive generationUses encoder and decoder blocksUses decoder blocksExample: translationGeneral text generation
Conceptually:
Original Transformer
Input
↓
Encoder
↓
Decoder
↓
Output
GPT
Input Context
↓
Decoder Blocks
↓
Next Token
The book uses GPT as the primary architecture because its decoder-only structure is central to the implementation that follows.
23. What Happens When Context Length Increases?
At the Part 1 level:
More Tokens
↓
More Computation
↓
More Memory
The deeper explanation comes later when studying self-attention, where the relationship between sequence length and attention computation becomes clearer.
A good interview answer is:
The architectural consequence of increasing context length is that the system needs more computation and memory. The exact attention complexity becomes clearer when we study self-attention.
This is better than jumping ahead without explaining the dependency.
24. How Would You Build a GPT Model From Scratch?
Give the interviewer this roadmap:
1. Prepare Corpus
↓
2. Tokenizer
↓
3. Token IDs
↓
4. Dataset / Context Windows
↓
5. Token Embeddings
↓
6. Positional Embeddings
↓
7. Transformer Blocks
↓
8. Final Normalization
↓
9. Vocabulary Projection
↓
10. Next-Token Loss
↓
11. Backpropagation
↓
12. Optimization
↓
13. Evaluation
↓
14. Generation
This is the architectural roadmap established in Part 1 and expanded by the subsequent parts.
25. The Most Important High-Level Trade-Offs
At a High level, you should be able to discuss these trade-offs without prompting.
Design DecisionTrade-OffBigger modelBetter capacity vs higher costMore training dataBetter coverage vs higher data-processing costLonger contextMore context vs compute/memoryLarger batchBetter utilization vs memoryMore layersHigher capacity vs latencyLarger hidden dimensionHigher capacity vs computeMore attention headsRepresentation flexibility vs computeFull fine-tuningFlexibility vs costSmaller modelLower cost vs lower capabilityHigher precisionNumerical fidelity vs memoryQuantizationMemory/latency gains vs potential quality lossShared servingBetter utilization vs isolationDedicated servingBetter isolation vs utilization
These trade-offs are critical when moving from model-level thinking to platform-level architecture.
26. What I Would Expect You to Say
Explain how an LLM works.
Your answer should progress through multiple levels.
Level 1 — Objective
It predicts the next token.
Level 2 — Architecture
It uses a decoder-only Transformer with causal self-attention.
Level 3 — Tensor Flow
[B,T]
→
[B,T,D]
→
Transformer Blocks
→
[B,T,V]
Level 4 — Training
We optimize next-token cross-entropy over large-scale text.
Level 5 — Scaling
Increasing model size, data, and compute can change capability, but also increases memory and training cost.
Level 6 — Inference
Generation is autoregressive, so serving becomes a scheduling, memory, throughput, and latency problem.
Level 7 — Production
I would design the platform around tokenizer and model versioning, request scheduling, batching, GPU memory, evaluation, observability, reliability, and cost.
That progression distinguishes an expert answer from a textbook definition.
27. Part 1 — What You Should Memorize
If you have an interview tomorrow, remember this:
LLM
│
├── Objective
│ └── Next-Token Prediction
│
├── Input
│ └── Token IDs [B,T]
│
├── Representation
│ └── Embeddings [B,T,D]
│
├── Architecture
│ └── Decoder-only Transformer
│
├── Core Mechanism
│ └── Causal Self-Attention
│
├── Output
│ └── Logits [B,T,V]
│
├── Training
│ └── Cross-Entropy + Backpropagation
│
├── Scale
│ ├── Parameters
│ ├── Data
│ └── Compute
│
├── Capabilities
│ ├── Generation
│ ├── Translation
│ ├── QA
│ └── Classification
│
└── Production
├── Inference
├── Latency
├── Throughput
├── Memory
├── Batching
├── Evaluation
└── Cost
Part 1 is intentionally foundational rather than a deep mathematical treatment. It establishes the GPT architecture and the major stages that the subsequent parts explore in greater detail.
The 10 Questions I Would Practice
If you are preparing for a AI/ML system-design interview, make sure you can answer these deeply:
- What exactly does an LLM learn during pretraining?
- Why is next-token prediction sufficient to learn useful capabilities?
- Why did the industry move toward decoder-only architectures?
- Encoder vs decoder vs encoder-decoder — when would you choose each?
- Explain the complete tensor flow through GPT.
- How does model size affect training and inference?
- What happens when context length increases?
- How would you design an LLM pretraining pipeline?
- How would you design an LLM inference platform?
- What are the major bottlenecks when scaling an LLM from 100M to billions of parameters?
If you can answer these questions deeply — including tensor shapes, complexity, memory, and trade-offs — you have extracted the important High-level system-design value of Part 1 rather than merely memorizing its contents.
Final Takeaway
The most important lesson from Part 1 is that an LLM should not be viewed simply as a neural network that generates text.
At the model level:
Text
↓
Tokens
↓
Embeddings
↓
Transformer
↓
Logits
↓
Next Token
At the training level:
Data
↓
Cleaning
↓
Tokenization
↓
Dataset
↓
Pretraining
↓
Evaluation
At the production level:
Request
↓
Routing
↓
Scheduling
↓
Batching
↓
GPU Inference
↓
Streaming
↓
Monitoring
And at the architecture level, you must reason about all of them together.
The progression is:
Understand the objective → understand the architecture → understand the tensors → understand training → understand scaling → understand inference → design the production platform.
That is the foundation for everything that follows in the LLM system-design journey.
Key Concepts to Remember
LLM
A probabilistic model that predicts the next token based on preceding context.
GPT
A decoder-only Transformer architecture built around causal self-attention and autoregressive next-token prediction.
Training
Parallel next-token prediction followed by loss computation and parameter updates.
Inference
Sequential autoregressive token generation.
Scaling
Model capability depends on the interaction between parameters, data, compute, optimization, and architecture.
System Design
A production LLM requires much more than model weights — it needs data pipelines, model and tokenizer versioning, scheduling, batching, GPU infrastructure, evaluation, observability, reliability, and cost controls.
Next Part: Working with Text Data — how raw human language becomes tokens, token IDs, training sequences, and embeddings that a Transformer can consume.