Building LLM System

How LLMs Actually Work

deepakcse2k5@gmail.com20 min read


Large Language Models (LLMs) have transformed how we build software systems, search engines, developer tools, conversational applications, and AI-powered products.

But understanding an LLM requires much more than knowing that it is a neural network trained on text.

You need to understand:

  • What an LLM actually learns
  • Why next-token prediction works
  • How Transformers enable contextual understanding
  • Why GPT uses a decoder-only architecture
  • How data, model size, and compute interact
  • How training differs from inference
  • How model architecture affects memory and latency
  • How to design an end-to-end LLM platform
  • What trade-offs matter when scaling an LLM

This Part builds that foundation.

1. The Most Important Mental Model

If an interviewer asks:

What is a Large Language Model?

A weak answer would be:

An LLM is a neural network trained on lots of text.

A stronger definition is:

An LLM is a parameterized probabilistic model that learns the distribution of token sequences from large-scale text and uses that learned distribution to predict the next token given the preceding context.

At a high level, the process looks like this:

Text
  ↓
Tokenizer
  ↓
Token IDs
  ↓
Embeddings
  ↓
Transformer
  ↓
Contextual Representations
  ↓
Vocabulary Logits
  ↓
Probability Distribution
  ↓
Next Token
  ↓
Append Token to Context
  ↓
Repeat

Mathematically, the probability of a sequence can be represented as:

P(x₁, x₂, ..., xₙ)
=
∏ P(xₜ | x₁, ..., xₜ₋₁)

The key idea is that the model does not directly learn:

“How to answer questions.”

At its fundamental level, it learns:

“Given this context, what token is likely to come next?”

Modern LLMs can process and generate language in ways that appear coherent and contextual. However, this should not be interpreted as human-like consciousness or comprehension.

Pretraining vs Instruction Following

This distinction becomes extremely important later:

Pretraining
    ↓
Next-Token Prediction
    ↓
General Language Capability
    ↓
Instruction Tuning
    ↓
Instruction Following
    ↓
Application-Specific Behavior

Therefore:

Pretraining ≠ Instruction Following

2. Why Did LLMs Become So Powerful?

Modern LLM capabilities are closely connected to advances in deep learning and the ability to train models on enormous quantities of text.

For an expert-level discussion, think about three major scaling dimensions:

LLM Capability
                         │
          ┌──────────────┼──────────────┐
          ↓              ↓              ↓
     Model Scale      Data Scale    Compute Scale
          │              │              │
      Parameters        Tokens      GPU/TPU Hours
      Layers            Quality     Distributed Training
      Hidden Size       Diversity   Optimized Kernels

A weak answer is:

More parameters make the model smarter.

A better answer is:

Model capability depends on the interaction between model capacity, training-data quantity and quality, optimization, and compute. Increasing parameters without sufficient high-quality data or compute does not automatically produce proportional gains.

What Happens If You Double the Model Size?

Do not immediately say:

Performance doubles.

Instead, discuss:

  • Parameter count
  • Training tokens
  • Compute budget
  • GPU memory
  • Optimizer state
  • Communication overhead
  • Training time
  • Inference latency
  • Inference memory
  • Serving cost
  • Scaling behavior

This is where an LLM discussion starts becoming a systems discussion.

3. Why Is Next-Token Prediction Enough to Learn Useful Behavior?

This is one of the most important questions in Part 1.

Suppose the training corpus contains:

Paris is the capital of France.

The model learns relationships such as:

Paris
  → is
Paris is
  → the
Paris is the
  → capital
Paris is the capital
  → of

But across billions of examples, the model also encounters patterns such as:

Question:
What is the capital of France?
Answer:
Paris

It may also encounter:

Translate:
Hello → Bonjour

or:

def fibonacci(n):
    ...

or:

Article → Summary

The model is not necessarily given a separate training objective for every possible task.

Instead, these patterns exist within the training distribution.

The model learns statistical relationships between tokens and contexts. When a prompt activates a particular pattern, generation can behave like task execution even though the underlying objective remains next-token prediction.

The book describes capabilities such as translation emerging from this process.

Expert Interview Answer

The next-token objective is deceptively general because the training corpus contains many latent task structures. The model learns statistical relationships between tokens and contexts. When a prompt activates a particular pattern, generation can behave like task execution even though the underlying objective remains next-token prediction.

4. LLM vs Traditional Machine Learning

Another common interview question is:

How is an LLM different from traditional machine learning?

Traditional ML often follows a pipeline like:

Problem
  ↓
Feature Engineering
  ↓
Task-Specific Model
  ↓
Prediction

For example:

Email
  ↓
TF-IDF Features
  ↓
Classifier
  ↓
Spam / Not Spam

Traditional machine-learning systems frequently depend on humans identifying and engineering useful features.

An LLM follows a very different paradigm:

Raw Text
  ↓
Tokenizer
  ↓
Large Neural Network
  ↓
Learned Representations
  ↓
Next-Token Prediction

One foundation model can support many downstream tasks.

That is a major architectural shift.

5. LLM Applications

Part 1 discusses applications including:

  • Text generation
  • Translation
  • Sentiment analysis
  • Question answering
  • Contextual text generation

For system-design interviews, however, it is more useful to classify applications by workload.

A. Generation

Prompt
  ↓
LLM
  ↓
Generated Text

Examples:

  • Chat
  • Coding
  • Writing
  • Summarization

B. Classification

Text
  ↓
LLM
  ↓
Class

Examples:

  • Spam detection
  • Sentiment analysis
  • Intent classification

C. Transformation

Input Text
  ↓
LLM
  ↓
Transformed Text

Examples:

  • Translation
  • Rewriting
  • Summarization

D. Reasoning and Tool Use

A broader system-design extension is:

User
  ↓
LLM
  ↓
Tool / Retrieval
  ↓
LLM
  ↓
Answer

This goes beyond the basic scope of Part 1 and should therefore be treated as an interview extension rather than a direct claim of the Part itself.

6. The LLM Lifecycle

One of the most important system-design lessons from Part 1 is that building an LLM is not simply:

Train a neural network.

Think about the complete lifecycle:

DATA
  ↓
Data Collection
  ↓
Data Cleaning
  ↓
Tokenization
  ↓
Dataset Creation
  ↓
PRETRAINING
  ↓
Base LLM
  ↓
Fine-Tuning
  ↓
Instruction Following
  ↓
Evaluation
  ↓
Model Registry
  ↓
Deployment
  ↓
Inference
  ↓
Monitoring / Feedback

This is where LLM theory transitions into system design.

Designing an LLM-Building Platform

If an interviewer asks:

Design a system for building an LLM.

Your architecture should contain at least:

                  Data Pipeline
                         │
              ┌──────────┼──────────┐
              ↓          ↓          ↓
          Ingestion   Cleaning   Deduplication
                         │
                     Filtering
                         │
                    Tokenization
                         ↓
                  Training Pipeline
                         │
                  ┌──────┴──────┐
                  ↓             ↓
             Checkpoints     Evaluation
                  │             │
                  └──────┬──────┘
                         ↓
                      Registry
                         ↓
                      Serving
                         │
        ┌────────┬────────┼────────┬────────┐
        ↓        ↓        ↓        ↓        ↓
     Routing  Batching  Inference Caching Monitoring

This architecture demonstrates that an LLM is not merely a model artifact.

It is a complete data, training, evaluation, deployment, and serving system.

7. Transformer Architecture

The Transformer architecture is probably the single most important technical topic in Part 1.

The original Transformer architecture contains:

Encoder + Decoder

The encoder converts input text into contextual numerical representations, while the decoder generates output text.

Self-attention allows the model to capture relationships between tokens.

Consider:

“I went to the bank to deposit money.”

The representation of:

bank

should depend strongly on:

deposit money

rather than only on nearby words.

That is the core motivation behind attention.

8. Encoder vs Decoder vs Encoder-Decoder

This is a very common system-design interview question.

| **Architecture** | **Main Purpose**                    |
| ---------------- | ----------------------------------- |
| Encoder-only     | Understand / represent input        |
| Decoder-only     | Generate continuation               |
| Encoder-decoder  | Transform one sequence into another |

Encoder-Only

Input
  ↓
Encoder
  ↓
Representation
  ↓
Classifier

Good for:

  • Classification
  • Embeddings
  • Representation learning

Encoder-Decoder

Input
  ↓
Encoder
  ↓
Context
  ↓
Decoder
  ↓
Output

Good for:

  • Translation
  • Sequence-to-sequence generation

Decoder-Only

Prompt
  ↓
Decoder
  ↓
Next Token
  ↓
Next Token
  ↓
Next Token

This is the GPT-style architecture emphasized in the book.

9. Why Decoder-Only GPT?

The original Transformer was designed with encoder and decoder blocks, particularly for sequence-to-sequence tasks such as translation.

GPT simplifies this into:

Decoder-Only
     +
Causal Self-Attention
     +
Next-Token Prediction

The decoder-only architecture provides a simple universal interface:

Context → Continuation

Many different tasks can be represented as text generation.

For example:

Translate:
English: Hello
French:

Or:

Question:
What is 2 + 2?
Answer:

Or:

Summarize:
<document>
Summary:

This makes the architecture remarkably general.

10. GPT Architecture — Know This Diagram

For an interview, remember the following architecture:

Token IDs
    │
    ▼
Token Embeddings
    +
Positional Embeddings
    │
    ▼
┌───────────────────────┐
│ Transformer Block 1   │
├───────────────────────┤
│ Causal Self-Attention │
│ Feed Forward          │
│ Residual + Norm       │
└───────────────────────┘
    │
    ▼
Transformer Block 2
    │
    ▼
    ...
    │
    ▼
Transformer Block N
    │
    ▼
Final LayerNorm
    │
    ▼
Output Projection
    │
    ▼
Vocabulary Logits
    │
    ▼
Softmax
    │
    ▼
Next Token

GPT models are built from repeated Transformer blocks.

The core architecture remains the same while model size can change through:

  • Embedding dimension
  • Number of layers
  • Number of attention heads

11. Tensor Shapes — Expert Coding Interview Level

Suppose:

B  = Batch Size
T  = Sequence Length
D  = Embedding Dimension
V  = Vocabulary Size
H  = Number of Attention Heads
Dh = D / H

The tensor flow is:

Token IDs
[B, T]
   │
   ▼
Embedding
[B, T, D]
   │
   ▼
Transformer × N
[B, T, D]
   │
   ▼
Linear Projection
[B, T, V]

The final vocabulary logits have shape:

[B, T, V]

Why [B, T, V] Instead of [B, V]?

Because during training, the model predicts the next token at every sequence position simultaneously.

Therefore, we need a vocabulary distribution for every one of the T positions.

During autoregressive generation, we typically use the logits corresponding to the final position.

12. Model Scaling

The book introduces GPT-2 configurations ranging from approximately 124M to 1.558B parameters.

The important design insight is that model size is influenced by:

Embedding Dimension
        +
Number of Layers
        +
Attention Heads
        +
Feed-Forward Dimensions
        +
Vocabulary

Increasing model size affects both training and inference.

Training

More Parameters
      ↓
More Compute
      ↓
More GPU Memory
      ↓
More Training Time

Inference

More Parameters
      ↓
More Weight Memory
      ↓
More Compute / Token
      ↓
Potentially Higher Latency

Distributed Systems

Large Model
    ↓
Single GPU Insufficient
    ↓
Model Parallelism
    ↓
Communication Overhead

Therefore, increasing model size is not simply a capability decision.

It is also an infrastructure decision.

13. Parameter Count — Coding and Design Interview

Suppose an interviewer asks:

How would you estimate the size of a GPT model?

Reason component by component.

Embedding Parameters

Approximately:

V × D

where:

  • V = vocabulary size
  • D = embedding dimension

Attention Parameters

The major projections are:

Q
K
V
Output

For a basic dense formulation, these contribute approximately:

4D²

per Transformer block.

Feed-Forward Parameters

If:

Dff = 4D

then the feed-forward component contributes approximately:

2D × Dff

which becomes:

8D²

Therefore, a rough block-level estimate is:

≈ 12D²

plus:

  • Normalization parameters
  • Biases
  • Other architecture-specific parameters

The exact architecture can differ, so always state your assumptions during an interview.

14. Why Large Datasets Matter

Part 1 emphasizes training LLMs on vast quantities of text.

Large-scale training allows models to capture deeper contextual information and subtleties of human language.

However:

Data quantity is not enough. Data quality is equally important.

Think about the data pipeline as:

Raw Web
   ↓
Filtering
   ↓
Deduplication
   ↓
Quality Scoring
   ↓
Language Identification
   ↓
PII / Safety Filtering
   ↓
Contamination Checks
   ↓
Training Corpus

Consider this interview question:

Would you rather have 10T low-quality tokens or 2T high-quality tokens?

Do not answer automatically.

A mature answer is:

I would evaluate data quality, diversity, duplication, domain coverage, contamination, and the target model’s scaling regime. Token count alone is not a sufficient measure of training value.

That demonstrates architectural maturity.

15. Data Quality Becomes a System-Design Problem

Imagine you have:

10 PB Web Corpus

and need to process it for LLM training.

A High-level design might look like:

Object Storage
     ↓
Distributed ETL
     ↓
Normalization
     ↓
Language Detection
     ↓
Quality Filtering
     ↓
Deduplication
     ↓
PII / Safety Filtering
     ↓
Tokenization
     ↓
Sharded Dataset
     ↓
Distributed Training

At this point, interviewers can ask:

  • How do you deduplicate?
  • How do you prevent data leakage?
  • How do you shard the dataset?
  • How do you resume after a worker failure?
  • How do you prevent training-data contamination of evaluation sets?

These are natural system-design extensions of Part 1.

16. Training vs Inference

This distinction is essential.

Training

Input Tokens
     ↓
Transformer
     ↓
Logits
     ↓
Loss
     ↓
Backpropagation
     ↓
Gradient
     ↓
Weight Update

Inference

Prompt
  ↓
Transformer
  ↓
Logits
  ↓
Select Next Token
  ↓
Append Token
  ↓
Repeat

GPT generation repeatedly predicts a subsequent token and appends it to the input context.

The Critical Difference

Training:

Parallel Across Positions

Inference:

Autoregressive / Sequential

This difference drives much of modern LLM infrastructure.

17. Why Inference Is a Systems Problem

At first glance, serving an LLM seems simple:

User → LLM

Production inference is much more complicated:

Request
   ↓
Tokenizer
   ↓
Queue
   ↓
Scheduler
   ↓
GPU
   ↓
Prefill
   ↓
Decode
   ↓
Streaming

Now you need to solve:

  • Batching
  • GPU utilization
  • Memory
  • KV cache
  • Latency
  • Throughput
  • Concurrency
  • Streaming
  • Cancellation
  • Timeouts
  • Model routing
  • Autoscaling

These topics go beyond the narrow implementation covered in Part 1, but they are exactly where Part 1 becomes relevant to expert LLM system-design interviews.

18. A High-Level System-Design Question

Consider the interview question:

Design a platform for serving an LLM to 100,000 concurrent users.

Do not immediately start drawing boxes.

First, clarify the workload.

Step 1 — Understand the Workload

Ask:

  • What is the model size?
  • What is the context length?
  • What is the requests-per-second requirement?
  • What is the average input-token count?
  • What is the average output-token count?
  • What is the peak traffic?
  • What is the latency SLO?
  • Is streaming required?
  • What availability is required?
  • Is tenant isolation required?

Step 2 — Design the Architecture

A high-level architecture could look like:

API Gateway
                         │
                         ▼
                 Auth / Rate Limit
                         │
                         ▼
                   Load Balancer
                         │
                         ▼
                   Request Router
                         │
                         ▼
                     Scheduler
                         │
              ┌──────────┴──────────┐
              ▼                     ▼
          GPU Pool A            GPU Pool B
            Model A               Model B
              │                     │
              └──────────┬──────────┘
                         ▼
                     Streaming
                         │
                         ▼
                       Client

Then discuss:

  • Model Registry
  • Tokenizer Registry
  • Prompt Registry
  • Metrics
  • Tracing
  • Logging
  • Evaluation
  • Canary Deployment
  • Autoscaling
  • Cost Controls

That is the difference between:

“I know Transformers.”

and:

“I can design an LLM platform.”

19. Expert Coding Interview Questions from Part 1

Q1. Explain an LLM in one minute.

Strong Answer

A decoder-only LLM such as GPT is an autoregressive Transformer that models the conditional probability of the next token given previous tokens. Text is tokenized into IDs, mapped to embeddings, processed through repeated Transformer blocks using causal self-attention and feed-forward transformations, and projected into vocabulary logits. During training, next-token cross-entropy drives parameter updates; during inference, the model generates tokens autoregressively.

Q2. What is the shape of the GPT output?

[B, T, V]

where:

  • B = batch size
  • T = sequence length
  • V = vocabulary size

Q3. Why doesn’t GPT output a single token during training?

Because training predicts every next-token position simultaneously.

For example:

Input:
I love machine learning
Targets:
love machine learning <next>

Conceptually:

I
  → love
I love
  → machine
I love machine
  → learning

Therefore, one forward pass generates multiple training predictions.

20. Why Does GPT Need Positional Information?

Self-attention operates on token representations and does not inherently encode sequence order.

Without positional information:

A B C

and:

C B A

would not have sufficient explicit positional distinction.

GPT models use positional information by adding positional embeddings to token embeddings.

This allows the model to distinguish tokens based on their positions in the sequence.

21. Why Can GPT Perform Translation If It Wasn’t Specifically Trained for Translation?

This is one of the best Part 1 interview questions.

A strong answer is:

Because the model learns statistical relationships across large multilingual corpora. If the training data contains translation-like patterns and aligned linguistic structures, next-token prediction can implicitly learn those mappings. This capability can emerge even though translation is not the fundamental training objective.

The important concept is that the model’s training distribution contains many different linguistic patterns.

22. What Is the Difference Between GPT and the Original Transformer?

Original TransformerGPTEncoder + DecoderDecoder-onlyDesigned for sequence-to-sequence tasksDesigned around autoregressive generationUses encoder and decoder blocksUses decoder blocksExample: translationGeneral text generation

Conceptually:

Original Transformer

Input
  ↓
Encoder
  ↓
Decoder
  ↓
Output

GPT

Input Context
  ↓
Decoder Blocks
  ↓
Next Token

The book uses GPT as the primary architecture because its decoder-only structure is central to the implementation that follows.

23. What Happens When Context Length Increases?

At the Part 1 level:

More Tokens
    ↓
More Computation
    ↓
More Memory

The deeper explanation comes later when studying self-attention, where the relationship between sequence length and attention computation becomes clearer.

A good interview answer is:

The architectural consequence of increasing context length is that the system needs more computation and memory. The exact attention complexity becomes clearer when we study self-attention.

This is better than jumping ahead without explaining the dependency.

24. How Would You Build a GPT Model From Scratch?

Give the interviewer this roadmap:

1. Prepare Corpus
       ↓
2. Tokenizer
       ↓
3. Token IDs
       ↓
4. Dataset / Context Windows
       ↓
5. Token Embeddings
       ↓
6. Positional Embeddings
       ↓
7. Transformer Blocks
       ↓
8. Final Normalization
       ↓
9. Vocabulary Projection
       ↓
10. Next-Token Loss
       ↓
11. Backpropagation
       ↓
12. Optimization
       ↓
13. Evaluation
       ↓
14. Generation

This is the architectural roadmap established in Part 1 and expanded by the subsequent parts.

25. The Most Important High-Level Trade-Offs

At a High level, you should be able to discuss these trade-offs without prompting.

Design DecisionTrade-OffBigger modelBetter capacity vs higher costMore training dataBetter coverage vs higher data-processing costLonger contextMore context vs compute/memoryLarger batchBetter utilization vs memoryMore layersHigher capacity vs latencyLarger hidden dimensionHigher capacity vs computeMore attention headsRepresentation flexibility vs computeFull fine-tuningFlexibility vs costSmaller modelLower cost vs lower capabilityHigher precisionNumerical fidelity vs memoryQuantizationMemory/latency gains vs potential quality lossShared servingBetter utilization vs isolationDedicated servingBetter isolation vs utilization

These trade-offs are critical when moving from model-level thinking to platform-level architecture.

26. What I Would Expect You to Say

Explain how an LLM works.

Your answer should progress through multiple levels.

Level 1 — Objective

It predicts the next token.

Level 2 — Architecture

It uses a decoder-only Transformer with causal self-attention.

Level 3 — Tensor Flow

[B,T]
  →
[B,T,D]
  →
Transformer Blocks
  →
[B,T,V]

Level 4 — Training

We optimize next-token cross-entropy over large-scale text.

Level 5 — Scaling

Increasing model size, data, and compute can change capability, but also increases memory and training cost.

Level 6 — Inference

Generation is autoregressive, so serving becomes a scheduling, memory, throughput, and latency problem.

Level 7 — Production

I would design the platform around tokenizer and model versioning, request scheduling, batching, GPU memory, evaluation, observability, reliability, and cost.

That progression distinguishes an expert answer from a textbook definition.

27. Part 1 — What You Should Memorize

If you have an interview tomorrow, remember this:

LLM
│
├── Objective
│   └── Next-Token Prediction
│
├── Input
│   └── Token IDs [B,T]
│
├── Representation
│   └── Embeddings [B,T,D]
│
├── Architecture
│   └── Decoder-only Transformer
│
├── Core Mechanism
│   └── Causal Self-Attention
│
├── Output
│   └── Logits [B,T,V]
│
├── Training
│   └── Cross-Entropy + Backpropagation
│
├── Scale
│   ├── Parameters
│   ├── Data
│   └── Compute
│
├── Capabilities
│   ├── Generation
│   ├── Translation
│   ├── QA
│   └── Classification
│
└── Production
    ├── Inference
    ├── Latency
    ├── Throughput
    ├── Memory
    ├── Batching
    ├── Evaluation
    └── Cost

Part 1 is intentionally foundational rather than a deep mathematical treatment. It establishes the GPT architecture and the major stages that the subsequent parts explore in greater detail.

The 10 Questions I Would Practice

If you are preparing for a AI/ML system-design interview, make sure you can answer these deeply:

  1. What exactly does an LLM learn during pretraining?
  2. Why is next-token prediction sufficient to learn useful capabilities?
  3. Why did the industry move toward decoder-only architectures?
  4. Encoder vs decoder vs encoder-decoder — when would you choose each?
  5. Explain the complete tensor flow through GPT.
  6. How does model size affect training and inference?
  7. What happens when context length increases?
  8. How would you design an LLM pretraining pipeline?
  9. How would you design an LLM inference platform?
  10. What are the major bottlenecks when scaling an LLM from 100M to billions of parameters?

If you can answer these questions deeply — including tensor shapes, complexity, memory, and trade-offs — you have extracted the important High-level system-design value of Part 1 rather than merely memorizing its contents.

Final Takeaway

The most important lesson from Part 1 is that an LLM should not be viewed simply as a neural network that generates text.

At the model level:

Text
  ↓
Tokens
  ↓
Embeddings
  ↓
Transformer
  ↓
Logits
  ↓
Next Token

At the training level:

Data
  ↓
Cleaning
  ↓
Tokenization
  ↓
Dataset
  ↓
Pretraining
  ↓
Evaluation

At the production level:

Request
  ↓
Routing
  ↓
Scheduling
  ↓
Batching
  ↓
GPU Inference
  ↓
Streaming
  ↓
Monitoring

And at the architecture level, you must reason about all of them together.

The progression is:

Understand the objective → understand the architecture → understand the tensors → understand training → understand scaling → understand inference → design the production platform.

That is the foundation for everything that follows in the LLM system-design journey.

Key Concepts to Remember

LLM

A probabilistic model that predicts the next token based on preceding context.

GPT

A decoder-only Transformer architecture built around causal self-attention and autoregressive next-token prediction.

Training

Parallel next-token prediction followed by loss computation and parameter updates.

Inference

Sequential autoregressive token generation.

Scaling

Model capability depends on the interaction between parameters, data, compute, optimization, and architecture.

System Design

A production LLM requires much more than model weights — it needs data pipelines, model and tokenizer versioning, scheduling, batching, GPU infrastructure, evaluation, observability, reliability, and cost controls.

Next Part: Working with Text Data — how raw human language becomes tokens, token IDs, training sequences, and embeddings that a Transformer can consume.