TPU Deep Dive
Ten chapters over five days, plus a synthesis chapter, on TPU hardware, interconnect, sharding, inference and the software stack, written for engineers who already know GPUs.
0 of 11 chapters readQuiz average none yetTime spent 0 min0 cards due, 222 new
Day 1
Rooflines and the chip
Day 2
The network and the model
Day 3
Sharding and training
Day 4
Inference
Day 5
Compiler, kernels, profiling
First block
Day 1, Chapter 1TPU lineage and roofline thinkingQuiz not taken55 min plannedDay 2, Chapter 3ICI, OCS, slices and podsQuiz not taken55 min plannedDay 3, Chapter 5Sharded matmuls and collectives on a torusQuiz not taken55 min plannedDay 4, Chapter 7Inference rooflinesQuiz not taken55 min plannedDay 5, Chapter 9From Python to the MXUQuiz not taken55 min plannedSecond block
Day 1, Chapter 2Anatomy of a TensorCoreQuiz not taken55 min plannedDay 2, Chapter 4Transformer accounting on TPUQuiz not taken55 min plannedDay 3, Chapter 6Choosing a mesh for trainingQuiz not taken55 min plannedDay 4, Chapter 8The TPU serving stackQuiz not taken55 min plannedDay 5, Chapter 10Pallas, XProf and a design reviewQuiz not taken55 min plannedSynthesis
Chapter 11, after Day 5TPU and GPU architecture, side by sideQuiz not takenEach cell has three marks: chapter read, quiz passed at 80% or more, teach-back saved.
Reference
- Sourcesevery citation, with access date
- Your dataexport, import, flashcard export