The Invisible Bottleneck
A modern AI training cluster is a precision instrument. Thousands of accelerators, wired together across a high-speed fabric, must all advance in lockstep — each iteration waiting for its next batch of data before any useful computation can happen. Stall that data feed for even a fraction of a second, and the economics collapse: you are paying for GPU time you are not using. Storage is, quietly, the constraint that determines whether a hundred-million-dollar cluster runs at 95% utilization or 60%.
The challenge is not simply capacity — it is bandwidth, sustained in parallel, at scale. A single high-end GPU can consume training data faster than a conventional hard-drive array can deliver it. Multiply that across thousands of nodes reading simultaneously, and the storage tier faces an aggregate demand that can reach hundreds of gigabytes per second of raw throughput, all flowing east-west through the same fabric that carries gradient traffic between accelerators. "Just add more disks" misses the point entirely: spinning disks are bandwidth-constrained, not just slow in latency terms, and a larger pile of them does not linearly solve a problem rooted in parallel access patterns.
The challenge is not simply capacity — it is bandwidth, sustained in parallel, at scale.
The Data Pipeline, Stage by Stage
Think of an AI training run as a pipeline with four distinct storage moments, each with different requirements.
Ingest and raw storage. Before a model ever sees a token, petabytes of raw data — text, images, video, code — must be collected and held somewhere. This is bulk object storage: cheap, durable, highly scalable, optimized for large sequential writes and occasional reads. Cloud object stores and on-premises object platforms both fit here. Latency requirements are loose; throughput and cost per terabyte dominate.
Preprocessing. Raw data is filtered, tokenized, shuffled and written out in training-ready formats. This stage is I/O-intensive in a bursty, pipeline fashion — a CPU-heavy workload that writes large batches sequentially. Intermediate scratch storage at moderate speed handles this well; it does not need to sit on the fastest tier.
Training reads. This is the hot path, and the one that breaks naïve storage designs. During active training, every node in the cluster issues randomized reads — different samples, different offsets — simultaneously. The workload is highly parallel and largely random at the file level, even if individual reads are sequential chunks. What is required is a parallel distributed file system — Lustre and GPFS (now IBM Spectrum Scale) are the workhorses in HPC-heritage deployments; newer purpose-built systems like WekaIO target AI specifically — backed by NVMe flash rather than spinning media. These systems stripe data across many flash devices and many storage nodes, so aggregate bandwidth scales with the number of storage nodes rather than being limited by any single device. The goal is to saturate the accelerators, not the storage.
Checkpointing. As a training run progresses, the cluster must periodically snapshot the entire model state — optimizer parameters, gradients, weights — so that if a node fails, the run resumes from the last good checkpoint rather than from scratch. Modern large models can have hundreds of billions of parameters; a single checkpoint can run to many terabytes. These writes are bursty, large, and time-sensitive: the cluster is paused while the checkpoint lands. Fast shared storage and increasingly asynchronous checkpointing strategies — writing to a fast local NVMe tier first, then flushing to bulk storage in the background — are both essential to limiting the dead time.
- Stage 1 — IngestRaw data lands in bulk object storage: cheap, durable, sequential.
- Stage 2 — PrepFiltered, tokenized, shuffled into training-ready shards.
- Stage 3 — Training readsThe hot path: parallel file systems saturating thousands of accelerators.
- Stage 4 — CheckpointsThe model’s state, tens of terabytes, written out periodically.
Tiering Is the Architecture
The right storage architecture for AI is explicitly tiered: a fast, expensive, NVMe-backed parallel layer close to the compute for active training and checkpoints; a moderate-performance scratch tier for preprocessing; a deep, cheap bulk layer for raw datasets and long-term checkpoint archives. Data moves between these tiers under policy — pulled forward when a training run begins, pushed back when it ends.
Getting this wrong is expensive in both directions. Under-size the fast tier and GPUs stall; over-build it and you are paying NVMe prices for data that could live on object storage at a tenth of the cost. The storage architecture, unglamorous as it is, deserves exactly the same engineering attention as the GPU cluster it feeds.