Post Syndicated from Jeronimo De Leon original https://www.backblaze.com/blog/scaling-ai-data-storage-and-delivery-with-nvidia-aistore-and-b2/

AI workloads generate plenty of data you need to keep: source datasets, curated versions, and checkpoints you may need to revisit. A given training job only needs to access part of that data footprint, but it needs that access fast. 

Your storage decisions need to address two conflicting needs: durable, affordable retention and high-throughput delivery. Keeping your full data footprint on storage near your GPUs can get expensive. Reading directly from remote object storage introduces a different set of performance considerations. Pairing Backblaze B2 Cloud Storage with NVIDIA AIStore lets you retain datasets in B2 and serve the active working set from local storage near compute—two tiers with one source of truth.

Backblaze B2 Cloud Storage is built to be the capacity tier holding the data of record, with durable, S3-compatible storage priced for retaining large datasets. The fast tier asks for something different: applications should use it without a rewrite, it should stay consistent with the record, and it should understand how training workloads read.

NVIDIA AIStore is that fast tier. It is an open-source, scale-out object store built for AI data loading, and three of its capabilities matter most here. 

1. It attaches a remote S3 bucket as a backend, so the first read of any object pulls it from the record and later reads come from local disk while it stays cached. 
2. Supported S3 workflows can use AIStore through endpoint and compatibility configuration, without replacing their storage access code. 
3. It adds operations shaped for training: prefetch jobs that stage a dataset revision before a run starts, write-through so results land in the record as part of the same request, and GetBatch, which returns many small objects as one ordered stream instead of one request per object.

We built two samples on this split for NVIDIA workloads: video curation with NeMo Curator and robot policy training with Isaac GR00T. In both, Backblaze B2 Cloud Storage holds the durable system of record, and AIStore serves the active working set from local storage near compute.

The archive grows while the working set moves

What a job reads at any moment is a small, shifting slice of the overall dataset. Curation passes over raw data again each time its parameters change. Experiments have several engineers rereading the same revisions. Training reads one revision for many epochs and writes checkpoints back as it goes.

That is the case for a durable capacity tier that holds everything, paired with a fast tier that follows whichever subset is active. Every stage reads from B2 and publishes back to it.

Layer	Responsibility
Backblaze B2 (capacity tier)	Retain source datasets, derived assets, and checkpoints as the durable system of record.
NVIDIA AIStore with local storage (fast tier)	Fetch remote objects, retain reusable data near compute, and serve application reads.
Application and GPU memory	Decode inputs, prepare batches, and execute the model.

Reads move up from B2 through AIStore to the application. Results and checkpoints move down and land in B2.

Two ways applications reach durable data

The two sample repos cover two common access patterns: a pipeline that reads objects over S3, and a trainer that reads files from the GPU node’s local disk. In both, AIStore sits between the application and B2, and results publish back through it.

Example	Data path	What it shows
Video curation	B2 → AIStore → NeMo Curator → local results → AIStore → B2	An S3-consuming pipeline can use a nearby data tier while B2 retains inputs and outputs.
GR00T training	B2 → AIStore → verified local dataset → GR00T → checkpoints → AIStore → B2	A trainer that requires local files can use AIStore during staging without changing its data loader.

Data processing: reading object storage through the fast tier

Data processing pipelines often read straight from object storage and make several passes over the same source as parameters change, so every pass after the first can come from the fast tier. Because AIStore exposes an S3-compatible API, a pipeline that already reads over S3 can switch to it with configuration rather than a rewrite.

Our example is video curation with NVIDIA NeMo Curator, an open-source, GPU-accelerated library for preparing large datasets for generative AI training. Full runs prefetch source videos into AIStore before Curator starts. Runs with a video limit or disabled prefetch read uncached inputs on demand. Curator writes clips and Cosmos-Embed1 embeddings, and the sample publishes them through AIStore to B2. Rerunning curation can reuse source videos from AIStore while they remain cached and valid.

The client configuration and the full workflow are in the NeMo Curator video curation repo.

https://github.com/backblaze-b2-samples/aistore-b2-cosmos-video-curation

Model training: staging a dataset to local disk through the fast tier

Some trainers expect a dataset directory on local disk and a data loader you would rather not change. For them, the fast tier does its work before training starts, staging the dataset onto the GPU node, and again whenever another job or node needs the same revision.

Our example is robot policy fine-tuning with NVIDIA Isaac GR00T, an open vision-language-action (VLA) model for humanoid robot skills. Robot demonstrations arrive as many small files: per-episode tables and camera video, plus metadata. The sample stages a dataset revision from AIStore onto the GPU node and verifies it against the dataset manifest before the trainer starts, so the data loader stays untouched. Checkpoints publish back through AIStore to B2 during training. Staging the same revision again reads from AIStore, not B2.

The staging steps and the full workflow are in the GR00T robotics training repo.

https://github.com/backblaze-b2-samples/aistore-b2-gr00t-robotics-training

Five rules that keep the layers independent

These keep each tier scaling on its own terms and make results checkable.

1. Size the fast tier for concurrent working sets and expected reuse. B2 holds the total. AIStore holds the revisions jobs read at once, plus headroom so prefetched data stays resident until staging finishes. For remote buckets like B2, AIStore’s LRU eviction is on by default and starts when disk usage crosses a cluster-wide high watermark (90% by default), so set that watermark with the working set in mind.
2. Prefetch when startup predictability matters. A prefetch job stages a known revision before the application starts, so processing begins on warm data.
3. Treat dataset prefixes as write-once. Write each revision to a new prefix, such as raw/revision-001/, and never write into an existing one. AIStore can check cached objects against B2 on every read with versioning.validate_warm_get (AIStore out-of-band updates); For completed revisions whose contents never change, unique prefixes avoid the need to check each cached object for remote updates. For revisions you need to keep for audits or reproduction, B2 Object Lock prevents each locked object version from being deleted during its retention period.
4. Publish results with a verifiable manifest. Both samples write a _SUCCESS.json record as pending before upload and mark it complete, with every file’s size and SHA-256, after all objects land. When an input manifest exists, output provenance includes its hash to identify the source dataset revision.
5. Measure each phase separately. Initial loading, warm reads, processing, and publication respond to different levers. Publication can be the largest transfer in a run: checkpoints include model weights and optimizer state, so even the 20-step GR00T demo can publish tens of gigabytes. Both repos benchmark direct B2 reads against cold and warm AIStore reads, and their integration tests confirm warm reads make zero B2 GETs.

B2’s S3-Compatible API makes the fit direct: AIStore attaches the B2 bucket through its S3 backend, and Curator and standard S3 tools already speak S3 (B2 S3-Compatible API). Curator and GR00T child processes do not receive the B2 key; AIStore targets and direct-to-B2 utilities require it.

Retention for the next model, delivery for this one.

The fast tier pays off in proportion to reuse. Jobs sharing an AIStore cluster can reuse cached objects without downloading them from B2 again. That keeps heavy reuse economical, since fewer repeat reads count against B2’s free egress allowance of three times your average monthly storage. Egress to compute partners such as CoreWeave is free without limit. For a single sequential pass, streaming sharded data from B2 straight into the data loader is the simpler fit.

Plan for what that takes. AIStore is infrastructure your team operates. Kubernetes with ais-k8s is one production deployment option. Write-through adds a local write before each upload, and NVIDIA’s benchmarks show that overhead shrinking as object size grows (AIStore PUT performance). In return, results published through AIStore stay cached for the next job that reads them, so weigh that reuse against the upload cost for each workload.

The bigger opportunity is what you get to keep. Every curation pass, dataset revision, and checkpoint can be input for a model you haven’t built yet, whether that’s a new embedding model re-curating the same video or a new VLA backbone fine-tuning on the same demonstrations. Plan the capacity tier around everything your team retains, and the fast tier around the working sets its jobs share and reuse.

The post Scaling AI Data Storage and Delivery with NVIDIA AIStore and B2 appeared first on Backblaze Blog | Cloud Storage & Cloud Backup