Post Syndicated from David Johnson original https://www.backblaze.com/blog/neoclouds-your-storage-layer-just-got-a-public-grade/

SemiAnalysis published ClusterMAX 3.0, and storage helps decide whether a neocloud lands in Platinum, Gold, Silver, Bronze, or the new Participation Ribbon tier. Your customers can read that grade before they take your call.

Why it matters: Only 19 of the 77 neoclouds tested in depth earned a Medallion rating, and plenty that held Silver or Gold last round lost it. For most of them, the storage grade doesn’t reflect a decision anyone made. It reflects a default.

Storage decides whether you get rated at all

Storage isn’t something the evaluation gets to eventually. You can’t be rated without it.

• Every provider has to supply at least 10TB of high-performance file storage and 10TB of S3-compliant object storage before testing begins.
• Then storage is tested twice: once for speed, once for whether that speed holds through 15 minutes of sustained mixed traffic.
• Some providers lost nearly 40% of their bandwidth in the second test.

Every neocloud is in one of three states

Size isn’t what matters here. AWS and Azure both had storage problems this round. TensorWave, far smaller than either, had storage SemiAnalysis called exceptional. The difference is whether anyone deliberately decided what storage is for, what it runs on and who owns it.

1. Not decided yet. You rent GPUs by the hour. Customers bring data in, run a job, and take the results home. Storage is scratch space, and there’s nothing to migrate if you change your mind.
2. Decided by default. Someone stood something up once, and nobody’s job includes going back to check it. It was fine at the time, and it has stayed “fine at the time” for two years.
3. Decided deliberately. A person or a contract is responsible for whether it still works, and you pay for that one way or another.

Most neoclouds are in the second state and think they’re in the first. The first state is fine. The second is the one that gets graded.

Why the second state is so common

Decision by default isn’t negligence. If you sell compute by the hour, customers manage their own data, and nobody has to decide what your storage offering is. GPUs are what customers evaluate, so that’s where the attention goes.

But that changes when customers start leaving their data on your platform. Now you have to decide where it lives, who can reach it, what happens when a machine dies and what to charge for it. That decision usually arrives while the business is growing too fast to stop and make it.

What “decision by default” looks like from the outside

SemiAnalysis found four recurring storage failures:

• Storage servers not enforcing RBAC correctly on volumes
• Storage misconfigured on the overlay network, so customers could reach the underlay
• Backend storage sharing a network with no isolation
• A BlueField data processing unit (DPU) left in its default host-trusted mode

The DPU case shows the pattern most clearly. 

A DPU is a small computer built into the network card. In NVIDIA’s reference design, it handles networking, storage, and security work, including the rules that keep tenants apart. Out of the box, it trusts anyone with root access on the server. That’s fine in NVIDIA’s model, where the server admin is trusted. But in a GPU cloud, the customer often has root. That means the customer can reach the DPU itself, which is the component running the rules that separate them from everyone else on that hardware. Anyone who gets that far is in a position to weaken or turn off the boundary they’re supposed to be sitting behind. 

SemiAnalysis found at least one host where that path was open. Nobody chose that setting. It’s how the card ships, and nobody went back to change it.

Read the reviews in tier order and the pattern holds: the further down the tiers, the less anyone has looked at storage.

• At the top, the complaints are about ergonomics. CoreWeave’s only storage note was a wish for one-click local GPUDirect Storage, which SemiAnalysis itself called nitpicky.
• In the middle, there are configuration gaps. Azure’s default StorageClass couldn’t attach, and a model load from NFS timed out because no parallel filesystem was attached to the Slurm cluster.
• At Bronze and below, storage often isn’t set up at all. One Seattle cluster had no shared filesystem to test. Another had no RWX StorageClass, so there was nowhere to put /shared. At a third, NVMe existed but nothing exposed it, and PVCs hung in Pending.

One case sums it up best. A provider’s default StorageClass provisioned and bound, then left the pod stuck in ContainerCreating. Meanwhile, the Lustre tier on the same worker nodes hit 91.7 GB/s aggregate read across 32 clients. The hardware was fast, but anyone who used the default would have concluded the storage didn’t work.

Customers will ask these questions at any size

The tenant-isolation failures SemiAnalysis found hit providers big enough to be testing GB300 racks:

• A shared Kubernetes control plane exposing other tenants’ namespaces
• A default InfiniBand partition key surfacing 532 hostnames
• A Grafana dashboard that separated tenants on screen while the key behind it read every tenant’s logs

That reads like someone else’s problem, but it isn’t. It’s a preview of the diligence review you’ll get the first time a serious customer evaluates you. 

The exposed customers in the report included banks, telcos, universities, research institutions, AI labs, and a national intelligence agency. Buyers like these ask:

• What isolates my workloads from other tenants: containers, VMs, or dedicated nodes?
• Is the control plane shared?
• Who can read my volumes, and what enforces that: the storage layer or the interface in front of it?
• Is backend storage on its own isolated network?
• Can anyone at your company reach my logs and monitoring data without my permission?
• Are machines reprovisioned from scratch between customers, or recycled?

It’s important to have answers to these questions before a customer asks, not while they’re waiting.

Three ways out of the second state

Getting out of the second state means picking one of three paths, and committing to it:

• Build on Ceph. You get full control over the design and no dependence on anyone else’s release schedule. The cost is a permanent, dedicated headcount line: Ceph specialists aren’t interchangeable with your compute engineers.
• Build your own hardware. You design, rack, and run your own storage servers, shaped to your customers’ workloads, and the revenue stays in-house. The cost is procurement, refresh cycles, and capital committed years ahead of demand you can’t forecast.
• Partner. Storage works and stays working without anyone on your team owning it, and your engineers stay on compute. What you’re trading is some control: a partner’s pricing, roadmap and deprecation timelines are decisions you’re part of rather than decisions you make alone, and when a customer’s job stalls on storage, they still call you. The right partner expects those questions early and answers them plainly. It’s worth asking before you sign.

All three are legitimate. The option most neoclouds are actually running is none of them.

Decide before the grade, not after

Every provider SemiAnalysis tested had storage. The ones who had decided what it was for, and kept checking that it worked, had an answer when customers asked. The rest had whatever the default gave them, and now that’s published.

If you rent GPUs by the hour today, your customers will eventually stop taking their data home. Decide on your storage before that happens, while there’s less data to migrate, no customers waiting on an answer, and no grade already in front of your prospects.

Read the full report here. And visit us to explore how Backblaze partners with neoclouds to ensure more open, AI-ready storage architectures.

The post Neoclouds: Your Storage Layer Just Got a Public Grade appeared first on Backblaze Blog | Cloud Storage & Cloud Backup

Post Syndicated from David Johnson original https://www.backblaze.com/blog/why-the-best-ai-clouds-dont-run-on-flash-alone/

Neoclouds are built around GPUs, and GPUs can only run as fast as the data coming in. At the speeds AI training demands, conventional cloud storage solutions can’t keep up. 

That’s where flash storage comes in. Flash is fast and reliable, purpose-built for workloads that need data delivered at extreme speed and low latency. But much of the storage work that surrounds active training—saving checkpoints, storing datasets, keeping finished models—doesn’t need flash-grade speed (or flash-grade prices).

That’s an issue for neoclouds. Their infrastructure is built around flash. So for storage that doesn’t need flash, they often point customers to hyperscaler object storage solutions instead. It’s cheaper, it works, and it keeps the neocloud focused on what it does best.

The problem is data gravity: the more data you have stored somewhere, the harder it becomes to leave. Hyperscalers make this worse by design. They let you put data in for free but charge prohibitively expensive egress fees to take it out. So once your data is entrenched on a hyperscaler’s platform, the cost of leaving keeps climbing. And wherever your data lives, that’s where you tend to buy more services. So while the neocloud keeps getting paid for GPU computing, the hyperscaler quietly takes over the broader account. What started as a sensible storage decision ends up with the hyperscaler owning everything except the GPU contract.

This article explains how a sensible storage decision can slowly hand a customer to a hyperscaler, and how a tiered storage architecture, with flash where it belongs and cheaper high-capacity storage everywhere else, lets neoclouds compete for the whole account.

Each lifecycle stage carries a different storage profile

To see why flash alone isn’t the answer, it helps to walk through what each stage of the AI lifecycle actually needs from storage.

• It starts with data ingestion. Before training begins, raw datasets have to be cleansed, labeled, and prepared. This stage needs high-capacity storage that can move large volumes of data efficiently. Speed matters, but not to the degree that flash is necessary.

• Active training is the stage where flash earns its place. Data moves through several storage tiers before it reaches the GPU, but at that final step, the model needs it delivered continuously and at very high speed. Only flash is fast enough to keep thousands of GPUs running without interruption.

• Checkpointing runs throughout training. Because AI training is a synchronous process with every node in the cluster working in lockstep, a single component failure can bring the entire training run to a halt. And at the scale of a large modern cluster, failures are not rare. The failure rate also scales exponentially with cluster size. Checkpoints are the solution: frequent snapshots of the model’s current state that allow training to resume from where it left off. At large scale, a checkpoint may need to capture 15TB of data in under five seconds, repeated every few minutes, around the clock. This stage needs storage that is fast enough to keep up with that pace, and reliable enough that nothing gets lost.

• Once training finishes, the outputs—model weights, logs, and metadata—go into longer-term storage. Fine-tuning experiments and additional training runs add more over time. This stage needs capacity and durability, not speed.

• Finally, inference: serving a deployed model requires repeated access to model weights on demand. The data volumes are far smaller than training, and while latency-sensitive deployments may still benefit from flash, the storage demands are far less intensive.

Flash storage is purpose-built for only one of these stages (active training). At every other stage, what’s needed is a cost-effective, high-capacity storage tier, which is precisely what neoclouds are currently sending to hyperscalers. (NVIDIA’s certified storage program validates storage solutions separately for each of these stages, recognizing that a single storage architecture can’t serve all of them well.)

Flash is the wrong tool for most of the job

Flash storage is expensive because it’s built for speed. But you pay for that speed whether you need it or not. Using flash to store datasets, checkpoints, and finished model artifacts is like shipping all of your mail overnight when most of it could go by ground; you’re paying a premium for a capability you don’t need.

And there’s a second problem beyond the cost. Flash that’s tied up storing checkpoints and datasets isn’t available for GPUs, and GPUs are what neoclouds sell. Every terabyte of flash used in the wrong place is capacity taken away from the core product.

The economics have always favored a tiered approach: fast storage where speed is needed, cheaper high-capacity storage everywhere else. In fact, Google, Meta, and Microsoft have run tiered storage architectures for years (Meta’s engineering team describes their approach in four explicit tiers, each selected for the job it does best).

But building out a full storage tier below flash is a significant undertaking, and most neoclouds have understandably stayed focused on their core GPU product. With SSD prices up 257% in under a year, that gap has become too expensive to ignore.

How a tiered stack maps to the lifecycle

The solution is to follow the same logic the hyperscalers have used for years: match the storage to the workload, not the other way around.

For active training, flash is still the answer. VAST, WEKA, and DDN handle this tier with fast, GPU-adjacent storage that keeps training running at full speed.

For everything else, B2 Neo can fill the gap. Datasets live here before training starts. Checkpoints write here continuously during training. Finished model weights and records accumulate here over time. The storage is fast and durable, but priced for capacity workloads rather than flash workloads. B2 Neo connects directly to the hot tier, so data moves between them without a detour through a hyperscaler.

The best part is the neocloud doesn’t have to build any of this. B2 Neo is available as a white-label product, deployed under the neocloud’s own brand. The customer only sees one seamless platform; their data stays on the neocloud’s infrastructure from raw dataset to deployed model and never touches a hyperscaler. 

The full picture

Flash is the right storage for one stage of the AI lifecycle. At every other stage, it’s an expensive tool for a job that doesn’t need it.

Neoclouds that only serve the active training tier leave a gap, and customers often fill that gap by sending their data to a hyperscaler. Once it’s there, egress fees keep it there. The neocloud ends up handing the broader customer relationship to whoever holds the data.

The good news is that closing the gap doesn’t require building anything from scratch. The tiered architecture that Google and Meta run at scale is available to neoclouds today as white-label infrastructure. 

In this market, the GPU contract is just the beginning. Keeping the data is what keeps the customer. 

Hit a wall trying to scale your storage infrastructure? B2 Neo gets you the rest of the way there, no rebuild required. Let’s talk.

The post Why the Best AI Clouds Don’t Run on Flash Alone appeared first on Backblaze Blog | Cloud Storage & Cloud Backup

Post Syndicated from Maddie Presland original https://www.backblaze.com/blog/why-your-gpu-is-sitting-idle-the-data-pipeline-problem-no-one-talks-about/

At NVIDIA’s GTC conference, Adobe’s CTO Ely Greenfield walked the audience through the company’s three-year journey building frontier generative AI models from scratch. And the AI training data pipeline required to keep thousands of GPUs productive.

Along the way, he showed a profiler readout from their early training runs—a visualization of exactly how much time each GPU was spending on actual computation versus sitting idle. It revealed that roughly two-thirds of GPU time was spent simply waiting for data. And that idle time had a price tag. 

“If we were putting a million dollars into training,” he told the room, “that was $600,000 we were burning away on GPUs sitting and doing nothing.” In other words, roughly sixty cents of every dollar spent on GPU compute was being wasted.

Greenfield’s team traced the waste to two culprits, neither of which is unique to Adobe: whether data can reach your GPUs fast enough, and whether the work it represents is distributed evenly once it arrives.

One culprit is a storage problem. The other isn’t, but you won’t be able to fix it until you’ve solved the first one.

Culprit #1: Storage and retrieval speed

Adobe’s training data lived in petabytes of distributed cloud storage and had to be shipped out to thousands of GPUs constantly, over standard Ethernet. The dataset itself was enormous and varied—images and video, low-res and high-res, simple formats and expensive codecs, all moving at once. Standard networking, which was built for retrieving individual files on request, wasn’t designed for that kind of sustained, parallel, petabyte-scale movement. It became a massive bottleneck.

Checkpointing compounded the problem. As a safeguard, the training run would periodically write a complete copy of the model back to storage. Most of those checkpoints were never needed again. But writing and reading them still consumed real GPU time, which cost a lot of money whether they were used or not. 

How Adobe fixed it 

Adobe’s solution had two parts. 

First, they replaced standard Ethernet with a high-performance networking fabric designed for the petabyte-scale traffic distributed AI training generates. Now, data could finally move at the pace their GPUs needed. 

Second, they changed how they saved checkpoints. Instead of writing one giant file containing the entire model, they began breaking the model into smaller pieces and saving pipeline fragments to many places at once. Saving and loading checkpoints now takes significantly less time than it used to. If a checkpoint ever needs to be reassembled, that’s slightly slower, but it’s a rare event. The savings on every other write happen continuously, across thousands of GPUs, around the clock.

The underlying lesson is that for model training, parallel access and sustained high throughput are baseline requirements. Training data has to be immediately accessible at the pace your GPUs consume it, not tucked away in a storage tier that takes minutes or hours to retrieve.

Culprit #2: The data loader problem

A balanced data loader can only do its job if the data it’s balancing is actually available the moment it’s needed. That makes fast storage a precondition for everything else in the pipeline. But even after fixing the storage and retrieval speed culprit, Adobe still had a problem: the way training data was divided across the cluster meant some GPUs were doing far more work than others.

Their pipeline used a straightforward data-parallelism approach: slice the training data into equal-sized chunks and assign one chunk to each GPU in the cluster. But equal-sized chunks weren’t equal work. Some GPUs got simple, low-resolution assets that processed in seconds; others got large, complex files that took minutes. The fast GPUs finished early and sat idle waiting for the slow ones to catch up. And then all of them waited again while their results were merged into one updated model before the next round could begin. 

How Adobe fixed it

Adobe stopped treating all data as equivalent. They custom-built a balanced data loader that understood the processing cost of each asset and distributed work so every GPU finished at roughly the same time. Then they restructured how computation was divided across the cluster to make the merging step dramatically cheaper.

A perfectly balanced data loader still idles if the data it’s waiting on hasn’t arrived yet. This is why storage can’t be an afterthought. The loader optimizes what happens once data is there, but storage determines whether it’s there at all. 

And building something like Adobe’s balanced data loader takes real engineering investment, including time spent profiling workloads, testing distribution strategies, and tuning until every GPU finishes at roughly the same pace. Teams still fighting storage bottlenecks rarely get to that work. Their engineers are busy figuring out why GPUs are idle in the first place, not optimizing how work gets distributed once data arrives. Removing the storage bottleneck frees up the engineering time needed to tackle the data loader problem properly.

From 40 to 80 cents

After addressing both culprits, Adobe’s GPUs ran at roughly 80% utilization. After accounting for the coordination overhead inherent to running thousands of machines together, 80% is close to the practical ceiling. That’s the difference between a GPU cluster that’s mostly waiting and one that’s mostly working.

The Storage Side of the Solution

Backblaze B2 Overdrive addresses the storage side of exactly the problem Adobe ran into. B2 is always-hot object storage, with no tiering, no retrieval delays, and no waiting for data to be promoted from a cold tier before training can begin. Training data stays immediately accessible whether it was written an hour ago or six months ago.

But availability alone isn’t enough if data can’t move fast enough to keep up with the cluster. B2 Overdrive adds the throughput layer: at up to 1Tbps, it’s designed to keep petabyte-scale GPU clusters fed continuously, not just handle occasional bursts of traffic. Both layers are S3-compatible, so they drop into existing PyTorch or TensorFlow pipelines without a rewrite. And because egress is free, moving training data between storage and GPU compute—across regions or providers—doesn’t add a cost penalty on top of a performance one.

Checkpointing benefits from the same foundation. Adobe’s solution of breaking the model into fragments and saving them to many places at once only works if the underlying storage is fast and parallel enough to make it pay off. Always-hot, high-throughput storage makes checkpoint writes and recoveries faster across the board, so they cost less GPU time whether they happen rarely or often.

Backblaze can’t write your data loader for you, but it can make sure that once you’ve built one, it isn’t waiting on storage to do its job.

Ready to remove the storage bottleneck from your training pipeline? Learn more about Backblaze B2 Overdrive.

The post Why Your GPU Is Sitting Idle: The Data Pipeline Problem No One Talks About appeared first on Backblaze Blog | Cloud Storage & Cloud Backup

Post Syndicated from Maddie Presland original https://www.backblaze.com/blog/90-of-enterprise-data-is-unstructured-and-genai-cant-use-it-without-better-storage/

At GTC this year, Jensen Huang put a number on a problem that most enterprise AI teams already feel but struggle to articulate: about 90% of what gets generated every year is unstructured data. This includes PDFs, emails, video, contracts, support tickets, and voice recordings. 

Until recently, that data was, in his words, almost entirely useless. You could store it or search it by keyword, but machines couldn’t read it, understand what it meant, or draw any connections between it and other content. 

GenAI has changed that. RAG pipelines, vector databases, and embedding workflows give machines a way to process unstructured content, comprehend its meaning, and synthesize information across the entire enterprise.

The problem is that these tools put a completely different set of demands on storage than anything that came before them. And most enterprise storage wasn’t built for it.

The access pattern no one planned for

Traditional enterprise storage assumes that data gets written and then occasionally retrieved. The architecture for it was built with hierarchical file systems and tiered storage (hot archives for data that gets accessed regularly, cold archives for anything that hasn’t been touched in a while). It worked well for decades, until RAG pipelines broke it.

A RAG pipeline works by reading your documents and converting them into a format the AI can search. But that conversion process is not a one-time event. It has to be run repeatedly as you test, adjust, and update. Every adjustment means re-reading the same content, re-processing it, and re-writing the results back to storage.

For example, the pipeline breaks documents into chunks because the AI can only search effectively when content is divided into manageable pieces. The size of those chunks directly affects how well the AI answers questions. Too large, and the AI retrieves too much irrelevant content. Too small, and it loses context.

Whenever you have to update your underlying documents, the pipeline has to re-read and re-process everything affected by the change. The same is true when upgrading to a newer AI model, because the new model produces different representations of the same content and requires everything to be re-converted.

While the pipeline is reading, it’s also saving converted content, updated search indices, and checkpoints. And what it saves is tiny. Embeddings, vector chunks, and metadata objects are often just a few kilobytes each, but a production RAG pipeline generates millions of them. Legacy storage treats every save operation the same regardless of file size, so saving a million tiny files costs nearly as much as saving a million large ones.

Tiered storage exacerbates this even more. These systems judge which data is still needed and which can be moved to cheaper, slower storage based on how recently something was accessed. But a RAG pipeline might go quiet for weeks between runs and then suddenly need everything at once. Or it might re-index documents on a schedule that the storage system doesn’t recognize as regular access. Either way, the tiered system sees cold data and moves it to a slower tier. When the pipeline needs that data back, it waits and pays a retrieval fee before it can proceed. That cost and delay compounds across millions of objects and dozens of pipeline runs.

The governance problem hiding inside your index

When a RAG pipeline indexes documents into a vector database, it takes a snapshot of your data at that moment in time. The embeddings it generates reflect the content of those documents as they existed when the pipeline ran. When the original documents are updated or changed, the snapshot doesn’t change with them. The vector index keeps pointing to the old version, and the AI keeps retrieving it and presenting it as current fact.

To a user, that looks like hallucination. The model is confidently citing a document that no longer says what the model thinks it says. But the failure isn’t happening inside the model; it’s happening at the storage layer. Specifically, in the gap between when source documents change and when the index gets updated.

Most enterprises running RAG pipelines today have no systematic way to close that gap. Document management systems aren’t built to notify downstream AI pipelines when content changes. Vector databases aren’t built to automatically invalidate and re-index affected embeddings. The result is an AI system that’s architecturally incapable of knowing when its knowledge has gone stale, so it presents outdated information with the same confidence as current information.

What GenAI storage actually requires

Closing the gap between what RAG pipelines demand and what legacy storage delivers requires three things:

1. Sustained high throughput. RAG pipelines generate continuous, mixed traffic at every stage of the process—ingesting documents, processing them, writing the results to an index, and querying that index—often all running simultaneously. Storage must be able to move data in and out fast enough to match the speed at which the pipeline generates requests, and sustain that speed across the entire duration of a run, not just handle short bursts.

2. Always-hot architecture with no tiering. Every document needs to be immediately accessible regardless of when it was last touched. This is not just for pipeline performance, but also because slow re-indexing is what produces stale outputs. When source documents change, the pipeline needs to re-index immediately. Storage that puts documents behind retrieval delays or tiering penalties widens the window in which the AI is working from outdated information and presenting it as fact.

3. Free data movement. RAG workflows span multiple GPU providers, MLOps platforms, and inference environments across multiple regions. Storage that charges egress fees for every transfer turns architecture decisions into cost calculations and discourages the experimentation that gets AI systems from prototype to production.

The storage layer that doesn’t get in the way

Backblaze B2 Overdrive is built around exactly these requirements. It is always-hot object storage with no tiering, no retrieval delays, and no waiting for data to be promoted before a pipeline can run. Every document is always available, regardless of how recently it was written, which means when the pipeline needs to re-index, storage never slows it down.

At up to 1Tbps sustained throughput, it keeps high-volume RAG pipelines fed continuously through the constant mixed traffic of reads and writes they generate. Because egress is free, teams can move data between storage and compute across regions and providers without a cost penalty. And S3 compatibility lets B2 Overdrive drop into existing LangChain, LlamaIndex, and vector database pipelines without rewrites.

Ninety percent of enterprise data is sitting in formats that, until recently, machines couldn’t do anything useful with. The tools to unlock it now exist. How well they perform and what they cost to run depends largely on the storage underneath them.

Ready to build a RAG pipeline that doesn’t stall on storage? Learn more about Backblaze B2 Overdrive.

The post 90% of Enterprise Data Is Unstructured, and GenAI Can’t Use It Without Better Storage appeared first on Backblaze Blog | Cloud Storage & Cloud Backup

Post Syndicated from Maddie Presland original https://www.backblaze.com/blog/ai-data-pipeline-101-ingest-archive/

Extensive news coverage and analyst reports on AI missing productivity and ROI targets mean that AI failure is something of a hot topic. There’s no arguing that some AI initiatives are misguided, including replacing entire specialist teams with AI. For others, the issue actually lies with data knowledge and readiness–Gartner predicts that through 2026, organizations will abandon 60% of AI projects unsupported by AI-ready data, and S&P Global recently highlighted the importance of ingesting previously overlooked or unknown data to discover interdependencies in risk management.

The pressure to move at the perceived speed of AI makes it easy to skip or rush important prep work. Now that major AI pilots have been up and running, supplementing or sometimes entirely replacing select business functions with mixed results, this exposes an already known problem among AI experts: implementing AI too quickly and ignoring the importance of keeping human experts in the loop increases your threshold for error. This is especially prevalent for both internal AI tools that are meant to augment key roles, and customer-facing AI applications that are supposed to increase accessibility to an outcome, such as generating a lifelike video based on natural language prompts.

What’s causing this? It’s not just the LLMs–it’s the data. Now that human experts are more aware of what AI can get wrong, we’re going back to basics to help you get it right. This starts at the very beginning: curating, ingesting, and storing data using infrastructure that’s actually designed for moving data quickly, and without financial penalty.

The Stages of the AI Model Training Data Pipeline

Our in-house AI experts have split the AI data pipeline into five essential key phases for model training, and highlighting how storage factors into each stage (including what’s being stored.)

Data collection ≠ data ingest

Data ingest is the process of any type of data being added to a designated collection destination, whether that’s a specific file folder, database, or object storage bucket. For a lot of applications, data ingest is frequent or nearly constant–busy e-commerce sites with a constant flow of customer transactions and feedback, live video feeds, and combining real-time data sources like pairing security footage with physical building security sensors. This is called streaming ingestion–and streaming ingestion being the foundation of data collection for various types of AI is one of the key reasons why storage is an AI infrastructure problem that often flies under the radar until there’s a serious problem.

• Streaming ingestion requires constant low-latency access to the data storage repository to prevent data upload lags and errors.
• Streaming ingestion for video and other large files requires high rate limits and high-throughput networking capabilities to optimize upload times, especially when an application requires a file to be ready for processing in seconds/minutes instead of hours/days.
• Running out of storage capacity is not an option for model performance, and for compliance and auditing purposes–streaming ingestion requires constant access to storage that is as close to infinite as possible.

Automation from the start

In addition to the files themselves, setting up a highly effective AI data pipeline involves building automation from the very beginning to immediately allocate files to the right bucket using taxonomy and collect and store file metadata to begin data aggregation critical for future labeling and processing.

Why taxonomy is critical for AI

The file itself is the data source. Navigating your dataset starts with implementing a taxonomy that makes your dataset highly searchable as your data grows from a few thousand for your first round of training to millions for an AI application operating in production.

Taxonomy is essentially your file storage structure, or how files are automatically “nested” and relationships between files are built immediately upon ingest. When you’re just getting started, developing your taxonomy helps you stay organized and ready to go searching for a specific file when your coworker doesn’t believe what the data is saying. When you’re working in established teams, implementing a new taxonomy or showing that you understand the importance of following an established taxonomy builds a contract of understanding between you and your data engineering or ML teams. (AKA, changing taxonomy mid-project is a big undertaking with ripple effects, and should not be taken lightly.)

A simple example taxonomy for ingesting raw files can look like this:

/<source>/<modality>/<status>/<date>/filename

Taxonomy should also reflect what the data is actually doing or will do, not just the file type and data source. The goal is to make every object self-describing at write time, so downstream training pipelines can filter, version, and partition without touching the data itself.

• Partition by date/time at the prefix level so you can use time-range queries and lifecycle rules without scanning everything
• Include modality (video, audio, image, text) as a top-level segment so cross-modal datasets stay logically separated but co-located in the same bucket
• Use a UUID or content hash as the filename — never rely on source filenames, which are inconsistent and collision-prone at scale
• Use B2 bucket policies or object tagging rules to reject objects written to non-conforming prefixes
• Maintain a human-readable taxonomy manifest (taxonomy.json at bucket root) that documents each top-level prefix and its schema
• For video specifically, consider a separate prefix segment for resolution or codec: …/video/4k/h264/… — this pays off quickly when training jobs need to filter by input spec

Retain all the metadata

Write metadata as object tags and custom headers at ingest using per-object user-defined metadata at PUT, such as:

s3.put_object(
    Bucket="for-training",
    Key=object_key,
    Body=video_bytes,
    Metadata={
        "source-id": "warehouse-camera-feed-8",
        "capture-timestamp": "2026-07-08T14:32:00Z",
        "frame-rate": "30",
        "resolution": "3840x2160",
        "modality": "video",
        "label-status": "unlabeled",
        "ingest-pipeline-version": "v2.3.1"
    }
)

This way, the metadata travels with the object and is always returned on HEAD requests without a separate lookup. 

But to build out a rich dataset, you will need even more metadata in the form of sidecar metadata files. Retain annotations, bounding boxes, ground-truth labels, licensing info, and consent flag metadata by creating a sidecar JSON with a .meta.json suffix. For the warehouse camera video example listed above, you would end up with two files that look like this:

for-training/video/warehouse-camera-feed-8/2026/07/08/14/a3f9c1d2.mp4

for-training/video/warehouse-camera-feed-8/2026/07/08/14/a3f9c1d2.meta.json

Enable object versioning on training buckets. If a labeling pipeline updates an annotation, it should write a new version (or a new sidecar) rather than overwriting — training reproducibility depends on knowing which metadata was present at the time a dataset was compiled.

Expose metadata to training pipelines via a manifest

Rather than having training jobs scan bucket prefixes directly, generate a manifest file (JSONL or Parquet) at the end of each ingest batch that enumerates every object key + its full metadata. Tools like PyTorch’s WebDataset and HuggingFace datasets can load directly from these manifests, and it decouples the training job from needing object storage credentials for discovery.

Deciding on storage while evaluating data

To summarize, this is why choosing your storage destination based on your current (or, if you’re already undergoing a data management transformation, future-state) scenario is critical:

• Ingest type: Does your storage provide enough network bandwidth to capture the correct type of data in real time (especially for video?)
• Data type: Will large files potentially incur large upfront costs with upload fees?
• Data processing workflow: Does your storage have capacity headroom for file multiplication during processing, and does the cost structure allow this to happen without potentially draining infrastructure budgets?

Even with Backblaze B2’s hot storage at cold storage pricing, it may be beneficial for you to tier your storage based on your ingestion type, and how frequently the data will be accessed for training.

• Batch ingestion is better suited for mid to lower performance storage, as this is typically used for historical datasets or a set schedule of pre-determined data updates, such as jobs pulling from relational databases or CSV uploads once a day or once per week.
• Streaming ingestion is well-suited for hot storage to support a continuous stream of real-time (or near-real-time) data processing, such as from social media feeds and high-volume e-commerce AI helper agents.
• Hybrid ingestion uses a combination of batch and streaming ingestion to handle both historical and real-time data requirements for AI models.

Surprise retrieval penalties for model training can happen even while building out your MVP or proof of concept–so avoid a storage headache and start building your pipeline at data ingest with Backblaze B2 $6.95 per TB/month. Create an account to get started with 10GB for free, or contact our storage experts for assistance with migrations and more.

Looking for more info on data ingest? Watch the on-demand webinar that dives into more details about data ingest with Backblaze’s Director of Applied AI Jeronimo De Leon.

The post AI Data Pipeline 101: Ingest & Archive appeared first on Backblaze Blog | Cloud Storage & Cloud Backup