Post Syndicated from Maddie Presland original https://www.backblaze.com/blog/why-your-gpu-is-sitting-idle-the-data-pipeline-problem-no-one-talks-about/

At NVIDIA’s GTC conference, Adobe’s CTO Ely Greenfield walked the audience through the company’s three-year journey building frontier generative AI models from scratch. And the AI training data pipeline required to keep thousands of GPUs productive.

Along the way, he showed a profiler readout from their early training runs—a visualization of exactly how much time each GPU was spending on actual computation versus sitting idle. It revealed that roughly two-thirds of GPU time was spent simply waiting for data. And that idle time had a price tag. 

“If we were putting a million dollars into training,” he told the room, “that was $600,000 we were burning away on GPUs sitting and doing nothing.” In other words, roughly sixty cents of every dollar spent on GPU compute was being wasted.

Greenfield’s team traced the waste to two culprits, neither of which is unique to Adobe: whether data can reach your GPUs fast enough, and whether the work it represents is distributed evenly once it arrives.

One culprit is a storage problem. The other isn’t, but you won’t be able to fix it until you’ve solved the first one.

Culprit #1: Storage and retrieval speed

Adobe’s training data lived in petabytes of distributed cloud storage and had to be shipped out to thousands of GPUs constantly, over standard Ethernet. The dataset itself was enormous and varied—images and video, low-res and high-res, simple formats and expensive codecs, all moving at once. Standard networking, which was built for retrieving individual files on request, wasn’t designed for that kind of sustained, parallel, petabyte-scale movement. It became a massive bottleneck.

Checkpointing compounded the problem. As a safeguard, the training run would periodically write a complete copy of the model back to storage. Most of those checkpoints were never needed again. But writing and reading them still consumed real GPU time, which cost a lot of money whether they were used or not. 

How Adobe fixed it 

Adobe’s solution had two parts. 

First, they replaced standard Ethernet with a high-performance networking fabric designed for the petabyte-scale traffic distributed AI training generates. Now, data could finally move at the pace their GPUs needed. 

Second, they changed how they saved checkpoints. Instead of writing one giant file containing the entire model, they began breaking the model into smaller pieces and saving pipeline fragments to many places at once. Saving and loading checkpoints now takes significantly less time than it used to. If a checkpoint ever needs to be reassembled, that’s slightly slower, but it’s a rare event. The savings on every other write happen continuously, across thousands of GPUs, around the clock.

The underlying lesson is that for model training, parallel access and sustained high throughput are baseline requirements. Training data has to be immediately accessible at the pace your GPUs consume it, not tucked away in a storage tier that takes minutes or hours to retrieve.

Culprit #2: The data loader problem

A balanced data loader can only do its job if the data it’s balancing is actually available the moment it’s needed. That makes fast storage a precondition for everything else in the pipeline. But even after fixing the storage and retrieval speed culprit, Adobe still had a problem: the way training data was divided across the cluster meant some GPUs were doing far more work than others.

Their pipeline used a straightforward data-parallelism approach: slice the training data into equal-sized chunks and assign one chunk to each GPU in the cluster. But equal-sized chunks weren’t equal work. Some GPUs got simple, low-resolution assets that processed in seconds; others got large, complex files that took minutes. The fast GPUs finished early and sat idle waiting for the slow ones to catch up. And then all of them waited again while their results were merged into one updated model before the next round could begin. 

How Adobe fixed it

Adobe stopped treating all data as equivalent. They custom-built a balanced data loader that understood the processing cost of each asset and distributed work so every GPU finished at roughly the same time. Then they restructured how computation was divided across the cluster to make the merging step dramatically cheaper.

A perfectly balanced data loader still idles if the data it’s waiting on hasn’t arrived yet. This is why storage can’t be an afterthought. The loader optimizes what happens once data is there, but storage determines whether it’s there at all. 

And building something like Adobe’s balanced data loader takes real engineering investment, including time spent profiling workloads, testing distribution strategies, and tuning until every GPU finishes at roughly the same pace. Teams still fighting storage bottlenecks rarely get to that work. Their engineers are busy figuring out why GPUs are idle in the first place, not optimizing how work gets distributed once data arrives. Removing the storage bottleneck frees up the engineering time needed to tackle the data loader problem properly.

From 40 to 80 cents

After addressing both culprits, Adobe’s GPUs ran at roughly 80% utilization. After accounting for the coordination overhead inherent to running thousands of machines together, 80% is close to the practical ceiling. That’s the difference between a GPU cluster that’s mostly waiting and one that’s mostly working.

The Storage Side of the Solution

Backblaze B2 Overdrive addresses the storage side of exactly the problem Adobe ran into. B2 is always-hot object storage, with no tiering, no retrieval delays, and no waiting for data to be promoted from a cold tier before training can begin. Training data stays immediately accessible whether it was written an hour ago or six months ago.

But availability alone isn’t enough if data can’t move fast enough to keep up with the cluster. B2 Overdrive adds the throughput layer: at up to 1Tbps, it’s designed to keep petabyte-scale GPU clusters fed continuously, not just handle occasional bursts of traffic. Both layers are S3-compatible, so they drop into existing PyTorch or TensorFlow pipelines without a rewrite. And because egress is free, moving training data between storage and GPU compute—across regions or providers—doesn’t add a cost penalty on top of a performance one.

Checkpointing benefits from the same foundation. Adobe’s solution of breaking the model into fragments and saving them to many places at once only works if the underlying storage is fast and parallel enough to make it pay off. Always-hot, high-throughput storage makes checkpoint writes and recoveries faster across the board, so they cost less GPU time whether they happen rarely or often.

Backblaze can’t write your data loader for you, but it can make sure that once you’ve built one, it isn’t waiting on storage to do its job.

Ready to remove the storage bottleneck from your training pipeline? Learn more about Backblaze B2 Overdrive.

The post Why Your GPU Is Sitting Idle: The Data Pipeline Problem No One Talks About appeared first on Backblaze Blog | Cloud Storage & Cloud Backup

Post Syndicated from Maddie Presland original https://www.backblaze.com/blog/90-of-enterprise-data-is-unstructured-and-genai-cant-use-it-without-better-storage/

At GTC this year, Jensen Huang put a number on a problem that most enterprise AI teams already feel but struggle to articulate: about 90% of what gets generated every year is unstructured data. This includes PDFs, emails, video, contracts, support tickets, and voice recordings. 

Until recently, that data was, in his words, almost entirely useless. You could store it or search it by keyword, but machines couldn’t read it, understand what it meant, or draw any connections between it and other content. 

GenAI has changed that. RAG pipelines, vector databases, and embedding workflows give machines a way to process unstructured content, comprehend its meaning, and synthesize information across the entire enterprise.

The problem is that these tools put a completely different set of demands on storage than anything that came before them. And most enterprise storage wasn’t built for it.

The access pattern no one planned for

Traditional enterprise storage assumes that data gets written and then occasionally retrieved. The architecture for it was built with hierarchical file systems and tiered storage (hot archives for data that gets accessed regularly, cold archives for anything that hasn’t been touched in a while). It worked well for decades, until RAG pipelines broke it.

A RAG pipeline works by reading your documents and converting them into a format the AI can search. But that conversion process is not a one-time event. It has to be run repeatedly as you test, adjust, and update. Every adjustment means re-reading the same content, re-processing it, and re-writing the results back to storage.

For example, the pipeline breaks documents into chunks because the AI can only search effectively when content is divided into manageable pieces. The size of those chunks directly affects how well the AI answers questions. Too large, and the AI retrieves too much irrelevant content. Too small, and it loses context.

Whenever you have to update your underlying documents, the pipeline has to re-read and re-process everything affected by the change. The same is true when upgrading to a newer AI model, because the new model produces different representations of the same content and requires everything to be re-converted.

While the pipeline is reading, it’s also saving converted content, updated search indices, and checkpoints. And what it saves is tiny. Embeddings, vector chunks, and metadata objects are often just a few kilobytes each, but a production RAG pipeline generates millions of them. Legacy storage treats every save operation the same regardless of file size, so saving a million tiny files costs nearly as much as saving a million large ones.

Tiered storage exacerbates this even more. These systems judge which data is still needed and which can be moved to cheaper, slower storage based on how recently something was accessed. But a RAG pipeline might go quiet for weeks between runs and then suddenly need everything at once. Or it might re-index documents on a schedule that the storage system doesn’t recognize as regular access. Either way, the tiered system sees cold data and moves it to a slower tier. When the pipeline needs that data back, it waits and pays a retrieval fee before it can proceed. That cost and delay compounds across millions of objects and dozens of pipeline runs.

The governance problem hiding inside your index

When a RAG pipeline indexes documents into a vector database, it takes a snapshot of your data at that moment in time. The embeddings it generates reflect the content of those documents as they existed when the pipeline ran. When the original documents are updated or changed, the snapshot doesn’t change with them. The vector index keeps pointing to the old version, and the AI keeps retrieving it and presenting it as current fact.

To a user, that looks like hallucination. The model is confidently citing a document that no longer says what the model thinks it says. But the failure isn’t happening inside the model; it’s happening at the storage layer. Specifically, in the gap between when source documents change and when the index gets updated.

Most enterprises running RAG pipelines today have no systematic way to close that gap. Document management systems aren’t built to notify downstream AI pipelines when content changes. Vector databases aren’t built to automatically invalidate and re-index affected embeddings. The result is an AI system that’s architecturally incapable of knowing when its knowledge has gone stale, so it presents outdated information with the same confidence as current information.

What GenAI storage actually requires

Closing the gap between what RAG pipelines demand and what legacy storage delivers requires three things:

1. Sustained high throughput. RAG pipelines generate continuous, mixed traffic at every stage of the process—ingesting documents, processing them, writing the results to an index, and querying that index—often all running simultaneously. Storage must be able to move data in and out fast enough to match the speed at which the pipeline generates requests, and sustain that speed across the entire duration of a run, not just handle short bursts.

2. Always-hot architecture with no tiering. Every document needs to be immediately accessible regardless of when it was last touched. This is not just for pipeline performance, but also because slow re-indexing is what produces stale outputs. When source documents change, the pipeline needs to re-index immediately. Storage that puts documents behind retrieval delays or tiering penalties widens the window in which the AI is working from outdated information and presenting it as fact.

3. Free data movement. RAG workflows span multiple GPU providers, MLOps platforms, and inference environments across multiple regions. Storage that charges egress fees for every transfer turns architecture decisions into cost calculations and discourages the experimentation that gets AI systems from prototype to production.

The storage layer that doesn’t get in the way

Backblaze B2 Overdrive is built around exactly these requirements. It is always-hot object storage with no tiering, no retrieval delays, and no waiting for data to be promoted before a pipeline can run. Every document is always available, regardless of how recently it was written, which means when the pipeline needs to re-index, storage never slows it down.

At up to 1Tbps sustained throughput, it keeps high-volume RAG pipelines fed continuously through the constant mixed traffic of reads and writes they generate. Because egress is free, teams can move data between storage and compute across regions and providers without a cost penalty. And S3 compatibility lets B2 Overdrive drop into existing LangChain, LlamaIndex, and vector database pipelines without rewrites.

Ninety percent of enterprise data is sitting in formats that, until recently, machines couldn’t do anything useful with. The tools to unlock it now exist. How well they perform and what they cost to run depends largely on the storage underneath them.

Ready to build a RAG pipeline that doesn’t stall on storage? Learn more about Backblaze B2 Overdrive.

The post 90% of Enterprise Data Is Unstructured, and GenAI Can’t Use It Without Better Storage appeared first on Backblaze Blog | Cloud Storage & Cloud Backup

Post Syndicated from Maddie Presland original https://www.backblaze.com/blog/ai-data-pipeline-101-ingest-archive/

Extensive news coverage and analyst reports on AI missing productivity and ROI targets mean that AI failure is something of a hot topic. There’s no arguing that some AI initiatives are misguided, including replacing entire specialist teams with AI. For others, the issue actually lies with data knowledge and readiness–Gartner predicts that through 2026, organizations will abandon 60% of AI projects unsupported by AI-ready data, and S&P Global recently highlighted the importance of ingesting previously overlooked or unknown data to discover interdependencies in risk management.

The pressure to move at the perceived speed of AI makes it easy to skip or rush important prep work. Now that major AI pilots have been up and running, supplementing or sometimes entirely replacing select business functions with mixed results, this exposes an already known problem among AI experts: implementing AI too quickly and ignoring the importance of keeping human experts in the loop increases your threshold for error. This is especially prevalent for both internal AI tools that are meant to augment key roles, and customer-facing AI applications that are supposed to increase accessibility to an outcome, such as generating a lifelike video based on natural language prompts.

What’s causing this? It’s not just the LLMs–it’s the data. Now that human experts are more aware of what AI can get wrong, we’re going back to basics to help you get it right. This starts at the very beginning: curating, ingesting, and storing data using infrastructure that’s actually designed for moving data quickly, and without financial penalty.

The Stages of the AI Model Training Data Pipeline

Our in-house AI experts have split the AI data pipeline into five essential key phases for model training, and highlighting how storage factors into each stage (including what’s being stored.)

Data collection ≠ data ingest

Data ingest is the process of any type of data being added to a designated collection destination, whether that’s a specific file folder, database, or object storage bucket. For a lot of applications, data ingest is frequent or nearly constant–busy e-commerce sites with a constant flow of customer transactions and feedback, live video feeds, and combining real-time data sources like pairing security footage with physical building security sensors. This is called streaming ingestion–and streaming ingestion being the foundation of data collection for various types of AI is one of the key reasons why storage is an AI infrastructure problem that often flies under the radar until there’s a serious problem.

• Streaming ingestion requires constant low-latency access to the data storage repository to prevent data upload lags and errors.
• Streaming ingestion for video and other large files requires high rate limits and high-throughput networking capabilities to optimize upload times, especially when an application requires a file to be ready for processing in seconds/minutes instead of hours/days.
• Running out of storage capacity is not an option for model performance, and for compliance and auditing purposes–streaming ingestion requires constant access to storage that is as close to infinite as possible.

Automation from the start

In addition to the files themselves, setting up a highly effective AI data pipeline involves building automation from the very beginning to immediately allocate files to the right bucket using taxonomy and collect and store file metadata to begin data aggregation critical for future labeling and processing.

Why taxonomy is critical for AI

The file itself is the data source. Navigating your dataset starts with implementing a taxonomy that makes your dataset highly searchable as your data grows from a few thousand for your first round of training to millions for an AI application operating in production.

Taxonomy is essentially your file storage structure, or how files are automatically “nested” and relationships between files are built immediately upon ingest. When you’re just getting started, developing your taxonomy helps you stay organized and ready to go searching for a specific file when your coworker doesn’t believe what the data is saying. When you’re working in established teams, implementing a new taxonomy or showing that you understand the importance of following an established taxonomy builds a contract of understanding between you and your data engineering or ML teams. (AKA, changing taxonomy mid-project is a big undertaking with ripple effects, and should not be taken lightly.)

A simple example taxonomy for ingesting raw files can look like this:

/<source>/<modality>/<status>/<date>/filename

Taxonomy should also reflect what the data is actually doing or will do, not just the file type and data source. The goal is to make every object self-describing at write time, so downstream training pipelines can filter, version, and partition without touching the data itself.

• Partition by date/time at the prefix level so you can use time-range queries and lifecycle rules without scanning everything
• Include modality (video, audio, image, text) as a top-level segment so cross-modal datasets stay logically separated but co-located in the same bucket
• Use a UUID or content hash as the filename — never rely on source filenames, which are inconsistent and collision-prone at scale
• Use B2 bucket policies or object tagging rules to reject objects written to non-conforming prefixes
• Maintain a human-readable taxonomy manifest (taxonomy.json at bucket root) that documents each top-level prefix and its schema
• For video specifically, consider a separate prefix segment for resolution or codec: …/video/4k/h264/… — this pays off quickly when training jobs need to filter by input spec

Retain all the metadata

Write metadata as object tags and custom headers at ingest using per-object user-defined metadata at PUT, such as:

s3.put_object(
    Bucket="for-training",
    Key=object_key,
    Body=video_bytes,
    Metadata={
        "source-id": "warehouse-camera-feed-8",
        "capture-timestamp": "2026-07-08T14:32:00Z",
        "frame-rate": "30",
        "resolution": "3840x2160",
        "modality": "video",
        "label-status": "unlabeled",
        "ingest-pipeline-version": "v2.3.1"
    }
)

This way, the metadata travels with the object and is always returned on HEAD requests without a separate lookup. 

But to build out a rich dataset, you will need even more metadata in the form of sidecar metadata files. Retain annotations, bounding boxes, ground-truth labels, licensing info, and consent flag metadata by creating a sidecar JSON with a .meta.json suffix. For the warehouse camera video example listed above, you would end up with two files that look like this:

for-training/video/warehouse-camera-feed-8/2026/07/08/14/a3f9c1d2.mp4

for-training/video/warehouse-camera-feed-8/2026/07/08/14/a3f9c1d2.meta.json

Enable object versioning on training buckets. If a labeling pipeline updates an annotation, it should write a new version (or a new sidecar) rather than overwriting — training reproducibility depends on knowing which metadata was present at the time a dataset was compiled.

Expose metadata to training pipelines via a manifest

Rather than having training jobs scan bucket prefixes directly, generate a manifest file (JSONL or Parquet) at the end of each ingest batch that enumerates every object key + its full metadata. Tools like PyTorch’s WebDataset and HuggingFace datasets can load directly from these manifests, and it decouples the training job from needing object storage credentials for discovery.

Deciding on storage while evaluating data

To summarize, this is why choosing your storage destination based on your current (or, if you’re already undergoing a data management transformation, future-state) scenario is critical:

• Ingest type: Does your storage provide enough network bandwidth to capture the correct type of data in real time (especially for video?)
• Data type: Will large files potentially incur large upfront costs with upload fees?
• Data processing workflow: Does your storage have capacity headroom for file multiplication during processing, and does the cost structure allow this to happen without potentially draining infrastructure budgets?

Even with Backblaze B2’s hot storage at cold storage pricing, it may be beneficial for you to tier your storage based on your ingestion type, and how frequently the data will be accessed for training.

• Batch ingestion is better suited for mid to lower performance storage, as this is typically used for historical datasets or a set schedule of pre-determined data updates, such as jobs pulling from relational databases or CSV uploads once a day or once per week.
• Streaming ingestion is well-suited for hot storage to support a continuous stream of real-time (or near-real-time) data processing, such as from social media feeds and high-volume e-commerce AI helper agents.
• Hybrid ingestion uses a combination of batch and streaming ingestion to handle both historical and real-time data requirements for AI models.

Surprise retrieval penalties for model training can happen even while building out your MVP or proof of concept–so avoid a storage headache and start building your pipeline at data ingest with Backblaze B2 $6.95 per TB/month. Create an account to get started with 10GB for free, or contact our storage experts for assistance with migrations and more.

Looking for more info on data ingest? Watch the on-demand webinar that dives into more details about data ingest with Backblaze’s Director of Applied AI Jeronimo De Leon.

The post AI Data Pipeline 101: Ingest & Archive appeared first on Backblaze Blog | Cloud Storage & Cloud Backup

Post Syndicated from Jeronimo De Leon original https://www.backblaze.com/blog/data-orchestration-in-the-age-of-autonomous-agents-architectural-patterns-building-on-nemoclaw-openclaw/

OpenClaw crossed 250,000 GitHub stars in 60 days, surpassing React’s decade-long record to become the most-starred software project on GitHub. At GTC 2026, NVIDIA CEO Jensen Huang declared it “the operating system for personal AI” and told the room: “For the CEOs, the question is, what’s your OpenClaw strategy?”

At NVIDIA’s Hack for Impact hackathon at GTC, I built and watched engineers build autonomous agents on NemoClaw, OpenClaw, and Nemotron. Wildfire detection ingesting NASA satellite data. Crime pattern analysis across police jurisdictions. Energy grid anomaly forecasting.

The same architectural question surfaced in every project: agents that collect and generate data at scale need a deliberate strategy for archiving, retaining, and surfacing that data. Without one, agent-produced artifacts become dark data, generated but inaccessible, unversioned, and invisible to the rest of the organization. That question only gets more consequential in production.

Beyond NemoClaw’s runtime governance: Architecting for data persistence

NVIDIA’s NemoClaw wraps OpenClaw with security through OpenShell, a runtime that sandboxes each agent at the kernel level. Network requests, file access, and inference calls are governed by declarative YAML policy, enforced outside the agent’s process so the agent itself can never override them.

OpenClaw agents create workspace files (SOUL.md, USER.md, IDENTITY.md) that define the agent’s personality, preferences, and behavioral context. Inside a NemoClaw sandbox, this state lives in a Kubernetes Persistent Volume Claim inside an embedded K3s cluster, and the community is already asking for better backup and restore workflows on the NemoClaw GitHub repo.

At fleet scale, with dozens of agents each maintaining persistent memory, conversation history, and skill artifacts, a durable storage layer beneath the runtime is what keeps agent state from becoming disposable. What that layer looks like depends on the type of data your agents produce.

The agentic data layer

Two categories of data define the storage requirements for autonomous agents.

Operational artifacts

Autonomous agents generate reports, analyses, transformed datasets, alerts, and increasingly, multimodal outputs like processed video, audio, and images. Inside NemoClaw’s sandbox, filesystem access is confined to /sandbox and /tmp, both ephemeral by design.

Cloud storage decouples the artifact from the runtime, enables scoped access via URLs, and plugs into every major orchestration framework. Bucket-level permissions and scoped application keys extend governance into the storage layer, so each agent or agent class gets write access only to its designated output path.

Lineage matters here too: Each artifact should trace back to which agent, model, inputs, and policy produced it. Our GTC project, FireWatch, used Backblaze B2 exactly this way, uploading wildfire risk reports with a bucket-scoped key, generating shareable URLs, and embedding them directly in stakeholder alert emails.

State and compliance data

Agent memory, skill artifacts, and audit logs from policy decisions all require durable, long-term retention. NemoClaw’s privacy router splits inference between local and cloud models based on policy, generating routing metadata that compliance teams will want to retain and query. Cloud storage brings high durability, append-only immutability for audit trails, and lifecycle policies for tiered retention as data ages.

We built an open source OpenClaw plugin (openclaw-b2-backup) around this: Encrypted snapshots of agent config, memory, and sessions pushed to B2 on a daily cron, before compaction events, and on gateway shutdown. Three fields to configure, rollback from chat, one-command migration to a new machine.

The agent landscape is expanding. The storage pattern is consistent.

The open-source, autonomous AI agent ecosystem now spans at least 16 variants, each optimized for a different deployment context: NanoClaw for container-isolated security, ZeroClaw for edge deployment in a 3.4MB Rust binary, IronClaw for regulated industries through Trusted Execution Environments, managed platforms like ClawCloud and Maxclaw, and Qwen-Agent from Alibaba for the Chinese developer ecosystem.

Whether self-hosted or managed, all of them produce artifacts that need to persist beyond the runtime. Teams building autonomous agents for their organizations will need durable output sharing, state backup, and cross-agent data access regardless of which runtime they choose.

What enterprise AI leaders should build toward

Architect your agent data orchestration on cloud storage. As organizations scale from initial agent deployments to multi-team production workloads, data volume grows with every agent added, every week they run, and every modality they process. Agents gather, generate, and transform data continuously. Cloud storage gives you a durable layer for managing that lifecycle: ingestion and collection, versioned outputs, long-term archival, lifecycle policies for retention, and portability across agent platforms as your organization’s runtime choices evolve. Establishing this now, while the ecosystem is still forming, is the strategic move.

Automate agent state backup as part of your deployment standard. Agents building context across customer data, internal systems, and team workflows for weeks carry real operational value. Automated workspace snapshots protect that investment, create a disaster recovery path, and enable migration across environments.

Design for lineage and audit from day one. Policy decisions, tool invocations, inference routing, and multimodal processing chains all generate metadata. For enterprises operating under SOC 2, HIPAA, or GDPR, storing lineage and audit data alongside your artifacts in cloud storage means your compliance posture is ready before the audit, not after.

NemoClaw brought governance to the agentic stack. If your organization is deploying autonomous agents today, data orchestration and lineage are your next architectural decisions. Get them right early, and your agents scale with durable state, shareable outputs, and auditable history from day one.

The post Data Orchestration in the Age of Autonomous Agents: Architectural Patterns Building on NemoClaw & OpenClaw appeared first on Backblaze Blog | Cloud Storage & Cloud Backup