Post Syndicated from Maddie Presland original https://www.backblaze.com/blog/why-your-gpu-is-sitting-idle-the-data-pipeline-problem-no-one-talks-about/

At NVIDIA’s GTC conference, Adobe’s CTO Ely Greenfield walked the audience through the company’s three-year journey building frontier generative AI models from scratch. And the AI training data pipeline required to keep thousands of GPUs productive.

Along the way, he showed a profiler readout from their early training runs—a visualization of exactly how much time each GPU was spending on actual computation versus sitting idle. It revealed that roughly two-thirds of GPU time was spent simply waiting for data. And that idle time had a price tag. 

“If we were putting a million dollars into training,” he told the room, “that was $600,000 we were burning away on GPUs sitting and doing nothing.” In other words, roughly sixty cents of every dollar spent on GPU compute was being wasted.

Greenfield’s team traced the waste to two culprits, neither of which is unique to Adobe: whether data can reach your GPUs fast enough, and whether the work it represents is distributed evenly once it arrives.

One culprit is a storage problem. The other isn’t, but you won’t be able to fix it until you’ve solved the first one.

Culprit #1: Storage and retrieval speed

Adobe’s training data lived in petabytes of distributed cloud storage and had to be shipped out to thousands of GPUs constantly, over standard Ethernet. The dataset itself was enormous and varied—images and video, low-res and high-res, simple formats and expensive codecs, all moving at once. Standard networking, which was built for retrieving individual files on request, wasn’t designed for that kind of sustained, parallel, petabyte-scale movement. It became a massive bottleneck.

Checkpointing compounded the problem. As a safeguard, the training run would periodically write a complete copy of the model back to storage. Most of those checkpoints were never needed again. But writing and reading them still consumed real GPU time, which cost a lot of money whether they were used or not. 

How Adobe fixed it 

Adobe’s solution had two parts. 

First, they replaced standard Ethernet with a high-performance networking fabric designed for the petabyte-scale traffic distributed AI training generates. Now, data could finally move at the pace their GPUs needed. 

Second, they changed how they saved checkpoints. Instead of writing one giant file containing the entire model, they began breaking the model into smaller pieces and saving pipeline fragments to many places at once. Saving and loading checkpoints now takes significantly less time than it used to. If a checkpoint ever needs to be reassembled, that’s slightly slower, but it’s a rare event. The savings on every other write happen continuously, across thousands of GPUs, around the clock.

The underlying lesson is that for model training, parallel access and sustained high throughput are baseline requirements. Training data has to be immediately accessible at the pace your GPUs consume it, not tucked away in a storage tier that takes minutes or hours to retrieve.

Culprit #2: The data loader problem

A balanced data loader can only do its job if the data it’s balancing is actually available the moment it’s needed. That makes fast storage a precondition for everything else in the pipeline. But even after fixing the storage and retrieval speed culprit, Adobe still had a problem: the way training data was divided across the cluster meant some GPUs were doing far more work than others.

Their pipeline used a straightforward data-parallelism approach: slice the training data into equal-sized chunks and assign one chunk to each GPU in the cluster. But equal-sized chunks weren’t equal work. Some GPUs got simple, low-resolution assets that processed in seconds; others got large, complex files that took minutes. The fast GPUs finished early and sat idle waiting for the slow ones to catch up. And then all of them waited again while their results were merged into one updated model before the next round could begin. 

How Adobe fixed it

Adobe stopped treating all data as equivalent. They custom-built a balanced data loader that understood the processing cost of each asset and distributed work so every GPU finished at roughly the same time. Then they restructured how computation was divided across the cluster to make the merging step dramatically cheaper.

A perfectly balanced data loader still idles if the data it’s waiting on hasn’t arrived yet. This is why storage can’t be an afterthought. The loader optimizes what happens once data is there, but storage determines whether it’s there at all. 

And building something like Adobe’s balanced data loader takes real engineering investment, including time spent profiling workloads, testing distribution strategies, and tuning until every GPU finishes at roughly the same pace. Teams still fighting storage bottlenecks rarely get to that work. Their engineers are busy figuring out why GPUs are idle in the first place, not optimizing how work gets distributed once data arrives. Removing the storage bottleneck frees up the engineering time needed to tackle the data loader problem properly.

From 40 to 80 cents

After addressing both culprits, Adobe’s GPUs ran at roughly 80% utilization. After accounting for the coordination overhead inherent to running thousands of machines together, 80% is close to the practical ceiling. That’s the difference between a GPU cluster that’s mostly waiting and one that’s mostly working.

The Storage Side of the Solution

Backblaze B2 Overdrive addresses the storage side of exactly the problem Adobe ran into. B2 is always-hot object storage, with no tiering, no retrieval delays, and no waiting for data to be promoted from a cold tier before training can begin. Training data stays immediately accessible whether it was written an hour ago or six months ago.

But availability alone isn’t enough if data can’t move fast enough to keep up with the cluster. B2 Overdrive adds the throughput layer: at up to 1Tbps, it’s designed to keep petabyte-scale GPU clusters fed continuously, not just handle occasional bursts of traffic. Both layers are S3-compatible, so they drop into existing PyTorch or TensorFlow pipelines without a rewrite. And because egress is free, moving training data between storage and GPU compute—across regions or providers—doesn’t add a cost penalty on top of a performance one.

Checkpointing benefits from the same foundation. Adobe’s solution of breaking the model into fragments and saving them to many places at once only works if the underlying storage is fast and parallel enough to make it pay off. Always-hot, high-throughput storage makes checkpoint writes and recoveries faster across the board, so they cost less GPU time whether they happen rarely or often.

Backblaze can’t write your data loader for you, but it can make sure that once you’ve built one, it isn’t waiting on storage to do its job.

Ready to remove the storage bottleneck from your training pipeline? Learn more about Backblaze B2 Overdrive.

The post Why Your GPU Is Sitting Idle: The Data Pipeline Problem No One Talks About appeared first on Backblaze Blog | Cloud Storage & Cloud Backup

Post Syndicated from Maddie Presland original https://www.backblaze.com/blog/90-of-enterprise-data-is-unstructured-and-genai-cant-use-it-without-better-storage/

At GTC this year, Jensen Huang put a number on a problem that most enterprise AI teams already feel but struggle to articulate: about 90% of what gets generated every year is unstructured data. This includes PDFs, emails, video, contracts, support tickets, and voice recordings. 

Until recently, that data was, in his words, almost entirely useless. You could store it or search it by keyword, but machines couldn’t read it, understand what it meant, or draw any connections between it and other content. 

GenAI has changed that. RAG pipelines, vector databases, and embedding workflows give machines a way to process unstructured content, comprehend its meaning, and synthesize information across the entire enterprise.

The problem is that these tools put a completely different set of demands on storage than anything that came before them. And most enterprise storage wasn’t built for it.

The access pattern no one planned for

Traditional enterprise storage assumes that data gets written and then occasionally retrieved. The architecture for it was built with hierarchical file systems and tiered storage (hot archives for data that gets accessed regularly, cold archives for anything that hasn’t been touched in a while). It worked well for decades, until RAG pipelines broke it.

A RAG pipeline works by reading your documents and converting them into a format the AI can search. But that conversion process is not a one-time event. It has to be run repeatedly as you test, adjust, and update. Every adjustment means re-reading the same content, re-processing it, and re-writing the results back to storage.

For example, the pipeline breaks documents into chunks because the AI can only search effectively when content is divided into manageable pieces. The size of those chunks directly affects how well the AI answers questions. Too large, and the AI retrieves too much irrelevant content. Too small, and it loses context.

Whenever you have to update your underlying documents, the pipeline has to re-read and re-process everything affected by the change. The same is true when upgrading to a newer AI model, because the new model produces different representations of the same content and requires everything to be re-converted.

While the pipeline is reading, it’s also saving converted content, updated search indices, and checkpoints. And what it saves is tiny. Embeddings, vector chunks, and metadata objects are often just a few kilobytes each, but a production RAG pipeline generates millions of them. Legacy storage treats every save operation the same regardless of file size, so saving a million tiny files costs nearly as much as saving a million large ones.

Tiered storage exacerbates this even more. These systems judge which data is still needed and which can be moved to cheaper, slower storage based on how recently something was accessed. But a RAG pipeline might go quiet for weeks between runs and then suddenly need everything at once. Or it might re-index documents on a schedule that the storage system doesn’t recognize as regular access. Either way, the tiered system sees cold data and moves it to a slower tier. When the pipeline needs that data back, it waits and pays a retrieval fee before it can proceed. That cost and delay compounds across millions of objects and dozens of pipeline runs.

The governance problem hiding inside your index

When a RAG pipeline indexes documents into a vector database, it takes a snapshot of your data at that moment in time. The embeddings it generates reflect the content of those documents as they existed when the pipeline ran. When the original documents are updated or changed, the snapshot doesn’t change with them. The vector index keeps pointing to the old version, and the AI keeps retrieving it and presenting it as current fact.

To a user, that looks like hallucination. The model is confidently citing a document that no longer says what the model thinks it says. But the failure isn’t happening inside the model; it’s happening at the storage layer. Specifically, in the gap between when source documents change and when the index gets updated.

Most enterprises running RAG pipelines today have no systematic way to close that gap. Document management systems aren’t built to notify downstream AI pipelines when content changes. Vector databases aren’t built to automatically invalidate and re-index affected embeddings. The result is an AI system that’s architecturally incapable of knowing when its knowledge has gone stale, so it presents outdated information with the same confidence as current information.

What GenAI storage actually requires

Closing the gap between what RAG pipelines demand and what legacy storage delivers requires three things:

1. Sustained high throughput. RAG pipelines generate continuous, mixed traffic at every stage of the process—ingesting documents, processing them, writing the results to an index, and querying that index—often all running simultaneously. Storage must be able to move data in and out fast enough to match the speed at which the pipeline generates requests, and sustain that speed across the entire duration of a run, not just handle short bursts.

2. Always-hot architecture with no tiering. Every document needs to be immediately accessible regardless of when it was last touched. This is not just for pipeline performance, but also because slow re-indexing is what produces stale outputs. When source documents change, the pipeline needs to re-index immediately. Storage that puts documents behind retrieval delays or tiering penalties widens the window in which the AI is working from outdated information and presenting it as fact.

3. Free data movement. RAG workflows span multiple GPU providers, MLOps platforms, and inference environments across multiple regions. Storage that charges egress fees for every transfer turns architecture decisions into cost calculations and discourages the experimentation that gets AI systems from prototype to production.

The storage layer that doesn’t get in the way

Backblaze B2 Overdrive is built around exactly these requirements. It is always-hot object storage with no tiering, no retrieval delays, and no waiting for data to be promoted before a pipeline can run. Every document is always available, regardless of how recently it was written, which means when the pipeline needs to re-index, storage never slows it down.

At up to 1Tbps sustained throughput, it keeps high-volume RAG pipelines fed continuously through the constant mixed traffic of reads and writes they generate. Because egress is free, teams can move data between storage and compute across regions and providers without a cost penalty. And S3 compatibility lets B2 Overdrive drop into existing LangChain, LlamaIndex, and vector database pipelines without rewrites.

Ninety percent of enterprise data is sitting in formats that, until recently, machines couldn’t do anything useful with. The tools to unlock it now exist. How well they perform and what they cost to run depends largely on the storage underneath them.

Ready to build a RAG pipeline that doesn’t stall on storage? Learn more about Backblaze B2 Overdrive.

The post 90% of Enterprise Data Is Unstructured, and GenAI Can’t Use It Without Better Storage appeared first on Backblaze Blog | Cloud Storage & Cloud Backup

Post Syndicated from Maddie Presland original https://www.backblaze.com/blog/ai-data-pipeline-101-ingest-archive/

Extensive news coverage and analyst reports on AI missing productivity and ROI targets mean that AI failure is something of a hot topic. There’s no arguing that some AI initiatives are misguided, including replacing entire specialist teams with AI. For others, the issue actually lies with data knowledge and readiness–Gartner predicts that through 2026, organizations will abandon 60% of AI projects unsupported by AI-ready data, and S&P Global recently highlighted the importance of ingesting previously overlooked or unknown data to discover interdependencies in risk management.

The pressure to move at the perceived speed of AI makes it easy to skip or rush important prep work. Now that major AI pilots have been up and running, supplementing or sometimes entirely replacing select business functions with mixed results, this exposes an already known problem among AI experts: implementing AI too quickly and ignoring the importance of keeping human experts in the loop increases your threshold for error. This is especially prevalent for both internal AI tools that are meant to augment key roles, and customer-facing AI applications that are supposed to increase accessibility to an outcome, such as generating a lifelike video based on natural language prompts.

What’s causing this? It’s not just the LLMs–it’s the data. Now that human experts are more aware of what AI can get wrong, we’re going back to basics to help you get it right. This starts at the very beginning: curating, ingesting, and storing data using infrastructure that’s actually designed for moving data quickly, and without financial penalty.

The Stages of the AI Model Training Data Pipeline

Our in-house AI experts have split the AI data pipeline into five essential key phases for model training, and highlighting how storage factors into each stage (including what’s being stored.)

Data collection ≠ data ingest

Data ingest is the process of any type of data being added to a designated collection destination, whether that’s a specific file folder, database, or object storage bucket. For a lot of applications, data ingest is frequent or nearly constant–busy e-commerce sites with a constant flow of customer transactions and feedback, live video feeds, and combining real-time data sources like pairing security footage with physical building security sensors. This is called streaming ingestion–and streaming ingestion being the foundation of data collection for various types of AI is one of the key reasons why storage is an AI infrastructure problem that often flies under the radar until there’s a serious problem.

• Streaming ingestion requires constant low-latency access to the data storage repository to prevent data upload lags and errors.
• Streaming ingestion for video and other large files requires high rate limits and high-throughput networking capabilities to optimize upload times, especially when an application requires a file to be ready for processing in seconds/minutes instead of hours/days.
• Running out of storage capacity is not an option for model performance, and for compliance and auditing purposes–streaming ingestion requires constant access to storage that is as close to infinite as possible.

Automation from the start

In addition to the files themselves, setting up a highly effective AI data pipeline involves building automation from the very beginning to immediately allocate files to the right bucket using taxonomy and collect and store file metadata to begin data aggregation critical for future labeling and processing.

Why taxonomy is critical for AI

The file itself is the data source. Navigating your dataset starts with implementing a taxonomy that makes your dataset highly searchable as your data grows from a few thousand for your first round of training to millions for an AI application operating in production.

Taxonomy is essentially your file storage structure, or how files are automatically “nested” and relationships between files are built immediately upon ingest. When you’re just getting started, developing your taxonomy helps you stay organized and ready to go searching for a specific file when your coworker doesn’t believe what the data is saying. When you’re working in established teams, implementing a new taxonomy or showing that you understand the importance of following an established taxonomy builds a contract of understanding between you and your data engineering or ML teams. (AKA, changing taxonomy mid-project is a big undertaking with ripple effects, and should not be taken lightly.)

A simple example taxonomy for ingesting raw files can look like this:

/<source>/<modality>/<status>/<date>/filename

Taxonomy should also reflect what the data is actually doing or will do, not just the file type and data source. The goal is to make every object self-describing at write time, so downstream training pipelines can filter, version, and partition without touching the data itself.

• Partition by date/time at the prefix level so you can use time-range queries and lifecycle rules without scanning everything
• Include modality (video, audio, image, text) as a top-level segment so cross-modal datasets stay logically separated but co-located in the same bucket
• Use a UUID or content hash as the filename — never rely on source filenames, which are inconsistent and collision-prone at scale
• Use B2 bucket policies or object tagging rules to reject objects written to non-conforming prefixes
• Maintain a human-readable taxonomy manifest (taxonomy.json at bucket root) that documents each top-level prefix and its schema
• For video specifically, consider a separate prefix segment for resolution or codec: …/video/4k/h264/… — this pays off quickly when training jobs need to filter by input spec

Retain all the metadata

Write metadata as object tags and custom headers at ingest using per-object user-defined metadata at PUT, such as:

s3.put_object(
    Bucket="for-training",
    Key=object_key,
    Body=video_bytes,
    Metadata={
        "source-id": "warehouse-camera-feed-8",
        "capture-timestamp": "2026-07-08T14:32:00Z",
        "frame-rate": "30",
        "resolution": "3840x2160",
        "modality": "video",
        "label-status": "unlabeled",
        "ingest-pipeline-version": "v2.3.1"
    }
)

This way, the metadata travels with the object and is always returned on HEAD requests without a separate lookup. 

But to build out a rich dataset, you will need even more metadata in the form of sidecar metadata files. Retain annotations, bounding boxes, ground-truth labels, licensing info, and consent flag metadata by creating a sidecar JSON with a .meta.json suffix. For the warehouse camera video example listed above, you would end up with two files that look like this:

for-training/video/warehouse-camera-feed-8/2026/07/08/14/a3f9c1d2.mp4

for-training/video/warehouse-camera-feed-8/2026/07/08/14/a3f9c1d2.meta.json

Enable object versioning on training buckets. If a labeling pipeline updates an annotation, it should write a new version (or a new sidecar) rather than overwriting — training reproducibility depends on knowing which metadata was present at the time a dataset was compiled.

Expose metadata to training pipelines via a manifest

Rather than having training jobs scan bucket prefixes directly, generate a manifest file (JSONL or Parquet) at the end of each ingest batch that enumerates every object key + its full metadata. Tools like PyTorch’s WebDataset and HuggingFace datasets can load directly from these manifests, and it decouples the training job from needing object storage credentials for discovery.

Deciding on storage while evaluating data

To summarize, this is why choosing your storage destination based on your current (or, if you’re already undergoing a data management transformation, future-state) scenario is critical:

• Ingest type: Does your storage provide enough network bandwidth to capture the correct type of data in real time (especially for video?)
• Data type: Will large files potentially incur large upfront costs with upload fees?
• Data processing workflow: Does your storage have capacity headroom for file multiplication during processing, and does the cost structure allow this to happen without potentially draining infrastructure budgets?

Even with Backblaze B2’s hot storage at cold storage pricing, it may be beneficial for you to tier your storage based on your ingestion type, and how frequently the data will be accessed for training.

• Batch ingestion is better suited for mid to lower performance storage, as this is typically used for historical datasets or a set schedule of pre-determined data updates, such as jobs pulling from relational databases or CSV uploads once a day or once per week.
• Streaming ingestion is well-suited for hot storage to support a continuous stream of real-time (or near-real-time) data processing, such as from social media feeds and high-volume e-commerce AI helper agents.
• Hybrid ingestion uses a combination of batch and streaming ingestion to handle both historical and real-time data requirements for AI models.

Surprise retrieval penalties for model training can happen even while building out your MVP or proof of concept–so avoid a storage headache and start building your pipeline at data ingest with Backblaze B2 $6.95 per TB/month. Create an account to get started with 10GB for free, or contact our storage experts for assistance with migrations and more.

Looking for more info on data ingest? Watch the on-demand webinar that dives into more details about data ingest with Backblaze’s Director of Applied AI Jeronimo De Leon.

The post AI Data Pipeline 101: Ingest & Archive appeared first on Backblaze Blog | Cloud Storage & Cloud Backup

Post Syndicated from Maddie Presland original https://www.backblaze.com/blog/why-coreweaves-object-storage-launch-is-good-for-ai-and-everyone-building-it/

CoreWeave just launched their own AI Object Storage. Our take? We love to see it. 

At first glance, it might look like a competitive offering, but as far as we’re concerned, the more storage options out there, the better for builders. It’s another sign that object storage has officially arrived as a key ingredient in the AI stack. 

Now, your AI stack can look like this: Fast, flexible storage close to your GPUs from CoreWeave (essential for training and inference). And when the run’s over? Move it to Backblaze B2 Overdrive to keep it ready for your next run at the right temperature and price-to-performance ratio.

More options mean more ways to build smart, cost-efficient pipelines that let teams train faster and iterate more without getting locked in. We’ll always cheer for that. 

Why object storage matters in the AI stack

Every AI model depends on moving massive datasets through training, inference, and retraining cycles. Each stage requires fast, reliable access to data. That’s where object storage comes in.

Object storage enables this by offering:

• Elastic scalability for petabyte-scale data.
• Reliability and durability across long model lifecycles.
• Lifecycle management features to balance cost, performance, and accessibility.

As AI projects scale, smart data management becomes just as important as GPU performance. High-end GPUs can only deliver full value when they’re continuously fed the right data at the right time. When data sits in the wrong tier or takes too long to retrieve, compute resources go underused. And that means wasted time and money.

Balancing performance and cost in AI workloads

CoreWeave’s Local Object Transport Accelerator (LOTA) delivers up to 7GB/s throughput per GPU, helping data move quickly between storage and compute. With pricing around $110 per terabyte (about $60 with discounts) and regional capacity up to 10TiB, it’s built for performance-critical workloads where proximity to GPUs makes a measurable difference.

Its launch adds more choice to the ecosystem and highlights the growing demand for storage built specifically for AI. As more specialized options emerge, organizations are thinking carefully about how to right-size their infrastructure for each stage of the AI lifecycle.

When maximum performance is the goal, GPU-adjacent storage like CoreWeave’s can help teams squeeze out every last bit of speed during intensive training cycles. But for most AI workloads, B2 Overdrive provides the right balance of cost and performance. It offers the throughput and durability needed to support active training while keeping pricing predictable and scalable.

Many AI builders combine these strengths through a multi-cloud setup. Teams might use CoreWeave Object Storage when latency and proximity to GPUs deliver measurable gains, and then keep the rest of their AI pipeline on B2 Overdrive so datasets remain readily available for retraining, testing, or deployment.

Example configuration:

• CoreWeave Object Storage for specialized, compute-intensive training where every millisecond counts. It’s ideal for short bursts of high-throughput processing, such as large-scale model fine-tuning or time-sensitive inferencing.

• B2 Overdrive for the broader AI workflow, including day-to-day training, staging, versioning, and long-term dataset management. It provides the performance needed for ongoing model development while keeping data costs predictable and accessible across teams and environments. 

B2 Overdrive offers: 

• Storage at roughly $15 per terabyte
• High throughput and rapid access for post-training workflows
• Simple APIs and event notifications to automate data movement across environments

This kind of architecture gives teams the freedom to use each platform where it shines. Backblaze handles the heavy lifting for most workloads, while CoreWeave adds targeted acceleration when raw GPU performance is the top priority. The result is a flexible, cost-aware workflow that supports both innovation and scale.

AI infrastructure that plays to every strength

The most effective AI setups use the right cloud for the right job. They run training where GPUs can perform at their peak, and store data where it stays organized and ready to move when needed.

B2 Overdrive provides a foundation for this strategy, offering a layer of object storage that keeps data secure, accessible, and easy to integrate across environments. Teams can combine each platform’s strengths to achieve speed when it’s needed, scalability that endures, and freedom from lock-in and runaway costs.

The AI ecosystem is expanding, and with the right partners, so are the possibilities.

See how Backblaze B2 Overdrive keeps AI data fast, flexible, and affordable.

The post Why CoreWeave’s Object Storage Launch is Good for AI—and Everyone Building It appeared first on Backblaze Blog | Cloud Storage & Cloud Backup

Post Syndicated from Maddie Presland original https://www.backblaze.com/blog/three-hidden-costs-in-ai-video-storage/

Generative AI video is exploding. Platforms can turn prompts into polished clips, and models churn through massive training sets of images and footage. Behind the magic, though, is the unavoidable reality of storing and moving petabytes of data. 

Training runs require archiving colossal datasets, then pulling them back in full when it’s time to retrain. Once models go live, the output itself becomes another major workload to manage, whether that’s endless libraries of user-generated videos or fast-cycling streams of ephemeral content. These challenges are part of life for every GenAI company, but the costs of handling them vary widely depending on the provider.Those cloud storage costs can spiral quickly out of control. The big cloud providers lure teams in with low headline rates, but the fine print tells a different story. Pricing depends on which storage tier you pick, how often you move data between regions, and how many API requests your pipeline makes. Founders end up building workflows around cloud quirks instead of what’s fastest and simplest for their teams.

Hidden cost #1: Storage tiers and complexity

AI video data doesn’t behave neatly. Training sets might sit untouched for long stretches before being needed again all at once. User-facing content might accumulate forever, or spike and crash depending on the latest trend. For lean engineering teams, predicting these swings is nearly impossible.

On major cloud providers, the stakes are high. Choose a hot tier and you’ll overpay when data goes cold. Pick an archive tier and you’ll face delays and penalties when you suddenly need that dataset tomorrow. Constantly shifting petabytes between tiers adds both operational overhead and surprise costs.

The numbers tell the story: a 5PB archive costs about $120K a month on AWS S3 Standard for storage alone, before any egress charges. The same capacity runs closer to $30K on Backblaze B2 Cloud Storage—a $90K delta that could fund another GPU cluster or extend a startup’s runway.

Backblaze B2 comes in at around one-fifth the cost of S3, with no tiering games to manage. And when workloads demand maximum throughput, B2 Overdrive scales while delivering a stronger price-to-performance ratio than others offer. That means less time modeling cost scenarios and more time iterating on product and model design.

Hidden cost #2: Egress fees

AI development thrives on iteration. Training and retraining cycles shuffle enormous datasets across clusters, often more than once a month. Each transfer can rival the cost of storage itself. And the faster a team wants to move, the more those bills stack up.

The big cloud providers introduce friction at every step. They charge not only when data exits their cloud but also when it crosses between their own regions. At petabyte scale, those tolls can reach five or even six figures in a single month, forcing founders into an impossible tradeoff: experiment less or drain the budget.

Consider that moving just 1PB once per month on AWS in the US East (N. Virginia) region racks up around $53.8K. Double that transfer frequency and you’re staring at over $100K in egress fees. That’s budget better spent on hiring, acquiring customers, or building better products.

Backblaze removes this bottleneck. Backblaze B2 already includes free egress to leading GPU and CDN partners. For companies operating at AI scale, B2 Overdrive goes further with unlimited free egress to any destination. That means models can be trained, tuned, and distributed globally without a single surprise charge standing in the way of progress.

Mirage, an AI video platform, experienced this firsthand. By eliminating egress costs, they cut storage-related expenses by up to 95% compared to their previous provider—freeing resources to reinvest in growth and product innovation.

Hidden cost #3: API requests and transaction fees

Not every AI video workflow interacts with storage the same way. Some stream large video files in big chunks, keeping the number of calls manageable. Others slice data into millions or billions of tiny objects—frames, embeddings, or metadata—and rely heavily on listing and indexing operations. In those cases, what looks like spare change per request quickly compounds into thousands of dollars in charges every month.

Major cloud storage providers are relentless here. Every PUT, GET, LIST, or HEAD operation comes with a fee, no matter how small. At scale, those fractions of a cent add up fast, leaving engineers designing around billing quirks instead of choosing the cleanest solution for their pipelines.

Picture a pipeline that generates one billion writes and two billion reads in a single month. On AWS, the tab for those transactions alone would run close to $5.8K. On Backblaze B2, writes are free and reads cost just $0.004 per 10,000 requests, bringing the same workload down to about $800. And the first 2,500 Class B and Class C transactions each day are free, further shrinking the bill. On B2 Overdrive, all API calls are included at no additional cost.

Whether your architecture leans toward billions of tiny objects or more efficient streaming, Backblaze keeps request charges predictable and manageable. That makes API calls something your team doesn’t need to obsess over, which is exactly how it should be.

Bringing it together: Simple, predictable economics

Taken together, these hidden costs show why storing AI video on “the big three” often feels like playing a rigged game. The pricing looks straightforward until the bills arrive, padded with charges for tiers, transfers, and transactions. Each one eats away at budget and slows the pace of innovation.

Backblaze offers a different path. By stripping out the fine print and focusing on price-to-performance, it makes storage a stable foundation instead of a moving target. Mirage proves what that means in practice: eliminating egress fees drove huge savings and freed resources to reinvest in their product.

For founders, that kind of predictability turns storage from a frustrating line item into the fuel for faster iteration, bolder experimentation, and sustainable growth.

The post Three Hidden Costs in AI Video Storage appeared first on Backblaze Blog | Cloud Storage & Cloud Backup