Tag Archives: Cloud Storage

An Inside Look at the Backblaze Storage Pod Museum

Post Syndicated from Andy Klein original https://www.backblaze.com/blog/backblaze-storage-pod-museum/

image of the back of a Backblaze Storage Pod

Merriam-Webster defines a museum as “an institution devoted to the procurement, care, study, and display of objects of lasting interest or value.” With that definition in mind, we’d like to introduce the Backblaze Storage Pod Museum. While some folks think of a museum as a place of static, outdated artifacts, others realize that those artifacts can tell a story over time of experimentation, evolution, and innovation. That is certainly the case with our Storage Pods. Modesty prevents from us saying that we changed the storage industry with our Storage Pod design, so let’s say we added a lot of red to the picture.

Over the years, Larry, our data center manager, has stashed away the various versions of our Storage Pods as they were removed from service. He also kept drives, SATA cards, power supplies, cables, and more. Thank goodness. With the equipment that Larry’s pack-rat tendencies saved, and a couple of current Storage Pods we borrowed (shhhh, don’t tell Larry), we were able to start the Backblaze Storage Pod Museum. Let’s take a quick photo trip through the years.

Before Storage Pod 1.0

Before we announced Storage Pod 1.0 to the world nearly 10 years ago, we had already built about twenty or so Storage Pods. These early pods used Western Digital 1.0 TB Green drives. There were multiple prototypes, but once we went into production, we had settled on the 45-drive design with 3 rows of 15 vertically mounted drives. We ordered the first batch of ten chassis to be built and then discovered we did not spec a hole for the on/off switch. We improvised.

Storage Pod 1.0 — Petabytes on a Budget

We introduced the storage world to inexpensive cloud storage with Storage Pod 1.0. Funny thing, we didn’t refer to this innovation as version 1.0 — just a Backblaze Storage Pod. We not only introduced the Storage Pod, we also open-sourced the design, publishing the design specs, parts list, and more. People took notice. We introduced the design with Seagate 1.5 TB drives for a total of 67 TB of storage. This version also had an Intel Desktop motherboard (DG43NB) and 4 GB of memory.

Storage Pod 2.0 — More Petabytes on a Budget

Storage Pod 2.0 was basically twice the system that 1.0 was. It had twice the memory, twice the speed, and twice the storage, but it was in the same chassis with the same number of drives. All of this combined to reduce the cost per GB of the Storage Pod system over 50%: from $0.117/GB in version 1 to $0.055/GB in version 2.

Among the changes: the desktop motherboard in V1 was upgraded to a server class motherboard, we simplified things by using three four-port SATA cards, and reduced the cost of the chassis itself. In addition, we used Hitachi (HGST) 3 TB drives in Storage Pod 2.0 to double the total amount of storage to 135 TB. Over their lifetime, these HGST drives had an annualized failure rate of 0.82%, with the last of them being replaced in Q2 2017.

Storage Pod 3.0 — Good Vibrations

Storage Pod 3.0 brought the first significant chassis redesign in our efforts to make the design easier to service and provide the opportunity to use a wider variety of components. The most noticeable change was the introduction of drive lids — one for each row of 15 drives. Each lid was held in place by a pair of steel rods. The drive lids held the drives below in place and replaced the drive bands used previously. The motherboard and CPU were upgraded and we went with memory that was Supermicro certified. In addition, we added standoffs to the chassis to allow for Micro ATX motherboards to be used if desired, and we added holes where needed to allow for someone to use one or two 2.5” drives as boot drives — we use one 3.5” drive.

Storage Pod 4.0 — Direct Wire

Up through Storage Pod 3.0, Protocase helped design and then build our Storage Pods. During that time, they also designed and produced a direct wire version, which replaced the nine backplanes with direct wiring to the SATA cards. Storage Pod 4.0 was based on the direct wire technology. We deployed a small number of these systems but we fought driver problems between our software and the new SATA cards. In the end, we went back to our backplanes and Protocase continued forward with direct wire systems that they continued to deploy successfully. Conclusion: there are multiple ways you can be successful with the Storage Pod design.

Storage Pod 4.5 — Backplanes are Back

This version started with the Storage Pod 3.0 design and introduced new 5-port backplanes and upgraded to SATA III cards. Both of these parts were built on Marvel chipsets. The backplanes we previously used were being phased out, which prompted us to examine other alternatives like the direct wire pods. Now we had a ready supply of 5-port backplanes and Storage Pod 4.5 was ready to go.

We also began using Evolve Manufacturing to build these systems. They were located near Backblaze and were able to scale to meet our ever increasing production needs. In addition, they were full of great ideas on how to improve the Storage Pod design.

Storage Pod 5.0 — Evolution from the Chassis on Up

While Storage Pod 3.0 was the first chassis redesign, Storage Pod 5.0 was, to date, the most substantial. Working with Evolve Manufacturing, we examined everything down to the rivets and stand-offs, looking for a better, more cost efficient design. Driving many of the design decisions was the introduction of Backblaze B2 Cloud Storage that was designed to run on our Backblaze Vault architecture. From a performance point-of-view we upgraded the motherboard and CPU, increased memory fourfold, upgraded the networking to 10 GB on the motherboard, and moved from SATA II to SATA III. We also completely redid the drive enclosures, replacing the 15-drive clampdown lids to nine five-drive compartments with drive guides.

Storage Pod 6.0 — 60 Drives

Storage Pod 6.0 increased the amount of storage from 45 to 60 drives. We had a lot of questions when this idea was first proposed, like would we need: bigger power supplies (answer: no), more memory (no), a bigger CPU (no), or more fans (no). We did need to redesign our SATA cable routes from the SATA cards to the backplanes as we needed to stay under the one meter spec length for the SATA cables. We also needed to update our power cable harness, and, of course, add length to the chassis to accommodate the 15 additional drives, but nothing unexpected cropped up — it just worked.

What’s Next?

We’ll continue to increase the density of our storage systems. For example, we unveiled a Backblaze Vault full of 14 TB drives in our 2018 Drive Stats report. Each Storage Pod in that vault contains 840 terabytes worth of hard drives, meaning the 20 Storage Pods that make up the Backblaze Vault bring 16.8 petabytes of storage online when the vault is activated. As higher density drives and new technologies like HAMR and MAMR are brought to market, you can be sure we’ll be testing them for inclusion in our environment.

Nearly 10 years after the first Storage Pod altered the storage landscape, the innovation continues to deliver great returns to the market. Many other companies, from 45Drives to Dell and HP, have leveraged the Storage Pod’s concepts to make affordable, high-density storage systems. We think that’s awesome.

The post An Inside Look at the Backblaze Storage Pod Museum appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

How Cloud-Based MAMs Can Make End-to-End Cloud Workflows a Reality

Post Syndicated from Janet Lafleur original https://www.backblaze.com/blog/how-to-migrate-mam-to-cloud/

Create, Capture, Distribute, Archive

Ever since commercial cloud services were launched over 12 years ago, media and entertainment professionals have debated how and where cloud services best fit in their workflows. Archive and delivery were seen as the most natural fits. But complete, that is, end-to-end cloud workflows were seen as improbable due to the network bandwidth required to edit full-resolution content. Now, with new cloud-oriented creative tools on the market, cloud is now playing a role at every step of creative workflows.

Of course, it’s one thing to talk about complete cloud workflows and it’s another thing to show how the cloud has transformed an actual customer’s workflow from end-to-end. But that’s exactly what healthcare content provider Everwell did by building a streamlined work from anywhere workflow with cloud storage and cloud-delivered asset management. The best part was that rolling out the new cloud workflow was just as painless as it was transformative for their business.

Where On-Site Asset Management Fails: Scaling Up and Remote Access

Everwell was founded on the idea that millions of TVs in medical office lobbies and waiting rooms could deliver compelling, well-produced healthcare educational content. Hospitals, medical groups, and medical practitioners that sign up with Everwell receive media players pre-loaded with an extensive library of Everwell’s educational videos along with software that allows each practice to customize the service with their own information.

As the number of subscribers and demand for their content grew, Everwell COO Loren Goldfarb realized that their production workflow needed to adapt quickly or they wouldn’t be able to scale their business to meet growth. The production workflow was centered around an on-site media asset management (MAM) server with on-site storage that had served them well for several years. But as the volume of raw footage grew and the file sizes increased from HD to 4K, their MAM struggled to keep up with production deadlines.

At the same time, Everwell’s content producers and editors needed to work more efficiently from remote locations. Having to travel to the main production office to check content into the media asset manager became a critical bottleneck. Their existing MAM was designed for teams working in a single location, and remote team members struggled to maintain access to it. And the off-site team members and Everwell’s IT support staff were spending far too much time managing VPNs and firewall access.

Workarounds Were Putting Their Content Library at Risk

Given the pain of a distributed team trying to use systems designed for a single office, it was no surprise that off-site producers resorted to shipping hard drives directly to editors, bypassing the asset management system altogether. Content was extremely vulnerable to loss while being shipped around on hard drives. And making editorial changes to content afterward without direct access to the original source files wasn’t practical. Content was becoming increasingly disorganized and hard for users to find or repurpose. Loren knew that installing servers and storage at every remote production site was not an option.

What Loren needed was an asset management solution that could keep productions moving smoothly and content organized and protected, even with remote producers and editors, so that his team could stay focused on creating content. He soon realized that most available MAMs weren’t built for that.

Everwell remote workflow

Everwell’s distributed workflow

A Cloud-Based MAM Designed for the Complete Workflow

After reviewing and rejecting several vendors on his own, Loren met with Jason Perr of Workflow Intelligence Nexus. Jason proposed a complete cloud workflow solution with iconik for asset management and B2 for cloud storage. Built by established MAM provider Cantemo, iconik takes an entirely new approach by delivering asset management with integrated workflow tools as an on-demand service. With iconik, everything is available through a web browser.

Jason helped Everwell migrate existing content, then deploy a complete, cloud-based production system. Remote producers can easily ingest content into iconik, making it immediately available to other team members anywhere on the planet. As soon as content is added, iconik’s cloud-based compute resources capture the files’ asset metadata, generate proxies, then seamlessly store both the proxies and full-resolution content to the cloud. What’s more, iconik provides in-the-cloud processing for advanced metadata extraction and other artificial intelligence (AI) analysis to enrich assets and allow intelligent searching across the entire content library.

Another critical iconik feature for Everwell is the support for cloud-based proxy editing. Proxies stored in the cloud can be pulled directly into Adobe Premiere, allowing editors to work on their local machine with lower resolution proxies, rather than having every editor download the full-resolution content and generate their own proxy. After the proxy editing is complete, full-resolution sequences are rendered using the full-resolution originals stored in B2 cloud storage and then returned to the cloud. Iconik also offers cloud-based compute resources that can perform quality checks, transcoding, and other processing its customers need to prepare the content for delivery.

Cloud Storage That Goes Beyond Archive

Working behind the scenes, cloud storage seamlessly supports the iconik asset management system, hosting and delivering proxy and full-resolution content while keeping it instantly available for editing, metadata extraction, and AI or other processing. And because cloud storage is built with object storage instead of RAID, it offers the extreme durability needed to keep valuable content highly protected with the infinite scalability needed to grow capacity on demand.

Backblaze B2’s combination of data integrity, dramatically lower pricing than other leading cloud storage options, and full integration with iconik made it an obvious choice for Everwell. With B2, they no longer have to pay for or manage on-site production storage servers, tape, or disk-based archives — all their assets are securely stored in the cloud.

This was the seamless, real-time solution that Loren had envisioned, with all of the benefits of a truly cloud-delivered and cloud-enabled solution. Both iconik and Backblaze services can be scaled up in minutes and the pricing is transparent and affordable. He doesn’t pay for services or storage he doesn’t use and he was able to phase out his on-site servers.

Migrating Existing Content Archive to the Cloud

Everwell’s next challenge was migrating their enormous content library of raw material and existing asset metadata without impacting production. With Jason of Workflow Intelligence Nexus guiding them, they signed up for Backblaze’s B2 Fireball, the rapid ingest service that avoids time-consuming internet transfers by delivering content directly to their cloud-based iconik library.

As part of the service, Backblaze sent Everwell the 70TB Fireball. Everwell connected it to their local network and copied archived content onto it. Meanwhile, Jason and Loren’s team exported the metadata records from their existing asset manager and with a migration tool from Workflow Intelligence Nexus, they automatically created new placeholder records in iconik with all of that metadata.

Everwell then shipped the Fireball back to the Backblaze data center where all of the content was securely uploaded to their B2 account. iconik then scanned and identified the content and linked it to the existing iconik records. The result was an extremely fast migration of an existing content archive to a new cloud-based MAM that was immediately ready for production work.

Workflow diagram of Everwell media archive to B2 cloud storage

Everwell’s media ingest workflow

Cloud Simplicity and Efficiency, with Growth for the Future

With a cloud-based asset management and storage solution in place, production teams like Loren’s can have creative freedom and add significant new capabilities. They can be free to add new editors and producers on the fly and at a moment’s notice, and let them ingest new content from any location and use a single interface to keep track of every project in their expanding asset library.

Production teams can use new AI-powered discovery tools to find content quickly and can always access the original raw source files to create new videos at any time. And they’ll have more time to add new features to their service and take on new productions and customers when they wish.

Best of all for Loren, he’s now free to grow Everwell’s production operations as fast as possible without having to worry about running out of storage, managing servers, negotiating expensive maintenance contracts, or paying for staff to run it all. Their workflow is more nimble, their workforce is more productive, and Loren finally has the modern cloud-delivered production he’s always wanted.

•  •  •

We invite you to view our demo on integrating iconik with B2, 3 Steps to Making Your Cloud Media Archive Active with iconik and Backblaze B2.

Backblaze will be exhibiting at NAB 2019 in Las Vegas on April 8-11, 2019.NABShow logoSchedule a meeting with our cloud storage experts to learn how B2 Cloud Storage can streamline your workflow today!

The post How Cloud-Based MAMs Can Make End-to-End Cloud Workflows a Reality appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

B2 on Your Desktop — Cloud Storage Made Easy

Post Syndicated from Roderick Bauer original https://www.backblaze.com/blog/cloud-storage-made-easy/

B2 on your Desktop

People have lots of different ways that they work with files in B2 Cloud Storage, and there’s a wide range of integrations for different platforms and different uses.

Sometimes, though, being able to use B2 as if it were just another drive on your desktop is the easiest way to go. The applications we’ll be covering in this post make working with B2 as easy as dragging and dropping files from a file manager on your computer directly to B2, or from B2 to your computer. In other cases, you can drag files from a file manager to the application, or between panes inside the application. There’s something for every platform, too, whether you’re on Windows, Macintosh, or Linux. Some of these tools are even free.

Let’s take a look at the applications that make working with B2 a piece of cake! (Or, as easy as pie.)

Use B2 As a Drive on the Desktop

Our first group of applications let you use B2 as if it were a local drive on your computer. The files on B2 are available for you from (depending on platform) File Explorer on Windows, the Finder on Mac, or the File Manager on Linux (as well as the command-line). Some of the applications are free and some require purchase (marked with $).

Most of these apps are simple for anyone to set up. If you are a more advanced user, and comfortable working with the command-line in your OS’s terminal, there are a number of free command-line tools for mounting B2 as a drive, including restic, Rclone, and HashBackup. See their docs for how to mount restic, Rclone, or HashBackup as a drive. We previously wrote about using restic with B2 in our Knowledge Base.

When would dragging and dropping files on the desktop be useful? If you just need to move one or a few files, this could be the fastest way to do that. You can load the application when you need to transfer files, or have it start with your computer so your B2 files and buckets are always just a click away. If you keep archived documents or media in B2 and often need to browse to find a file, this makes that much faster. You can even use shortcuts, search, and other tools you have available for your desktop to find and manage files on B2.

We’ve grouped the applications by platform that let you use B2 as a drive.

Apps for Mounting B2 as a Drive

Macintosh

Windows

Linux

Some Screenshots Showing Applications That Let You Use B2 as a Drive

screenshot of Mountain Duck interface for saving to B2 Cloud Storage

Mountain Duck

screenshot of B2 mounted on the desktop with Mountain Duck

B2 mounted on the desktop with Mountain Duck

screenshot of ExpanDrive saving to B2 cloud storage

ExpanDrive

Cloudmounter

Cloudmounter

screenshot of Cloudmounter with B2 open in Mac Finder

Cloudmounter with B2 open in Mac Finder

Use B2 From a Desktop Application

These applications allow you to use B2 from within the application, and also often work with the local OS’s file manager for drag and drop. They support not just B2, but other cloud and sync services, plus FTP, SFTP, Webdav, SSH, SMB, and other protocols for networking and transferring files.

All of the applications below require purchase, but they have demo periods when you can try them out before you decide you’re ready to purchase.

Apps for Using B2 from the Desktop

Macintosh

Windows

Linux

Screenshots of Using B2 From Desktop Applications

Filezilla Pro

Filezilla Pro browsing photos on B2

screenshot of Transmit with B2 files

Transmit with B2 files

screenshot of Cyberduck transmitting files to B2

Cyberduck

screenshot of odrive cloud storage integration

odrive

SmartFTP

SmartFTP

The Cloud on Your Desktop

We hope these applications make you think of B2 as easy and always available on your desktop whenever you need to move files to or from cloud storage. Easy Peasy Lemon Squeezy, right?

If you’ve used any of these applications, or others we didn’t mention in this post, please tell us in the comments how they worked for you.

The post B2 on Your Desktop — Cloud Storage Made Easy appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

What’s the Diff: DAM vs MAM

Post Syndicated from Janet Lafleur original https://www.backblaze.com/blog/whats-the-diff-dam-vs-mam/

What's the Diff: DAM vs MAM

There’s a reason digital asset management (DAM) and media asset management (MAM) seem to be used interchangeably. Both help organizations centrally organize and manage assets —  images, graphics, documents, video, audio — so that teams can create content efficiently and securely. Both simplify managing those assets through the content life cycle, from raw source files through editing, to distribution, to archive. And, as a central repository, they enable teams to collaborate by giving team members direct access to shared assets.

A quick answer to the difference is that MAM is considered a subset of the broader DAM, with MAMs providing more video capabilities. But since most DAMs can manage videos, and MAMs vary widely in what kind of video-oriented features they offer, it’s worth diving deeper to understand these different asset management solutions.

What to Expect From Any Asset Manager

Before we focus on the differences, let’s outline the basic structure and the capabilities of any asset manager.  The best place to start is with the understanding that any given asset a team might want to work with — a video clip, a document, an image —  is usually presented by the asset manager as a single item to the user, but is actually composed of three elements: the master source file, a thumbnail or proxy that’s displayed, and metadata about the object itself. Note that in the context of asset management, metadata is more than simple file attributes (i.e. owner, date created, last modified date, size). It’s a broader set of attributes, including details about the actual content of the file. We’ll spell out more on that later. As far as capabilities, any DAM or MAM worth being called an asset manager should offer:

  • Collaboration — Members of content creation teams all should have direct access to assets in the asset management system from their own workstations.
  • Access control — Access to specific assets or groups of assets should be allowed or restricted based on the user’s rights and permission settings. This is particularly important if teams work in different departments or for different external clients.
  • Browse — Assets should be easily identifiable by more than their file name, such as thumbnails or proxies for videos, and browsable in the asset manager’s graphical interface.
  • Metadata search —  Assets should be searchable by attributes assigned to them, known as metadata. Metadata assignment capabilities should be flexible and extensible over time.
  • Preview — For larger or archived assets, a preview or quick review capability should be provided, such as playing video proxies or mouse-over zoom for thumbnails.
  • Versions — Based on permissions, team members should be able to add new versions of existing assets or add new assets so that material can be easily repurposed for future projects.

Why Metadata Matters So Much

Metadata is a critical element that distinguishes asset managers from file browsers. Without metadata, file names end up doing the heavy lifting with long names like 20190118-gbudman-broll-01-lv-0001.mp4, which strings together a shoot date, subject, camera number, clip number, and more. Structured file naming is not a bad practice, but it doesn’t scale easily to larger teams of contributors and creators. And metadata is not used only to search for assets, it can be fed into other workflow applications integrated with the asset manager for use there.

Metadata is particularly important for images and video because, unlike text-based documents, they can’t be searched for keywords. Metadata can describe in detail what’s in the image or video. For example, metadata for an image could be: male, beard, portrait, blue shirt, dark hair, fair skin, middle-aged, outdoors. And since videos are streams of images, their metadata goes one step further to describe elements at precise moments or ranges of time in the video, known as timecodes. For example, video of a football game could include metadata tags such as 00:10:30 kickoff, 00:15:37 interception, and 00:21:04 touchdown.

iconik MAM example displaying meta data for a BMW M635CSi

iconik MAM

Workflow Integration and Archive Support

More robust DAMs and MAMs go beyond the basic capabilities and offer a range of advanced features that simplify or otherwise support the creation process, also known as the workflow. These can include features for editorial review, automated metadata extraction (e.g. transcription for facial recognition), multilingual support, automated transcode, and much, much more. This is where different asset management solutions diverge the most and show their customization for a particular type of workflow or industry.

Regardless of whether you need all the bells and whistles in your asset manager, as your content library grows it will need storage management features, starting with archive. Archiving completed projects and assets that are infrequently used can conserve disk space on your server by moving them off to less expensive storage, such as cloud storage or digital tape. In particular, images and video are huge storage hogs, and the higher the resolution, the more storage capacity they consume. Regular archiving can keep costs down and keep you from having to upgrade your expensive storage server every year.

Asset managers with built-in archiving make moving content into and out of an archive seamless and straightforward. For most asset managers, assets can be archived directly from the graphical interface. After archive, the thumbnails or proxies of the archived assets continue to appear as before, with a visual indication that they’re archived on secondary storage. Users can retrieve the asset as before, albeit with some time delay that depends on the archive storage and network connection chosen.

A good asset manager will offer multiple choices for archive storage, from cloud storage to LTO tape to inexpensive disk, and from different vendors.  An excellent one will let you automatically make multiple copies to different archive storage for added data protection.

What is a MAM?

With all these common characteristics, what makes a media asset manager different than other asset managers is that it’s created for video production. While DAMs can generally manage video assets, and MAMs can manage images and documents, MAMs are designed from the ground up for creating and managing video content in a video production workflow. That means metadata creation and management, application integrations, and workflow orchestration are all video-oriented.

Metadata for video starts when it’s shot, with camera data, shoot notes or basic logging captured on set.  More detailed metadata cataloging happens when the content is ingested from the camera into the MAM for post-production. Nearly all MAMs offer some type of manual logging to create timecode-based metadata. MAMs built for live broadcast events like sports provide shortcut buttons for key events, such as a face off or slap shot in a hockey game.

More advanced systems offer additional tools for automated metadata extraction. For example, some will use facial recognition to automatically identify actors or public figures.

There is also metadata related to how, where, and how many times the asset has been used and what kinds of edits have been made from the original. There’s no end to what you can describe and categorize with metadata. Defining it for a content library of any reasonable size can be a major undertaking.

MAMs Integrate Video Production Applications

Unlike the more general-purpose DAMs, MAMs will integrate tools built specifically for video production. These widely ranging integrated applications include ingest tools, video editing suites, visual effects, graphics tools, transcode, quality assurance, file transport, specific distribution systems, and much more.

Modern MAM solutions integrate cloud storage throughout the workflow, and not just for archive, but also for creating content through proxy editing. In proxy editing, video editors work using a lower-resolution of the video stored locally, then those edits are applied later to the full-resolution version stored in the cloud when the final cut in rendered.

MAMs May be Tailored for Specific Industry Niches and Workflows

To sum up, the longer explanation for DAM vs MAM is that MAMs focus on video production, with better MAMs offering all the integrations needed for complex video workflows. And because video workflows are as varied as they are complex, MAMs often fall into specific niches within the industry: news, sports, post-production, film production, etc. The size of the organization or team matters too. To stay within their budget, a small post house may select a MAM with fewer of the advanced features that may be basic requirements for a larger multinational post-production facility.

That’s why there are so many MAMs on the market, and why choosing one can be a daunting task with a long evaluation process. And it’s why migrating from one asset manager to another is more common than you’d think. Pro tip: working with a trusted system integrator that serves your industry niche can save you a lot of heartache and money in the long run.

Finally, keep in mind that for legacy reasons, sometimes what’s marketed as a DAM will have all the video capabilities you’d expect from MAM.  So don’t let the name throw you off. Instead, look for an asset manager that fits your workflow with the features and integrated tools you need today, while also providing the  flexibility you need as your business changes in the future.

Backblaze will be exhibiting at NAB 2019 in Las Vegas on April 8-11, 2019.NABShow logoSchedule a meeting with our cloud storage experts to learn how B2 Cloud Storage can streamline your workflow today!

The post What’s the Diff: DAM vs MAM appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Backblaze Hard Drive Stats for 2018

Post Syndicated from Andy Klein original https://www.backblaze.com/blog/hard-drive-stats-for-2018/

Backblaze Hard Drive Stats for 2018

We published our first “Hard Drive Stats” report just over 5 years ago on January 21, 2014. We titled that report “What Hard Drive Should I Buy.” In hindsight, that might have been a bit of an overreach, but we were publishing data that was basically non-existent otherwise.

Many people like our reports, some don’t, and some really don’t — and that’s fine. From the beginning, the idea was to share our experience and use our data to shine a light on the otherwise opaque world of hard disk drives. We hope you have enjoyed reading our reports and we look forward to publishing them for as long as people find them useful.

Thank you.

As of December 31, 2018, we had 106,919 spinning hard drives. Of that number, there were 1,965 boot drives and 104,954 data drives. This review looks at the hard drive failure rates for the data drive models in operation in our data centers. In addition, we’ll take a look at the new hard drive models we’ve added in 2018 including our 12 TB HGST and 14 TB Toshiba drives. Along the way we’ll share observations and insights on the data presented and we look forward to you doing the same in the comments.

2018 Hard Drive Failure Rates: What 100,000+ Hard Drives Tell Us

At the end of 2018 Backblaze was monitoring 104,954 hard drives used to store data. For our evaluation we remove from consideration those drives that were used for testing purposes and those drive models for which we did not have at least 45 drives (see why below). This leaves us with 104,778 hard drives. The table below covers what happened just in 2018.

2018 annualized hard drive failure rates

Notes and Observations

If a drive model has a failure rate of 0%, it means there were no drive failures of that model during 2018.

For 2018, the Annualized Failure Rate (AFR) stated is usually pretty solid. The exception is when a given drive model has a small number of drives (fewer than 500) and/or a small number of drive days (fewer than 50,000). In these cases, the APR can be too wobbly to be used reliably for buying or retirement decisions.

There were 176 drives (104,954 minus 104,778) that were not included in the list above. These drives were either used for testing or we did not have at least 45 drives of a given model. We use 45 drives of the same model as the minimum number when we report quarterly, yearly, and lifetime drive statistics. This is a historical number based on the number of drives needed to fill one Backblaze Storage Pod (version 5 or earlier).

The Annualized Failure Rate (AFR) for 2018 for all drive models was just 1.25%, well below the rates from previous years as we’ll discuss later on in this review.

What’s New in 2018

In 2018 the big trend was hard drive migration: replacing lower density 2, 3, and 4 TB drives, with 8, 10, 12, and in Q4, 14 TB drives. In 2018 we migrated 13,720 hard drives and we added another 13,389 hard drives as we increased our total storage from about 500 petabytes to over 750 petabytes. So in 2018, our data center techs migrated or added 75 drives a day on average, every day of the year.

Here’s a quick review of what’s new in 2018.

  • There are no more 4 TB Western Digital drives; the last of them was replaced in Q4. This leaves us with only 383 Western Digital drives remaining — all 6 TB drives. That’s 0.37% of our drive farm. We do have plenty of drives from HGST (owned by WDC), but over the years we’ve never been able to get the quantity of Western Digital drives we need at a reasonable price.
  • Speaking of HGST drives, in Q4 we added 1,200 HGST 12 TB drives (model: HUH721212ALN604). We had previously tested these drives in Q3 with no failures, so we have filled a Backblaze Vault with 1,200 drives. After about one month we’ve only had one failure, so they are off to a good start.
  • The HGST drives have a ways to go as in Q4 we also added 6,045 Seagate 12 TB drives (model: ST12000NM0007) to bring us to 31,146 of this drive model. That’s 29.7% of our drive farm.
  • Finally in Q4, we added 1,200 Toshiba 14 TB drives (model: MG07ACA14TA). These are helium-filled PMR (perpendicular magnetic recording) drives. The initial annualized failure rate (AFR) is just over 3%, which is similar to the other new models and we would expect the AFR to drop over time as the drives settle in.

Comparing Hard Drive Failure Rates Over Time

When we compare Hard Drive stats for 2018 to previous years two things jump out. First, the migration to larger drives, and second, the improvement in the overall annual failure rate each year. The chart below compares each of the last three years. The data for each year is inclusive of that year only.

Annualized Hard Drive Failure Rates by Year

Notes and Observations

  • In 2016 the average size of hard drives in use was 4.5 TB. By 2018 the average size had grown to 7.7 TB.
  • The 2018 annualized failure rate of 1.25% was the lowest by far of any year we’ve recorded.
  • None of the 45 Toshiba 5 TB drives (model: MD04ABA500V) has failed since Q2 2016. While the drive count is small, that’s still a pretty good run.
  • The Seagate 10 TB drives (model: ST10000NM0086) continue to impress as their AFR for 2018 was just 0.33%. That’s based on 1,220 drives and nearly 500,000 drive days, making the AFR pretty solid.

Lifetime Hard Drive Stats

While comparing the annual failure rates of hard drives over multiple years is a great way to spot trends, we also look at the lifetime annualized failure rates of our hard drives. The chart below is the annualized failure rates of all of the drives currently in production.

Annualized Hard Drive Failure Rates for Active Drives

Hard Drive Stats Webinar

We’ll be presenting the webinar “Backblaze Hard Drive Stats for 2018” on Thursday, January 24, 2019 at 10:00 Pacific time. The webinar will dig deeper into the quarterly, yearly, and lifetime hard drive stats and include the annual and lifetime stats by drive size and manufacturer. You will need to subscribe to the Backblaze BrightTALK channel to view the webinar. Sign up today.

The Hard Drive Stats Data

The complete data set used to create the information used in this review is available on our Hard Drive Test Data page. You can download and use this data for free for your own purpose. All we ask are three things: 1) you cite Backblaze as the source if you use the data, 2) you accept that you are solely responsible for how you use the data, and 3) you do not sell this data to anyone; it is free.

If you just want the summarized data used to create the tables and charts in this blog post you can download the ZIP file containing the CSV file.

Good luck and let us know if you find anything interesting.

The post Backblaze Hard Drive Stats for 2018 appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Breaking the Cycle of Archive Migrations With B2 Cloud Storage

Post Syndicated from Janet Lafleur original https://www.backblaze.com/blog/cloud-data-archiving/

Assorted tapes

Back in the 1980s, my family and I took a trip to visit our friends, the Bremers. We all used to live next door, but the Bremers had moved away a decade prior. As our parents were reminiscing on old times, one of the Bremer teens pulled out a 8mm movie projector and we watched home movies his dad had shot of us playing together in the backyard: on the swings, the see-saw, and running about. What I wouldn’t give to see that footage today! It would be the only video of my sisters and me as kids.

Perhaps Mr. Bremer digitized his home movie collection before he passed away. But it’s more likely his children inherited the box of reels, and it’s now buried and decaying in a closet (or gone entirely). And, if they had the tape, would they have a projector or anything to play it? What a pity. Those precious moments captured once upon a time on film are probably lost forever.

Obsolescence isn’t just a concern for home video enthusiasts. Professional content creators likely have content stored on obsolete technology, whether it’s videotape, LTO digital tape, or external drives. And unlike the simplicity of Mr. Brehmer’s film reels and projectors, there are many more factors that can make digital content inaccessible.

Common Causes of Data Obsolescence

Media Failure

The most obvious issue is storage media degradation. If film is carefully stored in a cold, dry environment, it can last an extremely long time. Yet for both videotape and digital tape, there are a myriad of pitfalls: magnetic particles can lose their charge; the tape substrate can deteriorate; and heavily used tapes can stretch. Tapes over 15 years old are at greatest risk, even if stored in the ideal conditions of low-heat and low-humidity.

Hard disk drives have shortfalls too: mechanical failure, overheating, and power spikes. External drives in particular, are at risk of shock damage from being dropped. Even a drive standing on its side, then tipping over, can generate enough shock to damage the drive internals. At our Backblaze data centers, we replace disk drives after four years, and earlier for drive models that show higher-than-usual failure rates. We have ~100,000 drives in our data centers, and document which ones are more likely to fail in our quarterly drive stats posts.

Obsolete Technology

Even if the storage media remains intact and the data uncorrupted, the data format can become obsolete, often more quickly than you’d expect. For example, manufacturers of the commonly used LTO digital tape are now shipping LTO-8 and only guarantee two generations of backward compatibility. That means if you upgrade your tape system for higher-capacity 12TB LTO-8 tapes, you won’t be able to read the LTO-6 tapes that were introduced just six years ago.

Also, if the file data itself was encoded in a proprietary format, you’ll likely need proprietary software installed on a computer running a potentially outdated operating system version to be able to read its data. This is a bigger topic than we’ll cover today, because there can be layers of encoding involved: backup formats, graphics formats, codecs, etc. But suffice to say that you might find yourself having to hunt down a Mac that’s still running macOS X Leopard to migrate some content.

Museum of Obsolete Media

Not sure how much your content is at risk? The Museum of Obsolete Media rates all imaginable media types on both media stability and obsolescence, from Endangered to In Use.

Spoiler alert:  VHS tapes are rated Endangered for media stability and rated Vulnerable for obsolescence.

Migrate…Then Migrate Again

The only way to combat this sort of media decay and obsolescence and maintain access to your content is to migrate it to newer media and/or a newer technology. This unglamorous task sounds simple — read the data off the old media and copy it to new media — but the devil is in the details. Here is a checklist for trying to maintain your physical media:

The Eight Steps of Data Migration

  1. Determine which content is obsolete or at risk. Choose a media and format for the new archive, and calculate whether you can afford to migrate everything. If not, decide what you can afford to lose forever.
  2. Gather all the tapes or drives to be migrated. Are you sure you have the complete set? Your content spreadsheet might not be up to date. You might need to interview team members to gather any unwritten tribal knowledge about the backup sets.
  3. Identify a migration workstation or server that can run the application that wrote the archived media files. Attach the tape drive or disk device and test it. Can it still properly read, write, and then restore test files?
  4. Using a checklist system, feed tapes into the drive or attach the external drive in order. You might need to track down obscure adapters for older technologies like a SATA to EIDE adapter for parallel port disk drives, or a SCSI card and cables.
  5. Initiate the copy of all files to local storage. Hope you have enough space.
  6. Carefully monitor the entire process and make sure that all files are copied completely, and only then can you check the tape or disk off of your migration list. Then repeat with the next tape or disk.
  7. When you’re done extracting all the old files (or earlier if you’re pinched for disk space), reverse the process. Attach any needed devices and write the files to the new media. Cross your fingers that you bought enough tapes or disk drives (but not too many).
  8. Repeat again in 4-7 years before the new media ages or technologies change.

If all of that sounds too painful, you can pay a transfer service to migrate your whole archive for you, but that’s not cheap, and remember you’ll have to pay to do it again sooner than you think. Alternatively, you can migrate content on-demand and cross your fingers that it’s still readable and that you can retrieve it fast enough. The longer you wait, the greater the risk of media failure. You might only get one shot at reading an old tape or film. Few find that an acceptable risk.

Why Data Archiving to the Cloud Is a Better Solution

Migrate Once with Backblaze B2 Cloud Storage

You can break this migration cycle by migrating once to Backblaze B2 Cloud Storage. We’ll take over from there, moving your data to newer storage technologies as needed over time. Backblaze’s erasure coding technology that protects your data from loss happens to make upgrading technologies easier for us. Not that you need to worry about it; it’s included in our service.

No New Media or Hardware

Moving to B2 Cloud Storage for your archive means you won’t have any hardware or media to purchase, manage, or house. No tapes or disks to buy, no clearing off shelf space as your archive grows. You won’t have to feed tapes into an autoloader every time you want to write or retrieve content from the archive. And moving to B2 Cloud Storage gives you the benefit of only paying for what you’re actually using. Pay-as-you-go means your storage costs move from a capital expense to an operating expense.

B2 is Less Expensive than LTO

Did you know that Backblaze B2 is the first cloud storage that’s more affordable than LTO storage solutions? If you want to see the math, check out our LTO vs B2 calculator. Enter the size of your existing archive and how much you expect to add each year and it will show you cost differences after 1-10 years. To understand its cost and operational assumptions, read our recent blog post, LTO Versus Cloud Storage Costs — the Math Revealed. It details the many factors for storage costs that many media professionals don’t always consider.

Data That’s Always Accessible

The only thing worse than having a tape or disk you can’t read is having one that you can read go missing in action. Your content database or spreadsheet is only as accurate as what’s on the shelf. You may believe that an external drive is still in your archive closet when it went home over the weekend with a staff member and never came back. With B2 Cloud Storage, your archived content is stored in a central location that’s not only always accessible, it’s accessible from anywhere through a web browser.

B2 is Proven Technology

With Backblaze, you get a partner with over a decade of cloud storage experience. The erasure coding we use to encode data gives B2 customers a 99.999999999% durability (11 nines) rating for their data stored in our cloud. As NASA says, there’s higher probability of an asteroid destroying the planet than you losing a file with B2.

Make Your Final Migration Painless and Smart

Of course, you’ll still have to migrate once, but we can help make that final migration as painless and smart as possible. B2 Cloud Storage has several options for moving dataAPIs, Web UI, CLIplus our Fireball rapid ingest service for large data sets. We’ve also partnered with vendors and system integrators who have deep experience in managing media archives.

Streamlined LTO Migration

If your current archive is on LTO tapes, we have a newly announced partnership with StorageDNA that can speed migration of LTFS archives. The Storage DNA Smart Migration bundle combines the latest version of their DNAfabric storage with Backblaze B2 cloud storage, plus an autoloading LTO library so you won’t waste time manually loading tapes. To learn more about how it works, register for our upcoming webinar, From LTO to the Cloud: Your Last Data Migration with Backblaze and StorageDNA, on Friday, December 14.

Organize Content with a MAM

Archive migrations are a great time to evaluate your asset management strategy. If you haven’t rolled out a media asset manager (MAM) yet, or you’re dissatisfied with your current one, know that more and more MAMs are integrated with cloud storage and can simplify collaboration across remote teams. With a cloud-integrated MAM solution, your content can be easily searched, filtered, sorted and previewed all from a web browser, from anywhere. To see B2 in action with a cloud MAM solution, watch our recent webinar, Three Steps to Making Your Cloud Media Archive Active with iconik and Backblaze B2.

Automated Backup and Archive

Finally, B2 isn’t just an archive solution, it’s great for backup, too. Most of our customers who archive content to B2 also back up active production data to the same B2 account. We have a growing list of backup, sync and other tools integrated with B2 to make the data movement to the cloud seamless and to make retrieval intuitive and straightforward.

Pro Tip: syncing newly ingested footage or assets to B2 will spare you a big headache when someone accidentally deletes a critical file.

If you have content that’s on media or in a format that’s aging fast, now’s the time to plan for its migration. By migrating it to B2 Cloud Storage, you can not only make it your last migration, it’s priced so that you can afford to migrate ALL your content. You never know what you’ll need, or when you’ll need it. And some content, like Mr. Bremer’s home movies, simply can’t be re-created.

The post Breaking the Cycle of Archive Migrations With B2 Cloud Storage appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

LTO Versus Cloud Storage Costs — the Math Revealed

Post Syndicated from Andy Klein original https://www.backblaze.com/blog/lto-versus-cloud-storage/

B2 Cloud Storage $68,813 vs. LTO 8 Tape $119,873

A few months back we did a blog post titled, LTO versus Cloud Storage: Choosing the Model That Fits Your Business. In that post we presented our version of an LTO vs. B2 Cloud Storage calculator, a useful tool to determine whether or not it makes economic sense to consider using cloud storage over your LTO storage.

Rather than just saying, “trust us, it’s cheaper,” we thought it would be a good idea to show you what’s inside the model: the assumptions we used, the variables we defined, and the actual math we used to compute our answers. In fact, we’re making the underlying model available for download.

Our Model: LTO vs Cloud Storage

The LTO vs. B2 calculator that is on our website was based on a Microsoft Excel spreadsheet we built. The Excel file we’ve provided for download below is completely self-contained; there are no macros and no external data sources.

Download Excel file: Backblaze-LTO-Calculator-Public-Nov2018.xlsx

The spreadsheet is divided into multiple sections. In the first section, you enter the four values the model needs to calculate the LTO and B2 cloud storage costs. The website implementation is obviously much prettier, but the variables and math are the same as the spreadsheet. Let’s look at the remaining sections.

Entered Values Section

The second section is for organization and documentation of the data that is entered. You also can see the limits we imposed on the data elements.

One question you may have is why we limited the Daily Incremental Backup value to 10 TB. As the comment notes, that’s about as much traffic you can cram through a 1Gbps upload connection in a 24-hour period. If you have bigger (or smaller) pipes, adjust accordingly.

Don’t use the model for one-time archives. You may be tempted to enter zeros in both the Yearly Added Data and Daily Incremental Backup fields to compare the cost of a one-time archive. The model is not designed to compare the cost of a one-time archive. It will give you an answer, but the LTO costs will be overstated by anywhere from 10%-50%. The model was designed for the typical LTO use case where data is written to tape, typically daily, based on the data backup plan.

Variables Section

The third section stores all the variable values you can play with in the model. There is a short description for each variable, but let’s review some general concepts:

Tapes — We use LTO-8 tapes that will decrease in cost about 20% per year down to $60. Non-compressed, these tapes store 12 TB each and take about 9.5 hours to fully load. We use 24 TB for each tape assuming 2:1 compression. If some or all of your data is comprised of video or photos, then compression cannot be used, which makes actual tape capacity number much lower and increases the cost of the LTO solution.

Tapes Used — Based on the grandfather-father-son (GFS) model and assumes you replace tapes once a year.

Maintenance — Assumes you have no spare units, so you cannot miss more than one business day for backups. You could add a spare unit and remove the maintenance or just decide it is OK to miss a day or two while the unit is being repaired.

Off-site Storage — The cost of getting your tapes off-site (and back) assuming a once a week pick-up/drop-off.

Personnel — The cost of the person doing the LTO work, and how much time per week they spend doing the LTO related work, including data restoration. The cost of a person doing the cloud storage work is calculated from this value as described in the Time Savings paragraph below.

Data Restoration — How much of your data on average you will restore each month. The model is a bit limited here in that we use an average for all time periods when downloads are typically uneven across time. You are, of course, welcome to adjust the model. One thing to remember is that you’ll want to test your restore process from time to time, so make sure you allocate resources for that task.

Time Savings — We make the assumption that you will only spend 25% of the time working with cloud storage versus managing and maintaining an LTO system, i.e. no more buying, mounting, unmounting, labeling, cataloging, packaging, reading, or writing tapes.

Model Section

The last section is where the math gets done. Don’t change specific values in this section as they all originate in previous sections. If you decide to change a formula, remember to do so across all 10 years. It is quite possible that many of these steps can be combined into more complex formulas. We break them out to try to make an already complicated calculation somewhat easier to follow. Let’s look at the major subsections.

Data Storage — This section is principally used to organize the different data types and amounts. The model does not apply any corporate data retention policies such as deleting financial records after seven years. Data that is deleted is done so solely based on the GFS backup model, for example, deleting incremental data sets after 30 days.

LTO Costs — This starts with defining the amount of data to store, then calculates the quantity of tapes needed and their costs, along with the number of drive units and their annual unit cost and annual maintenance cost. The purchase price of a tape drive unit is divided evenly over a 10-year period.

Why 10 years? The LTO foundation, states is will support LTO tapes two versions back and expects to release a new version every two years. If you buy an LTO-8 system is 2018, in 2024 LTO-11 will not be able to read your LTO-8 tapes. You are now using obsolete hardware. We assume your LTO-8 hardware will continue to be supported through third party vendors for at least four years (to 2028) after it goes obsolete.

We finish up with calculating the cost of the off-site storage service and finally the personnel cost of managing the system and maintaining the tape library. Other models seem to forget this cost or just assume it is the same as your cloud storage personnel costs.

Cloud Storage Costs — We start with calculating the cost to store the data. This uses the amount of data at the end of the year, versus trying to compute monthly numbers throughout the year. This overstates the total amount a bit, but simplifies the math without materially changing the results. We then calculate the cost to download the data, again using the number at the end of the period. We calculate the incremental cost of enhancing the network to send and restore cloud data. This is an incremental cost, not the total cost. Finally, we add in the personnel cost to access and check on the cloud storage system as needed.

Result Tables — These are the totals from the LTO and cloud storage section in one place.

B2 Fireball Section

There is a small section and some variables associated with the B2 Fireball data transfer service. This service is useful to transfer large amounts of data from your organization to Backblaze. There is a cost for this service of $550 per month to rent the Fireball, plus $75 for shipping. Organizations with existing LTO libraries often don’t want to use their network bandwidth to transfer their entire library, so they end up keeping some LTO systems just to read their archived tapes. The B2 Fireball can move the data in the library quickly and let you move completely away from LTO if desired.

Summary

While we think the model is pretty good there is always room for improvement. If you have any thoughts you’d like to share, let us know in the comments. One more thing: the model is free to update and use within your organization, but if you publicize it anywhere please cite Backblaze as the original source.

The post LTO Versus Cloud Storage Costs — the Math Revealed appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

What’s the Diff: NAS vs SAN

Post Syndicated from Roderick Bauer original https://www.backblaze.com/blog/whats-the-diff-nas-vs-san/

What's the Diff? Network Attachd Storage (NAS) vs Storage Area Network (SAN)

Both network-attached storage (NAS) and storage area network (SAN) were developed to solve the problem of making stored data available to a lot of users at once. Each of them provides dedicated storage for a group of users, but they couldn’t be more different in their approach to achieving their mission.

A NAS is a single storage device that serves files over Ethernet and is relatively inexpensive and easy to set up, while a SAN is a tightly coupled network of multiple devices that work with block-based data and is more expensive and complex to set up and manage. From a user perspective, the biggest difference between NAS and SAN is that NAS devices look like volumes on a file server and use protocols like NFS and SMB/CIFS, while SAN-connected disks appear to the user as local drives.

We provide an overview of the differences between NAS and SAN below. We’ll also briefly cover solutions that combine NAS and SAN and offer many of the advanced benefits of SAN without its high cost.

Basic Definitions — What is NAS?

A NAS is a computer connected to a network that provides file-based data storage services to other devices on the network. The primary strength of NAS is how simple it is to set up and deploy. NAS volumes appear to the user as network mounted volume. The files to be served are typically contained on one or more storage drives, often arranged into logical, redundant storage containers or RAID. The device itself is a network node, much like computers and other TCP/IP devices, all of which maintain their own IP address and can effectively communicate with other networked devices. Although a NAS is usually not designed to be a general-purpose server, NAS vendors and third parties are increasingly offering other software to provide server-like functionality on a NAS.

NAS devices offer an easy way for multiple users in diverse locations to access data, which is valuable when uses are collaborating on projects or sharing information. NAS provides good access controls and security to support collaboration, while also enabling someone who is not an IT professional to administer and manage access to the data. It also offers good fundamental data security through the use of redundant data structures — often RAID — and automatic backup services to local devices and to the cloud.

Benefits of NAS

A NAS is frequently the next step up for a home office or small business that is using DAS (direct attached storage). The move up to NAS results from the desire to share files locally and remotely, having files available 24/7, data redundancy, the ability to replace and upgrade hard drives in the system, and and the availability of other services such as automatic backup.

Summary of NAS Benefits

  • Relatively inexpensive
  • 24/7 and remote data availability
  • Good expandability
  • Redundant storage architecture
  • Automatic backups to other devices and cloud
  • Flexibility

Network attached Storage (NAS)

Synology NAS

NAS with eight drive bays for 3.5″ disk drives

Limitations of NAS

The weaknesses of a NAS are related to scale and performance. As more users need access, the server might not be able to keep up and could require the addition of more server horsepower. The other weakness is related to the nature of Ethernet itself. By design, Ethernet transfers data from one place to another via packets, dividing the source into a number of segments and sending them along to their destination. Any of those packets could be delayed, or sent out of order, and might not be available to the user until all of the packets arrive and are put back in order.

Any latency (slow or retried connections) is usually not noticed by users for small files, but can be a major problem in demanding environments such as video production, where files are extremely large and latency of more than a few milliseconds can disrupt production steps such as rendering.

Basic Definitions — What is SAN?

A SAN is a way to provide users shared access to consolidated, block level data storage, even allowing multiple clients to access files at the same time with very high performance. A SAN enhances the accessibility of storage devices such as disk arrays and tape libraries by making them appear to users as if they were external hard drives on their local system. By providing a separate storage-based network for block data access over high-speed Fibre Channel, and avoiding the limitations of TCP/IP protocols and local area network congestion, a SAN provides the highest access speed available for media and mission critical stored data.

Storage area network (SAN)

SAN connecting yellow storage devices with orange servers via purple Fibre Channel switches

SAN connecting yellow storage devices with orange servers via purple Fibre Channel switches

Benefits of SAN

Because it’s considerably more complex and expensive than NAS, SAN is typically used by large corporations and requires administration by an IT staff. For some applications, such as video editing, it’s especially desirable due to its high speed and low latency. Video editing requires fair and prioritized bandwidth usage across the network, which is an advantage of SAN.

A primary strength of a SAN is that all of the file access negotiation happens over Ethernet while the files are served via extremely high speed Fibre Channel, which translates to very snappy performance on the client workstations, even for very large files. For this reason SAN is widely used today in collaborative video editing environments.

Summary of SAN Benefits

  • Extremely fast data access
  • Dedicated network for storage relieves stress on LAN
  • Highly expandable
  • OS level (block level) access to files
  • High quality-of-service for demanding applications such as video editing

Limitations of SAN

The challenge of SAN can be summed up in its cost and administration requirements — having to dedicate and maintain both a separate Ethernet network for metadata file requests and implement a Fibre Channel network can be a considerable investment. That being said, SANs are really the only way to provide very fast data access for a large number of users that also can scale to supporting hundreds of users at the same time.

What’s the Diff: NAS vs SAN

NAS SAN
Typically used in homes and small to medium sized businesses. Typically used in professional and enterprise environments.
Less expensive More expensive
Easier to manage Requires more administration
Data accessed as if it were a network-attached drive (files) Servers access data as if it were a local hard drive (blocks)
Speed dependent on local TCP/IP usually Ethernet network, typically 100 megabits to one gigabit per second. Generally slower throughput and higher latency due to slower file system layer. High speed using Fibre Channel, 2 gigabits to 128 gigabits per second. Some SANs use iSCSI as a less expensive but slower alternative to Fibre Channel.
I/O protocols: NFS, SMB/CIFS, HTTP SCSI, iSCSI, FCoE
Lower-end not highly scalable; high-end NAS scale to petabytes using clusters or scale-out nodes Network architecture enables admins to scale both performance and capacity as needed
Does not work with virtualization Works with virtualization
Requires no architectural changes Requires architectural changes
Entry level systems often have a single point of failure, e.g. power supply Fault tolerant network with redundant functionality
Susceptible to network bottlenecks Not affected by network traffic bottlenecks. Simultaneous access to cache, benefiting applications such as video editing.
File backups and snapshots economical and schedulable. Block backups and mirrors require more storage.

NAS/SAN Convergence

The benefits of SAN are motivating some vendors to offer SAN-like products at lower cost chiefly by avoiding the high expense of Fibre Channel networking. This has resulted in a partial convergence of NAS and SAN approaches to network storage at a lower cost than purely SAN.

One example is Fibre Channel over Ethernet (FCoE), which supports block level transfers over standard LAN at speeds of 10GB/sec+. For smaller deployments, iSCSI is even less expensive, allowing SCSI commands to be sent inside of IP packets on a LAN. Both of these approaches avoid expensive Fibre Channel completely, resulting in slower, but less expensive ways to get the block level access and other benefits of a SAN.

Are You Using NAS, SAN, or Both?

If you are using NAS or SAN, we’d love to hear from you about what you’re using and how you’re using them. Also, please feel free to suggest other topics for this series.

The post What’s the Diff: NAS vs SAN appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Buying a Hard Drive this Holiday Season? These Tips Will Help

Post Syndicated from Andy Klein original https://www.backblaze.com/blog/hard-drive-buying-guide/

Hard drives with bows
Over the last few years we’ve shared many observations in our quarterly Hard Drive Stats reports that go beyond the hard drive failure rates. We decided to consolidate some of these additional observations into one post just in time for the holiday buying season. If you have “buy a hard drive” on your shopping list this holiday season, here is just about everything we know about hard disk drives.

First, let’s establish that we are talking about hard disk drives (HDDs) here and not solid state drives (SSDs). Here’s a Backblaze “What’s the Diff” blog post where we discuss the differences between HDD and SSD drives.

How Will You Use Your HDD?

Hard drive manufacturers build drive models for different use cases; that is, a given drive model is optimized for a given purpose. For example, a consumer drive may spin slower to save energy and provides little if any access to tools that can adjust the firmware settings on the drive. An enterprise class drive, on the other hand, is typically much faster and provides the user with access to features they can tweak to adjust performance and/or power usage.

Each drive manufacturer has their own criteria for their use cases, but in general there are five categories: consumer, NAS (network attached storage), archiving/video recording, enterprise, and more recently, data center. The different drive manufacturers have different variations on these categories, so the first thing you should do is to know what you are going to do with the drive before you start looking.

Hard Drive Recording Technologies

For a long time, the recording technology a drive manufacturer used was not important. Then SMR (shingled magnetic recording) drives appeared a couple of years ago.

Let’s explain:

PMR: Perpendicular Magnetic Recording
This is the technology inside of most hard drives. With PMR data is written to and read from circular tracks on a spinning platter.
SMR: Shingled Magnetic Recording
This type of drive overlaps recording tracks to store data at a lower cost than PMR technology. The downside occurs when data is deleted and that space is reused. If existing data overlaps the space you want to reuse, this can mean delays in writing the new data. These drives are great for archive storage (write once, read many) use cases, but if your files turn over with some regularity, stick with PMR drives.

That sounds simple, but here are two things you should know:

  1. SMR drives are often the least expensive drives available when you consider the cost per gigabyte. If you are price sensitive, you may believe you are getting a great deal, but you may be buying the wrong drive for your use case. For example, buying SMR drives for your NAS device running RAID 6 would be ugly because of all the rewrites that may be involved.
  2. It is sometimes really hard to figure out if the drive you want to buy is an SMR or PMR drive. For example, based on the cost per gigabyte, the 8TB Seagate external drive (model: STEB8000100) is one of the least expensive external drives out there right now. But, the 8TB drive inside is an SMR drive, and that fact is not obvious to the buyer. To be fair, the manufacturers try to guide buyers to the right drive for their use case, but a lot of that guiding information is lost on reseller sites such as Amazon and Newegg, where the buyer is often blinded by price.

Over the next couple of years, HAMR (heat-assisted magnetic recording) by Seagate and MAMR (microwave-assisted magnetic recording) by Western Digital will be introduced, making the drive selection process even more complicated.

What About Refurbished Drives?

Refurbished drives are hard drives that have been returned to the manufacturer and repaired in some way to make them operational. Given the cost, repairs are often limited to what can be done in the software or firmware of the failed drive. For example, the repair may consist of identifying a section of bad media on a drive platter and telling the drive to read and write around it.

Once repaired, refurbished drives are tested and often marked certified by the manufacturer, e.g. “Certified Refurbished.” Refurbished drives are typically less expensive and come with a limited warranty, often one year or less. You can decide if you want to use these types of drives in your environment.

Helium-Filled versus Air-Filled Drives

Helium-filled drives are finally taking center stage after spending years as an experimental technology. Backblaze has in part used helium-filled drives since 2015, and over the years we’ve compared helium-filled drives to air-filled drives. Here’s what we know so far.

The first commercial helium-filled drives were 6TB; the transition to helium took hold at 8TB as we started seeing helium-filled 8TB drives from every manufacturer. Today helium-filled 12 and 14TB drives are now available at a reasonable price per terabyte.

Helium drives have two advantages over their air-filled cohorts: they create less heat and they use less power. Both of these are important in data centers, but may be less important to you, especially when you consider the primary two disadvantages: a higher cost and lack of experience. The street-price premium for a helium-filled drive is roughly 20% right now versus an air-filled drive of the same size. That premium is expected to decrease as time goes on.

While price is important, the lack of experience of helium-filled drives may be more interesting as these drives have only been in the field in quantity a little over four years. That said, we have had helium-filled drives in service for 3.5 years. They are solid performers with a 1.2% annualized failure rate and show no signs of hitting the wall.

Enterprise versus Consumer Drives

In our Q2 2018 Hard Drive Stats report we delved into this topic, so let’s just summarize some of the findings below.

We have both 8TB consumer and enterprise models to compare. Both models are from Seagate. The consumer drive is model ST800DM002 and the Enterprise drive model is ST800NM0055. The chart below, from the Q2 2018 report, shows the failure rates for each of these drive models at the same average age of all of the drives of the specified model.

Annualized Hard Drive Failure Rates by Time table

When you constrain for the average age of each of the drive models, the AFR (annualized failure rate) of the enterprise drive is consistently below that of the consumer drive for these two drive models — albeit not by much. By the way, conducting the same analysis at an average age of 15 months showed little change, with the consumer drive recording a 1.10% AFR and the enterprise drive holding at 0.97% AFR.

Whether every enterprise model is better than every corresponding consumer model is unknown, but below are a few reasons you might choose one class of drive over another:

Enterprise Class Drives

  • Longer Warranty: 5 years vs. 2 years
  • More Accessible Features, i.e. Seagate PowerChoice technology
  • Faster reads and writes

Consumer Class Drives

  • Lower Price: Up to 50% less
  • Similar annualized failure rates as enterprise drives
  • Uses less power and produces less heat

Hard Drive Failure Rates

As many of you know, each quarter Backblaze publishes our Hard Drive Stats report for the hard drives in our data centers. Here’s the lifetime chart from our most recent Q3 2018 report.

Backblaze Lifetime Hard Drive Failure Rates table

Along with the report, we also publish the data we used to create the reports. We are not alone. Let’s look at the various ways you can find hard drive failure rates for the drive you wish to purchase.

Backblaze AFR (annualized failure rate)
The failure rate of a given hard drive model based on the number of days a drive model is in use and the number of failures of that drive model. Here’s the formula:

( ( Drive Failures / ( Drive Days / 365 ) ) * 100 )
MTBF (mean time between failures)
TBF is the term some disk drive manufacturers use to quantify disk drive average failure rates. It is the average number of service hours between failures. This is similar to MTTF (mean time to failure), which is the average time to the first failure. MTBF has been superseded by AFR for some drive vendors as described below.
AFR (Seagate and Western Digital)
These manufacturers have decided to replace MTBF with AFR. Their definition of AFR is the probable percent of failures per year, based on the manufacturer’s total number of installed units of similar type. While Seagate and WD don’t give the specific formula for calculating AFR, Seagate notes that AFR is similar to MTBF and differs only in units. One way for converting MTBF to AFR can be found here.
Comparing Backblaze AFR to the Seagate/WD AFR
The Backblaze environment is a closed system, meaning we know with a high degree of certainty the variables we need to compute the Backblaze AFR percentage. We also know most, if not all, of the mitigating factors. The Seagate/WD AFR environment is made up of potentially millions of drives in the field (home, office, mobile, etc.) where the environmental variables can be quite varied and in some cases unknown. Either of the AFR calculations can be considered as part of your evaluation if you are comfortable with how they are calculated.
CDL (component design life)
This term is used by Western Digital in their support knowledge base although we don’t see it in their technical specifications yet. The example provided in the knowledge base article is, “The Component Design Life of the drive is 5 years and the Annualized Failure Rate is less than 0.8%.” With those two numbers you can calculate that no more than four out of 100 drives will die in a five-year period. The is really good information, but it is not readily available yet.

Which Hard Drive Do I Need?

While hard drive failure rates are interesting, we believe that our Hard Drive Stats reports are just one of the factors to consider in your hard drive buying decision. Here are some things you should think about, in no particular order:

  • Your use case
    • What you will do with the drive.
  • What size drive do you need?
    • Using it as a Time Machine backup? It should be 3-4 times the size of your internal hard drive. Using it as an archive for your photo collection? — bigger is better.
  • How long do you want the drive to last?
    • Forever is not a valid answer. We suggest starting with the warranty period and subtracting a year if you move the drive around a lot or if you fill it up and stuff it in the closet.
  • The failure rate of the drive
    • We talked about that above.
  • What your friends think
    • You might get some good advice.
  • What the community thinks
    • reddit, Hacker News, Spiceworks, etc.
  • Product reviews
    • I read them, but only to see if there is anything else worth investigating via other sources.
  • Product review sites
    • These days, many review sites on the internet are pay-to-play, although not all. Pay-to-play means the vendor pays the site either for their review or if the review leads to a sale. Sometimes, whoever pays the most gets to the top of the list. This isn’t true for all sites, but often it is really hard to tell the good guys. One of our favorite sites, Tom’s Hardware, has stopped doing HDD reviews, so if you have a site you trust for such reviews, share it in the comments, we’d all like to know.
  • The drive manufacturer
    • Most drive manufacturer websites provide information that can help you determine the right drive for your use case. Of course, they are also trying to sell you a drive, but the information, especially the technical specs, can be useful.

What about price? We left that out of our list as many people start and end their evaluation with just price and we wanted to mention a few other things we thought could be important. Speaking of price…

What’s a Good Price for a Hard Drive?

Below is our best guess as to what you could pay over the next couple of months for different sized internal drives. Of course, there are bound to be some great discounts on Black Friday, Cyber Monday, Hanukkah, Christmas, Kwanzaa, Boxing Day, Winter Solstice, and Festivus — to name a few holiday season reasons for a sale on hard disk drives.

Drive Size Price Cost per GB
1TB $35 $0.035
2TB $50 $0.25
3TB $75 $0.25
4TB $100 $0.25
6TB $170 $0.28
8TB $250 $0.31
10TB $300 $0.30
12TB $380 $0.32
14TB $540 $0.39

How Much Do External Hard Drives Cost?

We wanted to include the same information about external hard drives, but there is just too much unclear information to feel good about doing it. While researching this topic, we came across multiple complaints about a wide variety of external drive systems containing refurbished or used drives. In reviewing the advertisements and technical specs, the fact that the HDD inside an external drive sometimes is not new often gets left off the specifications. In addition, on Amazon and similar sites, many of the complaints were from purchases made via third party sellers and not the original external drive manufacturers, so check the “by” tag before buying.

Let’s make it easy: an external hard drive should have at least a two-year warranty and be available from a trusted source. The list price for the external drive should be about 10-15% higher than the same sized internal drive. What you will actually pay, the street price, is based on supply and demand and a host of other factors. Don’t be surprised if the cost of an external drive is sometimes less than a corresponding internal drive — that’s just supply and demand at work. Following this guidance doesn’t mean the drive won’t fail, it just means you’ll have better odds at getting a good external drive for your money.

One More Thing Before You Buy

The most important thing to consider when buying a hard drive is the value of the data on the drive and what it would cost to replace that data. If you have a good backup plan and practice the 3-2-1 backup strategy, then the value of a given drive is low and limited to the time and cost it takes to replace the drive that goes bad. That’s annoying, yes, but you still have your data. In other words, if you want to get the most for your money when buying a hard drive, have a good backup plan.

The post Buying a Hard Drive this Holiday Season? These Tips Will Help appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Modern Storage Workflows in the Age of Cloud

Post Syndicated from Skip Levens original https://www.backblaze.com/blog/cloud-based-video-production-workflows/

Video Production Workflow

Not too long ago, hardware storage vendors held an iron grip on what kinds of storage underpinned your creative, film, and broadcast workflows. This storage took many complex forms — RAIDs, JBODs, SANs, NAS systems, tape robots, and more. All of it was expensive, deeply complex, and carried fat vendor margins and high support costs.

How Storage Can Make Your Video Production Workflow More Efficient

But when you’re considering storage in today’s technology environment — whether it’s cloud, on-site storage, or a USB stick — the guiding principle in choosing storage for your creative production should simply be to choose the storage that best fits each workflow step.

Production Storage Maxim: Choose the storage that best fits each workflow step

Doing your best creative work is what builds your customer base, boosts your reputation, and earns you revenue and royalties. So any time sunk into legacy storage solutions, wrestling with complexity, unneeded production steps, refereeing competing vendors, and overpaying for, well, everything, just gets in the way of what you really want to do, create.

The right answer for your specific production needs is a function of the size of your production team and the complexity of your operating environment. Whatever that answer is, it should be as frictionless an environment as possible that helps you get your work done more efficiently and gives you the most flexibility.

An independent filmmaker can follow this production storage evaluation process for each stage of their workflow and decide to make do with a small deskside RAID system for primary production storage, and depend on the cloud for everything else.

A large, global production team will probably need multiple SANs in each production office and a complex series of cloud and dedicated playout applications and systems. If your environment falls somewhere between those two extremes, then your ideal solution mix does as well.

Traditional Content Production Workflow - Ingest > Work-in-Process > Deliver > Archive

The traditional content production workflow is thought of as a linear process. Content is ingested as raw camera files pulled into a shared work-in-process storage for editors, the final cut is then delivered to the client, and when the project is finished all files are saved off to an archive.

Simplified Production Workflow Steps

Let’s look at what the storage requirements and needs are for each of the common steps in a production workflow and where cloud can add value. Along the way, we’ll call out concrete examples of cloud capabilities at each stage with B2 cloud storage.

Ingest Stage - Ingest Stage Goals: Safely retrieve and protect files from capture media and move to production environment. Ingest Stage Needs: File data protection - Easy path to Production Storage. Where Cloud Can Add Value: Ingest and archive in one step

The Ingest Stage

Media copied in the ingest phase typically needs to get off of camera carts and flash drives as quickly and safely as possible and transported to the editing environment. Since those camera carts need to be used again for the next shot, pressure to get files copied over quickly (but safely) is intense.

Any time that critical content exists only in one place is dangerous. At this stage, lost or corrupted files mean a reshoot, which may not be practical or even possible.

Storage Needs for Ingest

Storage at the ingest stage can be very rudimentary and is often satisfied by just copying files from camera carts to an external drive, then to another drive as a safety, or by putting a RAID system on a crash cart on-set. Every team tends to come up with a different solution.

Where Cloud Can Add Value to Ingest

But even if your data wranglers aren’t ready to give up external hard drives here, one way cloud can help in the ingest stage is to help combine your ingest and archive for safety steps.

Instead of carrying carts from the shoot location to the production environment and copying them over to production storage, you could immediately start uploading content via the internet to your cloud storage, simultaneously copying over those files safely, and making them available to your entire team immediately.

When you restructure your workflow like this, you’ll get better than RAID-level protection for your content in the cloud. And by checking content into your archive first, your asset manager tools can immediately start processing those files by adding tags and generating lighter weight proxies. As soon as the files hit cloud storage, your entire team can start working on them. They can immediately begin tagging and reviewing files, and even mark edit points before handing off to editors, thereby speeding up production dramatically.

Some creatives have hit a roadblock in trying to take advantage of the cloud. Data transfer has historically been gated by the available upload bandwidth at your given location, but our customers have solved this in some interesting ways.

Producers, editors, and reporters are finding that even cellular 4G internet connections make it feasible to immediately start uploading raw shots to their cloud storage. Others make it routine to stop off at a data center or affiliate with excellent upload speeds on their way in from the field.

Either way, even novice shooters and freelancers can safely get content into your system quickly in a system that can be as simple as an upload bucket in your B2 account and making sure that your media or project manager tools are configured to watch those upload points.

Cloud Capability Example — Use a Backblaze Fireball to Rapidly Ingest Content

Backblaze offers a Rapid Ingest Service to help get large amounts of your content into your Backblaze account quickly. Backblaze ships you a 70TB storage system that you connect to your network and copy content to. When the system is shipped back to Backblaze, it is quickly moved directly into your B2 account, dramatically reducing ingest times.

 

Cloud Capability Example — Share Files Directly From Cloud

Archive.zip file in B2

An example of navigating to a file-review bucket in the B2 web interface to copy the direct sharing link to send to a reviewer

In addition to the archive on ingest technique, many customers share files for approval review or dailies directly from their Backblaze B2 account’s web interface.

If your B2 bucket for finished files is public, you can get a direct share link from the Backblaze account management website and simply send that to your customer, thereby eliminating a copy step.

You can even snapshot a folder of your content in B2, and have Backblaze ship it directly to your customer.

Work in Process Stage - WIP Stage Goals: Support collaborative, simultaneous editing of source files to finished content. WIP Stage Needs: Performance to support shared, collaborative editing access for many users. Very large file support. Where Cloud Can Add Value: Keeping expensive primary production storage running efficiently.

The Work-In-Process Stage

Work-in-process or primary production storage is the main storage used to support collaborative editing and production of content. The bulk of what’s thought of as collaborative editing happens in this stage.

For simplicity we’re combining several steps under the umbrella of work-in-process such as craft editing, voiceover, sound, ADR, special effects, and even color grading and finish etc. under a far simpler work-in-process step.

As audio, color grading and SFX steps get more complex, they sometimes need to be broken out into separate, extremely high performance storage such as more exotic (and expensive) flash-based storage that then feeds the result back to WIP storage.

Work-in-Process Stage Storage Needs

Storage performance requirements in this stage are extremely hard to meet, demanding the ability to serve multiple editors, each pulling multiple, extremely large streams of video files as they edit raw shots into a complex, visual story. Meeting this requirement usually requires either equipment intensive SAN, or a NAS that scales to eye-watering size and price.

Many production environments have gotten in the habit of keeping older projects and media assets on the shared production environment alongside current production files, knowing that if those files are needed they can be retrieved quickly. But this also means that production storage fills up quickly, and it’s tempting to let more and more users not involved in primary production have access to those files as well, both of which can slow down production storage and creation of your content.

Having to make a rush purchase to expand or add to your SAN is not fun, especially in the middle of a project, so regularly moving any files not needed for current production to your content archive is a great strategy to keep your production storage as light and small as possible so that it can last over several seasons.

Where Cloud Can Add Value to Work-in-Process

By regularly moving content from your production storage you keep it light, fast, and simpler to manage. But that content still needs to be readily available. Cloud is an excellent choice here as content is both immediately available and stored on highly resilient object storage. In effect, you’re lightening the burden on your primary storage, and using cloud as an always ready, expanding store for all of your content. We’ll explore this concept more in the archive stage.

Deliver Stage - Deliver Stage Goals: Securely deliver finished files to upstream/downstream clients. Deliver Stage Needs: High reliability. Separation from primary production storage. Where Cloud Can Add Value: Share files directly and securely from cloud without copying.

The Deliver Stage

The deliver stage, where your finished work is handed off to your customer, varies depending on what type of creative you are. Broadcast customers will almost always need dedicated playout server appliances, and others will simply copy files to where they’re needed by downstream customers, or upstream to a parent organization for distribution. But, at some level, we all have to deliver our work when it’s done.

Deliver Stage Storage Needs

Files for delivery should be moved off of your primary production storage and delivered in a separate workflow available to dedicated workflow or playout tools. Whatever the workflow, this storage needs to be extremely reliable and available for your customers whenever it is needed.

Where Cloud Can Add Value to Deliver

Whether content delivery in your workflow is met by copying files to a playout server or giving a finished file to a customer, cloud can help cut down on the number of steps to get the content to its final destination while giving you extreme reliability.

Cloud Capability Example — Serve Time-Limited Links to Content

Many customers use the Backblaze B2 API to add expiration limits that can last from seconds to a week to shared links:

B2 command-line

An example of using the B2 command-line tool to generate time-expiring tokens for content sharing and delivery

If your team is comfortable writing scripts to automate your workflow, this can be a powerful way to directly share files simply and quickly with tools provided by Backblaze.

For more information see this B2 Article: Get Download Authorization

 

Cloud Capability Example — Move Content Directly to Your Delivery and Distribution Servers

Serving your content to a wide audience via your website, content channel, or app is an increasingly popular way to deliver content. And thanks to our recent Cloudflare agreement, you can now move content from your B2 storage over to Cloudflare’s content delivery network at zero transfer cost for your content application or website.For more information see this B2 article: How to Allow Cloudflare to Fetch Backblaze B2 Content

Archive Stage - Archive Stage Goals: Securely deliver finished files to upstream/downstream clients. Archive Stage Needs: High reliability. Separation from primary prodcution storage. Where Cloud Can Add Value: Serve as your content backplane across all workflow steps.

The Archive Stage

At last, we come to the archive stage of content creation, traditionally thought of as the end of the traditional content creation chain, the source of the most frustration for creatives, and the hardest storage to size properly.

Traditionally, when a project or season of a show is finished, all of the files used to create the content are moved off of expensive primary production storage and stored on separate, highly reliable storage in case they are needed again.

Archive Stage Storage Needs

Archive storage needs to be a safe repository for all of the content that you’ve created. It should scale well at a sustainable price, and make all archived content available immediately when requested by your users and workflow tools like asset managers.

Tape was often chosen to store these archive files because it was cheaper than disk-based storage and offered good reliability. But choosing tape required a large investment in specialized tape systems, tape media, and the associated support contracts and maintenance.

Tape based archiving strategies usually rely on compressing content as it’s written to tape to hit the advertised storage capacity of tape media. But video content is already stored in a compressed container, so compressing those files as they’re written and retrieved from tape offers no advantage and only slows the process down.

Here we find the chief drawback of tape based content archives for many customers: the time required to retrieve content from those tape systems. As the pace of production has increased, many customers find they can no longer wait for tape systems to return archive sets or unarchive files.

Where Cloud Can Add Value to Archive

The archive stage is where cloud has the most impact on your entire workflow. The benefits of cloud itself are familiar: the ability to scale up or down instantly as your needs change, paying only for the storage you actually use, extremely high object storage file reliability, and availability anywhere there is a network connection.

Modern Content Production Workflow - Ingest > Archive as a Cloud Content Backplane ><Work-In-Process

Creating The Cloud Content Backplane

Having all of your content immediately available to your production storage and your asset management systems is emerging as the killer feature of cloud for production environments. By adding cloud, your content production goes from a linear process to a highly active one where content can freely check in and out of all of your other workflow steps as you’re producing content.

By shifting your content archives to cloud like Backblaze B2, you are creating, in effect, a cloud content backplane that supports your entire content creation and delivery process with these new capabilities:

  • New productions now have access to every file you might possibly need without waiting, letting you explore more creative choices
  • A single, authoritative content repository backing all of your creative production lets you phase out other storage and the associated management headaches and expense
  • You can now serve and deliver files directly from your cloud-based content archive with no impact on production storage
  • Having content in a single place means that your workflow tools like asset managers work better. You can find files across your entire content store instantly, and even archive or move files from your production storage to your cloud content archive automatically

The content not needed on your work-in-process storage is both highly protected and immediately available wherever you need it. Your entire workflow can get much simpler with fewer steps, and you can phase out storage you no longer need on-site.

Above all, you’ll have fewer steps between you and creating great content, and you’ll be able to explore new creative options faster while shifting to a pay-as-you-use-it model for all of your content storage.

In part two, we’ll explore the ways your new cloud-delivered content archive backplane can dramatically improve how you create, deliver, and monetize content with other cloud-based technologies in the age of cloud.

The post Modern Storage Workflows in the Age of Cloud appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Hard Drive Stats for Q3 2018: Less is More

Post Syndicated from Andy Klein original https://www.backblaze.com/blog/2018-hard-drive-failure-rates/

Backblaze Drive Stats Q3 2018

As of September 30, 2018 Backblaze had 99,636 spinning hard drives. Of that number, there were 1,866 boot drives and 97,770 data drives. This review looks at the quarterly and lifetime statistics for the data drive models in operation in our data centers. In addition, we’ll say goodbye to the last of our 3TB drives, hello to our new 12TB HGST drives, and we’ll explain how we have 584 fewer drives than last quarter, but have added over 40 petabytes of storage. Along the way, we’ll share observations and insights on the data presented and we look forward to you doing the same in the comments.

Hard Drive Reliability Statistics for Q3 2018

At the end of Q3 2018, Backblaze was monitoring 97,770 hard drives used to store data. For our evaluation, we remove from consideration those drives that were used for testing purposes and those drive models for which we did not have at least 45 drives (see why below). This leaves us with 97,600 hard drives. The table below covers what happened in Q3 2018.

Backblaze Q3 2018 Hard Drive Failure Rates chart

Notes and Observations

  • If a drive model has a failure rate of 0%, it only means there were no drive failures of that model during Q3 2018.
  • Quarterly failure rates can be volatile, especially for models that have a small number of drives and/or a small number of Drive Days.
  • There were 170 drives (97,770 minus 97,600) that were not included in the list above because we did not have at least 45 of a given drive model. We use 45 drives of the same model as the minimum number when we report quarterly, yearly, and lifetime drive statistics.

When to Replace a Hard Drive

As noted, at the end of Q3 that we had 584 fewer drives, but over 40 petabytes more storage space. We replaced 3TB, 4TB, and even a handful of 6TB drives with 3,600 new 12TB drives using the very same data center infrastructure, i.e. racks of Storage Pods. The drives we are replacing are about 4 years old. That’s plus or minus a few months depending on how much we paid for the drive and a number of other factors. Keeping lower density drives in service when higher density drives are both available and efficiently priced does not make economic sense.

Why Drive Migration Will Continue

Over the next several years, data growth is expected to explode. Hard drives are still expected to store the bulk of that data, meaning cloud storage companies like Backblaze will have to increase capacity by either increasing existing storage density and/or building, or building out, more data centers. Drive manufacturers, like Seagate and Western Digital, are looking at HDD storage densities of 40TB as early as 2023, just 5 years away. It is significantly less expensive to replace lower density operational drives in a data center versus building a new facility or even building out an existing facility to house the higher density drives.

Goodbye 3TB WD Drives

For the last couple of quarters, we had 180 Western Digital 3TB drives (model: WD30EFRX) remaining — the last of our 3TB drives. In early Q3, they were removed and replaced with 12TB drives. These 3TB drives were purchased in the aftermath of the Thailand drive crisis and installed in mid-2014 and were still hard at work when we replaced them. Sometime over the next couple of years we expect to say goodbye to all of our 4TB drives and upgrade them to 14, 16, or even 20TB drives. After that it will be time to “up-density” our 6TB systems, then our 8TB systems, and so on.

Hello 12TB HGST Drives

In Q3 we added 79 HGST 12TB drives (model: HUH721212ALN604) to the farm. While 79 may seem like an unusual number of drives to add, it represents “stage 2” of our drive testing process. Stage 1 uses 20 drives, the number of hard drives in one Backblaze Vault tome. That is, there are are 20 Storage Pods in a Backblaze Vault, and there is one “test” drive in each Storage Pod. This allows us to compare the performance, etc., of the test tome to the remaining 59 production tomes (which are running already-qualified drives). There are 60 tomes in each Backblaze Vault. In stage 2, we fill an entire Storage Pod with the test drives, adding 59 test drives to the one currently being tested in one of the 20 Storage Pods in a Backblaze Vault.

To date, none of the 79 HGST drives have failed, but as of September 30th, they were installed only 9 days. Let’s see how they perform over the next few months.

A New Drive Count Leader

For the last 4 years, the drive model we’ve deployed the most has been the 4TB Seagate drive, model ST4000DM000. In Q3 we had 24,208 of this drive model, which is now only good enough for second place. The 12TB Seagate drive, model ST12000NM0007, became our new drive count leader with 25,101 drives in Q3.

Lifetime Hard Drive Reliability Statistics

While the quarterly chart presented earlier gets a lot of interest, the real test of any drive model is over time. Below is the lifetime failure rate chart for all the hard drive models in operation as of September 30th, 2018. For each model, we compute their reliability starting from when they were first installed.

Backblaze Lifetime Hard Drive Failure Rates Chart

Notes and Observations

  • The failure rates of all of the larger drives (8, 10, and 12 TB) are very good: 1.21% AFR (Annualized Failure Rate) or less. In particular, the Seagate 10TB drives, which have been in operation for over 1 year now, are performing very nicely with a failure rate of 0.48%.
  • The overall failure rate of 1.71% is the lowest we have ever achieved, besting the previous low of 1.82% from Q2 of 2018.

The Hard Drive Stats Data

The complete data set used to create the information used in this review is available on our Hard Drive Test Data page. You can download and use this data for free for your own purpose. All we ask are three things: 1) you cite Backblaze as the source if you use the data, 2) you accept that you are solely responsible for how you use the data, and 3) you do not sell this data to anyone. It is free.

If you just want the summarized data used to create the tables and charts in this blog post you can download the ZIP file containing the MS Excel spreadsheet.

Good luck and let us know if you find anything interesting.

The post Hard Drive Stats for Q3 2018: Less is More appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Move Even Your Largest Archives to B2 with Fireball and Archiware P5

Post Syndicated from Skip Levens original https://www.backblaze.com/blog/archiware-p5-cloud-backup/

Archiware P5 and Fireball

Backblaze B2’s reliability, scalability, and affordable, “pay only for what you use” pricing means that it’s an increasingly popular storage option for all phases of content production, and that’s especially true for media archiving.

By shifting storage to B2, you can phase out hard-to-manage and expensive local backup storage and clear space on your primary storage. Having all of your content in a single place — and instantly available — can transform your production and keep you focused on the creative process.

Fireball Rapid Ingest to Speed Your First Migration to Backblaze B2

Once you sign up for Backblaze B2, one tool that can speed an initial content migration tremendously is Backblaze’s Fireball rapid ingest service. As part of the service, Backblaze ships you a 70TB storage system. You then copy over all the content that you want in B2 to the Fireball system: all at local network speeds. Once the system is shipped to Backblaze, it’s quickly moved to your B2 account, a process far faster than uploading those files over the internet.

Setting Up Your Media Archive

Since manually moving files to archive and backing up project folders can be very time-consuming, many customers choose software like Archiware P5 that can manage this automatically. In P5’s interface you can choose files to add to archive libraries, restore individual files to your local storage from B2, and even browse all of your archive content on B2 with thumbnail previews, and more.

However, many media and entertainment customers have terabytes and terabytes of content in “archive” — that is, project files and content not needed for a current production, but necessary to keep nearby, ready to pull into a new production.

They’d love to get that content into their Backblaze B2 account and then manage it with an archive, sync, backup solution like Archiware P5. But the challenge facing too many is how to get all these terabytes up to B2 through the existing bandwidth in the office. Once the large, initial archive is loaded, the incrementals aren’t a problem, but getting years of backlog pushed up efficiently is.

For anyone facing that challenge, we’re pleased to announce the Archiware P5 Fireball Integration. Our joint solution provides any customer with an easy way to get all of their archives loaded into their B2 account without having to worry about bandwidth bottlenecks.

Archiware P5 Fireball Integration

A backup and archive manager like Archiware P5 is a great way to get your workflow under control and automated while ensuring that your content is safely and reliably stored. By moving your archives offsite, you get the highest levels of data protection while keeping your data immediately available for use anytime, anywhere.

With the newest release, Archiware P5 can archive directly to Fireball at fast, local network speeds. Then, once your Fireball content has been uploaded to your Backblaze account, a few clicks are all that is needed to point Archiware at your Backblaze account as the new location of your archive.

Finally, you can clear out those closets of hard drives and tape sets!

Archiware P5 to B2 workflow

Archiware P5 can now archive directly to Fireball at local network speeds, which are then linked to their new locations in your B2 accounts. With a few clicks you can get your entire archive uploaded to the B2 cloud without suffering any downtime or bandwidth issues.

For detailed information about configuring Archiware to archive directly to Fireball:

For more information about Backblaze B2 Fireball Rapid Ingest Service:

Archiware on Synology and QNAP NAS Devices

Archiware, NAS and B2

Archiware P5 can also now run directly on several Synology, QNAP, and G-Tech NAS systems to archive and move content to your Backblaze B2 account over the internet

With their most recent releases Archiware now supports several NAS system devices from QNAP, Synology, and G-Tech as P5 clients or servers.

The P5 software is installed as an application from the NAS vendor’s app store and runs directly on the NAS system itself without having to install additional hardware.

This means that all of your offices or departments with these NAS systems can now fully participate in your sync, archive, and backup workflows, and each of them can archive off to your central Backblaze B2 account.

For more information:

Archiware plus Backblaze: A Complete Front-to-Back Media Solution

Archiware P5, Fireball, and Backblaze B2 are all important parts of a great backup, archive, and sync plan. By getting all of your content into archive and B2, you’ll know that it’s highly protected, instantly available for new production workflows, and also readily discoverable through thumbnail and search capability.

With the latest version of P5, you not only have your entire production and backup workflows managed, with Fireball you can get even the largest and hardest to move archive safely and quickly into B2, as well!

For more information about the P5 Software Suite: Archiware P5 Software Suite

And to order a Fireball as part of our Rapid Ingest Service, start here: Backblaze B2 Fireball


You might also be interested in reading our recent guest post written by Marc N. Batschkus of Archiware about how to save time, money, and gain peace of mind with an archive solution that combines Backblaze B2 and Archiware P5.

Creating a Media Archive Solution with Backblaze B2 and Archiware P5

 

The post Move Even Your Largest Archives to B2 with Fireball and Archiware P5 appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

iconik and Backblaze — The Cloud Production Solution You’ve Always Wanted

Post Syndicated from Skip Levens original https://www.backblaze.com/blog/iconik-and-backblaze-cloud-production-solution/

Cantemo iconik Plus Backblaze B2 for Media Cloud Production

Cantemo iconik Plus Backblaze B2 for Media Cloud Production

Many of our customers are archiving media assets in Backblaze B2, from long-running television productions, media distributors, AR/VR video creators, corporate video producers, houses of worship, and many more.

They are emptying their closets of USB hard drives, clearing off RAID arrays, and migrating LTO tapes to cloud storage. B2 has been proven to be the least expensive storage for their media archives, while keeping the archives online and accessible. Gone are the days of Post-its, clipboards, and cryptic drive labels defining whether old video footage can be found or not. Migrating archives from one form of storage to another will no longer suck up weeks and weeks of time.

So now that their archives are limitless, secure, always active, and available, the next step is making them actionable.

Our customers have been asking us — how can I search across all of my archives? Can I preview clips before I download the hi-res master, or share portions of the archive with collaborators around the world? Why not use the latest AI tools to intelligently tag my footage with metadata?

To meet all of those needs and more, we are excited to announce that Cantemo’s iconik cloud media management service now officially supports Backblaze B2.

iconik — A Media Management Service

iconik is an affordable and simple-to-use media management service that can read a Backblaze B2 bucket full of media and make it actionable. Your media assets are findable, sortable with full previews, and ready to pull into a new project or even right into your editor, such as Adobe Premiere, instantly.

Cantemo iconik user interface

iconik — Cantemo’s new media management service with AI features to find, sort, and even suggest assets for your project across your entire library

As a true media management service, iconik’s pricing model is a pay-as-you-go service, transparently priced per-user, per month. There are no minimum purchases, no servers to buy, and no large licensing fees to pay. To use iconik, all your users need is a web browser.

iconik Pricing

To get an idea of what “priced-per-user” might look like, most organizations will need at least one administrative user ($89/month), standard users ($49/month) who can organize content, create workflows, and ingest new media, and browse-only users ($19/month), who can search and download what they need. There’s also a “share-only” level that has no monthly charge that lets you incorporate customer and reviewer comments. This should accommodate teams of all kinds and all sizes.

Best of all, iconik is intelligent about how it uses storage, and while iconik charges small consumption fees for proxy storage, bandwidth, etc., they have found that for customers that bring media from Backblaze B2 buckets, consumption charges should be less than 5% of the monthly bill for user licenses.

As part of their launch promotion, if you get started in October, Cantemo will give Backblaze customers a $300 getting started credit!

You can sign up and get started here using the offer code of BBB22018.

Everwell’s Experience with iconik and Backblaze

One of the first customers to adopt iconik with Backblaze is Everwell, a video production company. Everwell creates a constant stream of videos for medical professionals to show in their waiting rooms. Rather than continuously buying upgrades for their in-house asset management system and local storage, iconik allows Everwell to shift their production to the cloud for all of their users. Their new solution will allow Everwell to manage their growing library of videos as new content constantly comes online, and kick off longer form productions with full access to all the assets they need across a fast-moving team that can be anywhere their production takes them.

collage of Everwll video images

Everwell is a fast-growing medical content developer for healthcare givers

To speed up their deployment of iconik, Everwell started with Backblaze’s data ingestion service, Fireball. Everwell copied their content to Fireball, and once back in the Backblaze data center, the data from Fireball was quickly added directly to Everwell’s B2 buckets. iconik could immediately start ingesting the content in place and make it available to every user.

Learn more about Backblaze B2 Fireball

With iconik and Backblaze, Everwell dramatically simplified their workflow as well, collapsing several critical workflow steps into one. For example, by uploading source files to Backblaze B2 as soon as they’re shot, Everwell not only reduces the need to stage local production storage at every site, they ingest and archive in a single step. Every user can immediately start work on their part of the project.

“The ‘everyone in the same production building’ model didn’t work for us any longer as our content service grew, with more editors and producers checking in content from remote locations that our entire team needed to use immediately. With iconik and Backblaze, we have what feels like the modern cloud-delivered production tool we’ve always wanted.”

— Loren Goldfarb, COO, Everwell

See iconik in Action at NAB NYC October 17-18

NAB Show New York - Media In Action October 17-18 2018

Backblaze is at NAB New York. Meet us there!

We’re excited to bring you several chances to see iconik and Backblaze working together.

The first is the NAB New York show, held October 17-18 at the Javits Center. iconik will be shown by Professional Video Technology in Booth N1432, directly behind Backblaze, Booth N1333.

Have you signed up for NAB NY yet? You can still receive a free exhibits pass by entering Backblaze’s Guest Code NY8842.

And be sure to sign up to meet with the Backblaze team at NAB by signing up on our calendar.

Attend the iconik and B2 Webinar on November 20

Soon after NAB NY, Backblaze and iconik will host a webinar to demo the solution called “3 Steps to Making Your Cloud Media Archive ‘active’ With iconik and Backblaze B2.” The webinar will be presented on November 20 and available on demand after November 20. Be sure to sign up for that too!

3 Steps Demo with: iconik and Backblaze B2 Cloud Storage

Sign up for the iconik/B2 Webinar

Don’t Miss the iconik October Launch Promotion

The demand for creative content is growing exponentially, putting more demands on your creative team. With iconik and B2, you can make all of your media instantly accessible within your workflows while adopting a infinitely scalable, pay only for what you use, storage solution.

To take advantage of the iconik October launch promotion and receive $300 free credit with iconik, sign up using the BBB22018 code.

The post iconik and Backblaze — The Cloud Production Solution You’ve Always Wanted appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Backblaze and Cloudflare Partner to Provide Free Data Transfer

Post Syndicated from Gleb Budman original https://www.backblaze.com/blog/backblaze-and-cloudflare-partner-to-provide-free-data-transfer/

 Backblaze B2 Free Data Transfer to Cloudflare

Today we are announcing that beginning immediately, Backblaze B2 customers will be able to download data stored in B2 to Cloudflare for zero transfer fees. This happens automatically once Cloudflare is configured to distribute your B2 files. This means that Backblaze B2 can now be used as an origin store for the Cloudflare CDN and edge network, providing customers enhanced performance and access to their content stored on B2. The result is that customers can save up to 75% on storage versus Amazon S3 to store their content in the cloud and deliver it worldwide.

The zero B2 transfer fees are available to all Cloudflare customers using any plan. Cloudflare customers can also use paid add-ons such as Argo and Workers to enhance the routing and security of the B2 files being delivered over the Cloudflare CDN. To implement this service, Backblaze and Cloudflare have directly connected, thereby allowing near-instant data transfers from B2 to Cloudflare.

Backblaze has prepared a guide on “Using Backblaze B2 storage with Cloudflare.” This guide provides step-by-step instructions on how to set up Backblaze B2 with Cloudflare to take advantage of this program.

The Bandwidth Alliance

The driving force behind the free transfer program is the Bandwidth Alliance. Backblaze and Cloudflare are two of the founding members of this group of forward-thinking cloud and networking companies that are committed to providing the best and most cost-efficient experience for our mutual customers. Additional founding members of the Bandwidth Alliance include Automattic (WordPress), DigitalOcean, IBM Cloud, Microsoft Azure, Packet, and other leading cloud and networking companies.

How Companies Can Leverage the Bandwidth Alliance

Below are examples of how Bandwidth Alliance partners can work together to save customers on their data transfer fees.

Hosting Website Assets

Whether you are a professional webmaster or just run a few homegrown sites, you’ve lived the frustration of having a slow website. Over the past few years these challenges have become more acute as video and other types of rich media have become core to the website experience. This new content has also translated to higher storage and bandwidth costs. That’s where Backblaze B2 and Cloudflare come in.diagram of zero cost data transfer from Backblaze B2 to Cloudflare CDN

Customers can store their videos, photos, and other assets in Backblaze B2’s pay-as-you-go cloud storage and serve the site with Cloudflare’s CDN and edge services. The result is an amazingly affordable cloud-based solution that dramatically improves web site performance and reliability. And customers pay each service for what they do best.

“I am extremely happy with my experience serving html/css/js and over 17 million images from B2 via Cloudflare Workers. Page load time has been great and costs are minimal.”

— Jacob Hands, Lead Developer, FactorioMaps.com

Media Content Distribution

The ability to download content from B2 cloud storage to the Cloudflare CDN for zero transfer cost is the just the beginning. A company needing to distribute media can now store original assets in Backblaze B2, send them to a compute service to transcode and transmux them, and forward the finished assets to be served up by Cloudflare. Backblaze and Packet previously announced zero transfer fees between Backblaze B2 storage and Packet compute services. This enabled customers to store data in B2 at 1/4th the price of competitive offerings and then process data for transcoding, AI, data analysis, and more inside of Packet without worrying about data transfer fees. Packet is also a member of the Bandwidth Alliance and will deliver content to Cloudflare for zero transfer fees as well.

diagram of zero cost data transfer flow from Backblaze B2 to Packet Compute to Cloudflare CDN

Process Now, Distribute Later

A variation of the example above is for a company to store the originals in B2, transcode and transmux the files in Packet, then put those versions back into B2, and finally serve them up via Cloudflare. All of this is done with zero transfer fees between Backblaze, Packet, and Cloudflare. The result is all originals and transmuxed versions are stored at 1/4th the prices of other storage, and served up efficiently via Cloudflare.diagram of data transfer flow between B2 to Packet back to B2 to Cloudflare

In all cases you would only pay for services you use and not for the cost to move data between those services. This results in a predictable and affordable cost for a given project using industry leading best-of-breed services.

Moving Forward

The members of the Bandwidth Alliance are committed to enabling the best and most cost efficient cloud services when it comes to working with data stored in the cloud. Backblaze has committed to a transfer fee of $0 to move content from B2 to either Cloudflare or Packet. We think that’s a great step in the right direction. And if you are cloud provider, let us know if you’d be interested in taking a step like this one with Backblaze.

The post Backblaze and Cloudflare Partner to Provide Free Data Transfer appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Backblaze B2 API Version 2 Beta is Now Open

Post Syndicated from Andy Klein original https://www.backblaze.com/blog/backblaze-b2-api-version-2-beta-is-now-open/

cloud storage workflow image

Since B2 cloud storage was introduced nearly 3 years ago, we’ve been adding enhancements and new functionality to the B2 API, including capabilities like CORS support and lifecycle rules. Today, we’d like to introduce the beta of version 2 of the B2 API, which formalizes rules on application keys, provides a consistent structure for all API calls returning information about files, and cleans up outdated request parameters and returned data. All version 1 B2 API calls will continue to work as is, so no changes are required to existing integrations and applications.

The API Versions section of the B2 documentation on the Backblaze website provides the details on how the V1 and V2 APIs differ, but in the meantime here’s an overview into the what, why, and how of the V2 API.

What Has Changed Between the B2 Cloud Storage Version 1 and Version 2 APIs?

The most obvious difference between a V1 and V2 API call is the version number in the URL. For example:

https://apiNNN.backblazeb2.com/b2api/v1/b2_create_bucket

https://apiNNN.backblazeb2.com/b2api/v2/b2_create_bucket

In addition, the V2 API call may have different required request parameters and/or required response data. For example, the V2 version of b2_hide_file always returns accountId and bucketId, while V1 returns accountId.

The documentation for each API call will show whether there are any differences between API versions for a given API call.

No Change is Required For V1 Applications

With the introduction of V2 of the B2 API there will be V1 and V2 versions for every B2 API call. All applications using V1 API calls will continue to work with no change in behavior. In some cases, a given V2 API call will be different from its companion V1 API call as noted in the B2 API documentation. For the remaining API calls a given V1 API call and its companion V2 call will be the same, have identical parameters, return the same data, and have the same errors. This provides a B2 developer the flexibility to choose how to upgrade to the V2 API.

Obviously, if you want to use the functionality associated with a V2 API version, then you must use the V2 API call and update your code accordingly.

One last thing: beginning today, if we create a new B2 API call it will be created in the current API version (V2) and most likely will not be created in V1.

Standardizing B2 File Related API Calls

As requested by many B2 developers, the V2 API now uses a consistent structure for all API calls returning information about files. To enable this there are some V2 API calls that return additional fields, for example:

Restricted Application Keys

In August we introduced the ability to create restricted applications keys using the B2 API. This capability allows an account owner the ability to restrict who, how, and when the data in a given bucket can be accessed. This changed the functionality of multiple B2 API calls such that a user could create a restricted application key that could break a 3rd party integration to Backblaze B2. We subsequently updated the affected V1 API calls, so they could continue to work with the existing 3rd party integrations.

The V2 API fully implements the expected behavior when it comes to working with restricted application keys. The V1 API calls continue to operate as before.

Here is an example of how the V1 API and the V2 API will act differently as it relates to restricted application keys.

Set-up

  • The B2 account owner has created 2 public buckets, “Backblaze_123” and “Backblaze_456”
  • The account owner creates a restricted application key that allows the user to read the files in the bucket named “Backblaze_456”
  • The account owner uses the restricted application key in an application that uses the b2_list_buckets API call

In Version 1 of the B2 API

  • Action: The account owner uses the restricted application key (for bucket Backblaze_456) to access/list all the buckets they own (2 public buckets).
  • Result: The results returned are just for Backblaze_456 as the restricted application key is just for that bucket. Data about other buckets is not returned.

While this result may seem appropriate, the data returned did not match the question asked, i.e. list all buckets. V2 of the API ensures the data returned is responsive to the question asked.

In Version 2 of the B2 API

  • Action: The account owner uses the restricted application key (for bucket Backblaze_456) to access/list all the buckets they own (2 public buckets).
  • Result: A “401 unauthorized” error is returned as the request for access to “all” buckets does not match the restricted application key, e.g. bucket Backblaze_456. To achieve the desired result, the account owner can specify the name of the bucket being requested in the API call that matches the restricted application key.

Cleaning up the API

There are a handful of API calls in V2 where we dropped fields that were deprecated in V1 of the B2 API, but were still required. So in V2:

  • b2_authorize_account: The response no longer contains minimumPartSize. Use partSize and absoluteMinimumPartSize instead.
  • b2_list_file_names: The response no longer contains size. Use contentLength instead.
  • b2_list_file_versions: The response no longer contains size. Use contentLength instead.
  • b2_hide_file: The response no longer contains size. Use contentLength instead.

Support for Version 1 of the B2 API

As noted previously, V1 of the B2 API continues to function. There are no plans to stop supporting V1. If at some point in the future we do deprecate the V1 API, we will provide advance notice of at least one year before doing so.

The B2 Java SDK and the B2 Command Tool

Both the B2 Java SDK and the B2 Command Line Tool, do not currently support Version 2 of B2 API. They are being updated and will support the V2 API at the time the V2 API exits Beta and goes GA. Both of these tools, and more, can be found in the Backblaze GitHub repository.

More About the Version 2 Beta Program

We introduced Version 2 of the B2 API as beta so that developers can provide us feedback before V2 goes into production. With every B2 integration being coded differently, we want to hear from as many developers as possible. Give the V2 API a try and if you have any comments you can email our B2 beta team at b2beta@backblaze.com or contact Backblaze B2 support. Thanks.

The post Backblaze B2 API Version 2 Beta is Now Open appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Moving Tape Content to Backblaze Fireball with Canister

Post Syndicated from Skip Levens original https://www.backblaze.com/blog/moving-tape-content-to-cloud-storage/


Canister for Fireball: LTO tape to Backblaze B2 migration made 'drag and drop' easy
If you shoot video on the run and wrangle video from multiple sources, you know that reliably offloading files from your camera carts, storage cards, or pluggable SSDs can be a logistical challenge. All of your source files need to be copied over, verified, and backed up before you can begin the rest of your post-production work.  It’s arguably the most critical step in your post-production workflow.

Knowing how critical this step is, videographers and data wranglers alike have long relied on an app for Mac and Windows called Hedge to take charge of their file copy and verification needs.


Hedge source and target progress

Hedge for Mac and Windows — drag and drop source file copy and verify tool

With an intuitive drag and drop interface, Hedge makes it simple to select your cards, disks, or other sources, identify your destination drives, then copy and verify using a custom “Fast Lane” engine to speed transfers dramatically. You can log when copies were completed, and even back up to multiple destinations in the same action, including your local SAN, NAS, or Backblaze Fireball, then on to your Backblaze B2 cloud storage.

But How Do You “Data-Wrangle” Tape Content to the Cloud?

But what if you have content, backup sets, or massive media archives on LTO tape?

You may find yourself in one of these scenarios:

  • You may have “inherited” an older LTO tape system that is having a hard time keeping up with your daily workflow, and you aren’t ready to sign up for more capital expense and support contracts.
  • You may have valuable content “stuck” on tape that you can’t easily access and want it on cloud for content monetization workflows that would overwhelm your tape system.
  • Your existing tape based workflow is working fine for now, but you want to get all of that content into the cloud quickly to get ready for future growth and new customers with a solution similar to Hedge.

While many people decide to move tape workflows to cloud for simple economic reasons, having all of that content securely stored in the cloud means that the individual files and entire folders can be instantly pulled into workflows and directly shared from Backblaze B2 with no need for copying, moving, restoring, or waiting.

For more information about how Backblaze B2 can replace LTO solutions, including an LTO calculator:  Backblaze LTO Replacement Calculator

Whichever scenario fits your need, getting tape content into the cloud involves moving a lot of content at once, and in a perfect world it would be as easy to drag and drop that content from tape to Backblaze B2!

Meet Canister for Fireball

To meet this exact need the team that developed Hedge have created an “LTO tape content to Fireball” solution called Canister for Fireball.

Fireball is Backblaze’s solution to help you quickly get massive amounts of data into Backblaze B2 Cloud Storage. When you sign up for the service, Backblaze sends you a 70TB Fireball that is yours to use for 30 days. Simply attach it to your local network and copy content over to the device at the speed of your local network. You’re free to fill up and send in your Fireball device as many times as needed. When Backblaze receives your Fireball with your files, all of the content is ingested directly into Backblaze’s data centers and appears in your Backblaze B2 online storage.

Backblaze B2 Fireball Rapid Ingest Service

Canister for Fireball makes it incredibly easy to move your content and archives from your tape device to your Backblaze B2 Fireball. With an intuitive interface similar to Hedge, Canister copies over and verifies files read from your tapes.

Using Canister with B2

flow chart for moving data from tape to the cloudInsert LTO tapes in your tape system and Canister for Backblaze will move them to your Backblaze B2 Fireball for rapid ingest into your B2 Cloud Storage


Cannister to Fireball user interfaceSelect from any tape devices with LTO media…

Cannister data progression screenshot…and watch the files on the tape copy and verify to your Backblaze B2 Fireball

Here’s how the solution works:

Steps to Migrate Your LTO Content to the Cloud with Canister for Fireball

  1. Order a Fireball system: As part of the signup step you will choose a B2 bucket that you’d like your Fireball content moved to.
  2. Connect your Fireball system to your network, making sure that the workstation that connects to your tape device can also mount the storage volume presented by your Backblaze Fireball.
  3. Install Canister for Fireball on your Mac workstation.
  4. Connect your tape device. Any tape system that can read your tapes and mount them as an LTFS volume will work. Canister will automatically mount tapes inside the app for you.
  5. Launch Canister for Fireball. You can now select the tape device volume as your source, the Fireball as your target, and copy the files over to your Fireball.
  6. Repeat as needed until you have copied and verified all of your tapes securely to your Fireball. You can fill and send in your Fireball as many times as needed during your 30 day period. (And you can always extend your loaner period.)
LTFS or Linear Tape File System is an industry adopted way to make the contents of an entire tape cartridge available as if it were a single volume of files. Typically, the tape stores a list of the files and their location on that tape in the beginning, or header of the tape. When a tape is read into your tape device, that directory section is read in and the tape system then presents it to you as a volume of files and folders. Say you want to select an individual file from that LTFS volume to copy to your desktop. When you move that to your desktop, the tape spools out to wherever that file is stored, reads the entire stream of tape containing that file, then finally copies it to your desktop. It can be a very slow process indeed and why many people choose to store content in cloud storage like Backblaze B2 so that they get instant access to every file.

Now — Put Your LTO Tape Ingest Plan Into Action

If you have content on tape that needs to get into your Backblaze B2 storage, Canister for Fireball and a Backblaze B2 Fireball are the perfect solution.

Canister for Fireball can be licensed for 30 days of use for $99 and includes priority support. The full version is $199. If you decide to upgrade from the 30 Day license you’ll pay only the difference to the full version.

Get more information about Canister for Fireball

And of course, make sure that you’ve ordered your Fireball:

Order a Backblaze B2 Fireball

Now with your content and archives no longer “trapped” on tape, you can browse them in your asset manager, share links directly from Backblaze B2, and have your content ready to pull into new content creation workflows by your team located anywhere in the world.

The post Moving Tape Content to Backblaze Fireball with Canister appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

LTO versus Cloud Storage: Choosing the Model That Fits Your Business

Post Syndicated from Andy Klein original https://www.backblaze.com/blog/lto-vs-cloud-storage-vs-hybrid/

Choose Your Solution: Cloud Storage, LTO, Hybrid Cloud Storage/LTO

Years ago, when I did systems administration for a small company, we used RAID 1 for in-house data redundancy and an LTO tape setup for offsite data backup. Yes, the LTO cataloging and versioning were a pain, so was managing the tapes, and sometimes a tape would be unreadable, but the setup worked. And given there were few affordable alternatives out there at the time you lived and died with your tapes.

Over the last few years, cloud storage has emerged as a viable alternative to using LTO for offsite backups. Improvements in network speed coupled with lower costs are a couple of the factors that have changed the calculus of cloud storage. To see if enough has changed to make cloud storage a viable competitor to LTO, we’ll start by comparing the current and ongoing cost of LTO versus cloud storage and then dig into assumptions underlying the cost model. We’ll finish up by reviewing the pluses and minuses of three potential outcomes: switching to cloud storage, staying with LTO, or using a hybrid LTO/cloud storage solution.

Comparing the Cost of LTO Versus Cloud Storage

Cost calculators for comparing LTO to Cloud Storage have a tendency to be very simple or very complex. The simple ones generally compare hardware and tape costs to cloud storage costs and neglect things like personnel costs, maintenance costs, and so on. In the complex models you might see references to the cost of capital, interest on leasing equipment, depreciation, and the tax implications of buying equipment versus paying for a monthly subscription service.

The Backblaze LTO vs Cloud calculator is somewhere in between. The underlying model takes into account many factors, which we’ll get into in a moment, but if you are a Fortune 500 company with a warehouse full of tape robots, this model is not for you.

Calculator: LTO vs B2

To use the Backblaze calculator you enter:

  1. the amount of Existing Data you have on LTO tape
  2. the amount of data you expect to add in a given year
  3. the amount of incremental data you backup each day

Then you can use the slider to compare your total cost from 1 to 10 years. You can run the model as many times as you like under different scenarios.

Assumptions Behind the Model

To see the assumptions that were made in creating the model, start on the LTO Replacement page and scroll down past the LTO vs. B2 calculator. Click on the following text which will display the “Cost and Operational Assumptions” page.

+ See details on Cost and Operational Assumptions

Let’s take a few minutes to review some of the most relevant points and how they affect the cost numbers reported:

  • LTO Backup Model: We used the Grandfather-Father-Son (GFS) model. There are several others, but this was the most prevalent. If you use the “Tower of Hanoi” model for example, it uses fewer tapes and would lower the cost of the total LTO cost by some amount.
  • Data Compression: We assumed a 2-1 compression ratio for the data stored on the LTO tapes. If your data is principally video or photos, you will most likely not use compression. As such, film studios and post-production houses will need to double the cost of the total LTO solution to compensate for the increased number of tapes, the increased number of LTO tape units, and increased personnel costs.
  • Data Retention: We used a 30 day retention period as this is common in the GFS model. If you keep your incremental tapes/data for 2 weeks, then you would lower the number of tapes needed for incremental backups, but you would also lower the amount of incremental data you keep in the cloud storage system.
  • Tape Units: There are a wide variety of LTO tape systems. You can increase or decrease the total LTO cost based on the systems you are using. For example, you are considering the purchase of an LTO tape system which reads/writes up to 5 tapes simultaneously. That system is more expensive and has higher maintenance costs, but it also would mean you would have to purchase fewer tape units.
  • LTO-8 Tape Units: We used LTO-8 tape units as they are the currently available LTO system most likely to be around in 10 years.
  • Tape Migration: We made no provision for migration from an unsupported LTO version to a supported LTO version. During the next 10 years, many users with older LTO systems will find it likely they will have to migrate to newer systems as LTO only supports 2 generations back and is currently offering a new generation every 2 years.
  • Pickup Cost: The cost of having your tapes picked up so they are offsite. This cost can vary widely based on geography and service level. Our assumption of the cost is $60 per week or $3,120/year. You can adjust the LTO total cost according to your particular circumstances.
  • Network Cost: Using cloud storage requires that you have a reasonable amount of network bandwidth available. The number we used is incremental to your existing monthly cost for bandwidth. Network costs vary widely, so depending on your circumstance you can increase or decrease to the total cost of the cloud storage solution.
  • Personnel Cost: This is the total cost of what you are paying someone to manage and operate your LTO system. This raises or lowers the cost of both the LTO and cloud storage solutions at the same rate, so adjusting this number doesn’t affect the comparison, just the total values for each.
  • Time Savings Versus LTO: With a cloud storage solution, there are no tapes or tape machines to deal with. This saves a significant amount of time for the person managing the backup process. Increasing this value will increase the cost of the cloud storage solution relative to the LTO solution.

As hinted at earlier, we don’t consider the cost of capital, depreciation, etc. in our calculations. The general model is that a company purchases a number of LTO systems and the cost is spread over a 10 year period. After 10 years a replacement unit is purchased. Other items such as tapes and equipment maintenance are purchased and expensed as needed.

Choosing a Data Backup Model

We noted earlier the three potential outcomes when evaluating LTO versus cloud storage for data backup: switching to cloud storage, staying with LTO, or using a hybrid LTO/cloud storage solution. Here’s a look at each.

Switching to Cloud Storage

After using the calculator you find cloud storage is less expensive for your business or organization versus LTO. You don’t have a large amount of existing data, 100 terabytes for example, and you’d rather get out of the tape business entirely.

Your first challenge is to move your existing data to the cloud — quickly. One solution is the Backblaze B2 Fireball data transfer service. You can move up to 70 TB of data each trip from your location to Backblaze in days. This saves your bandwidth and saves time as well.

As the existing data is being transferred to Backblaze, you’ll want to select a product or service to move your daily generated information to the cloud on a regular basis. Backblaze has a number of integration partners that perform data backup services to Backblaze B2

Staying with LTO

After using the calculator you find cloud storage is less expensive, but you are one of those unlucky companies that can’t get reasonably priced bandwidth in their area. Or perhaps, the new LTO-8 equipment you ordered arrived minutes before you read this blog post. Regardless, you are destined to use LTO for at least a while longer. Tried and true, LTO does work and has the added benefit of making the person who manages the LTO setup nearly indispensable. Still, when you are ready, you can look at moving to the hybrid model described next.

Hybrid LTO/Cloud Storage model

In practice, many organizations that use LTO for backup and archive often store some data in the cloud as well, even if haphazardly. For our purposes, Hybrid LTO/Cloud Storage is defined as one of the following:

  1. Date Hybrid: All backups and archives from prior to the cut over date remain stored in LTO; everything after the cut over date date forward is stored in cloud storage.
  2. Classic Hybrid: All of the incremental backups are stored in cloud storage and all full backups and archives are stored on LTO.
  3. Type Hybrid: All data of a given type, say employee data, is stored on LTO, while all customer data is stored in cloud storage. We see this hybrid use case occur as a function of convenience and occasionally compliance, although some regulatory requirements such as GDPR may not be accommodated by LTO solutions.

You can imagine there being other splits, but in essence, there may be situations where keeping the legacy system going in some capacity for some period of time is the prudent business option.

If you have a large tape library, it can be almost paralyzing to think about moving to the cloud, even if it is less expensive. Being open to the hybrid LTO/cloud model is a way to break the task down into manageable steps. For example, solutions like Starwind VTL and Archiware P5 allow you to start backing up to the cloud with minimal changes to your existing tape-based backup schemes.

Many companies that start down the hybrid road typically begin with moving their daily incremental files to the cloud. This immediately reduces the amount of “tape work” you have to do each day and it has the added benefit of making the files readily available should they need to be restored. Once a company is satisfied that their cloud based backups for their daily incremental files are under control, they can consider whether or not they need to move the rest of their data to the cloud.

Will Cloud Storage Replace LTO?

At some point, the LTO tapes you have will need to be migrated to something else as the equipment to read your old tapes will become outdated, then unsupported, and finally unavailable. Users with LTO 4 and, to some degree, LTO 5 are already feeling this pain. To migrate all of that data from your existing LTO system to LTO version “X,” cloud storage, or something else, will be a monumental task. It is probably a good idea to start planning for that now.

In summary, many people will find that they can now choose cloud storage over LTO as an affordable way to store their data going forward. But, having a hybrid environment of both LTO and cloud storage is not only possible, it is a practical way to reduce your overall backup cost while maximizing your existing LTO investment. The hybrid model creates an improved operational environment and provides a pathway forward should you decide to move exclusively to storing your data in the cloud at some point in the future.

The post LTO versus Cloud Storage: Choosing the Model That Fits Your Business appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

How to Leverage Your Amazon S3 Experience to Code the Backblaze B2 API

Post Syndicated from Andy Klein original https://www.backblaze.com/blog/how-to-code-backblaze-b2-api-interface/

Going from S3 to learning Backblaze B2

We wrote recently about how the Backblaze B2 and Amazon S3 APIs are different. What we neglected to mention was how to bridge those differences so a developer can create a B2 interface if they’ve already coded one for S3. John Matze, Founder of BridgeSTOR, put together his list of things consider when levering your S3 API experience to create a B2 interface. Thanks John.   — Andy
BackBlaze B2 to Amazon S3 Conversion
by John Matze, Founder of BridgeSTOR

BackBlaze B2 Cloud Storage Platform has developed into a real alternative to the Amazon S3 online storage platform with the same redundancy capabilities but at a fraction of the cost.

Sounds great — sign up today!

Wait. If you’re an application developer, it doesn’t come free. The Backblaze REST API is not compatible with Amazon S3 REST API. That is the bad news. The good news — it includes almost the entire set of functionality so converting from S3 to B2 can be done with minimal work once you understand the differences between the two platforms.

This article will help you shortcut the process by describing the differences between B2 and S3.

  1. Endpoints: AWS has a standard endpoint of s3.amazonaws.com which redirects to the region where the bucket is located or you may send requests directly to the bucket by a region endpoint. B2 does not have regions, but does have an initial endpoint called api.blackblazeb2.com. Every application must start by talking to this endpoint. B2 also requires two other endpoints. One for uploading an object and another one for downloading an object. The upload endpoint is generated on demand when uploading an object while the download API is returned during the authentication process and may be saved for download requests.
  1. Host: Unlike Amazon S3, the HTML header requires the host token. If it is not present, B2 will not respond with an error.
  1. JSON: Unlike S3, which uses XML, all B2 calls use JSON. Some API calls require data to be sent on the request. This data must be in JSON and all APIs return JSON as a result. Fortunately, the amount of JSON required is minimal or none at all. We just built a JSON request when required and made a simple JSON parser for returned data.
  1. Authentication: Amazon currently has two major authentication mechanisms with complicated hashing formulas. B2 simply uses the industry standard “HTTP basic auth” algorithm. It takes only a few minutes to get to speed on this algorithm.
  1. Keys: Amazon has the concept of an access key and a secret key. B2 has the equivalent with the access key being your key id (your account id) and the secret key being the application id (returned from the website) that maps to the secret key.
  1. Bucket ID: Unlike S3, almost every B2 API requires a bucket ID. There is a special list bucket call that will display bucket IDs by bucket name. Once you find your bucket name, capture the bucket ID and save it for future API calls.
  1. Head Call: The bottom line — there is none. There is, however, a list_file_names call that can be used to build your own HEAD call. Parse the JSON returned values and create your own HEAD call.
  1. Directory Listings: B2 Directories again have the same functionality as S3, but with a different API format. Again the mapping is easy: marker is startFileName, prefix is prefix, max-keys is maxFileCount and delimiter is delimiter. The big difference is how B2 handles markers. The Amazon S3 nextmarker is literally the next marker to be searched, the B2 nextmarker is the last file name that was searched. This means the next listing will also include the last marker name again. This means your routines must parse out the name or your listing will show the next marker twice. That’s a difference, but not a difficult one.
  1. Uploading an object: Uploading an object in B2 is quite different than S3. S3 just requires you to send the object to an endpoint and they will automatically place the object somewhere in their environment. In the B2 world, you must request a location for the object with an API call and then send the object to the returned location. The first API will send you a temporary key and you can continue to use this key for one hour without generating another, with the caveat that you have to monitor for failures from B2. The B2 environment may become full or some other issue will require you to request another key.
  1. Downloading an Object: Downloading an object in B2 is really easy. There is a download endpoint that is returned during the authentication process and you pass your request to that endpoint. The object is downloaded just like Amazon S3.
  1. Multipart Upload: Finally, multipart upload. The beast in S3 is just as much of a beast in B2. Again the good news is there is a one to one mapping.
    1. Multipart Init: The equivalent initialization returns a fileid. This ID will be used for future calls.
    2. Mulitpart Upload: Similar to uploading an object, you will need to get the API location to place the part. So use the fileid from “a” above and call B2 for the endpoint to place the part. Another difference is the upload also requires the payload to be hashed with a SHA1 algorithm. Once done, simply pass the SHA and the part number to the URL and the part is uploaded. This SHA1 component is equivalent to an etag in the S3 world so save it for later.
    3. Multipart Complete: Like S3, you will have to build a return structure for each part. B2 of course requires this structure to be in JSON but like S3, B2 requires the part number and the SHA1 (etag) for each part.

What Doesn’t Port

We found almost everything we required easily mapped from S3 to B2 except for a few issues. To be fair, BackBlaze is working on the following in future versions.

  1. Copy Object doesn’t exist: This could cause some issues with applications for copying or renaming objects. BridgeSTOR has a workaround for this situation so it wasn’t a big deal for our application.
  2. Directory Objects don’t exist: Unlike Amazon, where an object with that ends with a “/” is considered a directory, this does not port to B2. There is an undocumented object name that B2 applications use called .bzEmpty. Numerous 3rd party applications, including BridgeSTOR, treat an object ending with .bzEmpty as a directory name. This is also important for directory listings described above. If you choose to use this method, you will be required to replace the “.bzEmpty” with a “/.”

In conclusion, you can see the B2 API is different than the Amazon S3, but as far as functionality they are basically the same. For us at first it looked like it was going to be a large task, but once we took the time to understand the differences, porting to B2 was not a major job for our application. We created a S3 to B2 shim in a week followed by a few extra weeks of testing and bug fixes. I hope this document helps in your S3 to B2 conversion.

— John Matze, BridgeSTOR

The post How to Leverage Your Amazon S3 Experience to Code the Backblaze B2 API appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Creating a Media Archive Solution with Backblaze B2 and Archiware P5

Post Syndicated from Skip Levens original https://www.backblaze.com/blog/creating-a-media-archive-solution/

Backblaze B2 Cloud Storage + Archiware P5= 7 Ways to Save

B2 + P5 = 7 Ways to Save Time, Money and Gain Peace of Mind with an Archive Solution of Backblaze B2 and Archiware P5

by Dr. Marc M. Batschkus, Archiware

This week’s guest post comes to us from Marc M. Batschkus of Archiware, who is well-known to media and entertainment customers, and is a trusted authority and frequent speaker and writer on data backup and archiving.

— Editor

Archiving has been around almost forever.

Roman "Archivum"Roman “Archivum” where scrolls were stored for later reference.

The Romans used the word “archivum” for the building that stored scrolls no longer needed for daily work. Since then, files have replaced scrolls, but the process has stayed the same and so today, files that are no longer needed for daily production can be moved to an archive.

Backup and Archive

Backblaze and Archiware complement each other in accomplishing this and we’ll show you how to get the most from this solution. But before we look at the benefits of archiving, let’s take a step back and review the difference between backup and archive.

A backup of your production storage protects your media files by replicating the files to a secondary storage. This is a cyclical process, continually checking for changed and new files, and overwriting files after the specified retention time is reached.

Archiving, on the other hand is a data migration, moving files that are no longer needed for daily production to (long-term) storage, yet keeping them easily retrievable. This way, all completed productions are collected in one place and kept for later reference, compliance, and re-use.

Think of backup as a spare tire, archive as winter tiresThink of BACKUP as a spare tire, in case you need it, and ARCHIVE as a stored set of tires for different needs.

To use an analogy:

  • Think of backup as the spare tire in the trunk.
  • Think of archive as the winter tires in the garage.

Both are needed!

Editor’s note: For more insight on “backup vs archive” have a look at What’s the Diff: Backup vs Archive.

Building a Media Archive Solution with Archiware P5 and Backblaze B2

Now that the difference between backup and archive is clear, let’s have a look at what an archive can do to make your life easier.

Archiware archive catalog transfering to B2 cloud storageArchiware P5 can be your interface to locate and manage your files, with Backblaze B2 as your ready storage for all of those files

P5 Archive connects to Backblaze B2 and offers the interface for locating files.

B2 + P5 = 7 Ways to Save Time and Money and Gain Peace-of-Mind

  1. Free up expensive production storage
  2. Archive from macOS, Windows, and Linux
  3. Browse and search the archive catalog with thumbnails and proxies
  4. Re-use, re-purpose, reference and monetize files
  5. Customize the metadata schema to fit your needs and speed up search
  6. Reduce backup size and runtime by moving files from production storage
  7. Protect precious assets from local disaster and for the long-term (no further migration/upgrade needed)

Archive as Mini-MAM

The “Mini-MAM” features of Archiware P5 help you to browse and find files easier than ever. Browse the archive visually using the thumbnails and proxy clips in the archive catalog. Search for specific criteria or a combination of criteria such as location or description.

Since P5 Archive lets you easily expand and customize metadata fields and menus, you can build the individual metadata schema that works best for you.

Technical metadata (e.g. camera type, resolution, lens) can be automatically imported from the file header into the metadata fields of P5 archive using a script.

The archive becomes the file memory of the company saving time and energy because now there is only one place to browse and search for files.

Mini MAM screenshotArchiware as “Mini-MAM” —  thumbnails, proxies, even metadata all within Archiware P5

P5 offers maximum flexibility and supports all storage strategies, be it cloud, disk or tape and any combination of the above.

For more information on Archiving with Archiware: Archiving with Archiware P5. For macOS, P5 Archive offers integration with the Finder and Final Cut Pro X via the P5 Archive App. For more information on integrated archiving with Final Cut Pro X: macOS Finder and Final Cut Pro X Integrated Archiving

You can start building an archive immediately with Backblaze B2 cloud storage because it allows you to do this without any additional storage hardware and upfront investment.

Backblaze B2 is the Best of Cloud

  • ✓  Saves investment in storage hardware
  • ✓  Access from anywhere
  • ✓  Storage on demand
  • ✓  Perpetual storage – no migration or upgrade of hardware
  • ✓  Financially advantageous (OPEX vs CAPEX)
  • ✓  Best price in its category

Backblaze B2 offers flexible access so that the archive can be accessed from several physical locations with no storage hardware needing to be moved.

P5 Archive supports consumable files as archive format. This makes the single files accessible even if P5 Archive is not present at the other location. This opens up a whole new world of possibilities for collaborative workflows that were not possible before.

Save Money with OPEX vs CAPEX

CAPEX vs. OPEXCAPital EXpenditures are the money companies spend to purchase major physical goods that will be used for more than one year. Examples in our field are investments in hardware such as storage and servers.

OPerating EXpenses are the costs for a company to run its business operations on a daily basis. Examples are rent and monthly cost for cloud storage like B2.

By using Backblaze B2, companies can save CAPEX and instead have monthly payments only for the cloud storage they use, while also saving maintenance and migration cost. Furthermore, migrating files to B2 makes expansion of high performance and costly production storage unnecessary. Over time this alone will make the archive pay off.

Now that you know how to profit from archiving with Archiware P5 and Backblaze B2, let’s look at the steps to build the best archive for you.

Connecting B2 cloud storage screenshot

Backblaze B2 is already a built-in option in P5 and works with P5 Archive and P5 Backup.

For detailed setup and best practice see:

Cloud Storage Setup and Best Practice for Archiware

Steps in Planning a Media Archive

Depending on the size of the archive, the number of people working with and using it, and the number of files that are archived, planning might be extremely important. Thinking ahead and asking the right questions ensures that the archive later delivers the value that it was built for.

Including people that will configure, operate, and use the system guarantees a high level of acceptance and avoids blind spots in your planning.

  1. Define users: who administers, who uses and who archives?
  2. Decide and select: what goes into the archive, and when?
  3. Which metadata are needed to describe the data needed (what will be searched for)?
  4. Actual security: on what operating system, hardware, software, infrastructure, interfaces, network and medium will be archived?
  5. What security requirements should be fulfilled: off-site storage, duplication, storage duration, test cycles of media, generation migration, etc.
  6. Retrieval:
    • Who searches?
    • With what criteria?
    • Who is allowed to restore?
    • On what storage?
    • For what use?

Metadata is the key to the archive and enables complex searches for technical and descriptive criteria.

Naming Conventions or “What’s in a File Name?”

The most robust metadata you can have is the file name. It can travel through different operating systems and file systems. The file name is the only metadata that is available all the time. It is independent of any database, catalog, MAM system, application, or other mechanism that can keep or read metadata. With it, someone can instantly make sense of a file that gets isolated, left over, misplaced or transferred to another location. Building a solid and intelligent naming convention for media files is crucial. Consistency is key for metadata. Metadata is a solid foundation for the workflow, searching and sharing files with other parties. The filename is the starting point.

Wrapping Up

There is much more that can make a media archive extremely worthwhile and efficient. For further reading I’ve made this free eBook available for more tips on planning and implementation.

eBook:  Data Management, Backup and Archive for Media Professionals — How to Protect Valuable Video Data in All Stages of the Workflow by Marc M. Batschkus

Start looking into the benefits an archive can bring you today. There is a 30-day fully featured trial license for Archiware P5 that can be combined with the Backblaze B2 free trial storage.

Trial License:  About Archiware P5 and 30-Day Trial

And of course, if you’re not already a Backblaze B2 customer, sign up instantly at the link below.

B2 Cloud Storage:  Instant Signup

— Dr. Marc M. Batschkus, Archiware

The post Creating a Media Archive Solution with Backblaze B2 and Archiware P5 appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

cPanel Backup to B2 Cloud Storage

Post Syndicated from Roderick Bauer original https://www.backblaze.com/blog/cpanel-backup-to-b2-cloud-storage/

laptop on a desk with a cup of coffee, cell phone, and iPad

Anyone who’s managed a business or personal website is likely familiar with cPanel, the control panel that provides a graphical interface and tools that simplify the process of managing a website. IT professionals who’ve managed hosting servers might know cPanel’s big brother, WHM (Web Host Manager), which is used by server administrators to manage large web hosting servers and cPanels for their customers.

cPanel Dashboard WHM Dashboard
cPanel Dashboard   WHM Dashboard

Just as with any other online service, backup is critically important to safeguard user and business data from hardware failure, accidental loss, or unforeseen events. Both cPanel and WHM support a number of applications for backing up websites and servers.

JetApps’s JetBackup cPanel App

One of those cPanel applications is JetApps’s JetBackup, which supports backing up data to a number of destinations, including local, remote SSH, remote FTP, and public cloud services. Backblaze B2 Cloud Storage was added as a backup destination in version 3.2. Web hosts that support JetBackup for their cPanel and WHM users include Clook, FastComet, TMDHosting, Kualo, Media Street, ServerCake, WebHost.UK.net, MegaHost, MonkeyTree Hosting, and CloudBunny.

cPanel with JetBackup app

cPanel with JetBackup app

JetBackup configuration for B2

JetBackup configuration for B2

Directions for configuring JetBackup with B2 are available on their website.

Note:  JetBackup version 3.2+ supports B2 cloud storage, but that support does not currently include incremental backups. JetApps has told us that incremental backup support will be available in an upcoming release.

Interested in more B2 Support for cPanel and WHM?

JetBackup support for B2 was added to JetBackup because their users asked for it. Users have been vocal in asking vendors to add cPanel/WHM support for backing up to B2 in forums and online discussions, as evidenced on cPanel.net and elsewhere — here, here, and here. The old axiom that the squeaky wheel gets the grease is true when lobbying vendors to add B2 support — the best way to have B2 directly supported by an app is to express your interest directly to the backup app provider.

Other Ways to Back Up Website Data to B2

When a dedicated backup app for B2 is not available, some cPanel users are creating their own solutions using the B2 Command Line Interface (CLI), while others are using Rclone to back up to B2.

B2 CLI example:

#!/bin/bash
b2 authorize_account ACCOUNTID APIKEY
b2 sync –noProgress /backup/ b2://STORAGECONTAINER/

Rclone example:

rclone copy /backup backblaze:my-server-backups –transfers 16

Those with WordPress websites have other options for backing up their sites, which we highlighted in a post, Backing Up WordPress.

Having a Solid Backup Plan is What’s Important

If you’re using B2 for cPanel backup, or are using your own backup solution, please let us know what you’re doing in the comments.

The post cPanel Backup to B2 Cloud Storage appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.