Tag Archives: Cloud Storage

Breaking the Cycle of Archive Migrations With B2 Cloud Storage

Post Syndicated from Janet Lafleur original https://www.backblaze.com/blog/cloud-data-archiving/

Assorted tapes

Back in the 1980s, my family and I took a trip to visit our friends, the Bremers. We all used to live next door, but the Bremers had moved away a decade prior. As our parents were reminiscing on old times, one of the Bremer teens pulled out a 8mm movie projector and we watched home movies his dad had shot of us playing together in the backyard: on the swings, the see-saw, and running about. What I wouldn’t give to see that footage today! It would be the only video of my sisters and me as kids.

Perhaps Mr. Bremer digitized his home movie collection before he passed away. But it’s more likely his children inherited the box of reels, and it’s now buried and decaying in a closet (or gone entirely). And, if they had the tape, would they have a projector or anything to play it? What a pity. Those precious moments captured once upon a time on film are probably lost forever.

Obsolescence isn’t just a concern for home video enthusiasts. Professional content creators likely have content stored on obsolete technology, whether it’s videotape, LTO digital tape, or external drives. And unlike the simplicity of Mr. Brehmer’s film reels and projectors, there are many more factors that can make digital content inaccessible.

Common Causes of Data Obsolescence

Media Failure

The most obvious issue is storage media degradation. If film is carefully stored in a cold, dry environment, it can last an extremely long time. Yet for both videotape and digital tape, there are a myriad of pitfalls: magnetic particles can lose their charge; the tape substrate can deteriorate; and heavily used tapes can stretch. Tapes over 15 years old are at greatest risk, even if stored in the ideal conditions of low-heat and low-humidity.

Hard disk drives have shortfalls too: mechanical failure, overheating, and power spikes. External drives in particular, are at risk of shock damage from being dropped. Even a drive standing on its side, then tipping over, can generate enough shock to damage the drive internals. At our Backblaze data centers, we replace disk drives after four years, and earlier for drive models that show higher-than-usual failure rates. We have ~100,000 drives in our data centers, and document which ones are more likely to fail in our quarterly drive stats posts.

Obsolete Technology

Even if the storage media remains intact and the data uncorrupted, the data format can become obsolete, often more quickly than you’d expect. For example, manufacturers of the commonly used LTO digital tape are now shipping LTO-8 and only guarantee two generations of backward compatibility. That means if you upgrade your tape system for higher-capacity 12TB LTO-8 tapes, you won’t be able to read the LTO-6 tapes that were introduced just six years ago.

Also, if the file data itself was encoded in a proprietary format, you’ll likely need proprietary software installed on a computer running a potentially outdated operating system version to be able to read its data. This is a bigger topic than we’ll cover today, because there can be layers of encoding involved: backup formats, graphics formats, codecs, etc. But suffice to say that you might find yourself having to hunt down a Mac that’s still running macOS X Leopard to migrate some content.

Museum of Obsolete Media

Not sure how much your content is at risk? The Museum of Obsolete Media rates all imaginable media types on both media stability and obsolescence, from Endangered to In Use.

Spoiler alert:  VHS tapes are rated Endangered for media stability and rated Vulnerable for obsolescence.

Migrate…Then Migrate Again

The only way to combat this sort of media decay and obsolescence and maintain access to your content is to migrate it to newer media and/or a newer technology. This unglamorous task sounds simple — read the data off the old media and copy it to new media — but the devil is in the details. Here is a checklist for trying to maintain your physical media:

The Eight Steps of Data Migration

  1. Determine which content is obsolete or at risk. Choose a media and format for the new archive, and calculate whether you can afford to migrate everything. If not, decide what you can afford to lose forever.
  2. Gather all the tapes or drives to be migrated. Are you sure you have the complete set? Your content spreadsheet might not be up to date. You might need to interview team members to gather any unwritten tribal knowledge about the backup sets.
  3. Identify a migration workstation or server that can run the application that wrote the archived media files. Attach the tape drive or disk device and test it. Can it still properly read, write, and then restore test files?
  4. Using a checklist system, feed tapes into the drive or attach the external drive in order. You might need to track down obscure adapters for older technologies like a SATA to EIDE adapter for parallel port disk drives, or a SCSI card and cables.
  5. Initiate the copy of all files to local storage. Hope you have enough space.
  6. Carefully monitor the entire process and make sure that all files are copied completely, and only then can you check the tape or disk off of your migration list. Then repeat with the next tape or disk.
  7. When you’re done extracting all the old files (or earlier if you’re pinched for disk space), reverse the process. Attach any needed devices and write the files to the new media. Cross your fingers that you bought enough tapes or disk drives (but not too many).
  8. Repeat again in 4-7 years before the new media ages or technologies change.

If all of that sounds too painful, you can pay a transfer service to migrate your whole archive for you, but that’s not cheap, and remember you’ll have to pay to do it again sooner than you think. Alternatively, you can migrate content on-demand and cross your fingers that it’s still readable and that you can retrieve it fast enough. The longer you wait, the greater the risk of media failure. You might only get one shot at reading an old tape or film. Few find that an acceptable risk.

Why Data Archiving to the Cloud Is a Better Solution

Migrate Once with Backblaze B2 Cloud Storage

You can break this migration cycle by migrating once to Backblaze B2 Cloud Storage. We’ll take over from there, moving your data to newer storage technologies as needed over time. Backblaze’s erasure coding technology that protects your data from loss happens to make upgrading technologies easier for us. Not that you need to worry about it; it’s included in our service.

No New Media or Hardware

Moving to B2 Cloud Storage for your archive means you won’t have any hardware or media to purchase, manage, or house. No tapes or disks to buy, no clearing off shelf space as your archive grows. You won’t have to feed tapes into an autoloader every time you want to write or retrieve content from the archive. And moving to B2 Cloud Storage gives you the benefit of only paying for what you’re actually using. Pay-as-you-go means your storage costs move from a capital expense to an operating expense.

B2 is Less Expensive than LTO

Did you know that Backblaze B2 is the first cloud storage that’s more affordable than LTO storage solutions? If you want to see the math, check out our LTO vs B2 calculator. Enter the size of your existing archive and how much you expect to add each year and it will show you cost differences after 1-10 years. To understand its cost and operational assumptions, read our recent blog post, LTO Versus Cloud Storage Costs — the Math Revealed. It details the many factors for storage costs that many media professionals don’t always consider.

Data That’s Always Accessible

The only thing worse than having a tape or disk you can’t read is having one that you can read go missing in action. Your content database or spreadsheet is only as accurate as what’s on the shelf. You may believe that an external drive is still in your archive closet when it went home over the weekend with a staff member and never came back. With B2 Cloud Storage, your archived content is stored in a central location that’s not only always accessible, it’s accessible from anywhere through a web browser.

B2 is Proven Technology

With Backblaze, you get a partner with over a decade of cloud storage experience. The erasure coding we use to encode data gives B2 customers a 99.999999999% durability (11 nines) rating for their data stored in our cloud. As NASA says, there’s higher probability of an asteroid destroying the planet than you losing a file with B2.

Make Your Final Migration Painless and Smart

Of course, you’ll still have to migrate once, but we can help make that final migration as painless and smart as possible. B2 Cloud Storage has several options for moving dataAPIs, Web UI, CLIplus our Fireball rapid ingest service for large data sets. We’ve also partnered with vendors and system integrators who have deep experience in managing media archives.

Streamlined LTO Migration

If your current archive is on LTO tapes, we have a newly announced partnership with StorageDNA that can speed migration of LTFS archives. The Storage DNA Smart Migration bundle combines the latest version of their DNAfabric storage with Backblaze B2 cloud storage, plus an autoloading LTO library so you won’t waste time manually loading tapes. To learn more about how it works, register for our upcoming webinar, From LTO to the Cloud: Your Last Data Migration with Backblaze and StorageDNA, on Friday, December 14.

Organize Content with a MAM

Archive migrations are a great time to evaluate your asset management strategy. If you haven’t rolled out a media asset manager (MAM) yet, or you’re dissatisfied with your current one, know that more and more MAMs are integrated with cloud storage and can simplify collaboration across remote teams. With a cloud-integrated MAM solution, your content can be easily searched, filtered, sorted and previewed all from a web browser, from anywhere. To see B2 in action with a cloud MAM solution, watch our recent webinar, Three Steps to Making Your Cloud Media Archive Active with iconik and Backblaze B2.

Automated Backup and Archive

Finally, B2 isn’t just an archive solution, it’s great for backup, too. Most of our customers who archive content to B2 also back up active production data to the same B2 account. We have a growing list of backup, sync and other tools integrated with B2 to make the data movement to the cloud seamless and to make retrieval intuitive and straightforward.

Pro Tip: syncing newly ingested footage or assets to B2 will spare you a big headache when someone accidentally deletes a critical file.

If you have content that’s on media or in a format that’s aging fast, now’s the time to plan for its migration. By migrating it to B2 Cloud Storage, you can not only make it your last migration, it’s priced so that you can afford to migrate ALL your content. You never know what you’ll need, or when you’ll need it. And some content, like Mr. Bremer’s home movies, simply can’t be re-created.

The post Breaking the Cycle of Archive Migrations With B2 Cloud Storage appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

LTO Versus Cloud Storage Costs — the Math Revealed

Post Syndicated from Andy Klein original https://www.backblaze.com/blog/lto-versus-cloud-storage/

B2 Cloud Storage $68,813 vs. LTO 8 Tape $119,873

A few months back we did a blog post titled, LTO versus Cloud Storage: Choosing the Model That Fits Your Business. In that post we presented our version of an LTO vs. B2 Cloud Storage calculator, a useful tool to determine whether or not it makes economic sense to consider using cloud storage over your LTO storage.

Rather than just saying, “trust us, it’s cheaper,” we thought it would be a good idea to show you what’s inside the model: the assumptions we used, the variables we defined, and the actual math we used to compute our answers. In fact, we’re making the underlying model available for download.

Our Model: LTO vs Cloud Storage

The LTO vs. B2 calculator that is on our website was based on a Microsoft Excel spreadsheet we built. The Excel file we’ve provided for download below is completely self-contained; there are no macros and no external data sources.

Download Excel file: Backblaze-LTO-Calculator-Public-Nov2018.xlsx

The spreadsheet is divided into multiple sections. In the first section, you enter the four values the model needs to calculate the LTO and B2 cloud storage costs. The website implementation is obviously much prettier, but the variables and math are the same as the spreadsheet. Let’s look at the remaining sections.

Entered Values Section

The second section is for organization and documentation of the data that is entered. You also can see the limits we imposed on the data elements.

One question you may have is why we limited the Daily Incremental Backup value to 10 TB. As the comment notes, that’s about as much traffic you can cram through a 1Gbps upload connection in a 24-hour period. If you have bigger (or smaller) pipes, adjust accordingly.

Don’t use the model for one-time archives. You may be tempted to enter zeros in both the Yearly Added Data and Daily Incremental Backup fields to compare the cost of a one-time archive. The model is not designed to compare the cost of a one-time archive. It will give you an answer, but the LTO costs will be overstated by anywhere from 10%-50%. The model was designed for the typical LTO use case where data is written to tape, typically daily, based on the data backup plan.

Variables Section

The third section stores all the variable values you can play with in the model. There is a short description for each variable, but let’s review some general concepts:

Tapes — We use LTO-8 tapes that will decrease in cost about 20% per year down to $60. Non-compressed, these tapes store 12 TB each and take about 9.5 hours to fully load. We use 24 TB for each tape assuming 2:1 compression. If some or all of your data is comprised of video or photos, then compression cannot be used, which makes actual tape capacity number much lower and increases the cost of the LTO solution.

Tapes Used — Based on the grandfather-father-son (GFS) model and assumes you replace tapes once a year.

Maintenance — Assumes you have no spare units, so you cannot miss more than one business day for backups. You could add a spare unit and remove the maintenance or just decide it is OK to miss a day or two while the unit is being repaired.

Off-site Storage — The cost of getting your tapes off-site (and back) assuming a once a week pick-up/drop-off.

Personnel — The cost of the person doing the LTO work, and how much time per week they spend doing the LTO related work, including data restoration. The cost of a person doing the cloud storage work is calculated from this value as described in the Time Savings paragraph below.

Data Restoration — How much of your data on average you will restore each month. The model is a bit limited here in that we use an average for all time periods when downloads are typically uneven across time. You are, of course, welcome to adjust the model. One thing to remember is that you’ll want to test your restore process from time to time, so make sure you allocate resources for that task.

Time Savings — We make the assumption that you will only spend 25% of the time working with cloud storage versus managing and maintaining an LTO system, i.e. no more buying, mounting, unmounting, labeling, cataloging, packaging, reading, or writing tapes.

Model Section

The last section is where the math gets done. Don’t change specific values in this section as they all originate in previous sections. If you decide to change a formula, remember to do so across all 10 years. It is quite possible that many of these steps can be combined into more complex formulas. We break them out to try to make an already complicated calculation somewhat easier to follow. Let’s look at the major subsections.

Data Storage — This section is principally used to organize the different data types and amounts. The model does not apply any corporate data retention policies such as deleting financial records after seven years. Data that is deleted is done so solely based on the GFS backup model, for example, deleting incremental data sets after 30 days.

LTO Costs — This starts with defining the amount of data to store, then calculates the quantity of tapes needed and their costs, along with the number of drive units and their annual unit cost and annual maintenance cost. The purchase price of a tape drive unit is divided evenly over a 10-year period.

Why 10 years? The LTO foundation, states is will support LTO tapes two versions back and expects to release a new version every two years. If you buy an LTO-8 system is 2018, in 2024 LTO-11 will not be able to read your LTO-8 tapes. You are now using obsolete hardware. We assume your LTO-8 hardware will continue to be supported through third party vendors for at least four years (to 2028) after it goes obsolete.

We finish up with calculating the cost of the off-site storage service and finally the personnel cost of managing the system and maintaining the tape library. Other models seem to forget this cost or just assume it is the same as your cloud storage personnel costs.

Cloud Storage Costs — We start with calculating the cost to store the data. This uses the amount of data at the end of the year, versus trying to compute monthly numbers throughout the year. This overstates the total amount a bit, but simplifies the math without materially changing the results. We then calculate the cost to download the data, again using the number at the end of the period. We calculate the incremental cost of enhancing the network to send and restore cloud data. This is an incremental cost, not the total cost. Finally, we add in the personnel cost to access and check on the cloud storage system as needed.

Result Tables — These are the totals from the LTO and cloud storage section in one place.

B2 Fireball Section

There is a small section and some variables associated with the B2 Fireball data transfer service. This service is useful to transfer large amounts of data from your organization to Backblaze. There is a cost for this service of $550 per month to rent the Fireball, plus $75 for shipping. Organizations with existing LTO libraries often don’t want to use their network bandwidth to transfer their entire library, so they end up keeping some LTO systems just to read their archived tapes. The B2 Fireball can move the data in the library quickly and let you move completely away from LTO if desired.

Summary

While we think the model is pretty good there is always room for improvement. If you have any thoughts you’d like to share, let us know in the comments. One more thing: the model is free to update and use within your organization, but if you publicize it anywhere please cite Backblaze as the original source.

The post LTO Versus Cloud Storage Costs — the Math Revealed appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

What’s the Diff: NAS vs SAN

Post Syndicated from Roderick Bauer original https://www.backblaze.com/blog/whats-the-diff-nas-vs-san/

What's the Diff? Network Attachd Storage (NAS) vs Storage Area Network (SAN)

Both network-attached storage (NAS) and storage area network (SAN) were developed to solve the problem of making stored data available to a lot of users at once. Each of them provides dedicated storage for a group of users, but they couldn’t be more different in their approach to achieving their mission.

A NAS is a single storage device that serves files over Ethernet and is relatively inexpensive and easy to set up, while a SAN is a tightly coupled network of multiple devices that work with block-based data and is more expensive and complex to set up and manage. From a user perspective, the biggest difference between NAS and SAN is that NAS devices look like volumes on a file server and use protocols like NFS and SMB/CIFS, while SAN-connected disks appear to the user as local drives.

We provide an overview of the differences between NAS and SAN below. We’ll also briefly cover solutions that combine NAS and SAN and offer many of the advanced benefits of SAN without its high cost.

Basic Definitions — What is NAS?

A NAS is a computer connected to a network that provides file-based data storage services to other devices on the network. The primary strength of NAS is how simple it is to set up and deploy. NAS volumes appear to the user as network mounted volume. The files to be served are typically contained on one or more storage drives, often arranged into logical, redundant storage containers or RAID. The device itself is a network node, much like computers and other TCP/IP devices, all of which maintain their own IP address and can effectively communicate with other networked devices. Although a NAS is usually not designed to be a general-purpose server, NAS vendors and third parties are increasingly offering other software to provide server-like functionality on a NAS.

NAS devices offer an easy way for multiple users in diverse locations to access data, which is valuable when uses are collaborating on projects or sharing information. NAS provides good access controls and security to support collaboration, while also enabling someone who is not an IT professional to administer and manage access to the data. It also offers good fundamental data security through the use of redundant data structures — often RAID — and automatic backup services to local devices and to the cloud.

Benefits of NAS

A NAS is frequently the next step up for a home office or small business that is using DAS (direct attached storage). The move up to NAS results from the desire to share files locally and remotely, having files available 24/7, data redundancy, the ability to replace and upgrade hard drives in the system, and and the availability of other services such as automatic backup.

Summary of NAS Benefits

  • Relatively inexpensive
  • 24/7 and remote data availability
  • Good expandability
  • Redundant storage architecture
  • Automatic backups to other devices and cloud
  • Flexibility

Network attached Storage (NAS)

Synology NAS

NAS with eight drive bays for 3.5″ disk drives

Limitations of NAS

The weaknesses of a NAS are related to scale and performance. As more users need access, the server might not be able to keep up and could require the addition of more server horsepower. The other weakness is related to the nature of Ethernet itself. By design, Ethernet transfers data from one place to another via packets, dividing the source into a number of segments and sending them along to their destination. Any of those packets could be delayed, or sent out of order, and might not be available to the user until all of the packets arrive and are put back in order.

Any latency (slow or retried connections) is usually not noticed by users for small files, but can be a major problem in demanding environments such as video production, where files are extremely large and latency of more than a few milliseconds can disrupt production steps such as rendering.

Basic Definitions — What is SAN?

A SAN is a way to provide users shared access to consolidated, block level data storage, even allowing multiple clients to access files at the same time with very high performance. A SAN enhances the accessibility of storage devices such as disk arrays and tape libraries by making them appear to users as if they were external hard drives on their local system. By providing a separate storage-based network for block data access over high-speed Fibre Channel, and avoiding the limitations of TCP/IP protocols and local area network congestion, a SAN provides the highest access speed available for media and mission critical stored data.

Storage area network (SAN)

SAN connecting yellow storage devices with orange servers via purple Fibre Channel switches

SAN connecting yellow storage devices with orange servers via purple Fibre Channel switches

Benefits of SAN

Because it’s considerably more complex and expensive than NAS, SAN is typically used by large corporations and requires administration by an IT staff. For some applications, such as video editing, it’s especially desirable due to its high speed and low latency. Video editing requires fair and prioritized bandwidth usage across the network, which is an advantage of SAN.

A primary strength of a SAN is that all of the file access negotiation happens over Ethernet while the files are served via extremely high speed Fibre Channel, which translates to very snappy performance on the client workstations, even for very large files. For this reason SAN is widely used today in collaborative video editing environments.

Summary of SAN Benefits

  • Extremely fast data access
  • Dedicated network for storage relieves stress on LAN
  • Highly expandable
  • OS level (block level) access to files
  • High quality-of-service for demanding applications such as video editing

Limitations of SAN

The challenge of SAN can be summed up in its cost and administration requirements — having to dedicate and maintain both a separate Ethernet network for metadata file requests and implement a Fibre Channel network can be a considerable investment. That being said, SANs are really the only way to provide very fast data access for a large number of users that also can scale to supporting hundreds of users at the same time.

What’s the Diff: NAS vs SAN

NAS SAN
Typically used in homes and small to medium sized businesses. Typically used in professional and enterprise environments.
Less expensive More expensive
Easier to manage Requires more administration
Data accessed as if it were a network-attached drive (files) Servers access data as if it were a local hard drive (blocks)
Speed dependent on local TCP/IP usually Ethernet network, typically 100 megabits to one gigabit per second. Generally slower throughput and higher latency due to slower file system layer. High speed using Fibre Channel, 2 gigabits to 128 gigabits per second. Some SANs use iSCSI as a less expensive but slower alternative to Fibre Channel.
I/O protocols: NFS, SMB/CIFS, HTTP SCSI, iSCSI, FCoE
Lower-end not highly scalable; high-end NAS scale to petabytes using clusters or scale-out nodes Network architecture enables admins to scale both performance and capacity as needed
Does not work with virtualization Works with virtualization
Requires no architectural changes Requires architectural changes
Entry level systems often have a single point of failure, e.g. power supply Fault tolerant network with redundant functionality
Susceptible to network bottlenecks Not affected by network traffic bottlenecks. Simultaneous access to cache, benefiting applications such as video editing.
File backups and snapshots economical and schedulable. Block backups and mirrors require more storage.

NAS/SAN Convergence

The benefits of SAN are motivating some vendors to offer SAN-like products at lower cost chiefly by avoiding the high expense of Fibre Channel networking. This has resulted in a partial convergence of NAS and SAN approaches to network storage at a lower cost than purely SAN.

One example is Fibre Channel over Ethernet (FCoE), which supports block level transfers over standard LAN at speeds of 10GB/sec+. For smaller deployments, iSCSI is even less expensive, allowing SCSI commands to be sent inside of IP packets on a LAN. Both of these approaches avoid expensive Fibre Channel completely, resulting in slower, but less expensive ways to get the block level access and other benefits of a SAN.

Are You Using NAS, SAN, or Both?

If you are using NAS or SAN, we’d love to hear from you about what you’re using and how you’re using them. Also, please feel free to suggest other topics for this series.

The post What’s the Diff: NAS vs SAN appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Buying a Hard Drive this Holiday Season? These Tips Will Help

Post Syndicated from Andy Klein original https://www.backblaze.com/blog/hard-drive-buying-guide/

Hard drives with bows
Over the last few years we’ve shared many observations in our quarterly Hard Drive Stats reports that go beyond the hard drive failure rates. We decided to consolidate some of these additional observations into one post just in time for the holiday buying season. If you have “buy a hard drive” on your shopping list this holiday season, here is just about everything we know about hard disk drives.

First, let’s establish that we are talking about hard disk drives (HDDs) here and not solid state drives (SSDs). Here’s a Backblaze “What’s the Diff” blog post where we discuss the differences between HDD and SSD drives.

How Will You Use Your HDD?

Hard drive manufacturers build drive models for different use cases; that is, a given drive model is optimized for a given purpose. For example, a consumer drive may spin slower to save energy and provides little if any access to tools that can adjust the firmware settings on the drive. An enterprise class drive, on the other hand, is typically much faster and provides the user with access to features they can tweak to adjust performance and/or power usage.

Each drive manufacturer has their own criteria for their use cases, but in general there are five categories: consumer, NAS (network attached storage), archiving/video recording, enterprise, and more recently, data center. The different drive manufacturers have different variations on these categories, so the first thing you should do is to know what you are going to do with the drive before you start looking.

Hard Drive Recording Technologies

For a long time, the recording technology a drive manufacturer used was not important. Then SMR (shingled magnetic recording) drives appeared a couple of years ago.

Let’s explain:

PMR: Perpendicular Magnetic Recording
This is the technology inside of most hard drives. With PMR data is written to and read from circular tracks on a spinning platter.
SMR: Shingled Magnetic Recording
This type of drive overlaps recording tracks to store data at a lower cost than PMR technology. The downside occurs when data is deleted and that space is reused. If existing data overlaps the space you want to reuse, this can mean delays in writing the new data. These drives are great for archive storage (write once, read many) use cases, but if your files turn over with some regularity, stick with PMR drives.

That sounds simple, but here are two things you should know:

  1. SMR drives are often the least expensive drives available when you consider the cost per gigabyte. If you are price sensitive, you may believe you are getting a great deal, but you may be buying the wrong drive for your use case. For example, buying SMR drives for your NAS device running RAID 6 would be ugly because of all the rewrites that may be involved.
  2. It is sometimes really hard to figure out if the drive you want to buy is an SMR or PMR drive. For example, based on the cost per gigabyte, the 8TB Seagate external drive (model: STEB8000100) is one of the least expensive external drives out there right now. But, the 8TB drive inside is an SMR drive, and that fact is not obvious to the buyer. To be fair, the manufacturers try to guide buyers to the right drive for their use case, but a lot of that guiding information is lost on reseller sites such as Amazon and Newegg, where the buyer is often blinded by price.

Over the next couple of years, HAMR (heat-assisted magnetic recording) by Seagate and MAMR (microwave-assisted magnetic recording) by Western Digital will be introduced, making the drive selection process even more complicated.

What About Refurbished Drives?

Refurbished drives are hard drives that have been returned to the manufacturer and repaired in some way to make them operational. Given the cost, repairs are often limited to what can be done in the software or firmware of the failed drive. For example, the repair may consist of identifying a section of bad media on a drive platter and telling the drive to read and write around it.

Once repaired, refurbished drives are tested and often marked certified by the manufacturer, e.g. “Certified Refurbished.” Refurbished drives are typically less expensive and come with a limited warranty, often one year or less. You can decide if you want to use these types of drives in your environment.

Helium-Filled versus Air-Filled Drives

Helium-filled drives are finally taking center stage after spending years as an experimental technology. Backblaze has in part used helium-filled drives since 2015, and over the years we’ve compared helium-filled drives to air-filled drives. Here’s what we know so far.

The first commercial helium-filled drives were 6TB; the transition to helium took hold at 8TB as we started seeing helium-filled 8TB drives from every manufacturer. Today helium-filled 12 and 14TB drives are now available at a reasonable price per terabyte.

Helium drives have two advantages over their air-filled cohorts: they create less heat and they use less power. Both of these are important in data centers, but may be less important to you, especially when you consider the primary two disadvantages: a higher cost and lack of experience. The street-price premium for a helium-filled drive is roughly 20% right now versus an air-filled drive of the same size. That premium is expected to decrease as time goes on.

While price is important, the lack of experience of helium-filled drives may be more interesting as these drives have only been in the field in quantity a little over four years. That said, we have had helium-filled drives in service for 3.5 years. They are solid performers with a 1.2% annualized failure rate and show no signs of hitting the wall.

Enterprise versus Consumer Drives

In our Q2 2018 Hard Drive Stats report we delved into this topic, so let’s just summarize some of the findings below.

We have both 8TB consumer and enterprise models to compare. Both models are from Seagate. The consumer drive is model ST800DM002 and the Enterprise drive model is ST800NM0055. The chart below, from the Q2 2018 report, shows the failure rates for each of these drive models at the same average age of all of the drives of the specified model.

Annualized Hard Drive Failure Rates by Time table

When you constrain for the average age of each of the drive models, the AFR (annualized failure rate) of the enterprise drive is consistently below that of the consumer drive for these two drive models — albeit not by much. By the way, conducting the same analysis at an average age of 15 months showed little change, with the consumer drive recording a 1.10% AFR and the enterprise drive holding at 0.97% AFR.

Whether every enterprise model is better than every corresponding consumer model is unknown, but below are a few reasons you might choose one class of drive over another:

Enterprise Class Drives

  • Longer Warranty: 5 years vs. 2 years
  • More Accessible Features, i.e. Seagate PowerChoice technology
  • Faster reads and writes

Consumer Class Drives

  • Lower Price: Up to 50% less
  • Similar annualized failure rates as enterprise drives
  • Uses less power and produces less heat

Hard Drive Failure Rates

As many of you know, each quarter Backblaze publishes our Hard Drive Stats report for the hard drives in our data centers. Here’s the lifetime chart from our most recent Q3 2018 report.

Backblaze Lifetime Hard Drive Failure Rates table

Along with the report, we also publish the data we used to create the reports. We are not alone. Let’s look at the various ways you can find hard drive failure rates for the drive you wish to purchase.

Backblaze AFR (annualized failure rate)
The failure rate of a given hard drive model based on the number of days a drive model is in use and the number of failures of that drive model. Here’s the formula:

( ( Drive Failures / ( Drive Days / 365 ) ) * 100 )
MTBF (mean time between failures)
TBF is the term some disk drive manufacturers use to quantify disk drive average failure rates. It is the average number of service hours between failures. This is similar to MTTF (mean time to failure), which is the average time to the first failure. MTBF has been superseded by AFR for some drive vendors as described below.
AFR (Seagate and Western Digital)
These manufacturers have decided to replace MTBF with AFR. Their definition of AFR is the probable percent of failures per year, based on the manufacturer’s total number of installed units of similar type. While Seagate and WD don’t give the specific formula for calculating AFR, Seagate notes that AFR is similar to MTBF and differs only in units. One way for converting MTBF to AFR can be found here.
Comparing Backblaze AFR to the Seagate/WD AFR
The Backblaze environment is a closed system, meaning we know with a high degree of certainty the variables we need to compute the Backblaze AFR percentage. We also know most, if not all, of the mitigating factors. The Seagate/WD AFR environment is made up of potentially millions of drives in the field (home, office, mobile, etc.) where the environmental variables can be quite varied and in some cases unknown. Either of the AFR calculations can be considered as part of your evaluation if you are comfortable with how they are calculated.
CDL (component design life)
This term is used by Western Digital in their support knowledge base although we don’t see it in their technical specifications yet. The example provided in the knowledge base article is, “The Component Design Life of the drive is 5 years and the Annualized Failure Rate is less than 0.8%.” With those two numbers you can calculate that no more than four out of 100 drives will die in a five-year period. The is really good information, but it is not readily available yet.

Which Hard Drive Do I Need?

While hard drive failure rates are interesting, we believe that our Hard Drive Stats reports are just one of the factors to consider in your hard drive buying decision. Here are some things you should think about, in no particular order:

  • Your use case
    • What you will do with the drive.
  • What size drive do you need?
    • Using it as a Time Machine backup? It should be 3-4 times the size of your internal hard drive. Using it as an archive for your photo collection? — bigger is better.
  • How long do you want the drive to last?
    • Forever is not a valid answer. We suggest starting with the warranty period and subtracting a year if you move the drive around a lot or if you fill it up and stuff it in the closet.
  • The failure rate of the drive
    • We talked about that above.
  • What your friends think
    • You might get some good advice.
  • What the community thinks
    • reddit, Hacker News, Spiceworks, etc.
  • Product reviews
    • I read them, but only to see if there is anything else worth investigating via other sources.
  • Product review sites
    • These days, many review sites on the internet are pay-to-play, although not all. Pay-to-play means the vendor pays the site either for their review or if the review leads to a sale. Sometimes, whoever pays the most gets to the top of the list. This isn’t true for all sites, but often it is really hard to tell the good guys. One of our favorite sites, Tom’s Hardware, has stopped doing HDD reviews, so if you have a site you trust for such reviews, share it in the comments, we’d all like to know.
  • The drive manufacturer
    • Most drive manufacturer websites provide information that can help you determine the right drive for your use case. Of course, they are also trying to sell you a drive, but the information, especially the technical specs, can be useful.

What about price? We left that out of our list as many people start and end their evaluation with just price and we wanted to mention a few other things we thought could be important. Speaking of price…

What’s a Good Price for a Hard Drive?

Below is our best guess as to what you could pay over the next couple of months for different sized internal drives. Of course, there are bound to be some great discounts on Black Friday, Cyber Monday, Hanukkah, Christmas, Kwanzaa, Boxing Day, Winter Solstice, and Festivus — to name a few holiday season reasons for a sale on hard disk drives.

Drive Size Price Cost per GB
1TB $35 $0.035
2TB $50 $0.25
3TB $75 $0.25
4TB $100 $0.25
6TB $170 $0.28
8TB $250 $0.31
10TB $300 $0.30
12TB $380 $0.32
14TB $540 $0.39

How Much Do External Hard Drives Cost?

We wanted to include the same information about external hard drives, but there is just too much unclear information to feel good about doing it. While researching this topic, we came across multiple complaints about a wide variety of external drive systems containing refurbished or used drives. In reviewing the advertisements and technical specs, the fact that the HDD inside an external drive sometimes is not new often gets left off the specifications. In addition, on Amazon and similar sites, many of the complaints were from purchases made via third party sellers and not the original external drive manufacturers, so check the “by” tag before buying.

Let’s make it easy: an external hard drive should have at least a two-year warranty and be available from a trusted source. The list price for the external drive should be about 10-15% higher than the same sized internal drive. What you will actually pay, the street price, is based on supply and demand and a host of other factors. Don’t be surprised if the cost of an external drive is sometimes less than a corresponding internal drive — that’s just supply and demand at work. Following this guidance doesn’t mean the drive won’t fail, it just means you’ll have better odds at getting a good external drive for your money.

One More Thing Before You Buy

The most important thing to consider when buying a hard drive is the value of the data on the drive and what it would cost to replace that data. If you have a good backup plan and practice the 3-2-1 backup strategy, then the value of a given drive is low and limited to the time and cost it takes to replace the drive that goes bad. That’s annoying, yes, but you still have your data. In other words, if you want to get the most for your money when buying a hard drive, have a good backup plan.

The post Buying a Hard Drive this Holiday Season? These Tips Will Help appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Modern Storage Workflows in the Age of Cloud

Post Syndicated from Skip Levens original https://www.backblaze.com/blog/cloud-based-video-production-workflows/

Video Production Workflow

Not too long ago, hardware storage vendors held an iron grip on what kinds of storage underpinned your creative, film, and broadcast workflows. This storage took many complex forms — RAIDs, JBODs, SANs, NAS systems, tape robots, and more. All of it was expensive, deeply complex, and carried fat vendor margins and high support costs.

How Storage Can Make Your Video Production Workflow More Efficient

But when you’re considering storage in today’s technology environment — whether it’s cloud, on-site storage, or a USB stick — the guiding principle in choosing storage for your creative production should simply be to choose the storage that best fits each workflow step.

Production Storage Maxim: Choose the storage that best fits each workflow step

Doing your best creative work is what builds your customer base, boosts your reputation, and earns you revenue and royalties. So any time sunk into legacy storage solutions, wrestling with complexity, unneeded production steps, refereeing competing vendors, and overpaying for, well, everything, just gets in the way of what you really want to do, create.

The right answer for your specific production needs is a function of the size of your production team and the complexity of your operating environment. Whatever that answer is, it should be as frictionless an environment as possible that helps you get your work done more efficiently and gives you the most flexibility.

An independent filmmaker can follow this production storage evaluation process for each stage of their workflow and decide to make do with a small deskside RAID system for primary production storage, and depend on the cloud for everything else.

A large, global production team will probably need multiple SANs in each production office and a complex series of cloud and dedicated playout applications and systems. If your environment falls somewhere between those two extremes, then your ideal solution mix does as well.

Traditional Content Production Workflow - Ingest > Work-in-Process > Deliver > Archive

The traditional content production workflow is thought of as a linear process. Content is ingested as raw camera files pulled into a shared work-in-process storage for editors, the final cut is then delivered to the client, and when the project is finished all files are saved off to an archive.

Simplified Production Workflow Steps

Let’s look at what the storage requirements and needs are for each of the common steps in a production workflow and where cloud can add value. Along the way, we’ll call out concrete examples of cloud capabilities at each stage with B2 cloud storage.

Ingest Stage - Ingest Stage Goals: Safely retrieve and protect files from capture media and move to production environment. Ingest Stage Needs: File data protection - Easy path to Production Storage. Where Cloud Can Add Value: Ingest and archive in one step

The Ingest Stage

Media copied in the ingest phase typically needs to get off of camera carts and flash drives as quickly and safely as possible and transported to the editing environment. Since those camera carts need to be used again for the next shot, pressure to get files copied over quickly (but safely) is intense.

Any time that critical content exists only in one place is dangerous. At this stage, lost or corrupted files mean a reshoot, which may not be practical or even possible.

Storage Needs for Ingest

Storage at the ingest stage can be very rudimentary and is often satisfied by just copying files from camera carts to an external drive, then to another drive as a safety, or by putting a RAID system on a crash cart on-set. Every team tends to come up with a different solution.

Where Cloud Can Add Value to Ingest

But even if your data wranglers aren’t ready to give up external hard drives here, one way cloud can help in the ingest stage is to help combine your ingest and archive for safety steps.

Instead of carrying carts from the shoot location to the production environment and copying them over to production storage, you could immediately start uploading content via the internet to your cloud storage, simultaneously copying over those files safely, and making them available to your entire team immediately.

When you restructure your workflow like this, you’ll get better than RAID-level protection for your content in the cloud. And by checking content into your archive first, your asset manager tools can immediately start processing those files by adding tags and generating lighter weight proxies. As soon as the files hit cloud storage, your entire team can start working on them. They can immediately begin tagging and reviewing files, and even mark edit points before handing off to editors, thereby speeding up production dramatically.

Some creatives have hit a roadblock in trying to take advantage of the cloud. Data transfer has historically been gated by the available upload bandwidth at your given location, but our customers have solved this in some interesting ways.

Producers, editors, and reporters are finding that even cellular 4G internet connections make it feasible to immediately start uploading raw shots to their cloud storage. Others make it routine to stop off at a data center or affiliate with excellent upload speeds on their way in from the field.

Either way, even novice shooters and freelancers can safely get content into your system quickly in a system that can be as simple as an upload bucket in your B2 account and making sure that your media or project manager tools are configured to watch those upload points.

Cloud Capability Example — Use a Backblaze Fireball to Rapidly Ingest Content

Backblaze offers a Rapid Ingest Service to help get large amounts of your content into your Backblaze account quickly. Backblaze ships you a 70TB storage system that you connect to your network and copy content to. When the system is shipped back to Backblaze, it is quickly moved directly into your B2 account, dramatically reducing ingest times.

 

Cloud Capability Example — Share Files Directly From Cloud

Archive.zip file in B2

An example of navigating to a file-review bucket in the B2 web interface to copy the direct sharing link to send to a reviewer

In addition to the archive on ingest technique, many customers share files for approval review or dailies directly from their Backblaze B2 account’s web interface.

If your B2 bucket for finished files is public, you can get a direct share link from the Backblaze account management website and simply send that to your customer, thereby eliminating a copy step.

You can even snapshot a folder of your content in B2, and have Backblaze ship it directly to your customer.

Work in Process Stage - WIP Stage Goals: Support collaborative, simultaneous editing of source files to finished content. WIP Stage Needs: Performance to support shared, collaborative editing access for many users. Very large file support. Where Cloud Can Add Value: Keeping expensive primary production storage running efficiently.

The Work-In-Process Stage

Work-in-process or primary production storage is the main storage used to support collaborative editing and production of content. The bulk of what’s thought of as collaborative editing happens in this stage.

For simplicity we’re combining several steps under the umbrella of work-in-process such as craft editing, voiceover, sound, ADR, special effects, and even color grading and finish etc. under a far simpler work-in-process step.

As audio, color grading and SFX steps get more complex, they sometimes need to be broken out into separate, extremely high performance storage such as more exotic (and expensive) flash-based storage that then feeds the result back to WIP storage.

Work-in-Process Stage Storage Needs

Storage performance requirements in this stage are extremely hard to meet, demanding the ability to serve multiple editors, each pulling multiple, extremely large streams of video files as they edit raw shots into a complex, visual story. Meeting this requirement usually requires either equipment intensive SAN, or a NAS that scales to eye-watering size and price.

Many production environments have gotten in the habit of keeping older projects and media assets on the shared production environment alongside current production files, knowing that if those files are needed they can be retrieved quickly. But this also means that production storage fills up quickly, and it’s tempting to let more and more users not involved in primary production have access to those files as well, both of which can slow down production storage and creation of your content.

Having to make a rush purchase to expand or add to your SAN is not fun, especially in the middle of a project, so regularly moving any files not needed for current production to your content archive is a great strategy to keep your production storage as light and small as possible so that it can last over several seasons.

Where Cloud Can Add Value to Work-in-Process

By regularly moving content from your production storage you keep it light, fast, and simpler to manage. But that content still needs to be readily available. Cloud is an excellent choice here as content is both immediately available and stored on highly resilient object storage. In effect, you’re lightening the burden on your primary storage, and using cloud as an always ready, expanding store for all of your content. We’ll explore this concept more in the archive stage.

Deliver Stage - Deliver Stage Goals: Securely deliver finished files to upstream/downstream clients. Deliver Stage Needs: High reliability. Separation from primary production storage. Where Cloud Can Add Value: Share files directly and securely from cloud without copying.

The Deliver Stage

The deliver stage, where your finished work is handed off to your customer, varies depending on what type of creative you are. Broadcast customers will almost always need dedicated playout server appliances, and others will simply copy files to where they’re needed by downstream customers, or upstream to a parent organization for distribution. But, at some level, we all have to deliver our work when it’s done.

Deliver Stage Storage Needs

Files for delivery should be moved off of your primary production storage and delivered in a separate workflow available to dedicated workflow or playout tools. Whatever the workflow, this storage needs to be extremely reliable and available for your customers whenever it is needed.

Where Cloud Can Add Value to Deliver

Whether content delivery in your workflow is met by copying files to a playout server or giving a finished file to a customer, cloud can help cut down on the number of steps to get the content to its final destination while giving you extreme reliability.

Cloud Capability Example — Serve Time-Limited Links to Content

Many customers use the Backblaze B2 API to add expiration limits that can last from seconds to a week to shared links:

B2 command-line

An example of using the B2 command-line tool to generate time-expiring tokens for content sharing and delivery

If your team is comfortable writing scripts to automate your workflow, this can be a powerful way to directly share files simply and quickly with tools provided by Backblaze.

For more information see this B2 Article: Get Download Authorization

 

Cloud Capability Example — Move Content Directly to Your Delivery and Distribution Servers

Serving your content to a wide audience via your website, content channel, or app is an increasingly popular way to deliver content. And thanks to our recent Cloudflare agreement, you can now move content from your B2 storage over to Cloudflare’s content delivery network at zero transfer cost for your content application or website.For more information see this B2 article: How to Allow Cloudflare to Fetch Backblaze B2 Content

Archive Stage - Archive Stage Goals: Securely deliver finished files to upstream/downstream clients. Archive Stage Needs: High reliability. Separation from primary prodcution storage. Where Cloud Can Add Value: Serve as your content backplane across all workflow steps.

The Archive Stage

At last, we come to the archive stage of content creation, traditionally thought of as the end of the traditional content creation chain, the source of the most frustration for creatives, and the hardest storage to size properly.

Traditionally, when a project or season of a show is finished, all of the files used to create the content are moved off of expensive primary production storage and stored on separate, highly reliable storage in case they are needed again.

Archive Stage Storage Needs

Archive storage needs to be a safe repository for all of the content that you’ve created. It should scale well at a sustainable price, and make all archived content available immediately when requested by your users and workflow tools like asset managers.

Tape was often chosen to store these archive files because it was cheaper than disk-based storage and offered good reliability. But choosing tape required a large investment in specialized tape systems, tape media, and the associated support contracts and maintenance.

Tape based archiving strategies usually rely on compressing content as it’s written to tape to hit the advertised storage capacity of tape media. But video content is already stored in a compressed container, so compressing those files as they’re written and retrieved from tape offers no advantage and only slows the process down.

Here we find the chief drawback of tape based content archives for many customers: the time required to retrieve content from those tape systems. As the pace of production has increased, many customers find they can no longer wait for tape systems to return archive sets or unarchive files.

Where Cloud Can Add Value to Archive

The archive stage is where cloud has the most impact on your entire workflow. The benefits of cloud itself are familiar: the ability to scale up or down instantly as your needs change, paying only for the storage you actually use, extremely high object storage file reliability, and availability anywhere there is a network connection.

Modern Content Production Workflow - Ingest > Archive as a Cloud Content Backplane ><Work-In-Process

Creating The Cloud Content Backplane

Having all of your content immediately available to your production storage and your asset management systems is emerging as the killer feature of cloud for production environments. By adding cloud, your content production goes from a linear process to a highly active one where content can freely check in and out of all of your other workflow steps as you’re producing content.

By shifting your content archives to cloud like Backblaze B2, you are creating, in effect, a cloud content backplane that supports your entire content creation and delivery process with these new capabilities:

  • New productions now have access to every file you might possibly need without waiting, letting you explore more creative choices
  • A single, authoritative content repository backing all of your creative production lets you phase out other storage and the associated management headaches and expense
  • You can now serve and deliver files directly from your cloud-based content archive with no impact on production storage
  • Having content in a single place means that your workflow tools like asset managers work better. You can find files across your entire content store instantly, and even archive or move files from your production storage to your cloud content archive automatically

The content not needed on your work-in-process storage is both highly protected and immediately available wherever you need it. Your entire workflow can get much simpler with fewer steps, and you can phase out storage you no longer need on-site.

Above all, you’ll have fewer steps between you and creating great content, and you’ll be able to explore new creative options faster while shifting to a pay-as-you-use-it model for all of your content storage.

In part two, we’ll explore the ways your new cloud-delivered content archive backplane can dramatically improve how you create, deliver, and monetize content with other cloud-based technologies in the age of cloud.

The post Modern Storage Workflows in the Age of Cloud appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Hard Drive Stats for Q3 2018: Less is More

Post Syndicated from Andy Klein original https://www.backblaze.com/blog/2018-hard-drive-failure-rates/

Backblaze Drive Stats Q3 2018

As of September 30, 2018 Backblaze had 99,636 spinning hard drives. Of that number, there were 1,866 boot drives and 97,770 data drives. This review looks at the quarterly and lifetime statistics for the data drive models in operation in our data centers. In addition, we’ll say goodbye to the last of our 3TB drives, hello to our new 12TB HGST drives, and we’ll explain how we have 584 fewer drives than last quarter, but have added over 40 petabytes of storage. Along the way, we’ll share observations and insights on the data presented and we look forward to you doing the same in the comments.

Hard Drive Reliability Statistics for Q3 2018

At the end of Q3 2018, Backblaze was monitoring 97,770 hard drives used to store data. For our evaluation, we remove from consideration those drives that were used for testing purposes and those drive models for which we did not have at least 45 drives (see why below). This leaves us with 97,600 hard drives. The table below covers what happened in Q3 2018.

Backblaze Q3 2018 Hard Drive Failure Rates chart

Notes and Observations

  • If a drive model has a failure rate of 0%, it only means there were no drive failures of that model during Q3 2018.
  • Quarterly failure rates can be volatile, especially for models that have a small number of drives and/or a small number of Drive Days.
  • There were 170 drives (97,770 minus 97,600) that were not included in the list above because we did not have at least 45 of a given drive model. We use 45 drives of the same model as the minimum number when we report quarterly, yearly, and lifetime drive statistics.

When to Replace a Hard Drive

As noted, at the end of Q3 that we had 584 fewer drives, but over 40 petabytes more storage space. We replaced 3TB, 4TB, and even a handful of 6TB drives with 3,600 new 12TB drives using the very same data center infrastructure, i.e. racks of Storage Pods. The drives we are replacing are about 4 years old. That’s plus or minus a few months depending on how much we paid for the drive and a number of other factors. Keeping lower density drives in service when higher density drives are both available and efficiently priced does not make economic sense.

Why Drive Migration Will Continue

Over the next several years, data growth is expected to explode. Hard drives are still expected to store the bulk of that data, meaning cloud storage companies like Backblaze will have to increase capacity by either increasing existing storage density and/or building, or building out, more data centers. Drive manufacturers, like Seagate and Western Digital, are looking at HDD storage densities of 40TB as early as 2023, just 5 years away. It is significantly less expensive to replace lower density operational drives in a data center versus building a new facility or even building out an existing facility to house the higher density drives.

Goodbye 3TB WD Drives

For the last couple of quarters, we had 180 Western Digital 3TB drives (model: WD30EFRX) remaining — the last of our 3TB drives. In early Q3, they were removed and replaced with 12TB drives. These 3TB drives were purchased in the aftermath of the Thailand drive crisis and installed in mid-2014 and were still hard at work when we replaced them. Sometime over the next couple of years we expect to say goodbye to all of our 4TB drives and upgrade them to 14, 16, or even 20TB drives. After that it will be time to “up-density” our 6TB systems, then our 8TB systems, and so on.

Hello 12TB HGST Drives

In Q3 we added 79 HGST 12TB drives (model: HUH721212ALN604) to the farm. While 79 may seem like an unusual number of drives to add, it represents “stage 2” of our drive testing process. Stage 1 uses 20 drives, the number of hard drives in one Backblaze Vault tome. That is, there are are 20 Storage Pods in a Backblaze Vault, and there is one “test” drive in each Storage Pod. This allows us to compare the performance, etc., of the test tome to the remaining 59 production tomes (which are running already-qualified drives). There are 60 tomes in each Backblaze Vault. In stage 2, we fill an entire Storage Pod with the test drives, adding 59 test drives to the one currently being tested in one of the 20 Storage Pods in a Backblaze Vault.

To date, none of the 79 HGST drives have failed, but as of September 30th, they were installed only 9 days. Let’s see how they perform over the next few months.

A New Drive Count Leader

For the last 4 years, the drive model we’ve deployed the most has been the 4TB Seagate drive, model ST4000DM000. In Q3 we had 24,208 of this drive model, which is now only good enough for second place. The 12TB Seagate drive, model ST12000NM0007, became our new drive count leader with 25,101 drives in Q3.

Lifetime Hard Drive Reliability Statistics

While the quarterly chart presented earlier gets a lot of interest, the real test of any drive model is over time. Below is the lifetime failure rate chart for all the hard drive models in operation as of September 30th, 2018. For each model, we compute their reliability starting from when they were first installed.

Backblaze Lifetime Hard Drive Failure Rates Chart

Notes and Observations

  • The failure rates of all of the larger drives (8, 10, and 12 TB) are very good: 1.21% AFR (Annualized Failure Rate) or less. In particular, the Seagate 10TB drives, which have been in operation for over 1 year now, are performing very nicely with a failure rate of 0.48%.
  • The overall failure rate of 1.71% is the lowest we have ever achieved, besting the previous low of 1.82% from Q2 of 2018.

The Hard Drive Stats Data

The complete data set used to create the information used in this review is available on our Hard Drive Test Data page. You can download and use this data for free for your own purpose. All we ask are three things: 1) you cite Backblaze as the source if you use the data, 2) you accept that you are solely responsible for how you use the data, and 3) you do not sell this data to anyone. It is free.

If you just want the summarized data used to create the tables and charts in this blog post you can download the ZIP file containing the MS Excel spreadsheet.

Good luck and let us know if you find anything interesting.

The post Hard Drive Stats for Q3 2018: Less is More appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Move Even Your Largest Archives to B2 with Fireball and Archiware P5

Post Syndicated from Skip Levens original https://www.backblaze.com/blog/archiware-p5-cloud-backup/

Archiware P5 and Fireball

Backblaze B2’s reliability, scalability, and affordable, “pay only for what you use” pricing means that it’s an increasingly popular storage option for all phases of content production, and that’s especially true for media archiving.

By shifting storage to B2, you can phase out hard-to-manage and expensive local backup storage and clear space on your primary storage. Having all of your content in a single place — and instantly available — can transform your production and keep you focused on the creative process.

Fireball Rapid Ingest to Speed Your First Migration to Backblaze B2

Once you sign up for Backblaze B2, one tool that can speed an initial content migration tremendously is Backblaze’s Fireball rapid ingest service. As part of the service, Backblaze ships you a 70TB storage system. You then copy over all the content that you want in B2 to the Fireball system: all at local network speeds. Once the system is shipped to Backblaze, it’s quickly moved to your B2 account, a process far faster than uploading those files over the internet.

Setting Up Your Media Archive

Since manually moving files to archive and backing up project folders can be very time-consuming, many customers choose software like Archiware P5 that can manage this automatically. In P5’s interface you can choose files to add to archive libraries, restore individual files to your local storage from B2, and even browse all of your archive content on B2 with thumbnail previews, and more.

However, many media and entertainment customers have terabytes and terabytes of content in “archive” — that is, project files and content not needed for a current production, but necessary to keep nearby, ready to pull into a new production.

They’d love to get that content into their Backblaze B2 account and then manage it with an archive, sync, backup solution like Archiware P5. But the challenge facing too many is how to get all these terabytes up to B2 through the existing bandwidth in the office. Once the large, initial archive is loaded, the incrementals aren’t a problem, but getting years of backlog pushed up efficiently is.

For anyone facing that challenge, we’re pleased to announce the Archiware P5 Fireball Integration. Our joint solution provides any customer with an easy way to get all of their archives loaded into their B2 account without having to worry about bandwidth bottlenecks.

Archiware P5 Fireball Integration

A backup and archive manager like Archiware P5 is a great way to get your workflow under control and automated while ensuring that your content is safely and reliably stored. By moving your archives offsite, you get the highest levels of data protection while keeping your data immediately available for use anytime, anywhere.

With the newest release, Archiware P5 can archive directly to Fireball at fast, local network speeds. Then, once your Fireball content has been uploaded to your Backblaze account, a few clicks are all that is needed to point Archiware at your Backblaze account as the new location of your archive.

Finally, you can clear out those closets of hard drives and tape sets!

Archiware P5 to B2 workflow

Archiware P5 can now archive directly to Fireball at local network speeds, which are then linked to their new locations in your B2 accounts. With a few clicks you can get your entire archive uploaded to the B2 cloud without suffering any downtime or bandwidth issues.

For detailed information about configuring Archiware to archive directly to Fireball:

For more information about Backblaze B2 Fireball Rapid Ingest Service:

Archiware on Synology and QNAP NAS Devices

Archiware, NAS and B2

Archiware P5 can also now run directly on several Synology, QNAP, and G-Tech NAS systems to archive and move content to your Backblaze B2 account over the internet

With their most recent releases Archiware now supports several NAS system devices from QNAP, Synology, and G-Tech as P5 clients or servers.

The P5 software is installed as an application from the NAS vendor’s app store and runs directly on the NAS system itself without having to install additional hardware.

This means that all of your offices or departments with these NAS systems can now fully participate in your sync, archive, and backup workflows, and each of them can archive off to your central Backblaze B2 account.

For more information:

Archiware plus Backblaze: A Complete Front-to-Back Media Solution

Archiware P5, Fireball, and Backblaze B2 are all important parts of a great backup, archive, and sync plan. By getting all of your content into archive and B2, you’ll know that it’s highly protected, instantly available for new production workflows, and also readily discoverable through thumbnail and search capability.

With the latest version of P5, you not only have your entire production and backup workflows managed, with Fireball you can get even the largest and hardest to move archive safely and quickly into B2, as well!

For more information about the P5 Software Suite: Archiware P5 Software Suite

And to order a Fireball as part of our Rapid Ingest Service, start here: Backblaze B2 Fireball


You might also be interested in reading our recent guest post written by Marc N. Batschkus of Archiware about how to save time, money, and gain peace of mind with an archive solution that combines Backblaze B2 and Archiware P5.

Creating a Media Archive Solution with Backblaze B2 and Archiware P5

 

The post Move Even Your Largest Archives to B2 with Fireball and Archiware P5 appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

iconik and Backblaze — The Cloud Production Solution You’ve Always Wanted

Post Syndicated from Skip Levens original https://www.backblaze.com/blog/iconik-and-backblaze-cloud-production-solution/

Cantemo iconik Plus Backblaze B2 for Media Cloud Production

Cantemo iconik Plus Backblaze B2 for Media Cloud Production

Many of our customers are archiving media assets in Backblaze B2, from long-running television productions, media distributors, AR/VR video creators, corporate video producers, houses of worship, and many more.

They are emptying their closets of USB hard drives, clearing off RAID arrays, and migrating LTO tapes to cloud storage. B2 has been proven to be the least expensive storage for their media archives, while keeping the archives online and accessible. Gone are the days of Post-its, clipboards, and cryptic drive labels defining whether old video footage can be found or not. Migrating archives from one form of storage to another will no longer suck up weeks and weeks of time.

So now that their archives are limitless, secure, always active, and available, the next step is making them actionable.

Our customers have been asking us — how can I search across all of my archives? Can I preview clips before I download the hi-res master, or share portions of the archive with collaborators around the world? Why not use the latest AI tools to intelligently tag my footage with metadata?

To meet all of those needs and more, we are excited to announce that Cantemo’s iconik cloud media management service now officially supports Backblaze B2.

iconik — A Media Management Service

iconik is an affordable and simple-to-use media management service that can read a Backblaze B2 bucket full of media and make it actionable. Your media assets are findable, sortable with full previews, and ready to pull into a new project or even right into your editor, such as Adobe Premiere, instantly.

Cantemo iconik user interface

iconik — Cantemo’s new media management service with AI features to find, sort, and even suggest assets for your project across your entire library

As a true media management service, iconik’s pricing model is a pay-as-you-go service, transparently priced per-user, per month. There are no minimum purchases, no servers to buy, and no large licensing fees to pay. To use iconik, all your users need is a web browser.

iconik Pricing

To get an idea of what “priced-per-user” might look like, most organizations will need at least one administrative user ($89/month), standard users ($49/month) who can organize content, create workflows, and ingest new media, and browse-only users ($19/month), who can search and download what they need. There’s also a “share-only” level that has no monthly charge that lets you incorporate customer and reviewer comments. This should accommodate teams of all kinds and all sizes.

Best of all, iconik is intelligent about how it uses storage, and while iconik charges small consumption fees for proxy storage, bandwidth, etc., they have found that for customers that bring media from Backblaze B2 buckets, consumption charges should be less than 5% of the monthly bill for user licenses.

As part of their launch promotion, if you get started in October, Cantemo will give Backblaze customers a $300 getting started credit!

You can sign up and get started here using the offer code of BBB22018.

Everwell’s Experience with iconik and Backblaze

One of the first customers to adopt iconik with Backblaze is Everwell, a video production company. Everwell creates a constant stream of videos for medical professionals to show in their waiting rooms. Rather than continuously buying upgrades for their in-house asset management system and local storage, iconik allows Everwell to shift their production to the cloud for all of their users. Their new solution will allow Everwell to manage their growing library of videos as new content constantly comes online, and kick off longer form productions with full access to all the assets they need across a fast-moving team that can be anywhere their production takes them.

collage of Everwll video images

Everwell is a fast-growing medical content developer for healthcare givers

To speed up their deployment of iconik, Everwell started with Backblaze’s data ingestion service, Fireball. Everwell copied their content to Fireball, and once back in the Backblaze data center, the data from Fireball was quickly added directly to Everwell’s B2 buckets. iconik could immediately start ingesting the content in place and make it available to every user.

Learn more about Backblaze B2 Fireball

With iconik and Backblaze, Everwell dramatically simplified their workflow as well, collapsing several critical workflow steps into one. For example, by uploading source files to Backblaze B2 as soon as they’re shot, Everwell not only reduces the need to stage local production storage at every site, they ingest and archive in a single step. Every user can immediately start work on their part of the project.

“The ‘everyone in the same production building’ model didn’t work for us any longer as our content service grew, with more editors and producers checking in content from remote locations that our entire team needed to use immediately. With iconik and Backblaze, we have what feels like the modern cloud-delivered production tool we’ve always wanted.”

— Loren Goldfarb, COO, Everwell

See iconik in Action at NAB NYC October 17-18

NAB Show New York - Media In Action October 17-18 2018

Backblaze is at NAB New York. Meet us there!

We’re excited to bring you several chances to see iconik and Backblaze working together.

The first is the NAB New York show, held October 17-18 at the Javits Center. iconik will be shown by Professional Video Technology in Booth N1432, directly behind Backblaze, Booth N1333.

Have you signed up for NAB NY yet? You can still receive a free exhibits pass by entering Backblaze’s Guest Code NY8842.

And be sure to sign up to meet with the Backblaze team at NAB by signing up on our calendar.

Attend the iconik and B2 Webinar on November 20

Soon after NAB NY, Backblaze and iconik will host a webinar to demo the solution called “3 Steps to Making Your Cloud Media Archive ‘active’ With iconik and Backblaze B2.” The webinar will be presented on November 20 and available on demand after November 20. Be sure to sign up for that too!

3 Steps Demo with: iconik and Backblaze B2 Cloud Storage

Sign up for the iconik/B2 Webinar

Don’t Miss the iconik October Launch Promotion

The demand for creative content is growing exponentially, putting more demands on your creative team. With iconik and B2, you can make all of your media instantly accessible within your workflows while adopting a infinitely scalable, pay only for what you use, storage solution.

To take advantage of the iconik October launch promotion and receive $300 free credit with iconik, sign up using the BBB22018 code.

The post iconik and Backblaze — The Cloud Production Solution You’ve Always Wanted appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Backblaze and Cloudflare Partner to Provide Free Data Transfer

Post Syndicated from Gleb Budman original https://www.backblaze.com/blog/backblaze-and-cloudflare-partner-to-provide-free-data-transfer/

 Backblaze B2 Free Data Transfer to Cloudflare

Today we are announcing that beginning immediately, Backblaze B2 customers will be able to download data stored in B2 to Cloudflare for zero transfer fees. This happens automatically once Cloudflare is configured to distribute your B2 files. This means that Backblaze B2 can now be used as an origin store for the Cloudflare CDN and edge network, providing customers enhanced performance and access to their content stored on B2. The result is that customers can save up to 75% on storage versus Amazon S3 to store their content in the cloud and deliver it worldwide.

The zero B2 transfer fees are available to all Cloudflare customers using any plan. Cloudflare customers can also use paid add-ons such as Argo and Workers to enhance the routing and security of the B2 files being delivered over the Cloudflare CDN. To implement this service, Backblaze and Cloudflare have directly connected, thereby allowing near-instant data transfers from B2 to Cloudflare.

Backblaze has prepared a guide on “Using Backblaze B2 storage with Cloudflare.” This guide provides step-by-step instructions on how to set up Backblaze B2 with Cloudflare to take advantage of this program.

The Bandwidth Alliance

The driving force behind the free transfer program is the Bandwidth Alliance. Backblaze and Cloudflare are two of the founding members of this group of forward-thinking cloud and networking companies that are committed to providing the best and most cost-efficient experience for our mutual customers. Additional founding members of the Bandwidth Alliance include Automattic (WordPress), DigitalOcean, IBM Cloud, Microsoft Azure, Packet, and other leading cloud and networking companies.

How Companies Can Leverage the Bandwidth Alliance

Below are examples of how Bandwidth Alliance partners can work together to save customers on their data transfer fees.

Hosting Website Assets

Whether you are a professional webmaster or just run a few homegrown sites, you’ve lived the frustration of having a slow website. Over the past few years these challenges have become more acute as video and other types of rich media have become core to the website experience. This new content has also translated to higher storage and bandwidth costs. That’s where Backblaze B2 and Cloudflare come in.diagram of zero cost data transfer from Backblaze B2 to Cloudflare CDN

Customers can store their videos, photos, and other assets in Backblaze B2’s pay-as-you-go cloud storage and serve the site with Cloudflare’s CDN and edge services. The result is an amazingly affordable cloud-based solution that dramatically improves web site performance and reliability. And customers pay each service for what they do best.

“I am extremely happy with my experience serving html/css/js and over 17 million images from B2 via Cloudflare Workers. Page load time has been great and costs are minimal.”

— Jacob Hands, Lead Developer, FactorioMaps.com

Media Content Distribution

The ability to download content from B2 cloud storage to the Cloudflare CDN for zero transfer cost is the just the beginning. A company needing to distribute media can now store original assets in Backblaze B2, send them to a compute service to transcode and transmux them, and forward the finished assets to be served up by Cloudflare. Backblaze and Packet previously announced zero transfer fees between Backblaze B2 storage and Packet compute services. This enabled customers to store data in B2 at 1/4th the price of competitive offerings and then process data for transcoding, AI, data analysis, and more inside of Packet without worrying about data transfer fees. Packet is also a member of the Bandwidth Alliance and will deliver content to Cloudflare for zero transfer fees as well.

diagram of zero cost data transfer flow from Backblaze B2 to Packet Compute to Cloudflare CDN

Process Now, Distribute Later

A variation of the example above is for a company to store the originals in B2, transcode and transmux the files in Packet, then put those versions back into B2, and finally serve them up via Cloudflare. All of this is done with zero transfer fees between Backblaze, Packet, and Cloudflare. The result is all originals and transmuxed versions are stored at 1/4th the prices of other storage, and served up efficiently via Cloudflare.diagram of data transfer flow between B2 to Packet back to B2 to Cloudflare

In all cases you would only pay for services you use and not for the cost to move data between those services. This results in a predictable and affordable cost for a given project using industry leading best-of-breed services.

Moving Forward

The members of the Bandwidth Alliance are committed to enabling the best and most cost efficient cloud services when it comes to working with data stored in the cloud. Backblaze has committed to a transfer fee of $0 to move content from B2 to either Cloudflare or Packet. We think that’s a great step in the right direction. And if you are cloud provider, let us know if you’d be interested in taking a step like this one with Backblaze.

The post Backblaze and Cloudflare Partner to Provide Free Data Transfer appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Backblaze B2 API Version 2 Beta is Now Open

Post Syndicated from Andy Klein original https://www.backblaze.com/blog/backblaze-b2-api-version-2-beta-is-now-open/

cloud storage workflow image

Since B2 cloud storage was introduced nearly 3 years ago, we’ve been adding enhancements and new functionality to the B2 API, including capabilities like CORS support and lifecycle rules. Today, we’d like to introduce the beta of version 2 of the B2 API, which formalizes rules on application keys, provides a consistent structure for all API calls returning information about files, and cleans up outdated request parameters and returned data. All version 1 B2 API calls will continue to work as is, so no changes are required to existing integrations and applications.

The API Versions section of the B2 documentation on the Backblaze website provides the details on how the V1 and V2 APIs differ, but in the meantime here’s an overview into the what, why, and how of the V2 API.

What Has Changed Between the B2 Cloud Storage Version 1 and Version 2 APIs?

The most obvious difference between a V1 and V2 API call is the version number in the URL. For example:

https://apiNNN.backblazeb2.com/b2api/v1/b2_create_bucket

https://apiNNN.backblazeb2.com/b2api/v2/b2_create_bucket

In addition, the V2 API call may have different required request parameters and/or required response data. For example, the V2 version of b2_hide_file always returns accountId and bucketId, while V1 returns accountId.

The documentation for each API call will show whether there are any differences between API versions for a given API call.

No Change is Required For V1 Applications

With the introduction of V2 of the B2 API there will be V1 and V2 versions for every B2 API call. All applications using V1 API calls will continue to work with no change in behavior. In some cases, a given V2 API call will be different from its companion V1 API call as noted in the B2 API documentation. For the remaining API calls a given V1 API call and its companion V2 call will be the same, have identical parameters, return the same data, and have the same errors. This provides a B2 developer the flexibility to choose how to upgrade to the V2 API.

Obviously, if you want to use the functionality associated with a V2 API version, then you must use the V2 API call and update your code accordingly.

One last thing: beginning today, if we create a new B2 API call it will be created in the current API version (V2) and most likely will not be created in V1.

Standardizing B2 File Related API Calls

As requested by many B2 developers, the V2 API now uses a consistent structure for all API calls returning information about files. To enable this there are some V2 API calls that return additional fields, for example:

Restricted Application Keys

In August we introduced the ability to create restricted applications keys using the B2 API. This capability allows an account owner the ability to restrict who, how, and when the data in a given bucket can be accessed. This changed the functionality of multiple B2 API calls such that a user could create a restricted application key that could break a 3rd party integration to Backblaze B2. We subsequently updated the affected V1 API calls, so they could continue to work with the existing 3rd party integrations.

The V2 API fully implements the expected behavior when it comes to working with restricted application keys. The V1 API calls continue to operate as before.

Here is an example of how the V1 API and the V2 API will act differently as it relates to restricted application keys.

Set-up

  • The B2 account owner has created 2 public buckets, “Backblaze_123” and “Backblaze_456”
  • The account owner creates a restricted application key that allows the user to read the files in the bucket named “Backblaze_456”
  • The account owner uses the restricted application key in an application that uses the b2_list_buckets API call

In Version 1 of the B2 API

  • Action: The account owner uses the restricted application key (for bucket Backblaze_456) to access/list all the buckets they own (2 public buckets).
  • Result: The results returned are just for Backblaze_456 as the restricted application key is just for that bucket. Data about other buckets is not returned.

While this result may seem appropriate, the data returned did not match the question asked, i.e. list all buckets. V2 of the API ensures the data returned is responsive to the question asked.

In Version 2 of the B2 API

  • Action: The account owner uses the restricted application key (for bucket Backblaze_456) to access/list all the buckets they own (2 public buckets).
  • Result: A “401 unauthorized” error is returned as the request for access to “all” buckets does not match the restricted application key, e.g. bucket Backblaze_456. To achieve the desired result, the account owner can specify the name of the bucket being requested in the API call that matches the restricted application key.

Cleaning up the API

There are a handful of API calls in V2 where we dropped fields that were deprecated in V1 of the B2 API, but were still required. So in V2:

  • b2_authorize_account: The response no longer contains minimumPartSize. Use partSize and absoluteMinimumPartSize instead.
  • b2_list_file_names: The response no longer contains size. Use contentLength instead.
  • b2_list_file_versions: The response no longer contains size. Use contentLength instead.
  • b2_hide_file: The response no longer contains size. Use contentLength instead.

Support for Version 1 of the B2 API

As noted previously, V1 of the B2 API continues to function. There are no plans to stop supporting V1. If at some point in the future we do deprecate the V1 API, we will provide advance notice of at least one year before doing so.

The B2 Java SDK and the B2 Command Tool

Both the B2 Java SDK and the B2 Command Line Tool, do not currently support Version 2 of B2 API. They are being updated and will support the V2 API at the time the V2 API exits Beta and goes GA. Both of these tools, and more, can be found in the Backblaze GitHub repository.

More About the Version 2 Beta Program

We introduced Version 2 of the B2 API as beta so that developers can provide us feedback before V2 goes into production. With every B2 integration being coded differently, we want to hear from as many developers as possible. Give the V2 API a try and if you have any comments you can email our B2 beta team at b2beta@backblaze.com or contact Backblaze B2 support. Thanks.

The post Backblaze B2 API Version 2 Beta is Now Open appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Moving Tape Content to Backblaze Fireball with Canister

Post Syndicated from Skip Levens original https://www.backblaze.com/blog/moving-tape-content-to-cloud-storage/


Canister for Fireball: LTO tape to Backblaze B2 migration made 'drag and drop' easy
If you shoot video on the run and wrangle video from multiple sources, you know that reliably offloading files from your camera carts, storage cards, or pluggable SSDs can be a logistical challenge. All of your source files need to be copied over, verified, and backed up before you can begin the rest of your post-production work.  It’s arguably the most critical step in your post-production workflow.

Knowing how critical this step is, videographers and data wranglers alike have long relied on an app for Mac and Windows called Hedge to take charge of their file copy and verification needs.


Hedge source and target progress

Hedge for Mac and Windows — drag and drop source file copy and verify tool

With an intuitive drag and drop interface, Hedge makes it simple to select your cards, disks, or other sources, identify your destination drives, then copy and verify using a custom “Fast Lane” engine to speed transfers dramatically. You can log when copies were completed, and even back up to multiple destinations in the same action, including your local SAN, NAS, or Backblaze Fireball, then on to your Backblaze B2 cloud storage.

But How Do You “Data-Wrangle” Tape Content to the Cloud?

But what if you have content, backup sets, or massive media archives on LTO tape?

You may find yourself in one of these scenarios:

  • You may have “inherited” an older LTO tape system that is having a hard time keeping up with your daily workflow, and you aren’t ready to sign up for more capital expense and support contracts.
  • You may have valuable content “stuck” on tape that you can’t easily access and want it on cloud for content monetization workflows that would overwhelm your tape system.
  • Your existing tape based workflow is working fine for now, but you want to get all of that content into the cloud quickly to get ready for future growth and new customers with a solution similar to Hedge.

While many people decide to move tape workflows to cloud for simple economic reasons, having all of that content securely stored in the cloud means that the individual files and entire folders can be instantly pulled into workflows and directly shared from Backblaze B2 with no need for copying, moving, restoring, or waiting.

For more information about how Backblaze B2 can replace LTO solutions, including an LTO calculator:  Backblaze LTO Replacement Calculator

Whichever scenario fits your need, getting tape content into the cloud involves moving a lot of content at once, and in a perfect world it would be as easy to drag and drop that content from tape to Backblaze B2!

Meet Canister for Fireball

To meet this exact need the team that developed Hedge have created an “LTO tape content to Fireball” solution called Canister for Fireball.

Fireball is Backblaze’s solution to help you quickly get massive amounts of data into Backblaze B2 Cloud Storage. When you sign up for the service, Backblaze sends you a 70TB Fireball that is yours to use for 30 days. Simply attach it to your local network and copy content over to the device at the speed of your local network. You’re free to fill up and send in your Fireball device as many times as needed. When Backblaze receives your Fireball with your files, all of the content is ingested directly into Backblaze’s data centers and appears in your Backblaze B2 online storage.

Backblaze B2 Fireball Rapid Ingest Service

Canister for Fireball makes it incredibly easy to move your content and archives from your tape device to your Backblaze B2 Fireball. With an intuitive interface similar to Hedge, Canister copies over and verifies files read from your tapes.

Using Canister with B2

flow chart for moving data from tape to the cloudInsert LTO tapes in your tape system and Canister for Backblaze will move them to your Backblaze B2 Fireball for rapid ingest into your B2 Cloud Storage


Cannister to Fireball user interfaceSelect from any tape devices with LTO media…

Cannister data progression screenshot…and watch the files on the tape copy and verify to your Backblaze B2 Fireball

Here’s how the solution works:

Steps to Migrate Your LTO Content to the Cloud with Canister for Fireball

  1. Order a Fireball system: As part of the signup step you will choose a B2 bucket that you’d like your Fireball content moved to.
  2. Connect your Fireball system to your network, making sure that the workstation that connects to your tape device can also mount the storage volume presented by your Backblaze Fireball.
  3. Install Canister for Fireball on your Mac workstation.
  4. Connect your tape device. Any tape system that can read your tapes and mount them as an LTFS volume will work. Canister will automatically mount tapes inside the app for you.
  5. Launch Canister for Fireball. You can now select the tape device volume as your source, the Fireball as your target, and copy the files over to your Fireball.
  6. Repeat as needed until you have copied and verified all of your tapes securely to your Fireball. You can fill and send in your Fireball as many times as needed during your 30 day period. (And you can always extend your loaner period.)
LTFS or Linear Tape File System is an industry adopted way to make the contents of an entire tape cartridge available as if it were a single volume of files. Typically, the tape stores a list of the files and their location on that tape in the beginning, or header of the tape. When a tape is read into your tape device, that directory section is read in and the tape system then presents it to you as a volume of files and folders. Say you want to select an individual file from that LTFS volume to copy to your desktop. When you move that to your desktop, the tape spools out to wherever that file is stored, reads the entire stream of tape containing that file, then finally copies it to your desktop. It can be a very slow process indeed and why many people choose to store content in cloud storage like Backblaze B2 so that they get instant access to every file.

Now — Put Your LTO Tape Ingest Plan Into Action

If you have content on tape that needs to get into your Backblaze B2 storage, Canister for Fireball and a Backblaze B2 Fireball are the perfect solution.

Canister for Fireball can be licensed for 30 days of use for $99 and includes priority support. The full version is $199. If you decide to upgrade from the 30 Day license you’ll pay only the difference to the full version.

Get more information about Canister for Fireball

And of course, make sure that you’ve ordered your Fireball:

Order a Backblaze B2 Fireball

Now with your content and archives no longer “trapped” on tape, you can browse them in your asset manager, share links directly from Backblaze B2, and have your content ready to pull into new content creation workflows by your team located anywhere in the world.

The post Moving Tape Content to Backblaze Fireball with Canister appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

LTO versus Cloud Storage: Choosing the Model That Fits Your Business

Post Syndicated from Andy Klein original https://www.backblaze.com/blog/lto-vs-cloud-storage-vs-hybrid/

Choose Your Solution: Cloud Storage, LTO, Hybrid Cloud Storage/LTO

Years ago, when I did systems administration for a small company, we used RAID 1 for in-house data redundancy and an LTO tape setup for offsite data backup. Yes, the LTO cataloging and versioning were a pain, so was managing the tapes, and sometimes a tape would be unreadable, but the setup worked. And given there were few affordable alternatives out there at the time you lived and died with your tapes.

Over the last few years, cloud storage has emerged as a viable alternative to using LTO for offsite backups. Improvements in network speed coupled with lower costs are a couple of the factors that have changed the calculus of cloud storage. To see if enough has changed to make cloud storage a viable competitor to LTO, we’ll start by comparing the current and ongoing cost of LTO versus cloud storage and then dig into assumptions underlying the cost model. We’ll finish up by reviewing the pluses and minuses of three potential outcomes: switching to cloud storage, staying with LTO, or using a hybrid LTO/cloud storage solution.

Comparing the Cost of LTO Versus Cloud Storage

Cost calculators for comparing LTO to Cloud Storage have a tendency to be very simple or very complex. The simple ones generally compare hardware and tape costs to cloud storage costs and neglect things like personnel costs, maintenance costs, and so on. In the complex models you might see references to the cost of capital, interest on leasing equipment, depreciation, and the tax implications of buying equipment versus paying for a monthly subscription service.

The Backblaze LTO vs Cloud calculator is somewhere in between. The underlying model takes into account many factors, which we’ll get into in a moment, but if you are a Fortune 500 company with a warehouse full of tape robots, this model is not for you.

Calculator: LTO vs B2

To use the Backblaze calculator you enter:

  1. the amount of Existing Data you have on LTO tape
  2. the amount of data you expect to add in a given year
  3. the amount of incremental data you backup each day

Then you can use the slider to compare your total cost from 1 to 10 years. You can run the model as many times as you like under different scenarios.

Assumptions Behind the Model

To see the assumptions that were made in creating the model, start on the LTO Replacement page and scroll down past the LTO vs. B2 calculator. Click on the following text which will display the “Cost and Operational Assumptions” page.

+ See details on Cost and Operational Assumptions

Let’s take a few minutes to review some of the most relevant points and how they affect the cost numbers reported:

  • LTO Backup Model: We used the Grandfather-Father-Son (GFS) model. There are several others, but this was the most prevalent. If you use the “Tower of Hanoi” model for example, it uses fewer tapes and would lower the cost of the total LTO cost by some amount.
  • Data Compression: We assumed a 2-1 compression ratio for the data stored on the LTO tapes. If your data is principally video or photos, you will most likely not use compression. As such, film studios and post-production houses will need to double the cost of the total LTO solution to compensate for the increased number of tapes, the increased number of LTO tape units, and increased personnel costs.
  • Data Retention: We used a 30 day retention period as this is common in the GFS model. If you keep your incremental tapes/data for 2 weeks, then you would lower the number of tapes needed for incremental backups, but you would also lower the amount of incremental data you keep in the cloud storage system.
  • Tape Units: There are a wide variety of LTO tape systems. You can increase or decrease the total LTO cost based on the systems you are using. For example, you are considering the purchase of an LTO tape system which reads/writes up to 5 tapes simultaneously. That system is more expensive and has higher maintenance costs, but it also would mean you would have to purchase fewer tape units.
  • LTO-8 Tape Units: We used LTO-8 tape units as they are the currently available LTO system most likely to be around in 10 years.
  • Tape Migration: We made no provision for migration from an unsupported LTO version to a supported LTO version. During the next 10 years, many users with older LTO systems will find it likely they will have to migrate to newer systems as LTO only supports 2 generations back and is currently offering a new generation every 2 years.
  • Pickup Cost: The cost of having your tapes picked up so they are offsite. This cost can vary widely based on geography and service level. Our assumption of the cost is $60 per week or $3,120/year. You can adjust the LTO total cost according to your particular circumstances.
  • Network Cost: Using cloud storage requires that you have a reasonable amount of network bandwidth available. The number we used is incremental to your existing monthly cost for bandwidth. Network costs vary widely, so depending on your circumstance you can increase or decrease to the total cost of the cloud storage solution.
  • Personnel Cost: This is the total cost of what you are paying someone to manage and operate your LTO system. This raises or lowers the cost of both the LTO and cloud storage solutions at the same rate, so adjusting this number doesn’t affect the comparison, just the total values for each.
  • Time Savings Versus LTO: With a cloud storage solution, there are no tapes or tape machines to deal with. This saves a significant amount of time for the person managing the backup process. Increasing this value will increase the cost of the cloud storage solution relative to the LTO solution.

As hinted at earlier, we don’t consider the cost of capital, depreciation, etc. in our calculations. The general model is that a company purchases a number of LTO systems and the cost is spread over a 10 year period. After 10 years a replacement unit is purchased. Other items such as tapes and equipment maintenance are purchased and expensed as needed.

Choosing a Data Backup Model

We noted earlier the three potential outcomes when evaluating LTO versus cloud storage for data backup: switching to cloud storage, staying with LTO, or using a hybrid LTO/cloud storage solution. Here’s a look at each.

Switching to Cloud Storage

After using the calculator you find cloud storage is less expensive for your business or organization versus LTO. You don’t have a large amount of existing data, 100 terabytes for example, and you’d rather get out of the tape business entirely.

Your first challenge is to move your existing data to the cloud — quickly. One solution is the Backblaze B2 Fireball data transfer service. You can move up to 70 TB of data each trip from your location to Backblaze in days. This saves your bandwidth and saves time as well.

As the existing data is being transferred to Backblaze, you’ll want to select a product or service to move your daily generated information to the cloud on a regular basis. Backblaze has a number of integration partners that perform data backup services to Backblaze B2

Staying with LTO

After using the calculator you find cloud storage is less expensive, but you are one of those unlucky companies that can’t get reasonably priced bandwidth in their area. Or perhaps, the new LTO-8 equipment you ordered arrived minutes before you read this blog post. Regardless, you are destined to use LTO for at least a while longer. Tried and true, LTO does work and has the added benefit of making the person who manages the LTO setup nearly indispensable. Still, when you are ready, you can look at moving to the hybrid model described next.

Hybrid LTO/Cloud Storage model

In practice, many organizations that use LTO for backup and archive often store some data in the cloud as well, even if haphazardly. For our purposes, Hybrid LTO/Cloud Storage is defined as one of the following:

  1. Date Hybrid: All backups and archives from prior to the cut over date remain stored in LTO; everything after the cut over date date forward is stored in cloud storage.
  2. Classic Hybrid: All of the incremental backups are stored in cloud storage and all full backups and archives are stored on LTO.
  3. Type Hybrid: All data of a given type, say employee data, is stored on LTO, while all customer data is stored in cloud storage. We see this hybrid use case occur as a function of convenience and occasionally compliance, although some regulatory requirements such as GDPR may not be accommodated by LTO solutions.

You can imagine there being other splits, but in essence, there may be situations where keeping the legacy system going in some capacity for some period of time is the prudent business option.

If you have a large tape library, it can be almost paralyzing to think about moving to the cloud, even if it is less expensive. Being open to the hybrid LTO/cloud model is a way to break the task down into manageable steps. For example, solutions like Starwind VTL and Archiware P5 allow you to start backing up to the cloud with minimal changes to your existing tape-based backup schemes.

Many companies that start down the hybrid road typically begin with moving their daily incremental files to the cloud. This immediately reduces the amount of “tape work” you have to do each day and it has the added benefit of making the files readily available should they need to be restored. Once a company is satisfied that their cloud based backups for their daily incremental files are under control, they can consider whether or not they need to move the rest of their data to the cloud.

Will Cloud Storage Replace LTO?

At some point, the LTO tapes you have will need to be migrated to something else as the equipment to read your old tapes will become outdated, then unsupported, and finally unavailable. Users with LTO 4 and, to some degree, LTO 5 are already feeling this pain. To migrate all of that data from your existing LTO system to LTO version “X,” cloud storage, or something else, will be a monumental task. It is probably a good idea to start planning for that now.

In summary, many people will find that they can now choose cloud storage over LTO as an affordable way to store their data going forward. But, having a hybrid environment of both LTO and cloud storage is not only possible, it is a practical way to reduce your overall backup cost while maximizing your existing LTO investment. The hybrid model creates an improved operational environment and provides a pathway forward should you decide to move exclusively to storing your data in the cloud at some point in the future.

The post LTO versus Cloud Storage: Choosing the Model That Fits Your Business appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

How to Leverage Your Amazon S3 Experience to Code the Backblaze B2 API

Post Syndicated from Andy Klein original https://www.backblaze.com/blog/how-to-code-backblaze-b2-api-interface/

Going from S3 to learning Backblaze B2

We wrote recently about how the Backblaze B2 and Amazon S3 APIs are different. What we neglected to mention was how to bridge those differences so a developer can create a B2 interface if they’ve already coded one for S3. John Matze, Founder of BridgeSTOR, put together his list of things consider when levering your S3 API experience to create a B2 interface. Thanks John.   — Andy
BackBlaze B2 to Amazon S3 Conversion
by John Matze, Founder of BridgeSTOR

BackBlaze B2 Cloud Storage Platform has developed into a real alternative to the Amazon S3 online storage platform with the same redundancy capabilities but at a fraction of the cost.

Sounds great — sign up today!

Wait. If you’re an application developer, it doesn’t come free. The Backblaze REST API is not compatible with Amazon S3 REST API. That is the bad news. The good news — it includes almost the entire set of functionality so converting from S3 to B2 can be done with minimal work once you understand the differences between the two platforms.

This article will help you shortcut the process by describing the differences between B2 and S3.

  1. Endpoints: AWS has a standard endpoint of s3.amazonaws.com which redirects to the region where the bucket is located or you may send requests directly to the bucket by a region endpoint. B2 does not have regions, but does have an initial endpoint called api.blackblazeb2.com. Every application must start by talking to this endpoint. B2 also requires two other endpoints. One for uploading an object and another one for downloading an object. The upload endpoint is generated on demand when uploading an object while the download API is returned during the authentication process and may be saved for download requests.
  1. Host: Unlike Amazon S3, the HTML header requires the host token. If it is not present, B2 will not respond with an error.
  1. JSON: Unlike S3, which uses XML, all B2 calls use JSON. Some API calls require data to be sent on the request. This data must be in JSON and all APIs return JSON as a result. Fortunately, the amount of JSON required is minimal or none at all. We just built a JSON request when required and made a simple JSON parser for returned data.
  1. Authentication: Amazon currently has two major authentication mechanisms with complicated hashing formulas. B2 simply uses the industry standard “HTTP basic auth” algorithm. It takes only a few minutes to get to speed on this algorithm.
  1. Keys: Amazon has the concept of an access key and a secret key. B2 has the equivalent with the access key being your key id (your account id) and the secret key being the application id (returned from the website) that maps to the secret key.
  1. Bucket ID: Unlike S3, almost every B2 API requires a bucket ID. There is a special list bucket call that will display bucket IDs by bucket name. Once you find your bucket name, capture the bucket ID and save it for future API calls.
  1. Head Call: The bottom line — there is none. There is, however, a list_file_names call that can be used to build your own HEAD call. Parse the JSON returned values and create your own HEAD call.
  1. Directory Listings: B2 Directories again have the same functionality as S3, but with a different API format. Again the mapping is easy: marker is startFileName, prefix is prefix, max-keys is maxFileCount and delimiter is delimiter. The big difference is how B2 handles markers. The Amazon S3 nextmarker is literally the next marker to be searched, the B2 nextmarker is the last file name that was searched. This means the next listing will also include the last marker name again. This means your routines must parse out the name or your listing will show the next marker twice. That’s a difference, but not a difficult one.
  1. Uploading an object: Uploading an object in B2 is quite different than S3. S3 just requires you to send the object to an endpoint and they will automatically place the object somewhere in their environment. In the B2 world, you must request a location for the object with an API call and then send the object to the returned location. The first API will send you a temporary key and you can continue to use this key for one hour without generating another, with the caveat that you have to monitor for failures from B2. The B2 environment may become full or some other issue will require you to request another key.
  1. Downloading an Object: Downloading an object in B2 is really easy. There is a download endpoint that is returned during the authentication process and you pass your request to that endpoint. The object is downloaded just like Amazon S3.
  1. Multipart Upload: Finally, multipart upload. The beast in S3 is just as much of a beast in B2. Again the good news is there is a one to one mapping.
    1. Multipart Init: The equivalent initialization returns a fileid. This ID will be used for future calls.
    2. Mulitpart Upload: Similar to uploading an object, you will need to get the API location to place the part. So use the fileid from “a” above and call B2 for the endpoint to place the part. Another difference is the upload also requires the payload to be hashed with a SHA1 algorithm. Once done, simply pass the SHA and the part number to the URL and the part is uploaded. This SHA1 component is equivalent to an etag in the S3 world so save it for later.
    3. Multipart Complete: Like S3, you will have to build a return structure for each part. B2 of course requires this structure to be in JSON but like S3, B2 requires the part number and the SHA1 (etag) for each part.

What Doesn’t Port

We found almost everything we required easily mapped from S3 to B2 except for a few issues. To be fair, BackBlaze is working on the following in future versions.

  1. Copy Object doesn’t exist: This could cause some issues with applications for copying or renaming objects. BridgeSTOR has a workaround for this situation so it wasn’t a big deal for our application.
  2. Directory Objects don’t exist: Unlike Amazon, where an object with that ends with a “/” is considered a directory, this does not port to B2. There is an undocumented object name that B2 applications use called .bzEmpty. Numerous 3rd party applications, including BridgeSTOR, treat an object ending with .bzEmpty as a directory name. This is also important for directory listings described above. If you choose to use this method, you will be required to replace the “.bzEmpty” with a “/.”

In conclusion, you can see the B2 API is different than the Amazon S3, but as far as functionality they are basically the same. For us at first it looked like it was going to be a large task, but once we took the time to understand the differences, porting to B2 was not a major job for our application. We created a S3 to B2 shim in a week followed by a few extra weeks of testing and bug fixes. I hope this document helps in your S3 to B2 conversion.

— John Matze, BridgeSTOR

The post How to Leverage Your Amazon S3 Experience to Code the Backblaze B2 API appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Creating a Media Archive Solution with Backblaze B2 and Archiware P5

Post Syndicated from Skip Levens original https://www.backblaze.com/blog/creating-a-media-archive-solution/

Backblaze B2 Cloud Storage + Archiware P5= 7 Ways to Save

B2 + P5 = 7 Ways to Save Time, Money and Gain Peace of Mind with an Archive Solution of Backblaze B2 and Archiware P5

by Dr. Marc M. Batschkus, Archiware

This week’s guest post comes to us from Marc M. Batschkus of Archiware, who is well-known to media and entertainment customers, and is a trusted authority and frequent speaker and writer on data backup and archiving.

— Editor

Archiving has been around almost forever.

Roman "Archivum"Roman “Archivum” where scrolls were stored for later reference.

The Romans used the word “archivum” for the building that stored scrolls no longer needed for daily work. Since then, files have replaced scrolls, but the process has stayed the same and so today, files that are no longer needed for daily production can be moved to an archive.

Backup and Archive

Backblaze and Archiware complement each other in accomplishing this and we’ll show you how to get the most from this solution. But before we look at the benefits of archiving, let’s take a step back and review the difference between backup and archive.

A backup of your production storage protects your media files by replicating the files to a secondary storage. This is a cyclical process, continually checking for changed and new files, and overwriting files after the specified retention time is reached.

Archiving, on the other hand is a data migration, moving files that are no longer needed for daily production to (long-term) storage, yet keeping them easily retrievable. This way, all completed productions are collected in one place and kept for later reference, compliance, and re-use.

Think of backup as a spare tire, archive as winter tiresThink of BACKUP as a spare tire, in case you need it, and ARCHIVE as a stored set of tires for different needs.

To use an analogy:

  • Think of backup as the spare tire in the trunk.
  • Think of archive as the winter tires in the garage.

Both are needed!

Editor’s note: For more insight on “backup vs archive” have a look at What’s the Diff: Backup vs Archive.

Building a Media Archive Solution with Archiware P5 and Backblaze B2

Now that the difference between backup and archive is clear, let’s have a look at what an archive can do to make your life easier.

Archiware archive catalog transfering to B2 cloud storageArchiware P5 can be your interface to locate and manage your files, with Backblaze B2 as your ready storage for all of those files

P5 Archive connects to Backblaze B2 and offers the interface for locating files.

B2 + P5 = 7 Ways to Save Time and Money and Gain Peace-of-Mind

  1. Free up expensive production storage
  2. Archive from macOS, Windows, and Linux
  3. Browse and search the archive catalog with thumbnails and proxies
  4. Re-use, re-purpose, reference and monetize files
  5. Customize the metadata schema to fit your needs and speed up search
  6. Reduce backup size and runtime by moving files from production storage
  7. Protect precious assets from local disaster and for the long-term (no further migration/upgrade needed)

Archive as Mini-MAM

The “Mini-MAM” features of Archiware P5 help you to browse and find files easier than ever. Browse the archive visually using the thumbnails and proxy clips in the archive catalog. Search for specific criteria or a combination of criteria such as location or description.

Since P5 Archive lets you easily expand and customize metadata fields and menus, you can build the individual metadata schema that works best for you.

Technical metadata (e.g. camera type, resolution, lens) can be automatically imported from the file header into the metadata fields of P5 archive using a script.

The archive becomes the file memory of the company saving time and energy because now there is only one place to browse and search for files.

Mini MAM screenshotArchiware as “Mini-MAM” —  thumbnails, proxies, even metadata all within Archiware P5

P5 offers maximum flexibility and supports all storage strategies, be it cloud, disk or tape and any combination of the above.

For more information on Archiving with Archiware: Archiving with Archiware P5. For macOS, P5 Archive offers integration with the Finder and Final Cut Pro X via the P5 Archive App. For more information on integrated archiving with Final Cut Pro X: macOS Finder and Final Cut Pro X Integrated Archiving

You can start building an archive immediately with Backblaze B2 cloud storage because it allows you to do this without any additional storage hardware and upfront investment.

Backblaze B2 is the Best of Cloud

  • ✓  Saves investment in storage hardware
  • ✓  Access from anywhere
  • ✓  Storage on demand
  • ✓  Perpetual storage – no migration or upgrade of hardware
  • ✓  Financially advantageous (OPEX vs CAPEX)
  • ✓  Best price in its category

Backblaze B2 offers flexible access so that the archive can be accessed from several physical locations with no storage hardware needing to be moved.

P5 Archive supports consumable files as archive format. This makes the single files accessible even if P5 Archive is not present at the other location. This opens up a whole new world of possibilities for collaborative workflows that were not possible before.

Save Money with OPEX vs CAPEX

CAPEX vs. OPEXCAPital EXpenditures are the money companies spend to purchase major physical goods that will be used for more than one year. Examples in our field are investments in hardware such as storage and servers.

OPerating EXpenses are the costs for a company to run its business operations on a daily basis. Examples are rent and monthly cost for cloud storage like B2.

By using Backblaze B2, companies can save CAPEX and instead have monthly payments only for the cloud storage they use, while also saving maintenance and migration cost. Furthermore, migrating files to B2 makes expansion of high performance and costly production storage unnecessary. Over time this alone will make the archive pay off.

Now that you know how to profit from archiving with Archiware P5 and Backblaze B2, let’s look at the steps to build the best archive for you.

Connecting B2 cloud storage screenshot

Backblaze B2 is already a built-in option in P5 and works with P5 Archive and P5 Backup.

For detailed setup and best practice see:

Cloud Storage Setup and Best Practice for Archiware

Steps in Planning a Media Archive

Depending on the size of the archive, the number of people working with and using it, and the number of files that are archived, planning might be extremely important. Thinking ahead and asking the right questions ensures that the archive later delivers the value that it was built for.

Including people that will configure, operate, and use the system guarantees a high level of acceptance and avoids blind spots in your planning.

  1. Define users: who administers, who uses and who archives?
  2. Decide and select: what goes into the archive, and when?
  3. Which metadata are needed to describe the data needed (what will be searched for)?
  4. Actual security: on what operating system, hardware, software, infrastructure, interfaces, network and medium will be archived?
  5. What security requirements should be fulfilled: off-site storage, duplication, storage duration, test cycles of media, generation migration, etc.
  6. Retrieval:
    • Who searches?
    • With what criteria?
    • Who is allowed to restore?
    • On what storage?
    • For what use?

Metadata is the key to the archive and enables complex searches for technical and descriptive criteria.

Naming Conventions or “What’s in a File Name?”

The most robust metadata you can have is the file name. It can travel through different operating systems and file systems. The file name is the only metadata that is available all the time. It is independent of any database, catalog, MAM system, application, or other mechanism that can keep or read metadata. With it, someone can instantly make sense of a file that gets isolated, left over, misplaced or transferred to another location. Building a solid and intelligent naming convention for media files is crucial. Consistency is key for metadata. Metadata is a solid foundation for the workflow, searching and sharing files with other parties. The filename is the starting point.

Wrapping Up

There is much more that can make a media archive extremely worthwhile and efficient. For further reading I’ve made this free eBook available for more tips on planning and implementation.

eBook:  Data Management, Backup and Archive for Media Professionals — How to Protect Valuable Video Data in All Stages of the Workflow by Marc M. Batschkus

Start looking into the benefits an archive can bring you today. There is a 30-day fully featured trial license for Archiware P5 that can be combined with the Backblaze B2 free trial storage.

Trial License:  About Archiware P5 and 30-Day Trial

And of course, if you’re not already a Backblaze B2 customer, sign up instantly at the link below.

B2 Cloud Storage:  Instant Signup

— Dr. Marc M. Batschkus, Archiware

The post Creating a Media Archive Solution with Backblaze B2 and Archiware P5 appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

cPanel Backup to B2 Cloud Storage

Post Syndicated from Roderick Bauer original https://www.backblaze.com/blog/cpanel-backup-to-b2-cloud-storage/

laptop on a desk with a cup of coffee, cell phone, and iPad

Anyone who’s managed a business or personal website is likely familiar with cPanel, the control panel that provides a graphical interface and tools that simplify the process of managing a website. IT professionals who’ve managed hosting servers might know cPanel’s big brother, WHM (Web Host Manager), which is used by server administrators to manage large web hosting servers and cPanels for their customers.

cPanel Dashboard WHM Dashboard
cPanel Dashboard   WHM Dashboard

Just as with any other online service, backup is critically important to safeguard user and business data from hardware failure, accidental loss, or unforeseen events. Both cPanel and WHM support a number of applications for backing up websites and servers.

JetApps’s JetBackup cPanel App

One of those cPanel applications is JetApps’s JetBackup, which supports backing up data to a number of destinations, including local, remote SSH, remote FTP, and public cloud services. Backblaze B2 Cloud Storage was added as a backup destination in version 3.2. Web hosts that support JetBackup for their cPanel and WHM users include Clook, FastComet, TMDHosting, Kualo, Media Street, ServerCake, WebHost.UK.net, MegaHost, MonkeyTree Hosting, and CloudBunny.

cPanel with JetBackup app

cPanel with JetBackup app

JetBackup configuration for B2

JetBackup configuration for B2

Directions for configuring JetBackup with B2 are available on their website.

Note:  JetBackup version 3.2+ supports B2 cloud storage, but that support does not currently include incremental backups. JetApps has told us that incremental backup support will be available in an upcoming release.

Interested in more B2 Support for cPanel and WHM?

JetBackup support for B2 was added to JetBackup because their users asked for it. Users have been vocal in asking vendors to add cPanel/WHM support for backing up to B2 in forums and online discussions, as evidenced on cPanel.net and elsewhere — here, here, and here. The old axiom that the squeaky wheel gets the grease is true when lobbying vendors to add B2 support — the best way to have B2 directly supported by an app is to express your interest directly to the backup app provider.

Other Ways to Back Up Website Data to B2

When a dedicated backup app for B2 is not available, some cPanel users are creating their own solutions using the B2 Command Line Interface (CLI), while others are using Rclone to back up to B2.

B2 CLI example:

#!/bin/bash
b2 authorize_account ACCOUNTID APIKEY
b2 sync –noProgress /backup/ b2://STORAGECONTAINER/

Rclone example:

rclone copy /backup backblaze:my-server-backups –transfers 16

Those with WordPress websites have other options for backing up their sites, which we highlighted in a post, Backing Up WordPress.

Having a Solid Backup Plan is What’s Important

If you’re using B2 for cPanel backup, or are using your own backup solution, please let us know what you’re doing in the comments.

The post cPanel Backup to B2 Cloud Storage appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

The B2 Developers’ Community

Post Syndicated from Roderick Bauer original https://www.backblaze.com/blog/object-storage-developer-community/

Developers at Work Using Object Storage

When we launched B2 Cloud Storage in September of 2015, we were hoping that the low cost, reliability, and openness of B2 would result in developers integrating B2 object storage into their own applications and platforms.

We’ve continually strengthened and encouraged the development of more tools and resources for the B2 developer community. These resources include APIs, a Command-Line tool, a Java SDK, and code examples for Swift and C++. Backblaze recently added application keys for B2, which enable developers to restrict access to B2 data and control how an application interacts with that data.

An Active B2 Developer Community

It’s three years later and we are happy to see that an active developer community has sprung up around B2. Just a quick look at GitHub shows over 250 repositories for B2 code with projects in ten different languages that range from C# to Go to Ruby to Elixir. A recent discussion on Hacker News about a B2 Python Library resulted in 225 comments.

B2 coding languages - Java, Ruby, C#, Shell, PHP, R, JavaScript, C++, Elixir, Go, Python, Swift

What’s Happening in the B2 Developer Community?

We believe that the two major reasons for the developer activity supporting B2 are, 1) the user demand for inexpensive and reliable storage, and, 2) the ease of implementation of the B2 API. We discussed the B2 API design decisions in a recent blog post.

Sharing and transparency have been cornerstone values for Backblaze since our founding, and we believe openness and transparency breed trust and further innovation in the community. Since we ask customers to trust us with their data, we want our actions to show why we are worthy of that trust.

Here are Just Some of the Many B2 Projects Currently Underway

We’re excited about all the developer activity and all of the fresh and creative ways you are using Backblaze B2 storage. We want everyone to know about these developer projects so we’re spotlighting some of the exciting work that is being done to integrate and extend B2.

Rclone (Go) — In addition to being an open source command line program to sync files and directories to and from cloud storage systems, Rclone is being used in conjunction with other applications such as restic. See Rclone on GitHub, as well.

CORS (General web development) — Backblaze supports CORS for efficient cross-site media serving. CORS allows developers to store large or infrequently accessed files on B2 storage, and then refer to and serve them securely from another website without having to re-download the asset.

b2blaze (Python) — The b2blaze Python library for B2.

Laravel Backblaze Adapter (PHP) — Connect your Laravel project to Backblaze connector with this storage adapter with token caching.

Wal-E (Postgres) — Continuous archiving to Backblaze for your Postgres databases.

Phoenix (Elixir) — File upload utility for the Phoenix web dev framework.

ZFS Backup (Go) — Backup tool to move your ZFS snapshots to B2.

Django Storage (Python) — B2 storage for the Python Django web development framework.

Arq Backup (Mac and Windows application) — Arq Backup is an example of a single developer, Stefan Reitshamer, creating and supporting a successful and well-regarded application for cloud backup. Stefan also is known for being responsive to his users.

Go Client & Libraries (Go) — Go is a popular language that is being used for a number of projects that support B2, including restic, Minio, and Rclone.

How to Get Involved as a B2 Developer

If you’re considering developing for B2, we encourage you to give it a try. It’s easy to implement and your application and users will benefit from dependable and economical cloud storage.

Developers at workStart by checking out the B2 documentation and resources on our website. GitHub and other code repositories are also great places to look. If you follow discussions on Reddit, you could learn of projects in the works and maybe find users looking for solutions.

We’ve written a number of blog posts highlighting the integrations for B2. You can find those by searching for a specific integration on our blog or under the tag B2. Posts for developers are tagged developer.

Developers at work

If you have a B2 integration that you believe will appeal to a significant audience, you should consider submitting it to us. Those that pass our review are listed on the B2 Integrations page on our website. We’re adding more each week. When you’re ready, just review the B2 Integration Checklist and submit your application. We’re looking forward to showcasing your work!

Now’s a good time to join the B2 developers’ community. Jump on in — the water’s great!

P.S. We want to highlight and promote more developers working with B2. If you have a B2 integration or project that we haven’t mentioned in this post, please tell us what you’re working on in the comments.

The post The B2 Developers’ Community appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Backing UP FreeNAS and TrueNAS to Backblaze B2

Post Syndicated from Roderick Bauer original https://www.backblaze.com/blog/how-to-setup-freenas-cloud-storage/

FreeNAS and TrueNAS

Thanks to recent updates of FreeNAS and TrueNAS, backing up data to Backblaze B2 Cloud Storage is now available for both platforms. FreeNAS/TrueNAS v11.1 adds a feature called Cloud Sync, which lets you sync, move, or copy data to and from Backblaze B2.

What Are FreeNAS and TrueNAS?

FreeNAS and TrueNAS are two faces of a comprehensive NAS storage environment built on the FreeBSD OS and OpenZFS file system. FreeNAS is the open source and development platform, while TrueNAS is the supported and commercial product line offered by IXSystems.

FreeNAS logo

FreeNAS is for the DIY crowd. If you don’t mind working with bleeding-edge software and figuring out how to make your software and hardware work harmoniously, then FreeNAS could be a good choice for you.

TrueNAS logo

If you’re in a business or other environment with critical data, then a fully supported product like TrueNAS is likely the way you’ll want to go. IXsystems builds their TrueNAS commercial server appliances on the battle-tested, open source framework that FreeNAS and OpenZFS provide.

The software developed by the FreeNAS open source community forms the basis for both platforms, so we’ll talk specifically about FreeNAS in this post.

Working with FreeNAS

You can download FreeNAS directly from the open source project website, freenas.org. Once installed, FreeNAS is managed through a comprehensive web interface that is supplemented by a minimal shell console that handles essential administrative functions. The web interface supports storage pool configuration, user management, sharing configuration, and system maintenance.

FreeNAS web UI

FreeNAS supports Windows, macOS and Unix clients.

Syncing to B2 with FreeNAS

Files or directories can be synchronized to remote cloud storage providers, including B2, with the Cloud Sync feature.

Selecting Tasks ‣ Cloud Sync shows the screen below. This screen shows a single cloud sync called “backup-acctg” that “pushes” a file to cloud storage. The last run finished with a status of SUCCESS.

Existing cloud syncs can be run manually, edited, or deleted with the buttons that appear when a single cloud sync line is selected by clicking with the mouse.

FreeNAS Cloud Sync status

Cloud credentials must be defined before a cloud sync is created. One set of credentials can be used for more than one cloud sync. For example, a single set of credentials for Backblaze B2 can be used for separate cloud syncs that push different sets of files or directories.

A cloud storage area must also exist. With B2, these are called buckets and must be created before a sync task can be created.

After the credentials and receiving bucket have been created, a cloud sync task is created with Tasks ‣ Cloud Sync ‣ Add Cloud Sync. The Add Cloud Sync dialog is shown below.

FreeNAS Cloud Sync credentials

Cloud Sync Options

The table below shows the options for Cloud Sync.

Setting Value Type Description
Description string a descriptive name for this Cloud Sync
Direction string Push to send data to cloud storage, or Pull to pull data from the cloud storage
Provider drop-down
menu
select the cloud storage provider; the list of providers is defined by Cloud Credentials
Path browse
button
select the directories or files to be sent for Push syncs or the destinations for Pull syncs
Transfer Mode drop-down
menu
Sync (default): make files on destination system identical to those on the source; files removed from the source are removed from the destination (like rsync –delete)
Copy: copy files from the source to the destination, skipping files that are identical (like rsync)
Move: copy files from the source to the destination, deleting files from the source after the copy (like mv)
Minute slider or
minute selections
select Every N minutes and use the slider to choose a value, or select Each selected minute and choose specific minutes
Hour slider or
hour selections
select Every N hours and use the slider to choose a value, or select Each selected hour and choose specific hours
Day of month slider or
day of month
selections
select Every N days of month and use the slider to choose a value, or select Each selected day of month and choose specific days
Month checkboxes months when the Cloud Sync runs
Day of week checkboxes days of the week when the Cloud Sync runs
Enabled checkbox uncheck to temporarily disable this Cloud Sync

Take care when choosing a Direction. Most of the time, Push will be used to send data to the cloud storage. Pull retrieves data from cloud storage, but be careful: files retrieved from cloud storage will overwrite local files with the same names in the destination directory.

Provider is the name of the cloud storage provider. These providers are defined by entering credentials in Cloud Credentials.

After the Provider is chosen, a list of available cloud storage areas from that provider is shown. With B2, this is a drop-down with names of existing buckets.

Path is the path to the directories or files on the FreeNAS system. On Push jobs, this is the source location for files sent to cloud storage. On Pull jobs, the Path is where the retrieved files are written. Again, be cautious about the destination of Pull jobs to avoid overwriting existing files.

The Minute, Hour, Days of month, Months, and Days of week fields permit creating a flexible schedule of when the cloud synchronization takes place.

Finally, the Enabled field makes it possible temporarily disable a cloud sync job without deleting it.

FreeNAS Cloud Sync Example

This example shows a Push cloud sync which writes an accounting department backup file from the FreeNAS system to Backblaze B2 storage.

Before the new cloud sync was added, a bucket called “cloudsync-bucket” was created with the B2 web console for storing data from the FreeNAS system.

System ‣ Cloud Credentials ‣ Add Cloud Credential is used to enter the credentials for storage on a Backblaze B2 account. The credential is given the name B2, as shown in the image below:

FreeNAS Cloud Sync B2 credentials

Note on encryption: FreeNAS 11.1 Cloud Sync does not support client-side encryption of data and file names before syncing to the cloud, whether the destination is B2 or another public cloud provider. That capability will be available in FreeNAS v11.2, which is currently in beta.

Example: Adding Cloud Credentials

The local data to be sent to the cloud is a single file called accounting-backup.bin on the smb-storage dataset. A cloud sync job is created with Tasks ‣ Cloud Sync ‣ Add Cloud Sync.

The Description is set to “backup-acctg” to describe the job. This data is being sent to cloud storage, so this is a Push. The provider comes from the cloud credentials defined in the previous step, and the destination bucket “cloudsync-bucket” has been chosen.

The Path to the data file is selected.

The remaining fields are for setting a schedule. The default is to send the data to cloud storage once an hour, every day. The options provide great versatility in configuring when a cloud sync runs, anywhere from once a minute to once a year.

The Enabled field is checked by default, so this cloud sync will run at the next scheduled time.

The completed dialog is shown below:

FreeNAS Cloud Sync example

Dependable and Economical Disaster Recovery

In the event of an unexpected data-loss incident, the VMs, files, or other data stored in B2 from FreeNAS or TrueNAS are available for recovery. Having that data ready and available in B2 provides a dependable, easy, and cost effective offsite disaster recovery solution.

Are you using FreeNAS or TrueNAS? What tips do you have? Let us know in the comments.

The post Backing UP FreeNAS and TrueNAS to Backblaze B2 appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Minio as an S3 Gateway for Backblaze B2 Cloud Storage

Post Syndicated from Roderick Bauer original https://www.backblaze.com/blog/how-to-use-minio-with-b2-cloud-storage/

Minio + B2

While there are many choices when it comes to object storage, the largest provider and the most recognized is usually Amazon’s S3. Amazon’s set of APIs to interact with their cloud storage, often just called “S3,” is frequently the first integration point for an application or service needing to send data to the cloud.

One of the more frequent questions we get is “how do I jump from S3 to B2 Cloud Storage?” We’ve previously highlighted many of the direct integrations that developers have built on B2: here’s a full list.

Another way to work with B2 is to use what is called a “cloud storage gateway.” A gateway is a service that acts as a translation layer between two services. In the case of Minio, it enables customers to take something that was integrated with the S3 API and immediately use it with B2.

Before going further, you might ask “why didn’t Backblaze just create an S3 compatible service?” We covered that topic in a recent blog post, Design Thinking: B2 APIs (& The Hidden Costs of S3 Compatibility). The short answer is that our architecture enables some useful differentiators for B2. Perhaps most importantly, it enables us to sustainably offer cloud storage at a ¼ of the price of S3, which you will really appreciate as your application or service grows.

However, there are situations when a customer is already using the S3 APIs in their infrastructure and want to understand all the options for switching to B2. For those customers, gateways like Minio can provide an elegant solution.

What is Minio?

Minio is an open source, multi-cloud object storage server and gateway with an Amazon S3 compatible API. Having an S3-compatible API means once configured, Minio acts as a gateway to B2 and will automatically and transparently put or get data into a Backblaze B2 account.

Backup, archive or other software that supports the S3 protocol can be configured to point at Minio. Minio internally translates all the incoming S3 API calls into equivalent B2 storage API calls, which means that all Minio buckets and objects are stored as native B2 buckets and objects. The S3 object layer is transparent to the applications that use the S3 API. This enables the simultaneous use of both Amazon S3 and B2 APIs without compromising any features.

Minio has become a popular solution, with over 113.7M+ Docker pulls. Minio implements the Amazon S3 v2/v4 API in the Minio client, AWS SDK, and in the AWS CLI.

Minio and B2

To try it out, we configured a MacBook Pro with a Docker container for the latest version of Minio. It was a straightforward matter to install the community version of Docker on our Mac and then install the container for Minio.

You can follow the instructions on GitHub for configuring Minio on your system.

In addition to using Minio with S3-compatible applications and creating new integrations using their SDK, one can use Minio’s Command-line Interface (CLI) and the Minio Browser to access storage resources.

Command-line Access to B2

We installed the Minio client (mc), which provides a modern CLI alternative to UNIX coreutils such as ls, cat, cp, mirror, diff, etc. It supports filesystems and Amazon S3 compatible cloud storage services. The Minio client is supported on Linux, Mac, and Windows platforms.

We used the command below to add the alias “myb2” to our host to make it easy to access our data.

mc config host add myb2 \
 http://localhost:9000 b2_account_id b2_application_key

Minio client commands

Once configured, you can use mc subcommands like ls, cp, mirror to manage your data.

Here’s the Minio client command to list our B2 buckets:

mc ls myb2

And the result:

Minio client

Browsing Your B2 Buckets

Minio Gateway comes with an embedded web based object browser that makes it easy to access your buckets and files on B2.

Minio browser

Minio is a Great Way to Try Out B2

Minio is designed to be straightforward to deploy and use. If you’re using an S3-compatible integration, or just want to try out Backblaze B2 using your existing knowledge of S3 APIs and commands, then Minio can be a quick solution to getting up and running with Backblaze B2 and taking advantage of the lower cost of B2 cloud storage.

The post Minio as an S3 Gateway for Backblaze B2 Cloud Storage appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Design Thinking: B2 APIs (& The Hidden Costs of S3 Compatibility)

Post Syndicated from Brian Wilson original https://www.backblaze.com/blog/design-thinking-b2-apis-the-hidden-costs-of-s3-compatibility/

API Functions - Authorize, Download, List, Upload

When we get asked, “why did Backblaze make its own set of APIs for B2?” the question behind the question is most often “why didn’t Backblaze just implement an S3-compatible interface for B2?”

Either are totally reasonable questions to ask. The quick answer to either question? So our customers and partners can move faster while simultaneously enabling Backblaze to sustainably offer a cloud storage service that is ¼ of the price of S3.

But, before we jump all the way to the end, let me step you through our thinking.

The Four Major Functions of Cloud Storage APIs

Throughout cloud storage, S3 or otherwise, APIs are meant to mainly provide access to four major underlying functions:

  • Authorization — providing account/bucket/file access
  • Upload — Sending files to the cloud
  • Download — Data retrieval
  • List — Data checking / selection / comparison

The comparison between B2 and S3 on the List and Download functions is, candidly, not that interesting. Fundamentally, we ended up having similar approaches when solving those challenges. If the detail is of interest, I’m happy to get into that on a later post or answer questions in the comments below.

Backblaze and Amazon did take different approaches to how each service handles Authorization. The 5 step approach for S3 is well outlined here. B2’s architecture enables secure authorization in just 2 steps. My assertion is that a 2 step architecture is ~60% simpler than having a 5 step approach. To understand what we’re doing, I’d like to introduce the concept of Backblaze’s “Contract Architecture.”

The easiest way to understand B2’s Contract Architecture is to deep dive into how we handle the Upload process.

Uploads (Load Balancing vs Contract Architecture)

The interface to upload data into Amazon S3 is actually a bit simpler than Backblaze B2’s API. But it comes at a literal cost. It requires Amazon to have a massive and expensive choke point in their network: load balancers. When a customer tries to upload to S3, she is given a single upload URL to use. For instance, http://s3.amazonaws.com/<bucketname>. This is great for the customer as she can just start pushing data to the URL. But that requires Amazon to be able to take that data and then, in a second step behind the scenes, find available storage space and then push that data to that available location. The second step creates a choke point as it requires having high bandwidth load balancers. That, in turn, carries a significant customer implication; load balancers cost significant money.

When we were creating the B2 APIs, we faced a dilemma — do we go a simple but expensive route like S3? Or is there a way to remove significant cost even if it means introducing some slight complexity? We understood that there are perfectly great reasons to go either way — and there are customers at either end of this decision tree.

We realized the expense savings could be significant; we know load balancers well. We use them for our Download capabilities. They are expensive so, to run a sustainable service, we charge for downloads. That B2 download pricing is 1¢/GB while Amazon S3 starts at 9¢/GB is a subject we covered in a prior blog post.

Back to the B2 Upload function. With our existing knowledge of the “expensive” design, the next step was to understand the alternative path. We found a way to create significant savings by only introducing a modest level of complexity. Here’s how: When a “Client” wants to push data to the servers, it does not just start uploading data to a “well known URL” and have the SERVER figure out where to put the data. At the start, the Client contacts a “Dispatching Server” that has the job of knowing where there is optimally available space in a Backblaze data center.

The Dispatching Server (the API server answering the b2_get_upload_url call) tells the Client “there is space over on “Vault-8329.” This next step is our magic. Armed with the knowledge of the open vault, the Client ends its connection with the Dispatching Server and creates a brand new request DIRECTLY to Vault-8329 (calling b2_upload_file or b2_upload_part). No load balancers involved! This is guaranteed to scale infinitely for very little overhead cost. A side note is that the client can continue to directly call b2_upload_file repeatedly from now on (without asking the dispatch server ever again), up until it gets the response indicating that particular vault is full. In other words, this does NOT double the number of network requests.

The “Contract” concept emanates from a simple truth: all APIs are contracts between two entities (machines). Since the Client knows exactly where to go and exactly what authorization to bring with it, it can establish a secure “contract” with the Vault specified by the Dispatching Server.[1] The modest complexity only comes into play if Vault-8392 fills up, gets too busy, or goes offline. In that case, the Client will receive either a 500 or 503 error as notification that the contract has been terminated (in effect, it’s a firm message that says “stop uploading to Vault-8392, it doesn’t have room for more data”). When this happens, the Client is responsible to go BACK to the Dispatching Server, ask for a new vault, and retry the upload to a different vault. In the scenario where the Client has to go back to the Dispatching Server, the “two phase” process becomes more work for the Client versus S3’s singular “well known URL” architecture. Of course, this is all handled at the code level and is well documented. In effect, your code just needs to know that “if you receive a 500 block error, just retry.” It’s free, it’s easy, and it will work.

So while the Backblaze approach introduces some modest complexity, it can quickly and easily be reliably handled with code. Looking at S3’s approach, it is certainly simpler, but it results in three expensive consequences:

1) Expensive fixed costs. Amazon S3 has a single upload URL choke point that requires load balancers and extremely high bandwidth requirements. Backblaze’s architecture does not require moving data around internally; this lets us use commodity 10 GB/s connections that are affordable and will scale infinitely. Further, as discussed above, load balancing hardware is expensive. By removing it from our Upload system, we remove a significant cost center.

2) Expensive single URL availability issues. Amazon S3’s solution requires high availability of the single upload URL for massive amounts of data. The Contract concept from Backblaze works more reliably, but does add slight additional complexity when (rare) extra network round trips are needed.

3) Expensive, time consuming data copy needs (and “eventual consistency”). Amazon S3 requires the copying of massive amounts of data from one part of their network (the upload server) to wherever the data’s ultimate resting place will be. This is at the root of one of the biggest frustrations when dealing with S3: Amazon’s “eventual consistency.” It means that you can’t use your data until it has been pushed to all the places it needs to go. As the article notes, this is usually fast, but can be material amounts of time, anytime. The lack of predictability around access times is something anyone dealing with S3 is all too familiar with.

The B2 architecture offers what one could consider “strong consistency.” There are different definitions of that idea. Ours is that the client connects DIRECTLY with the correct final location for the data to land. Once our system has confirmed a write, the data has been written to enough places that we can guarantee that the data can be seen without delay.

Was our decision a good one? Customers will continue to vote on that, but it appears that the marginal complexity is more than offset by the fact that B2 is sustainable service offered at ¼ of S3’s price.

But Seriously, Why Aren’t You Just “S3 Compatible?”

The use of Object Storage requires some sort of interface. You can build it yourself by learning your vendor’s APIs or you can go through a third party integration. Regardless of what route you choose, somebody is becoming fluent in the vendor’s APIs. And beyond the difference in cost, there’s a reliability component.

This is a good time to clear up a common misconception. The S3 protocol is not a specification: it’s an API doc. Why does this matter? Because API docs leave many outcomes undocumented. For instance, when one uses S3’s list_files function, a developer canNOT know what is going to happen just by reading the API docs. Compounding this issue is the sheer scope of the S3 API; it is huge and expanding. Systems the purport to be “S3 compatible” are unlikely to implement the full API and have to document whatever subset they implement. Once that is done, they will have to work with integration partners and customers to communicate what subset they choose as “important.”

Ultimately, we have chosen to create robust documentation describing, among other things, the engineering specification (this input returns that output, here’s how B2 handles various error cases, etc).

With hundreds of integrations from third parties and hundreds of thousands of customers, it’s clear that our APIs have proven easy to implement. The reality is the first time anyone implements cloud storage into their application it can take weeks. The first move into the cloud can be particularly tough for legacy applications. But the marginal cloud implementation can be reliably completed in days, if not hours, if the documentation is clear and the APIs can be well understood. I’m pleased that we’ve been able to create a complete solution that is proving quite easy to use.

And it’s a big deal that B2 is free of the “load balancer problem.” It solves for a huge scaling issue. When we roll out new vaults in new data centers in new countries, the clients are contacting those vaults DIRECTLY (over whatever network path is shortest) and so there are fewer choke points in our architecture.

It all means that, over an infinite time horizon, our customers can rely on B2 as the most affordable, easiest to use cloud storage on the planet. And, at the end of the day, if we’re doing that, we’ve done the right thing.


[1] The Contract Architecture also explains how we got to a secure two step Authorization process. When you call the Dispatching Server, we run the authentication process and then give you a Vault for uploads and an Auth token. When you are establishing the contract with the Vault, the Auth token is required before any other operation can begin.

The post Design Thinking: B2 APIs (& The Hidden Costs of S3 Compatibility) appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

What’s New In B2: Application Keys + Java SDK

Post Syndicated from Yev original https://www.backblaze.com/blog/b2-application-keys/

B2 Application Keys

It’s been a few months since our last “What’s New In B2” blog post, so we wanted to highlight some goings on and also introduce a new B2 feature!

Reintroducing: Java SDK + Compute Partnerships

We wanted to highlight the official Backblaze B2 Java SDK which can be found in our GitHub repo. The official Java SDK came out almost a year ago, but we’ve been steadily updating it since then with help from the community.

We’ve also announced some Compute Partnerships which give folks all the benefits of Backblaze B2’s low-cost cloud storage with the computing capabilities of Packet and ServerCentral. Backblaze B2 Cloud storage has directly connected to the compute providers, which offers customers low latency and free data transfers with B2 Cloud Storage.

Application Keys

Application keys give developers more control over who can do what and for how long with their B2 data. We’ve had the B2 application key documentation out for a while, and we’re ready to take off the “coming soon” tag.

row of keys

What are Application Keys?

In B2, the main application key has root access to everything and essentially controls every single operation that can be done inside B2. With the introduction of additional application keys, developers now have more flexibility.

Application keys are scoped by three things: 1) what operations the key can do, 2) what path inside of B2 that key can take, and 3) for how long it has the ability to do so. For example you might use a “read-only” key that only has access to one B2 bucket. You’d use that read-only key in situations where you don’t actually need to write things to the bucket, only read or “display” them. Or, you might use a “write-only” key which can only write to a specific folder inside of a bucket. All of this leads to cleaner code with segmented operations, essentially acting as firewalls should something go awry.

Application keys dialog screenshot

Use Cases for Application Keys

One example of how you’d use an application key is for a standard backing up operation. If you’re backing up an SQL database, you do not need to use your root level key to do so. Simply creating a key that can only upload to a specified folder is good enough.

Another example is that of a developer building apps inside of a client. That developer would want to restrict access and limit privileges of each client to specific buckets and folders — usually based on the client that is doing the operation. Using more locked-down application keys limits the possibility that one rogue client can affect the entire system.

A final case could be a Managed Service Provider (MSP) who creates and uses different application key for each client. That way, neither the client nor the MSP can accidentally access the files of another client. In addition, an MSP could have multiple application keys for a given client that define different levels of data access for given groups or individuals within the client’s organization.

We Hope You Like It

Are you one of the people that’s been waiting for application key support? We’d love to hear your use cases so sound off in the comments below with what you’re working on!

The post What’s New In B2: Application Keys + Java SDK appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.