All posts by Roderick Bauer

How Reliable are SSDs?

Post Syndicated from Roderick Bauer original https://www.backblaze.com/blog/how-reliable-are-ssds/

an exploded view of a Samsung Solid State Drive

What’s not to love about solid state drives (SSDs)? They are faster than conventional hard disk drives (HDDs), more compact, have no moving parts, are immune to magnetic fields, and can withstand more shocks and vibration than conventional magnetic platter disks. And, they are becoming available in larger and larger capacities while their cost comes down.

If you’ve upgraded an older computer with an SSD, you no doubt instantly saw the benefits. Your computer booted in less time, your applications loaded faster, and even when you ran out of memory, and apps and data had to be swapped to disk, it felt like everything was much snappier.

We’re now seeing SSDs with capacities that used to be reserved for HDDs and at prices that no longer make our eyes water. 500 GB SSDs are now affordable (under $100), and 1 TB drives are reasonably priced ($100 to $150). Even 2 TB SSDs fall into a budget range for putting together a good performance desktop system ($300 to $400).

We’ve written a number of times on this blog about SSDs, and considered the best uses for SSDs compared to HDDs. We’ve also written about the future of SSDs and how we use them in our data centers and whether we plan on using more in the future.

Reliability

In this post we’re going to consider the issue of SSD reliability. For all their merits, can SSDs be trusted with your data and will they last as long or longer than if you were using an HDD instead? You might have read that SSDs are limited to a finite number of reads and writes before they fail. What’s that all about?

The bottom line question is: do SSD drives fail? Of course they do, as do all drives eventually. The important questions we really need to be asking are 1) do they fail faster than HDDs, and 2) how long can we reasonably expect them to last?

Backing Up Is Great To Do

Of course, as a data storage and backup company, you know what we’re going to say right off. We always recommend that no matter which storage medium you use, you should always have a backup copy of your data. Even if the disk is reliable and in good condition, it won’t do you any good if your computer is stolen, consumed by a flood, or lost in a fire or other act of nature. You might have heard that water damage is the most common computer accident, and few computer components can survive a thorough soaking, especially when powered.

SSD Reliability Factors to Consider

Generally, SSDs are more durable than HDDs in extreme and harsh environments because they don’t have moving parts such as actuator arms. SSDs can withstand accidental drops and other shocks, vibration, extreme temperatures, and magnetic fields better than HDDs. Add to that their small size and lower power consumption, and you can understand why they’re a great fit for laptop computers and mobile applications.

First, let’s cover the basics. Almost all types of today’s SSDs use NAND flash memory. NAND isn’t an acronym like a lot of computer terms. Instead, it’s a name that’s derived from its logic gate called “NOT AND.”

SSD part diagram including Cache, Controller, and NAND Flash Memory

The term following NAND, flash, refers to a non-volatile solid state memory that retains data even when the power source is removed. NAND storage has specific properties that affect how long it will last. When data is written to a NAND cell (also known as programming), the data must be erased before new data can be written to that same cell. NAND is programed and erased by applying a voltage to send electrons through an insulator. The location of those electrons (and their quantity) determine when current will flow between a source and a sink (called a voltage threshold), determining the data stored in that cell (the 1s and 0s). When writing and erasing NAND, it sends the electrons through the insulator and back, and the insulator starts to wear — the exact number of these cycles in each individual cell varies by NAND design. Eventually, the insulator wears to the point where it may have difficulty keeping the electrons in their correct (programmed) location, which makes it increasingly more difficult to determine if the electrons are where they should be, or if they have migrated on their own.

This means that flash type memory cells can only be programmed and erased a limited number of times. This is measured in P/E cycles, which stands for programmed and erased.

P/E cycles are an important measurement of SSD reliability, but there are other factors that are important to consider, as well. These are P/E cycles, TBW (terabytes written), and MTBF (mean time between failures).

The SSD manufacturer will have these specifications available for their products and they can help you understand how long your drive can be expected to last and whether a particular drive is suited to your application.

P/E cycles — A solid-state-storage program-erase cycle is a sequence of events in which data is written to solid-state NAND flash memory cell, then erased, and then rewritten. How many P/E cycles a SSD can endure varies with the technology used, somewhere between 500 to 100,000 P/E cycles.

TBW — Terabytes written is the total amount of data that can be written into an SSD before it is likely to fail. For example, here are the TBW warranties for the popular Samsung 860 EVO SSD: 150 TBW for 250 GB model, 300 TBW for 500 GB model, 600 TBW for 1 TB model, 1,200 TBW for 2 TB model and 2,400 TBW for 4 TB model. Note: these models are warrantied for 5 years or TBW, whichever comes first.

MTBF — MTBF (mean time between failures) is a measure of how reliable a hardware product or component is over its expected lifetime. For most components, the measure is typically in thousands or even tens of thousands of hours between failures. For example, a hard disk drive may have a mean time between failures of 300,000 hours, while an SSD might have 1.5 million hours.

This doesn’t mean that your SSD will last that many hours, what it means is, given a sample set of that model of SSD, errors will occur at a certain rate. A 1.2 million hour MTBF means that if the drive is used at an average of 8 hours a day, a sample size of 1,000 SSDs would be expected to have one failure every 150 days, or about twice a year.

SSD Types

There are a number of different types of SSD, and advancements to the technology continue at a brisk pace. Generally, SSDs are based on four different NAND cell technologies:

  • SLC (Single Level Cell) — one bit per cell
  • When one bit is stored (SLC), it’s not necessary to keep close tabs on electron locations, so a few electrons migrating isn’t much of a concern. Because only a 1 or a 0 is being stored, it’s necessary only to accurately determine if voltage flows or not.

  • MLC (Multi-Level Cell) — two bits per cell
  • MLC stores two bits per cell, so more precision is needed (determining voltage threshold is more complex). It’s necessary to distinguish among 00, 01, 10 or 11. Migrating electrons have more of an impact, so the insulator cannot be worn as much as with SLC.

  • TLC (Triple Level Cell) — three bits per cell
  • This trend continues with TLC where three bits are stored: 001, 010, 100, …110 and 111. Migrating electrons have more effect than in MLC, which further reduces tolerable insulator wear.

  • QLC (Quad Level Cell) — four bits per cell
  • QLC stores four bits (16 possible combinations of 1s and 0s). With QLC, migrating electrons have the most significant effect. Tolerable insulator wear is further reduced.

    QLC is a good fit for read-centric workloads because NAND cells are worn negligibly when reading data versus worn more when writing data (programming and erasing). When writing and rewriting a lot of data, the insulator wears more quickly. If a NAND cell can tolerate that wear, it is well suited to read/write mixed accesses. The less wear-tolerable NAND cells are, the better they are suited for read-centric workloads and applications.

Each subsequent technology for NAND allows it to store an extra bit. The fewer bits per NAND cell, the faster, more reliable, and more energy efficient the technology is — and also, more expensive. A SLC SSD would technically be the most reliable SSD as it can endure more writes, while a QLC is the least reliable. If you’re selecting an SSD for an application where it will be written more than read, than the selection of NAND cell technology could be a significant factor in your decision. If your application is general computer use, it likely will matter less to you.

How Reliability Factors Affect Your Choice of SSD

How important these factors are to you depends on how the SSD is used. The right question to ask is how a drive will perform in your application. There are different performance and reliability criteria depending on whether the SSD will be used in a home desktop computer, a data center, or an exploration vehicle on Mars.

Manufacturers sometimes specify the type of application workload for which an SSD is designed, such as write-intensive, read-intensive or mixed-use. Some vendors allow the customer to select the optimal level of endurance and capacity for a particular SSD. For instance, an enterprise user with a high-transaction database might opt for a higher number of drive writes at the expense of capacity. Or a user operating a database that does infrequent writes might choose a lower drive writes number and a higher capacity.

Signs of SSD Failure

SSDs will eventually fail, but there usually are advance warnings of when that’s going to happen. You’ve likely encountered the dreaded clicking sound that emanates from a dying HDD. As an SSD has no moving parts, so we won’t get an audible warning that an SSD is about to fail us. You should be paying attention for a number of indicators that your SSD is nearing its end of life, and take action by replacing that drive with a new one.

1) Errors Involving Bad Blocks

Much like bad sectors on HDDs, there are bad blocks on SSDs. This is typically a scenario where the computer attempts to read or save a file, but it takes an unusually long time and ends in failure, so the system eventually gives up with an error message.

2) Files Cannot Be Read or Written

There are two ways in which a bad block can affect your files, 1) the system detects the bad block while writing data to the drive, and thus refuses to write data, and 2), the system detects the bad block after the data has been written, and thus refuses to read that data.

3) The File System Needs Repair
Getting an error message on your screen can happen simply because the computer was not shut down properly, but it also could be a sign of an SSD developing bad blocks or other problems.

4) Crashing During Boot
A crash during the computer boot is a sign that your drive could be developing a problem. You should make sure you have a current backup of all your data before it gets worse and the drive fails completely.

5) The Drive Becomes Read-Only
Your drive might refuse to write any more data to disk and can only read data. Fortunately, you can still get your data off the disk.

SSDs Generally Will Last As Long As You Need Them To

Let’s go back to the two questions we asked above.

Q: Do SSDs fail faster than HDDs?

A: That depends on the technology of the drives and how they’re used. HDDs are better suited for some applications and SSDs for others. SSDs can be expected to last as long or longer than HDDs in most general applications.

and

Q: How long can we reasonably expect an SSD to last?

A: An SSD should last as long as its manufacturer expects it to last (e.g. five years), provided that the use of the drive is not excessive for the technology it employs (e.g. using a QLC in an application with a high number of writes). Consult the manufacturer’s recommendations to ensure that how you’re using the SSD matches its best use.

SSDs are a different breed of animal than a HDD and they have their strengths and weaknesses relative to other storage media. The good news is that their strengths — speed, durability, size, power consumption, etc. — are backed by pretty good overall reliability.

SSD users are far more likely to replace their storage drive because they’re ready to upgrade to a newer technology, higher capacity, or faster drive, than having to replace the drive due to a short lifespan. Under normal use we can expect an SSD to last years. If you replace your computer every three years, as most users do, then you probably needn’t worry about whether your SSD will outlast your computer. What’s important is whether the SSD will be sufficiently reliable that you won’t lose your data.

As we saw above, if you’re paying attention to your system, you will be given ample warning of an impending drive failure, and you can replace the drive before the data is not readable.

It’s good to understand how the different SSD technologies affect their reliability, and whether it’s worth it to spend extra money for SLC over MLC or QLC. However, unless you’re using an SSD in a specialized application with more writes than reads as we described above, just selecting a good quality SSD from a reputable manufacturer should be enough to make you feel confident that your SSD will have a useful life span.

Keep an eye out for any signs of failure or bad sectors, and, of course, be sure to have a solid backup plan no matter what type of drive you’re using.

The post How Reliable are SSDs? appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

B2 on Your Desktop — Cloud Storage Made Easy

Post Syndicated from Roderick Bauer original https://www.backblaze.com/blog/cloud-storage-made-easy/

B2 on your Desktop

People have lots of different ways that they work with files in B2 Cloud Storage, and there’s a wide range of integrations for different platforms and different uses.

Sometimes, though, being able to use B2 as if it were just another drive on your desktop is the easiest way to go. The applications we’ll be covering in this post make working with B2 as easy as dragging and dropping files from a file manager on your computer directly to B2, or from B2 to your computer. In other cases, you can drag files from a file manager to the application, or between panes inside the application. There’s something for every platform, too, whether you’re on Windows, Macintosh, or Linux. Some of these tools are even free.

Let’s take a look at the applications that make working with B2 a piece of cake! (Or, as easy as pie.)

Use B2 As a Drive on the Desktop

Our first group of applications let you use B2 as if it were a local drive on your computer. The files on B2 are available for you from (depending on platform) File Explorer on Windows, the Finder on Mac, or the File Manager on Linux (as well as the command-line). Some of the applications are free and some require purchase (marked with $).

Most of these apps are simple for anyone to set up. If you are a more advanced user, and comfortable working with the command-line in your OS’s terminal, there are a number of free command-line tools for mounting B2 as a drive, including restic, Rclone, and HashBackup. See their docs for how to mount restic, Rclone, or HashBackup as a drive. We previously wrote about using restic with B2 in our Knowledge Base.

When would dragging and dropping files on the desktop be useful? If you just need to move one or a few files, this could be the fastest way to do that. You can load the application when you need to transfer files, or have it start with your computer so your B2 files and buckets are always just a click away. If you keep archived documents or media in B2 and often need to browse to find a file, this makes that much faster. You can even use shortcuts, search, and other tools you have available for your desktop to find and manage files on B2.

We’ve grouped the applications by platform that let you use B2 as a drive.

Apps for Mounting B2 as a Drive

Macintosh

Windows

Linux

Some Screenshots Showing Applications That Let You Use B2 as a Drive

screenshot of Mountain Duck interface for saving to B2 Cloud Storage

Mountain Duck

screenshot of B2 mounted on the desktop with Mountain Duck

B2 mounted on the desktop with Mountain Duck

screenshot of ExpanDrive saving to B2 cloud storage

ExpanDrive

Cloudmounter

Cloudmounter

screenshot of Cloudmounter with B2 open in Mac Finder

Cloudmounter with B2 open in Mac Finder

Use B2 From a Desktop Application

These applications allow you to use B2 from within the application, and also often work with the local OS’s file manager for drag and drop. They support not just B2, but other cloud and sync services, plus FTP, SFTP, Webdav, SSH, SMB, and other protocols for networking and transferring files.

All of the applications below require purchase, but they have demo periods when you can try them out before you decide you’re ready to purchase.

Apps for Using B2 from the Desktop

Macintosh

Windows

Linux

Screenshots of Using B2 From Desktop Applications

Filezilla Pro

Filezilla Pro browsing photos on B2

screenshot of Transmit with B2 files

Transmit with B2 files

screenshot of Cyberduck transmitting files to B2

Cyberduck

screenshot of odrive cloud storage integration

odrive

SmartFTP

SmartFTP

The Cloud on Your Desktop

We hope these applications make you think of B2 as easy and always available on your desktop whenever you need to move files to or from cloud storage. Easy Peasy Lemon Squeezy, right?

If you’ve used any of these applications, or others we didn’t mention in this post, please tell us in the comments how they worked for you.

The post B2 on Your Desktop — Cloud Storage Made Easy appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Save Data Directly to B2 With Backblaze Cloud Backup 6.0

Post Syndicated from Roderick Bauer original https://www.backblaze.com/blog/save-data-directly-to-cloud-storage/

Save Restores to B2 screenshot

Customers have often told us that they’d love a way to save data directly from their Backblaze Computer Backup account to B2 Cloud Storage. Some want to freeze a set of records in time, others want to preserve the state of a directory or system as it existed at a specific moment. Still others simply want to remove data from their local drive but have the assurance that it is safely stored in the cloud.

We listened to these requests and are happy to say that we’ve added this capability in our just released 6.0 update of Backblaze Computer Backup. Users can now select B2 Cloud Storage as a destination to save Snapshots from their backup account during the restore process.

This capability lets customers do a number of new things, like keep a copy of their old computer’s data even when migrating to a new one, save a collection of files (e.g. last year’s emails, a completed work project, your novel draft, tax returns) in the cloud as an archive, or free up space on a hard drive by moving data to a Snapshot in B2 and then deleting the original copy. Just like files in Computer Backup, the B2 Snapshot can be downloaded over the internet or delivered anywhere on a USB flash or hard drive.

No More Connecting Your External Drives Every 30 Days

This new feature can particularly benefit users who have been using Computer Backup to back up data from multiple external drives. Often, these external drives are not always connected to their computers, and to maintain the backups they have been required to connect these drives at least once every 30 days so that they’re active and therefore maintained in their backup — a task they tell us they’d rather avoid.

Now, with the ability to save a restore to B2, these customers can take a Snapshot of the data already backed up from these drives and save it to a B2 account. They can save as many Snapshots as they wish, thereby saving the state of the drive as it existed in one moment for as long as they wish to retain it.

Snapshots are stored at economical B2 rates: $0.005 gigabyte/month and $0.01 gigabyte for downloads. Customers get an instant cost estimate when a Snapshot is prepared from Backblaze Backup to B2.

What is B2 Cloud Storage?

B2 is Backblaze’s low cost and high performance cloud storage. It can be used to store data for as short or as long a period as you require. The data in B2 is retrievable without delay from anywhere at any time.

B2 is different from Backblaze Computer Backup in that B2 can be used to store whatever data you want and you have complete control of how long it is retained. Our Computer Backup service offers unlimited backup of the data on your Mac or Windows computer using the Backblaze client software. B2, in contrast, can be accessed through the account dashboard or used with any of a number of applications chosen by the user, or accessed through various programming interfaces or from a computer’s command line. For more on pricing, see our pricing page and calculator for B2.

How Does Saving a Restore to B2 Work?

Files in your Computer Backup can be zipped and archived to a Snapshot that is stored in B2 Cloud Storage. These selected files will be safe in B2 until the Snapshot is removed by the user, even if the files have been deleted from the computer and the backup.

screenshot of the View/Restore Files options

Creating a Restore Snapshot in Backup account

The user gets an instant estimate of the cost to store the Snapshot in B2.

Name this Snapshot screenshot

Preparing Snapshot from Computer Backup account

The user receives a notice when the Snapshot is created and stored.

Your B2 Snapshot is Ready!

Notice that Snapshot has been created

An unlimited number of restores can be saved and retained as B2 Snapshots for any length of time desired.The user’s account dashboard shows all the Snapshots that have been created, and gives options to download or remove the Snapshot. A Snapshot can be downloaded directly from B2 to a user’s computer or shipped to customers on a USB flash or hard drive. And, when returned within 30 days, the cost of the flash or hard drive is completely refundable, just like with regular restores.

screenshot of user B2 Snapshots

User account page showing status of Snapshots in B2

Let Us Know How You’re Using Snapshots

We hope you’ll try out this new capability and let us know how you’re using it.

For more tips on saving data to B2 Snapshots, read our help article, Saving Files to B2 from Computer Backup, or sign up for our free webinar on Backblaze Backup v6.0 on January 30, 2019, at 11am PST.

The post Save Data Directly to B2 With Backblaze Cloud Backup 6.0 appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Best of the Blog 2018

Post Syndicated from Roderick Bauer original https://www.backblaze.com/blog/best-of-the-blog-2018/

Best of the Blog 2018
2018 was a great year for Backblaze. We hope it was a great year for you, too. We’d like to start off the year with a look back at what we believe are the top blog posts of 2018.

We had outstanding posts in our Entrepreneurship series by our CEO Gleb Budman and others. We discussed the costs and use of LTO vs the cloud in a number of posts. We wrote again about ransomware, which unfortunately is still with us and will be covered along with other data security topics in the coming year. We increased the number of posts aimed at developers. Our What’s the Diff series of posts introduced technical terms and topics that might be new to some readers. We also launched a Power Tips series, which we hope will provide useful information to both our Cloud Backup and B2 users. We also had guest posts from Archiware, Lensrentals, and others, and a series about Veeam. Finally, hybrid cloud and multi-cloud are showing up in more and more posts and we expect that trend to continue. And, Backblaze wouldn’t be Backblaze without a little bit of our signature humor.

Please let us know if we’ve left out something you think should be included. We’d also like your comments on which topics you enjoyed, and what type of blog posts you’d like to see more of (or less of) in 2019.

We are looking forward to a busy and productive 2019!

The Best from the Backblaze Blog in 2018

January 3

12 B2 Power Tips for New Users

February 1

Backblaze Hard Drive Stats for 2017

February 22

Ode to ‘Locate My Computer’

March 9

Backblaze Cuts B2 Download Price In Half

April 3

Backblaze Announces B2 Compute Partnerships

April 10

Cloud Empire: Meet the Rebel Alliance

April 19

Confused About the Hybrid Cloud? You’re Not Alone

May 3

The Helium Factor and Hard Drive Failure Rates

May 29

Getting Rid of Your Mac? Here’s How to Securely Erase a Hard Drive or SSD

July 17

Backblaze Durability is 99.999999999% — And Why It Doesn’t Matter

July 26

Five Tips For Creating a Predictable Cloud Storage Budget

August 2

What’s the Diff: Backup vs Archive

September 26

Backblaze and Cloudflare Partner to Provide Free Data Transfer

October 2

Backing Up for Small Business

October 9

iconik and Backblaze — The Cloud Production Solution You’ve Always Wanted

November 13

Making Lemonade: The Importance of Social Media and Community

December 4

LTO Versus Cloud Storage Costs — the Math Revealed

December 13

Bootstrapping to $30 Million ARR

December 18

2018 in the Rear View Mirror

Thanks again for reading our blog in 2018!

The post Best of the Blog 2018 appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

What’s the Diff: NAS vs SAN

Post Syndicated from Roderick Bauer original https://www.backblaze.com/blog/whats-the-diff-nas-vs-san/

What's the Diff? Network Attachd Storage (NAS) vs Storage Area Network (SAN)

Both network-attached storage (NAS) and storage area network (SAN) were developed to solve the problem of making stored data available to a lot of users at once. Each of them provides dedicated storage for a group of users, but they couldn’t be more different in their approach to achieving their mission.

A NAS is a single storage device that serves files over Ethernet and is relatively inexpensive and easy to set up, while a SAN is a tightly coupled network of multiple devices that work with block-based data and is more expensive and complex to set up and manage. From a user perspective, the biggest difference between NAS and SAN is that NAS devices look like volumes on a file server and use protocols like NFS and SMB/CIFS, while SAN-connected disks appear to the user as local drives.

We provide an overview of the differences between NAS and SAN below. We’ll also briefly cover solutions that combine NAS and SAN and offer many of the advanced benefits of SAN without its high cost.

Basic Definitions — What is NAS?

A NAS is a computer connected to a network that provides file-based data storage services to other devices on the network. The primary strength of NAS is how simple it is to set up and deploy. NAS volumes appear to the user as network mounted volume. The files to be served are typically contained on one or more storage drives, often arranged into logical, redundant storage containers or RAID. The device itself is a network node, much like computers and other TCP/IP devices, all of which maintain their own IP address and can effectively communicate with other networked devices. Although a NAS is usually not designed to be a general-purpose server, NAS vendors and third parties are increasingly offering other software to provide server-like functionality on a NAS.

NAS devices offer an easy way for multiple users in diverse locations to access data, which is valuable when uses are collaborating on projects or sharing information. NAS provides good access controls and security to support collaboration, while also enabling someone who is not an IT professional to administer and manage access to the data. It also offers good fundamental data security through the use of redundant data structures — often RAID — and automatic backup services to local devices and to the cloud.

Benefits of NAS

A NAS is frequently the next step up for a home office or small business that is using DAS (direct attached storage). The move up to NAS results from the desire to share files locally and remotely, having files available 24/7, data redundancy, the ability to replace and upgrade hard drives in the system, and and the availability of other services such as automatic backup.

Summary of NAS Benefits

  • Relatively inexpensive
  • 24/7 and remote data availability
  • Good expandability
  • Redundant storage architecture
  • Automatic backups to other devices and cloud
  • Flexibility

Network attached Storage (NAS)

Synology NAS

NAS with eight drive bays for 3.5″ disk drives

Limitations of NAS

The weaknesses of a NAS are related to scale and performance. As more users need access, the server might not be able to keep up and could require the addition of more server horsepower. The other weakness is related to the nature of Ethernet itself. By design, Ethernet transfers data from one place to another via packets, dividing the source into a number of segments and sending them along to their destination. Any of those packets could be delayed, or sent out of order, and might not be available to the user until all of the packets arrive and are put back in order.

Any latency (slow or retried connections) is usually not noticed by users for small files, but can be a major problem in demanding environments such as video production, where files are extremely large and latency of more than a few milliseconds can disrupt production steps such as rendering.

Basic Definitions — What is SAN?

A SAN is a way to provide users shared access to consolidated, block level data storage, even allowing multiple clients to access files at the same time with very high performance. A SAN enhances the accessibility of storage devices such as disk arrays and tape libraries by making them appear to users as if they were external hard drives on their local system. By providing a separate storage-based network for block data access over high-speed Fibre Channel, and avoiding the limitations of TCP/IP protocols and local area network congestion, a SAN provides the highest access speed available for media and mission critical stored data.

Storage area network (SAN)

SAN connecting yellow storage devices with orange servers via purple Fibre Channel switches

SAN connecting yellow storage devices with orange servers via purple Fibre Channel switches

Benefits of SAN

Because it’s considerably more complex and expensive than NAS, SAN is typically used by large corporations and requires administration by an IT staff. For some applications, such as video editing, it’s especially desirable due to its high speed and low latency. Video editing requires fair and prioritized bandwidth usage across the network, which is an advantage of SAN.

A primary strength of a SAN is that all of the file access negotiation happens over Ethernet while the files are served via extremely high speed Fibre Channel, which translates to very snappy performance on the client workstations, even for very large files. For this reason SAN is widely used today in collaborative video editing environments.

Summary of SAN Benefits

  • Extremely fast data access
  • Dedicated network for storage relieves stress on LAN
  • Highly expandable
  • OS level (block level) access to files
  • High quality-of-service for demanding applications such as video editing

Limitations of SAN

The challenge of SAN can be summed up in its cost and administration requirements — having to dedicate and maintain both a separate Ethernet network for metadata file requests and implement a Fibre Channel network can be a considerable investment. That being said, SANs are really the only way to provide very fast data access for a large number of users that also can scale to supporting hundreds of users at the same time.

What’s the Diff: NAS vs SAN

NAS SAN
Typically used in homes and small to medium sized businesses. Typically used in professional and enterprise environments.
Less expensive More expensive
Easier to manage Requires more administration
Data accessed as if it were a network-attached drive (files) Servers access data as if it were a local hard drive (blocks)
Speed dependent on local TCP/IP usually Ethernet network, typically 100 megabits to one gigabit per second. Generally slower throughput and higher latency due to slower file system layer. High speed using Fibre Channel, 2 gigabits to 128 gigabits per second. Some SANs use iSCSI as a less expensive but slower alternative to Fibre Channel.
I/O protocols: NFS, SMB/CIFS, HTTP SCSI, iSCSI, FCoE
Lower-end not highly scalable; high-end NAS scale to petabytes using clusters or scale-out nodes Network architecture enables admins to scale both performance and capacity as needed
Does not work with virtualization Works with virtualization
Requires no architectural changes Requires architectural changes
Entry level systems often have a single point of failure, e.g. power supply Fault tolerant network with redundant functionality
Susceptible to network bottlenecks Not affected by network traffic bottlenecks. Simultaneous access to cache, benefiting applications such as video editing.
File backups and snapshots economical and schedulable. Block backups and mirrors require more storage.

NAS/SAN Convergence

The benefits of SAN are motivating some vendors to offer SAN-like products at lower cost chiefly by avoiding the high expense of Fibre Channel networking. This has resulted in a partial convergence of NAS and SAN approaches to network storage at a lower cost than purely SAN.

One example is Fibre Channel over Ethernet (FCoE), which supports block level transfers over standard LAN at speeds of 10GB/sec+. For smaller deployments, iSCSI is even less expensive, allowing SCSI commands to be sent inside of IP packets on a LAN. Both of these approaches avoid expensive Fibre Channel completely, resulting in slower, but less expensive ways to get the block level access and other benefits of a SAN.

Are You Using NAS, SAN, or Both?

If you are using NAS or SAN, we’d love to hear from you about what you’re using and how you’re using them. Also, please feel free to suggest other topics for this series.

The post What’s the Diff: NAS vs SAN appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Ever Wish You Had a Backup Brain? The Mars Rover Has One

Post Syndicated from Roderick Bauer original https://www.backblaze.com/blog/mars-rover-backup-brain/

Mars Curiosity Rover at JPL in Pasadena

Have you ever had one of those days when even a second cup of coffee can’t jump-start your thinking and you just wished you had another brain you could switch to? If you’re the Mars Curiosity Rover, you do.

A recent glitch in its main computer required the Curiosity Rover team at NASA’s Jet Propulsion Laboratory (JPL) to switch to another computer in the rover while they worked to resolve problems with its main computer. The problem started around September 15 with the rover “failing to store science and some key engineering data,” according to NASA. The rover continued to send limited engineering data stored in short-term memory when it connected to a relay orbiter — it was otherwise healthy and receiving commands. But whatever was preventing Curiosity from storing science data in long-term memory was also preventing the storage of the rover’s event records, a journal of all its actions that engineers need in order to make a diagnosis. The computer swap allowed data and event records to be stored on the Curiosity‘s other computer, improving the rover’s operations and helping the engineers diagnose the problem.

Tweet from Mars Curiosity Rover @MarsCuriosity on October 3, 2018

Two Brains Are Better Than One

Like most spacecraft, NASA outfits its spacecraft with twin computers for redundancy in case any problems arise with its main computer. Curiosity‘s paired computers are called Side-A and Side-B. The rover began its stay on Mars in August of 2012 using Side-A but switched to Side-B in February of 2013 when a problem developed in the computer’s flash memory that caused the computer to continuously reboot in a loop. Engineers working from 33.9 million miles away on earth were eventually able to get the Side-A computer back in working order. That’s the computer Curiosity switched back to this past October while engineers continued to investigate the memory errors in the Side-B machine.

Curiosity continues to operate using its Side-A computer. According to Steven Lee, Curiosity‘s deputy project manager at JPL, “At this point, we’re confident we’ll be getting back to full operations, but it’s too early to say how soon. It’s certainly possible to run the mission on the Side-A computer if we really need to, but our plan is to switch back to Side-B as soon as we can fix the problem to utilize its larger memory size.”

Tweet from @MarsCuriosity on October 17, 2018

The computer problems haven’t prevented Curiosity from continuing to pursue its mission objectives, which include an investigation of the Martian climate and geology; assessment of whether the selected field site inside Gale Crater has ever offered environmental conditions favorable for microbial life, including investigation of the role of water; and planetary habitability studies in preparation for human exploration.

Inside the Curiosity’s Brains

Even though Curiosity‘s computers are specialized for space use, the circuit board and operating system will be familiar to many. The CPU is a RAD750, a version of the IBM PowerPC 750, which was used in many computers from Apple, including the original iMac. The datasheet for the RAD750 states that the processor, “is the best space microprocessor available today by any selection criterion — performance, cost, availability, or flight heritage.”

RAD750 radiation-hardened PowerPC space microprocessor

RAD750 radiation-hardened PowerPC space microprocessor

On-board memory includes 256MB of DRAM and 2 GB of Flash Memory (~8 times as much as Rovers Spirit or Opportunity), both with error detection and correction and 256kB of EEPROM. The microprocessor operates at up to 200 megahertz speed, 10 times the speed of earlier microprocessors in rovers Spirit and Opportunity.

Two British Aerospace RAD750 single board computers as used aboard the Curiosity rover

Two British Aerospace RAD750 single board computers as used aboard the Curiosity rover

For Curiosity‘s software, NASA stuck to proven solutions, selecting the VxWorks operating system. VxWorks, developed by Wind River Systems, is a real-time operating system used in a huge number of embedded systems. The previous Mars rovers (Sojourner, Spirit, Opportunity), Mars Reconnaissance Orbiter, and the SpaceX Dragon spacecraft all use VxWorks. VxWorks also powers many earth-bound device and vehicles, including BMW’s iDrive, the Apache Longbow helicopter, and the Apple Airport Extreme and Linksys WRT54G routers.

Shortly after landing on Mars, on August 8, 2012, NASA Mission Control began upgrading the rover’s dual computers by deleting the entry-descent-landing software, then uploading and installing the surface operation software. The switchover to the new software was completed by August 15.

Note: some of the software developed for the rovers is available from NASA on GitHub.

The Right Stuff for Space Exploration

It might sound like these units resemble what we use everyday at home or in offices, but they are designed to withstand the harsh environments that will be encountered by satellites and space exploration vehicles. The RAD750 can withstand temperatures of between -55 and 70C and radiation levels up to 1000 gray (a gray is defined as the absorption of one joule of radiation energy per kilogram of matter). Safely protected within Curiosity, the temperature and radiation should remain well below these levels.

The units are priced differently than their cousins on earth, too — in 2002, the RAD750 microprocessor was listed at $200,000, which is quite a bit more than the PowerPC used at the time in iMacs, which sold in quantity for about $520 each. The high price of the RAD750 is mainly due to radiation hardening revisions to the PowerPC 750 architecture, manufacturing costs, stringent quality control requirements, and extended testing of each processor chip produced.

Each of the pair of rover computers is inside a module called The Rover Compute Element (RCE). The RCEs are protected from exposure in the middle of the rover body.

Curiosity Rover Compute Elements (highlighted)

Curiosity Rover Compute Elements (highlighted)

Sojourner, Spirit, Opportunity, Curiosity, and Beyond

The Mars Rover family, clockwise from bottom left: Sojourner (1997), Spirit/Opportunity (2004), Curiosity (2012)

The Mars Rover family, clockwise from bottom left: Sojourner (1997), Spirit/Opportunity (2004), Curiosity (2012)

Curiosity has had a long sojourn on Mars since landing on Aeolis Palus in Gale Crater on August 6, 2012, and follows the success of earlier Mars explorers Sojourner, Spirit, and Opportunity. Despite starting out with only a two-year mission, the durability of Curiosity prompted NASA in December 2012 to extend Curiosity‘s mission indefinitely.

Curiosity‘s design will serve as the basis for the planned  Mars 2020 rover, which is scheduled to launch in July/August of 2020. The new rover will have a few upgrades, however, including more sophisticated hardware and new instruments to conduct geological assessments of the rover’s landing site, which will determine the potential habitability of the environment and directly search for signs of ancient Martian life.

We don’t have to wait that long for another exciting Mars landing like we had with Curiosity, however. NASA InSight is scheduled to land on Mars in less than two weeks, on November 26, 2018. Following that, ExoMars and NASA Mars 2020 will head to Mars in 2020 to continue a search for evidence of existing and past life.

2018 NASA InSight Mission: InSight is a robotic explorer designed to study Mars’ crust, mantle, and core. InSight will land on Mars on November 26, 2018. NASA InSight
2020 ESA ExoMars Rover Mission: ExoMars, a joint mission of the European Space Agency and the Russian space agency Roscosmos, will search for evidence of life on Mars. NASA is providing critical elements for the astrobiology instrument on the rover. ESA ExoMars Rover
NASA 2020 Rover Mission: Mars 2020 seeks to answer key questions about the potential for life on Mars. It will cache samples for possible future return to Earth. Mars 2020 Rover

Tweet from @NASAJPL on Nov 12 re InSight Mars landing on November 26, 2018

 

A Backup is a Good Idea on Both Earth and Mars

It turns out that having a backup doesn’t apply just to data or computing. Sometimes, a second brain can come in handy, too, especially when you’re on Mars.

Do you follow Curiosity‘s advice to always have redundant systems? Have you ever switched to using your Side-A brain? Would you like to go to Mars? (I would.) Let’s hear your thoughts in the comments.

Don’t forget to catch landing on Mars of InSight on Monday, November 26. We’ll be watching!

The post Ever Wish You Had a Backup Brain? The Mars Rover Has One appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Backblaze’s Custom Data Center PDU

Post Syndicated from Roderick Bauer original https://www.backblaze.com/blog/backblazes-custom-data-center-pdu/

Backblaze PDU
When Jon needed to open a Backblaze Storage Pod for maintenance in our Phoenix data center, it wasn’t as straightforward as one might think. With a steel case, 60 hard drives, backplanes, a pair of power supplies and other components, each pod can weigh up to 150 pounds.

However, there was even a bigger challenge than the pod’s weight. A Storage Pod is divided into two main sections, the drive section and the processing section, each with separate access panels. To replace a drive, you need to open the access panel at the front, which requires sliding the Storage Pod out of the front of the cabinet. To replace a power supply or perhaps reseat a SATA card or cable, you’d prefer to slide the pod out the back of the cabinet because that gives you better access to the panel at the rear of the pod.

Backblaze's 6.0 Storage Pod with 60 drives

Backblaze’s 6.0 Storage Pod with 60 drives (front)

The problem was that doing that was difficult, if not impossible, with all the power cables that connected the pods to the power distribution unit (PDU) at the rear of the cabinet. That left Jon with only one choice: slide the pod out of the front of the cabinet even when he wanted to access the rear access panel, which took more time and often required two people.

Identifying the Problem — the PDU

As Backblaze’s Phoenix data center site manager, Jon realized that the job would be much easier if he could change one component, the PDU. The Phoenix data center used vertically-mounted power distribution units (PDUs) at the back of the cabinets that ran all the way from the top to the bottom of the cabinet. All the cables from the ten pods to the PDU blocked access to the back of the pods in the cabinet.

Vertically-mounted PDU blocking rear access to Storage Pods

Vertically-mounted PDU blocking rear access to Storage Pods

What’s a PDU?

A power distribution unit (PDU) is a device fitted with multiple outputs designed to distribute electric power to racks of computers and networking equipment located within a data center. Some PDUs have additional capabilities, including power filtering, intelligent load balancing, and remote monitoring and control by LAN or SNMP.

Data center IT managers remotely monitor PDU performance to ensure continual service, improve efficiency, and plan for growth.

Jon knew that the vertical PDU forced his team to spend more time than needed getting into the pods for service. If he could find a better option, everyone on the team would have more time to focus on other data center matters, like setting up more cabinets to fill with Storage Pods and customers’ data.

The Backblaze Storage Pods and Cabinets

Backblaze’s Storage Pod racks are standard full size data center cabinets that are 42 rack units (U or RU) high — a rack unit is 44.50 millimeters (1.75 inches). Equipment that fits into these racks is typically 1U, 2U, 3U, or 4U high. Backblaze’s Storage Pods are 4U high, so ten of them can fit into a single rack. With a small switch at the top occupying one of those rack units, that leaves just 1U of space.

If Jon could use that 1U of space in the cabinet for a horizontally-mounted PDU, he could get rid of the vertically-mounted PDU that was causing the access problem. The PDU had more power outlets than needed, anyway, as well as extra monitoring circuitry that wasn’t required for Zabbix, the software monitoring suite we use to track the health of all the components in our data centers.

The vertically-mounted PDU made it more complex and expensive than was necessary for the task — two factors that go against Backblaze’s philosophy of keeping things as simple and inexpensive as possible to keep costs low for our customers. (For a bit of history on this, see this post on how Backblaze got started.)

A Better PDU

Jon made a list of the requirements he wanted in a PDU that would fit Backblaze’s needs. It didn’t seem to him that it would be that hard to find one ready to drop into the cabinet.

Jon’s PDU Requirements

  • 1 rack unit high
  • 3-phase power
  • Horizontally mounted
  • Metering to remotely monitor circuit loads
  • 12 C13 power outlets
    • 10 outlets for Storage Pods
    • 1 outlet for small switch
    • 1 outlet for crash cart to service the pods

Finding a PDU that fit the list turned out to be harder than he expected. Jon searched to see if anyone made a 3-phase 1U horizontal mount PDU, and the only one he could find didn’t have the right type of power outlets (C13) or monitoring circuitry.

The only remaining option was to design a custom PDU. Jon remembered that he and Larry, Backblaze’s data center manager, had run into a PDU manufacturer, Geist, at an IT trade show in San Jose. Jon contacted our vendor, Mirapath, whom Jon had successfully worked with on other projects for Backblaze. Mirapath got the project rolling with Geist, worked out all the kinks, and were instrumental in bringing the project to completion.

The Custom PDU

The result is a custom PDU that fits Jon’s requirements. The PDU fits horizontally in the center-back of the cabinets and doesn’t block access from the back of the cabinet. It takes up only 1U of cabinet space, which allows Jon to put ten Storage Pods in each cabinet — five above the PDU in the center of the cabinet and five below. It has the correct type (C13) and number (12) of power outlets, which support the ten pods, one switch, and the crash cart. It also contains the power monitoring circuitry needed to collect data for Zabbix.

Custom PDU Custom PDU (back) Custom PDU display

Custom PDU

Custom PDU (back)

Custom PDU display

Guido, A Valued Member of Backblaze’s Operations Team

Guido, a valued member of Backblaze's operations team

Sometimes we do have to completely remove heavy pods from a cabinet, but a special member of the team helps with that challenge. Our server lift Guido has no trouble lifting and moving 150 pound Storage Pods and IT gear when needed.

Our server lift, Guido (on the right), helping Joe with the heavy lifting in our data center

Our server lift, Guido (on the right), helping Joe with the heavy lifting in our Phoenix data center

The custom PDU enables Jon and his team to access the Storage Pods from the back of the cabinet. Jon estimates that the new PDU enables him to complete a boot drive replacement in a Storage Pod in half the time it used to take with the previous PDU, and he doesn’t need the help of our server lift Guido for the job. That saved time adds up, especially when you need to replace boot drives in forty Storage Pods, as Jon did recently.

Custom PDU in a cabinet between two Storage Pods

Custom PDU in a cabinet between two Storage Pods

Storage Pod open at rear of cabinet

Storage Pod open at top

We Value Our Culture of Doing Things Differently

If you’re a regular reader of this blog, you’re already familiar with Backblaze’s history. Backblaze’s founders started the company because they thought people should back up their computers and it could be done at $5 per month. The problem was that no storage system available at the time would enable a sustainable business at that price. They did what they had to do: designed and built their own solution. The Backblaze Storage Pods, vault architecture, and Reed-Solomon encoding enabled a globally scalable storage system. After eleven years, three data centers, and seven hundred petabytes of customer data, we’re still able to sustainably offer the most affordable storage available anywhere.

Continuing the Backblaze Tradition

Hardworking innovators like Jon and our operations team find new ways every day to make our operations more efficient. This allows us to continuously reduce our costs while driving our growing, global footprint.

Thanks Jon. Well done!

Jon with two Backblaze cabinets, each with 10 Storage Pods, one switch, and one custom PDU

Jon with two Backblaze cabinets, each with 10 Storage Pods, one switch, and one custom PDU


Editor’s Note:  Anyone interested in obtaining information about availability and pricing for the PDU described above can contact Mirapath.

The post Backblaze’s Custom Data Center PDU appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Stories of Camera and Data Catastrophes

Post Syndicated from Roderick Bauer original https://www.backblaze.com/blog/stories-of-camera-and-data-catastrophes/

Salt water damaged camera

This is the third post in a series of post exchanges with our friends at Lensrentals.com, a popular online site for renting photography, videography, and lighting equipment. Seeing as how Halloween is just a few days away, we thought it appropriate to offer some scary tales of camera and data catastrophes. Enjoy.

Note: You can read all of Lensrentals’ posts on our blog. Find all of our posts on the Lensrentals blog.

— Editor

Stories of Camera and Data Catastrophes
by Zach Sutton, Editor-in-chief, Lensrentals.com

As one of the largest photo and video gear rental companies in the world, Lensrentals.com ships out thousands of pieces of gear each day. It would be impossible to expect that all of our gear would return to us in the same condition it was in when we rented it out. More often than not, the damage is the result of things being dropped, but now and then some pretty interesting things happen to the gear we rent out.

We have an incredible customer base, and when this kind of damage happens, they’re more than happy to pay the necessary repair fees. Stuff happens, mistakes are made, and we have a full-service repair center to keep the costs low. And while we have insurance policies for accidental damage such as drops, dings, and other accidents, it doesn’t cover neglect, which accounts for the stories we’re going to share with you below. Let’s take a look at some of our more exciting camera and data catastrophe stories.

Camera Data Catastrophes

Data catastrophes happen more often than anything else, but aren’t exactly the most exciting stories we’ve gotten over the years. The stories are usually similar. Someone rents a memory card or SSD from us, uses the card/SSD, then sends it back without pulling the footage off of it. When we receive gear back into our warehouse, we inspect and format all the media. If you realize your mistake and call or email us before that happens, we can usually put a hold on the media and ship it back to you to pull the data off of it. If we’ve already formatted the media, we will perform a recovery on the data using software such as TestDisk and PhotoRec, and let you know if we had any success. We then give you the option whether or not you want to rent the product again to have it shipped to you so you can pull the files.

The Salty Sony A7sII

A common issue we run into — and have addressed a number of times on our blog — is the dubious term “weather resistant.” This term is often used by equipment marketers and doesn’t give you the protection that people might assume by its name.

One example of that was last year, when we received a nonfunctioning Sony a7sII back from the California coast, and had to disassemble it to determine what was wrong. Upon opening the camera, it was quite apparent that it had been submerged in salt water. Water isn’t good for electronics, but the real killer is impurities, such as salt. Salt builds up on electronics, is a conductor of electricity, and will fry electronics in no time when power is applied. So, once we saw the salt corrosion, we knew that the camera was irreparable. Still, we disassembled it for no other reason than to provide evidence to others on what salt water can do to your electronics. You can read more about this and see the full break down in our post, About Getting Your Camera Wet… Teardown of a Salty Sony A7sII.

Sony A7sII disassembled into parts Sony A7sII salt water damage

The Color Run Cleanup

Color runs are 5K running events that happen all over the world. If you haven’t seen one, participants and spectators toss colorful powders throughout the run, so that by the time the runners reach the finish line, they’re covered head to toe in colorful powder. This event sounds like a lot of fun, and one would naturally want to take photos of the spectacle, but any camera gear used for the event will definitely require a deep cleaning.

Color run damage to camera lens

Color run damage to camera

We’ve asked our clients multiple times not to take our cameras to color runs, but each year we get another system back that is covered in pink, green, and blue dust. The dust used for these events is incredibly fine, making it easy to get into every nook and cranny within the camera body and lenses. This requires the gear to be completely disassembled, cleaned, and reassembled. We have two photos in this post of the results of a color run, but you can view more on the post we did about Color runs back in 2013, How to Ruin Your (or Our) Gear in 5 Minutes (Without Water).

The Eclipse That Killed Cameras

About a year ago, we had the incredible phenomenon here in the United States of a total solar eclipse. It was the first total solar eclipse to occur in the continental United States since 1979, hence a pretty exciting moment for all of us, but we braced ourselves for the damage it would do to cameras.

Eclipse camera lens damage

For weeks leading up to the event, we sent out fliers with our rentals that encouraged people to not only wear eye protection, but to protect their camera lenses with high-density ND filters. Despite that, in the days following the eclipse, we had gear coming back to us with aperture blades melted and holes burnt into sensors.

Eclipse camera damage

Eclipse camera shutter damage

As one would expect, it’s not a good idea to point your camera directly at the sun, especially for long periods of time. Most of the damage done from the eclipse was caused by people who had set up their camera and lens on a tripod pointing at the sun while waiting for the eclipse. This prolonged exposure causes a lot of heat to build up and will eventually start burning through apertures, shutters, sensors and anything else in its way. Not only do we recommend ND filters for the front of your lens, but also black cards to stop light from entering the camera until it’s go time for the total eclipse. You can read about the whole experience in our blog post on the topic, Rental Camera Gear Destroyed by the Solar Eclipse of 2017.

Damage from Burning Man

While we have countless stories of gear being destroyed, we figured it’d be best to just leave you with this one. Burning Man is an annual event that takes place in the deserts of Nevada. Touted as an art installation and experience, tens of thousands of people spend a few days living in the remote desert with fellow Burners to create and participate in a wide range of activities. And where there is a desert, there always are sand, dust, and dust storms.

Burning Man camera damage

Burning Man dust damage

One might think that sand is the biggest nuisance for camera gear at Burning Man, but it’s actually the fine dust that the wind picks up. One of the more interesting phenomena that happens during Burning Man are the dust storms. The dust storms occur with little warning, kicking up the fine dust buried within the sand that can quickly cause damage to your electronics, your skin, and your lungs. Because it is so fine, it is easily able to enter your cameras and lenses.

Burning Man damage to Nikon camera

While Burning Man doesn’t always totally destroy gear, it does result in a lot of cleaning and disassembling of gear after the event. This takes time and patience and costs the customer money. While there are stories of people who bring camera gear to Burning Man wrapped in nothing more than plastic and gaffer tape, we don’t recommend that for good gear. It’s best to just leave your camera at home, or buy an old camera for cheap to document the week. To see more of what can happen to gear at Burning Man, you can read our blog post on the topic, Please, Don’t Take Our Photography and Video Gear to Burning Man.

Those are just a few stories of some of the data and camera catastrophes that we’ve experienced over the years. We hope this serves as a warning to those who might be considering putting their gear through some of the experiences above and hopefully sway them against it. If you have some of your own stories on data or gear catastrophes, feel free to share them below in the comments.

— Zach Sutton, Editor-in-chief, Lensrentals.com

The post Stories of Camera and Data Catastrophes appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Hard Disk Drive (HDD) vs Solid State Drive (SSD): What’s the Diff?

Post Syndicated from Roderick Bauer original https://www.backblaze.com/blog/hdd-versus-ssd-whats-the-diff/

whats the diff? SSD vs. HDD

HDDs and SSDs have changed in the two years since Peter Cohen wrote the original version of this post on March 8 of 2016. We thought it was time for an update. We hope you enjoy it.

— Editor

In This Corner: The Hard Disk Drive (HDD)

The traditional spinning hard drive has been a standard for many generations of personal computers. Constantly improving technology has enabled hard drive makers to pack more storage capacity than ever, at a cost per gigabyte that still makes hard drives the best bang for the buck.

IBM RamacAs sophisticated as they’ve become, hard drives have been around since 1956. The ones back then were two feet across and could store only a few megabytes of information, but technology has improved to the point where you can cram 10 terabytes into something about the same size as a kitchen sponge.

Inside a hard drive is something that looks more than a bit like an old record player: There’s a platter, or stacked platters, which spin around a central axis — a spindle — typically at about 5,400 to 7,200 revolutions per minute. Some hard drives built for performance work faster.

Hard Drive exploded viewInformation is written to and read from the drive by changing the magnetic fields on those spinning platters using an armature called a read-write head. Visually, it looks a bit like the arm of a record player, but instead of being equipped with a needle that runs in a physical groove on the record, the read-write head hovers slightly above the physical surface of the disk.

The two most common form factors for hard drives are 2.5-inch, common for laptops, and 3.5-inch, common for desktop machines. The size is standardized, which makes for easier repair and replacement when things go wrong.

The vast majority of drives in use today connect through a standard interface called Serial ATA (or SATA). Specialized storage systems sometimes use Serial Attached SCSI (SAS), Fibre Channel, or other exotic interfaces designed for special purposes.

Hard Disk Drives Cost Advantage

Proven technology that’s been in use for decades makes hard disk drives cheap — much cheaper, per gigabyte than solid state drives. HDD storage can run as low as three cents per gigabyte. You don’t spend a lot but you get lots of space. HDD makers continue to improve storage capacity while keeping costs low, so HDDs remain the choice of anyone looking for a lot of storage without spending a lot of money.

The downside is that HDDs can be power-hungry, generate noise, produce heat, and don’t work nearly as fast as SSDs. Perhaps the biggest difference is that HDDs, with all their similarities to record players, are ultimately mechanical devices. Over time, mechanical devices will wear out. It’s not a question of if, it’s a question of when.

HDD technology isn’t standing still, and price per unit stored has decreased dramatically. As we said in our post, HDD vs SSD: What Does the Future for Storage Hold? — Part 2, the cost per gigabyte for HDDs has decreased by two billion times in about 60 years.

HDD manufacturers have made dramatic advances in technology to keep storing more and more information on HD platters — referred to as areal density. As HDD manufacturers try to outdo each other, consumers have benefited from larger and larger drive sizes. One technique is to replace the air in drives with helium, which reduces reduces friction and supports greater areal density. Another technology that should be available soon uses heat-assisted magnetic recording (HAMR). HAMR records magnetically using laser-thermal assistance that ultimately could lead to a 20 terabyte drive by 2019. See our post on HAMR by Seagate’s CTO Mark Re, What is HAMR and How Does It Enable the High-Capacity Needs of the Future?

The continued competition and race to put more and more storage in the same familiar 3.5” HDD form factor means that it will be a relatively small, very high capacity choice for storage for many years to come.

In the Opposite Corner: The Solid State Drive (SSD)

Solid State Drives (SSDs) have become much more common in recent years. They’re standard issue across Apple’s laptop line, for example the MacBook, MacBook Pro and MacBook Air all come standard with SSDs. So does the Mac Pro.

Inside an SSDSolid state is industry shorthand for an integrated circuit, and that’s the key difference between an SSD and a HDD: there are no moving parts inside an SSD. Rather than using disks, motors and read/write heads, SSDs use flash memory instead — that is, computer chips that retain their information even when the power is turned off.

SSDs work in principle the same way the storage on your smartphone or tablet works. But the SSDs you find in today’s Macs and PCs work faster than the storage in your mobile device.

The mechanical nature of HDDs limits their overall performance. Hard drive makers work tirelessly to improve data transfer speeds and reduce latency and idle time, but there’s a finite amount they can do. SSDs provide a huge performance advantage over hard drives — they’re faster to start up, faster to shut down, and faster to transfer data.

A Range of SSD Form Factors

SSDs can be made smaller and use less power than hard drives. They also don’t make noise, and can be more reliable because they’re not mechanical. As a result, computers designed to use SSDs can be smaller, thinner, lighter and last much longer on a single battery charge than computers that use hard drives.

SSD Conversion KitMany SSD makers produce SSD mechanisms that are designed to be plug-and-play drop-in replacements for 2.5-inch and 3.5-inch hard disk drives because there are millions of existing computers (and many new computers still made with hard drives) that can benefit from the change. They’re equipped with the same SATA interface and power connector you might find on a hard drive.


Intel SSD DC P4500A wide range of SSD form factors are now available. Memory Sticks, once limited to 128MB maximum, now come in versions as large as 2 TB. They are used primarily in mobile devices where size and density are primary factor, such as cameras, phones, drones, and so forth. Other high density form factors are designed for data center applications, such as Intel’s 32 TB P4500. Resembling a standard 12-inch ruler, the Intel SSD DC P4500 has a 32 terabyte capacity. Stacking 64 extremely thin layers of 3D NAND, the P4500 is currently the world’s densest solid state drive. The price is not yet available, but given that the DC P4500 SSD requires only one-tenth the power and just one-twentieth the space of traditional hard disk storage, once the price comes out of the stratosphere you can be sure that there will be a market for it.

Nimbus ExaDrive 100TB SSDEarlier this year, Nimbus Data announced the ExaDrive D100 100TB SSD. This SSD by itself holds over twice as much data as Backblaze’s first Storage Pods. Nimbus Data has said that the drive will have pricing comparable to other business-grade SSDs “on a per terabyte basis.” That likely means a price in the tens of thousands of dollars.

SSD drive manufacturers also are chasing ways to store more data in ever smaller form factors and at greater speeds. The familiar SSD drive that looks like a 2.5” HDD drive is starting to become less common. Given the very high speeds that data can be read and copied to the memory chips inside SSDs, it’s natural that computer and storage designers want to take full advantage of that capability. Increasingly, storage is plugging directly into the computer’s system board, and in the process taking on new shapes.

Anand Lal Shimpi, anandtech.com -- http://www.anandtech.com/show/6293/ngff-ssds-putting-an-end-to-proprietary-ultrabook-ssd-form-factors

A size comparison of an mSATA SSD (left) and an M.2 2242 SSD (right)

Laptop makers adopted the mSATA, and then the M.2 standard, which can be as small as a few squares of chocolate but have the same capacity as any 2.5” SATA SSD.

Another interface technology called NvM Express or NVMe may start to move from servers in the data center to consumer laptops in the next few years. NVMe will push storage speeds in laptops and workstations even higher.

SSDs Fail Too

Just like hard drives, SSDs can wear out, though for different reasons. With hard drives, it’s often just the mechanical reality of a spinning motor that wears down over time. Although there are no moving parts inside an SSD, each memory bank has a finite life expectancy — a limit on the number of times it can be written to and read from before it stops working. Logic built into the drives tries to dynamically manage these operations to minimize problems and extend its life.

For practical purposes, most of us don’t need to worry about SSD longevity. An SSD you put in your computer today will likely outlast the computer. But it’s sobering to remember that even though SSDs are inherently more rugged than hard drives, they’re still prone to the same laws of entropy as everything else in the universe.

Planning for the Future of Storage

If you’re still using a computer with a SATA hard drive, you can see a huge performance increase by switching to an SSD. What’s more, the cost of SSDs has dropped dramatically over the course of the past couple of years, so it’s less expensive than ever to do this sort of upgrade.

Whether you’re using a HDD or an SSD, a good backup plan is essential because eventually any drive will fail. You should have a local backup combined with secure cloud-based backup like Backblaze, which satisfies the 3-2-1 backup strategy. To help get started, make sure to check out our Backup Guide.

Hopefully, we’ve given you some insight about HDDs and SSDs. And as always, we encourage your questions and comments, so fire away!


Editor’s note:  You might enjoy reading more about the future of HDDs and SSDs in our two-part series, HDD vs SSD: What Does the Future for Storage Hold?

The post Hard Disk Drive (HDD) vs Solid State Drive (SSD): What’s the Diff? appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Securely Managing Your Digital Media (SD, CF, SSD, and Beyond)

Post Syndicated from Roderick Bauer original https://www.backblaze.com/blog/securely-managing-your-digital-media-sd-cf-ssd-and-beyond/

3 rows of 3 memory cards

This is the second in our post exchange series with our friends Zach Sutton and Ryan Hill at Lensrentals.com, who have an online site for renting photography, videography, and lighting equipment. You can read our post from last month on their blog, 3-2-1 Backup Best Practices using Cloud Archiving, and all posts on our blog in this series at Lensrentals post series.

— Editor

Managing digital media securely is crucial for all photographers and videographers. At Lensrentals.com, we take media security very seriously, with dozens of rented memory cards, hard drives, and other data devices returned to our facility every day. All of our media is inspected with each and every rental customer. Most of the cards returned to us in rental shipments are not properly reformatted and erased, so it’s part of our usual service to clear all the data from returned media to keep each client’s identity and digital property secure.

We’ve gotten pretty good at the routine of managing data and formatting storage devices for our clients while making sure our media has a long life and remains free from corruption. Before we get too involved in our process of securing digital media, we should first talk fundamentals.

The Difference Between Erasing and Reformatting Digital Media

When you insert a card in the camera, you’re likely given two options, either erase the card or format the card. There is an important distinction between the two. Erasing images from a card does just that — erases them. That’s it. It designates the area the prior data occupied on the card as available to write over and confirms to you that the data has been removed.

The term erase is a bit misleading here. The underlying data, the 1’s and 0’s that are recorded on the media, are still there. What really happens is that the drive’s address table is changed to show that the space the previous file occupied is available for new data.

This is the reason that simply erasing a file does not securely remove it. Data recovery software can be used to recover that old data as long as it hasn’t been overwritten with new data.

Formatting goes further. When you format a drive or memory card, all of the files are erased (even files you’re designated as “protected”) and also usually adds a file system. This is a more effective method for removing all the data on the drive since all the space previously divided up for specific files has a brand new structure unencumbered by whatever size files were previously stored. Be beware, however, that it’s possible to retrieve older data even after a format. Whether that can happen depends on the formatting method and whether new data has overwritten what was previously stored.

To make sure that the older data cannot be recovered, a secure erase goes further. Rather than simply designating the data that can be overwritten with new data, a secure erase writes a random selection of 1s and 0s to the disk to make sure the old data is no longer available. This takes longer and is more taxing on the card because data is being overwritten rather than simply removed.

Always Format a Card for the Camera You’re Going to Be Using

If you’ve ever tried to use the same memory card on cameras of different makes without formatting it, you may have seen problems with how the data files are displayed. Each camera system handles its file structure a little differently.

For this reason it’s advisable to format the card for the specific camera you’re using. If this is not done, there is a risk of corrupting data on the card.

Our Process For Securing Data

Our inspection process for recording media varies a little depending on what kind of card we’re inspecting. For standardized media like SD cards or compact flash cards, we simply use a card reader to format the card to exFAT. This is done in Disk Utility on the Apple Macbooks that we issue to each of our Video Technicians. We use exFAT specifically because it’s recognizable by just about every device. Since these cards are used in a wide variety of different cameras, recorders, and accessories, and we have no way of knowing at the point of inspection what device they’ll be used with, we have to choose a format that will allow any camera to recognize the card. While our customer may still have to format a card in a camera for file structure purposes, the card will at least always come formatted in a way that the camera can recognize.

Sony SxS media
For proprietary media — things like REDMAGs, SxS, and other cards that we know will only be used in a particular camera — we use cameras to do the formatting. While the exFAT system would technically work, a camera-specific erase and format process saves the customer a step and allows us to more regularly double-check the media ports on our cameras. In fact, we actually format these cards twice at inspection. First, the Technician erases the card to clear out any customer footage that may have been left on it. Next, they record a new clip to the card, around 30 seconds, just to make sure everything is working as it’s supposed to. Finally, they format the card again, erasing the test footage before sending it to the shelf where it awaits use by another customer.

REDMAG Red Mini-Mag You’ll notice that at no point in this process do we do a full secure erase. This is both to save time and to prevent unnecessary wear and tear on the cards. About 75% of the media we get back from orders still has footage on it, so we don’t get the impression that many of our customers are overly concerned with keeping their footage private once they’re done shooting. However, if you are one of those 25% that may have a personal or professional interest in keeping your footage secure after shooting, we’d recommend that you securely erase the media before returning rented memory cards and drives. Or, if you’d rather we handle it, just send an email or note with your return order requesting that we perform a secure erase rather than simply formatting the cards, and we’ll be happy to oblige.

Managing your digital media securely can be easy if done right. Data management and backing up files, on the other hand, can be more involved and require more planning. If you have any questions on that topic, be sure to check out our recent blog post on proper data backup.

— Zach Sutton and Ryan Hill, lensrentals.com

The post Securely Managing Your Digital Media (SD, CF, SSD, and Beyond) appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Mac and iOS Users: Remember to Back Up Before You Upgrade!

Post Syndicated from Roderick Bauer original https://www.backblaze.com/blog/mac-and-ios-users-remember-to-back-up-before-you-upgrade/

macOS Mojave

New versions of Apple’s operating systems are coming to your iPhone and Mac in the next week! iOS 12 was released today, and macOS 10.14 “Mojave” is available a week from today on September 24. If you’re planning to upgrade your Mac or iOS devices with Apple’s newest software, you should make it a point to back up before you install anything new.

The new releases were announced in June at Apple’s annual Worldwide Developer Conference (WWDC), which gathers thousands of Apple developers from around the world each year. It’s a familiar annual processional: Apple introduces new versions of both the Mac and iOS operating systems. They’re tested by developers and the public throughout the summer.

Back up Early and Often

Changing your Mac or iPhone’s operating system isn’t like installing a new version of an app, even though Apple has tried to make it a relatively simple process. Operating system software is essential software for these devices, and how it works has a cascading effect on all the other apps and services you depend on.

If you’re not currently backing up, it’s easy to get started using our 3-2-1 Backup Strategy. The idea behind the 3-2-1 Backup Strategy is that there should be three copies of your data: The main one you use, a local backup copy, and a remote copy, stored at a secure offsite data center like Backblaze. It’s served us and thousands of our customers very well over the years, so we recommend it unabashedly. Also check out our Mac Backup Guide.

Our advice is to make sure to back up all of your systems before installing operating system software, even final released software. It’s better to be safe rather than sorry, especially where the safety and security of your data are concerned.

The post Mac and iOS Users: Remember to Back Up Before You Upgrade! appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

The Maltese MacBook

Post Syndicated from Roderick Bauer original https://www.backblaze.com/blog/the-maltese-macbook/

Still from the 1941 John Huston film, The Maltese Falcon, with Humphrey Bogart, Peter Lorre, Mary Astor, and Sydney Greenstreet

Last year we decided to use Apple’s big fall announcement day to talk about backing up Windows computers. This year we’re continuing the tradition of writing something that runs a bit counter to all the hoopla with a tongue-in-cheek post in the style of hardboiled detective fiction entitled, “The Maltese MacBook,” with apologies to Dashiell Hammett.

— Editor

It was a Wednesday and it would have been just like any other Wednesday except Apple was making its big fall product announcements. Just my luck, I had to work in the San Francisco store, which meant that I was the genius who got to answer all the questions.

I had just finished helping a customer who claimed that Siri was sounding increasingly impatient answering his questions when I looked up and saw her walk in the door.

Her blonde hair was streaked with amethyst highlights and she was wearing a black leather tutu and polished kneehigh Victorian boots. Brightly colored tattoos of Asian characters ran up both of her forearms and her neck. Despite all that, she wouldn’t particularly stand out in San Francisco, but her cobalt-blue eyes held me and wouldn’t let me go. She rapidly reduced the distance between the door and where I stood behind the counter at the back of the store.

She plopped a Surface Pro computer on the counter in front of me.

“I lost my data,” she said.

I knew I’d seen her before, but I couldn’t place where.

“That’s a Windows computer,” I said.

She leaned over the counter towards me. Her eyes were even brighter and bluer close up.

“Tell me something I don’t know, genius,” she replied.

Then I remembered where I’d seen her. She was on Press: Here a while back talking about her new startup. She was head of software engineering for a Google spinoff. Angels all over the valley were fighting to throw money at her project. I had been sitting in my boxers eating cold pizza and watching her talk on TV about AI for Blockchain ML.

She was way out of my league.

“I was in Valletta on a business trip using my MacBook Pro,” she said. “I was reading Verlaine on the beach when a wave came in and soaked Reggie. ‘Reggie’ is my MacBook Pro. Before I knew it, it was all over.”

Her eyes misted up.

“You know that there isn’t an Apple store in Malta, don’t you?” she said.

“We have a reseller there,” I replied.

“But they aren’t geniuses, are they?” she countered.

“No, they’re not.” She had me there.

“I had no choice but to buy this Surface Pro at a Windows shop on Strait Street to get me through the conference. It’s OK, but it’s not Reggie. I came in today to get everything made right. You can do that for me, can’t you?”

I looked down at the Surface Pro. We weren’t supposed to work on other makes of computers. It was strictly forbidden in the Genius Training Student Workbook. Alarms were going off in my head telling me to be careful:  this dame meant nothing but trouble.

“Well?” she said.

I made the mistake of looking at her and lingering just a little too long. Her eyes were both shy and probing at the same time. I felt myself falling head over heels into their inky-blue depths.

I shook it off and gradually crawled back to consciousness. I told myself that if a customer’s computer needs help, it doesn’t make any difference what you think of the computer, or which brand it is. She’s your customer, and you’re supposed to do something about it. That’s the way it works. Damn the Genius Training Student Workbook.

“OK,” I said. “Let’s take care of this.”

I asked her whether she had files on the Surface Pro she needed to save. She told me that she used Backblaze Cloud Backup on both the new Surface Pro and her old MacBook Pro. My instincts had been right. This lady was smart.

“That will make it much easier,” I told her. “We’ll just download the backed up files for both your old Macbook Pro and your Surface Pro from Backblaze and put them on a new MacBook Pro. We’ll be done in just a few minutes. You know about Backblaze’s Inherit Backup State, right? It lets you move your account to a new computer, restore all your files from your backups to the computer, and start backing up again without having to upload all your files again to the cloud.

“What do you think?” she asked.

I assumed she meant that she already knew all about Inherit Backup State, so I went ahead and configured her new computer.

I was right. It took me just a little while to get her new MacBook Pro set up and the backed up files restored from the Backblaze cloud. Before I knew it, I was done.

“Thanks” she said. “You’ve saved my life.”

Saved her life? My head was spinning.

She turned to leave. I wanted to stop her before she left. I wanted to tell her about my ideas for an AI-based intelligent customer support agent. Maybe she’d be impressed. But she was already on her way towards the door.

I thought she was gone forever but she stopped just before the door. She flipped her hair back over her shoulder as she turned to look at me.

“You really are a genius.”

She smiled and walked out of the store and out of my life. My eyes lingered on the swinging door as she crossed the street and disappeared into the anonymous mass of humanity.

I thought to myself: she’ll be back. She’ll be back to get a charger, or a Thunderbolt to USB-C adaptor, or Magsafe to USB-C, or Thunderbolt 3 to Thunderbolt 2, or USB-C to Lightning, or USB-A to USB-C, or DisplayPort to Mini DisplayPort, or HDMI to DisplayPort, or vice versa.

Yes, she’ll be back.

I panicked. Maybe she’ll take the big fall for Windows and I’ll never see her again. What if that happened?

Then I realized I was just being a sap. Snap out of it! I’ll wait for her no matter what happens.

She deserves that.

The Maltese Falcon

The post The Maltese MacBook appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

cPanel Backup to B2 Cloud Storage

Post Syndicated from Roderick Bauer original https://www.backblaze.com/blog/cpanel-backup-to-b2-cloud-storage/

laptop on a desk with a cup of coffee, cell phone, and iPad

Anyone who’s managed a business or personal website is likely familiar with cPanel, the control panel that provides a graphical interface and tools that simplify the process of managing a website. IT professionals who’ve managed hosting servers might know cPanel’s big brother, WHM (Web Host Manager), which is used by server administrators to manage large web hosting servers and cPanels for their customers.

cPanel Dashboard WHM Dashboard
cPanel Dashboard   WHM Dashboard

Just as with any other online service, backup is critically important to safeguard user and business data from hardware failure, accidental loss, or unforeseen events. Both cPanel and WHM support a number of applications for backing up websites and servers.

JetApps’s JetBackup cPanel App

One of those cPanel applications is JetApps’s JetBackup, which supports backing up data to a number of destinations, including local, remote SSH, remote FTP, and public cloud services. Backblaze B2 Cloud Storage was added as a backup destination in version 3.2. Web hosts that support JetBackup for their cPanel and WHM users include Clook, FastComet, TMDHosting, Kualo, Media Street, ServerCake, WebHost.UK.net, MegaHost, MonkeyTree Hosting, and CloudBunny.

cPanel with JetBackup app

cPanel with JetBackup app

JetBackup configuration for B2

JetBackup configuration for B2

Directions for configuring JetBackup with B2 are available on their website.

Note:  JetBackup version 3.2+ supports B2 cloud storage, but that support does not currently include incremental backups. JetApps has told us that incremental backup support will be available in an upcoming release.

Interested in more B2 Support for cPanel and WHM?

JetBackup support for B2 was added to JetBackup because their users asked for it. Users have been vocal in asking vendors to add cPanel/WHM support for backing up to B2 in forums and online discussions, as evidenced on cPanel.net and elsewhere — here, here, and here. The old axiom that the squeaky wheel gets the grease is true when lobbying vendors to add B2 support — the best way to have B2 directly supported by an app is to express your interest directly to the backup app provider.

Other Ways to Back Up Website Data to B2

When a dedicated backup app for B2 is not available, some cPanel users are creating their own solutions using the B2 Command Line Interface (CLI), while others are using Rclone to back up to B2.

B2 CLI example:

#!/bin/bash
b2 authorize_account ACCOUNTID APIKEY
b2 sync –noProgress /backup/ b2://STORAGECONTAINER/

Rclone example:

rclone copy /backup backblaze:my-server-backups –transfers 16

Those with WordPress websites have other options for backing up their sites, which we highlighted in a post, Backing Up WordPress.

Having a Solid Backup Plan is What’s Important

If you’re using B2 for cPanel backup, or are using your own backup solution, please let us know what you’re doing in the comments.

The post cPanel Backup to B2 Cloud Storage appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

The B2 Developers’ Community

Post Syndicated from Roderick Bauer original https://www.backblaze.com/blog/object-storage-developer-community/

Developers at Work Using Object Storage

When we launched B2 Cloud Storage in September of 2015, we were hoping that the low cost, reliability, and openness of B2 would result in developers integrating B2 object storage into their own applications and platforms.

We’ve continually strengthened and encouraged the development of more tools and resources for the B2 developer community. These resources include APIs, a Command-Line tool, a Java SDK, and code examples for Swift and C++. Backblaze recently added application keys for B2, which enable developers to restrict access to B2 data and control how an application interacts with that data.

An Active B2 Developer Community

It’s three years later and we are happy to see that an active developer community has sprung up around B2. Just a quick look at GitHub shows over 250 repositories for B2 code with projects in ten different languages that range from C# to Go to Ruby to Elixir. A recent discussion on Hacker News about a B2 Python Library resulted in 225 comments.

B2 coding languages - Java, Ruby, C#, Shell, PHP, R, JavaScript, C++, Elixir, Go, Python, Swift

What’s Happening in the B2 Developer Community?

We believe that the two major reasons for the developer activity supporting B2 are, 1) the user demand for inexpensive and reliable storage, and, 2) the ease of implementation of the B2 API. We discussed the B2 API design decisions in a recent blog post.

Sharing and transparency have been cornerstone values for Backblaze since our founding, and we believe openness and transparency breed trust and further innovation in the community. Since we ask customers to trust us with their data, we want our actions to show why we are worthy of that trust.

Here are Just Some of the Many B2 Projects Currently Underway

We’re excited about all the developer activity and all of the fresh and creative ways you are using Backblaze B2 storage. We want everyone to know about these developer projects so we’re spotlighting some of the exciting work that is being done to integrate and extend B2.

Rclone (Go) — In addition to being an open source command line program to sync files and directories to and from cloud storage systems, Rclone is being used in conjunction with other applications such as restic. See Rclone on GitHub, as well.

CORS (General web development) — Backblaze supports CORS for efficient cross-site media serving. CORS allows developers to store large or infrequently accessed files on B2 storage, and then refer to and serve them securely from another website without having to re-download the asset.

b2blaze (Python) — The b2blaze Python library for B2.

Laravel Backblaze Adapter (PHP) — Connect your Laravel project to Backblaze connector with this storage adapter with token caching.

Wal-E (Postgres) — Continuous archiving to Backblaze for your Postgres databases.

Phoenix (Elixir) — File upload utility for the Phoenix web dev framework.

ZFS Backup (Go) — Backup tool to move your ZFS snapshots to B2.

Django Storage (Python) — B2 storage for the Python Django web development framework.

Arq Backup (Mac and Windows application) — Arq Backup is an example of a single developer, Stefan Reitshamer, creating and supporting a successful and well-regarded application for cloud backup. Stefan also is known for being responsive to his users.

Go Client & Libraries (Go) — Go is a popular language that is being used for a number of projects that support B2, including restic, Minio, and Rclone.

How to Get Involved as a B2 Developer

If you’re considering developing for B2, we encourage you to give it a try. It’s easy to implement and your application and users will benefit from dependable and economical cloud storage.

Developers at workStart by checking out the B2 documentation and resources on our website. GitHub and other code repositories are also great places to look. If you follow discussions on Reddit, you could learn of projects in the works and maybe find users looking for solutions.

We’ve written a number of blog posts highlighting the integrations for B2. You can find those by searching for a specific integration on our blog or under the tag B2. Posts for developers are tagged developer.

Developers at work

If you have a B2 integration that you believe will appeal to a significant audience, you should consider submitting it to us. Those that pass our review are listed on the B2 Integrations page on our website. We’re adding more each week. When you’re ready, just review the B2 Integration Checklist and submit your application. We’re looking forward to showcasing your work!

Now’s a good time to join the B2 developers’ community. Jump on in — the water’s great!

P.S. We want to highlight and promote more developers working with B2. If you have a B2 integration or project that we haven’t mentioned in this post, please tell us what you’re working on in the comments.

The post The B2 Developers’ Community appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Backing UP FreeNAS and TrueNAS to Backblaze B2

Post Syndicated from Roderick Bauer original https://www.backblaze.com/blog/how-to-setup-freenas-cloud-storage/

FreeNAS and TrueNAS

Thanks to recent updates of FreeNAS and TrueNAS, backing up data to Backblaze B2 Cloud Storage is now available for both platforms. FreeNAS/TrueNAS v11.1 adds a feature called Cloud Sync, which lets you sync, move, or copy data to and from Backblaze B2.

What Are FreeNAS and TrueNAS?

FreeNAS and TrueNAS are two faces of a comprehensive NAS storage environment built on the FreeBSD OS and OpenZFS file system. FreeNAS is the open source and development platform, while TrueNAS is the supported and commercial product line offered by IXSystems.

FreeNAS logo

FreeNAS is for the DIY crowd. If you don’t mind working with bleeding-edge software and figuring out how to make your software and hardware work harmoniously, then FreeNAS could be a good choice for you.

TrueNAS logo

If you’re in a business or other environment with critical data, then a fully supported product like TrueNAS is likely the way you’ll want to go. IXsystems builds their TrueNAS commercial server appliances on the battle-tested, open source framework that FreeNAS and OpenZFS provide.

The software developed by the FreeNAS open source community forms the basis for both platforms, so we’ll talk specifically about FreeNAS in this post.

Working with FreeNAS

You can download FreeNAS directly from the open source project website, freenas.org. Once installed, FreeNAS is managed through a comprehensive web interface that is supplemented by a minimal shell console that handles essential administrative functions. The web interface supports storage pool configuration, user management, sharing configuration, and system maintenance.

FreeNAS web UI

FreeNAS supports Windows, macOS and Unix clients.

Syncing to B2 with FreeNAS

Files or directories can be synchronized to remote cloud storage providers, including B2, with the Cloud Sync feature.

Selecting Tasks ‣ Cloud Sync shows the screen below. This screen shows a single cloud sync called “backup-acctg” that “pushes” a file to cloud storage. The last run finished with a status of SUCCESS.

Existing cloud syncs can be run manually, edited, or deleted with the buttons that appear when a single cloud sync line is selected by clicking with the mouse.

FreeNAS Cloud Sync status

Cloud credentials must be defined before a cloud sync is created. One set of credentials can be used for more than one cloud sync. For example, a single set of credentials for Backblaze B2 can be used for separate cloud syncs that push different sets of files or directories.

A cloud storage area must also exist. With B2, these are called buckets and must be created before a sync task can be created.

After the credentials and receiving bucket have been created, a cloud sync task is created with Tasks ‣ Cloud Sync ‣ Add Cloud Sync. The Add Cloud Sync dialog is shown below.

FreeNAS Cloud Sync credentials

Cloud Sync Options

The table below shows the options for Cloud Sync.

Setting Value Type Description
Description string a descriptive name for this Cloud Sync
Direction string Push to send data to cloud storage, or Pull to pull data from the cloud storage
Provider drop-down
menu
select the cloud storage provider; the list of providers is defined by Cloud Credentials
Path browse
button
select the directories or files to be sent for Push syncs or the destinations for Pull syncs
Transfer Mode drop-down
menu
Sync (default): make files on destination system identical to those on the source; files removed from the source are removed from the destination (like rsync –delete)
Copy: copy files from the source to the destination, skipping files that are identical (like rsync)
Move: copy files from the source to the destination, deleting files from the source after the copy (like mv)
Minute slider or
minute selections
select Every N minutes and use the slider to choose a value, or select Each selected minute and choose specific minutes
Hour slider or
hour selections
select Every N hours and use the slider to choose a value, or select Each selected hour and choose specific hours
Day of month slider or
day of month
selections
select Every N days of month and use the slider to choose a value, or select Each selected day of month and choose specific days
Month checkboxes months when the Cloud Sync runs
Day of week checkboxes days of the week when the Cloud Sync runs
Enabled checkbox uncheck to temporarily disable this Cloud Sync

Take care when choosing a Direction. Most of the time, Push will be used to send data to the cloud storage. Pull retrieves data from cloud storage, but be careful: files retrieved from cloud storage will overwrite local files with the same names in the destination directory.

Provider is the name of the cloud storage provider. These providers are defined by entering credentials in Cloud Credentials.

After the Provider is chosen, a list of available cloud storage areas from that provider is shown. With B2, this is a drop-down with names of existing buckets.

Path is the path to the directories or files on the FreeNAS system. On Push jobs, this is the source location for files sent to cloud storage. On Pull jobs, the Path is where the retrieved files are written. Again, be cautious about the destination of Pull jobs to avoid overwriting existing files.

The Minute, Hour, Days of month, Months, and Days of week fields permit creating a flexible schedule of when the cloud synchronization takes place.

Finally, the Enabled field makes it possible temporarily disable a cloud sync job without deleting it.

FreeNAS Cloud Sync Example

This example shows a Push cloud sync which writes an accounting department backup file from the FreeNAS system to Backblaze B2 storage.

Before the new cloud sync was added, a bucket called “cloudsync-bucket” was created with the B2 web console for storing data from the FreeNAS system.

System ‣ Cloud Credentials ‣ Add Cloud Credential is used to enter the credentials for storage on a Backblaze B2 account. The credential is given the name B2, as shown in the image below:

FreeNAS Cloud Sync B2 credentials

Note on encryption: FreeNAS 11.1 Cloud Sync does not support client-side encryption of data and file names before syncing to the cloud, whether the destination is B2 or another public cloud provider. That capability will be available in FreeNAS v11.2, which is currently in beta.

Example: Adding Cloud Credentials

The local data to be sent to the cloud is a single file called accounting-backup.bin on the smb-storage dataset. A cloud sync job is created with Tasks ‣ Cloud Sync ‣ Add Cloud Sync.

The Description is set to “backup-acctg” to describe the job. This data is being sent to cloud storage, so this is a Push. The provider comes from the cloud credentials defined in the previous step, and the destination bucket “cloudsync-bucket” has been chosen.

The Path to the data file is selected.

The remaining fields are for setting a schedule. The default is to send the data to cloud storage once an hour, every day. The options provide great versatility in configuring when a cloud sync runs, anywhere from once a minute to once a year.

The Enabled field is checked by default, so this cloud sync will run at the next scheduled time.

The completed dialog is shown below:

FreeNAS Cloud Sync example

Dependable and Economical Disaster Recovery

In the event of an unexpected data-loss incident, the VMs, files, or other data stored in B2 from FreeNAS or TrueNAS are available for recovery. Having that data ready and available in B2 provides a dependable, easy, and cost effective offsite disaster recovery solution.

Are you using FreeNAS or TrueNAS? What tips do you have? Let us know in the comments.

The post Backing UP FreeNAS and TrueNAS to Backblaze B2 appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Minio as an S3 Gateway for Backblaze B2 Cloud Storage

Post Syndicated from Roderick Bauer original https://www.backblaze.com/blog/how-to-use-minio-with-b2-cloud-storage/

Minio + B2

While there are many choices when it comes to object storage, the largest provider and the most recognized is usually Amazon’s S3. Amazon’s set of APIs to interact with their cloud storage, often just called “S3,” is frequently the first integration point for an application or service needing to send data to the cloud.

One of the more frequent questions we get is “how do I jump from S3 to B2 Cloud Storage?” We’ve previously highlighted many of the direct integrations that developers have built on B2: here’s a full list.

Another way to work with B2 is to use what is called a “cloud storage gateway.” A gateway is a service that acts as a translation layer between two services. In the case of Minio, it enables customers to take something that was integrated with the S3 API and immediately use it with B2.

Before going further, you might ask “why didn’t Backblaze just create an S3 compatible service?” We covered that topic in a recent blog post, Design Thinking: B2 APIs (& The Hidden Costs of S3 Compatibility). The short answer is that our architecture enables some useful differentiators for B2. Perhaps most importantly, it enables us to sustainably offer cloud storage at a ¼ of the price of S3, which you will really appreciate as your application or service grows.

However, there are situations when a customer is already using the S3 APIs in their infrastructure and want to understand all the options for switching to B2. For those customers, gateways like Minio can provide an elegant solution.

What is Minio?

Minio is an open source, multi-cloud object storage server and gateway with an Amazon S3 compatible API. Having an S3-compatible API means once configured, Minio acts as a gateway to B2 and will automatically and transparently put or get data into a Backblaze B2 account.

Backup, archive or other software that supports the S3 protocol can be configured to point at Minio. Minio internally translates all the incoming S3 API calls into equivalent B2 storage API calls, which means that all Minio buckets and objects are stored as native B2 buckets and objects. The S3 object layer is transparent to the applications that use the S3 API. This enables the simultaneous use of both Amazon S3 and B2 APIs without compromising any features.

Minio has become a popular solution, with over 113.7M+ Docker pulls. Minio implements the Amazon S3 v2/v4 API in the Minio client, AWS SDK, and in the AWS CLI.

Minio and B2

To try it out, we configured a MacBook Pro with a Docker container for the latest version of Minio. It was a straightforward matter to install the community version of Docker on our Mac and then install the container for Minio.

You can follow the instructions on GitHub for configuring Minio on your system.

In addition to using Minio with S3-compatible applications and creating new integrations using their SDK, one can use Minio’s Command-line Interface (CLI) and the Minio Browser to access storage resources.

Command-line Access to B2

We installed the Minio client (mc), which provides a modern CLI alternative to UNIX coreutils such as ls, cat, cp, mirror, diff, etc. It supports filesystems and Amazon S3 compatible cloud storage services. The Minio client is supported on Linux, Mac, and Windows platforms.

We used the command below to add the alias “myb2” to our host to make it easy to access our data.

mc config host add myb2 \
 http://localhost:9000 b2_account_id b2_application_key

Minio client commands

Once configured, you can use mc subcommands like ls, cp, mirror to manage your data.

Here’s the Minio client command to list our B2 buckets:

mc ls myb2

And the result:

Minio client

Browsing Your B2 Buckets

Minio Gateway comes with an embedded web based object browser that makes it easy to access your buckets and files on B2.

Minio browser

Minio is a Great Way to Try Out B2

Minio is designed to be straightforward to deploy and use. If you’re using an S3-compatible integration, or just want to try out Backblaze B2 using your existing knowledge of S3 APIs and commands, then Minio can be a quick solution to getting up and running with Backblaze B2 and taking advantage of the lower cost of B2 cloud storage.

The post Minio as an S3 Gateway for Backblaze B2 Cloud Storage appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Protecting Your Data From Camera to Archive

Post Syndicated from Roderick Bauer original https://www.backblaze.com/blog/protecting-your-data-from-camera-to-archive/

Camera data getting backed up to Backblaze B2 cloud

Lensrentals.com is a highly respected company that rents photography and videography equipment. We’re a fan of their blog and asked Zach Sutton and Ryan Hill of Lensrentals to contribute something for our audience. We also contributed a post to their blog that was posted today: 3-2-1 Backup Best Practices using Cloud Archiving.

Enjoy!

— Editor

At Lensrentals.com we get a number of support calls, but unfortunately one of them is among the most common: data catastrophes.

The first of the frequent calls is from someone who thought they transferred over their footage or photos before returning their rental and discovered later that they were missing some images or footage. If we haven’t already gone through an inspection of those cards, it’s usually not a problem to send the cards back to them so they can collect their data. But if our techs have inspected the memory cards, then there isn’t much we can do. Our team at Lensrentals.com perform a full and secure reformatting of the cards to keep each customer’s data safe from the next renter. Once that footage is gone, it is unrecoverable and gone forever. This is never a fun conversation to have.

The second scenario is when a customer calls to tell us that they did manage to transfer all the footage over, but one or more of the clips or images were corrupted in the transferring process. Typically, people don’t discover this until after they’ve sent back the memory cards, and after we’ve already formatted the original media. This is another tough phone call to have. On occasion, data corruption happens in camera, but more often than not, the file gets corrupted during the transfer from the media to the computer or hard drive.

These kinds of problems aren’t entirely avoidable and are inherent risks users take when working with digital media. However, as with all risks, you can take proper steps to assure that your data is safe. If a problem arises, there are techniques you can use to work around it.

We’ve summarized our best suggestions for protecting your data from camera to archive in the following sections. We hope you find them useful.

How to Protect Your Digital Assets

Before Your Shoot

The first and most obvious step to take to assure your data is safe is to make sure you use reliable media. For us, we recommend using cards from brands you trust, such as Sandisk, Lexar or ProGrade Digital (a company that took the reins from Lexar). For hard drives, SanDisk, Samsung, Western Digital, and Intel are all considered incredibly reliable. These brands may be more expensive than bargain brands but have been proven time and time again to be more reliable. The few extra dollars spent on reliable media will potentially save you thousands in the long run and will assure that your data is safe and free of corruption.

One of the most important things you should do before any shoot is format your memory card in the camera. Formatting in camera is a great way to minimize file corruption as it keeps the card’s file structure conforming to that camera manufacturer’s specifications, and it should be done every time before every shoot. Equally important, if the camera gives you an option to do a complete or secure format, take that option over the other low-level formatting options available. In the same vein, it’s essential to also take the time to research and see if your camera needs to unmount or “eject” the media before removing it physically. While this option applies more for video camera recording systems, like those found on the RED camera platform and the Odyssey 7Q, it’s always worth checking into to avoid any corruption of the data. More often than not, preventable data corruption happens when the users turn off the camera system before the media has been unmounted.

Finally, if you’re shooting for the entire day, you’ll want to make sure you have enough media on hand for the entire day, so that you do not need to back up and reformat cards throughout the shoot. While it’s possible to take footage off of the card, reformat it, and use it again for the same day, that is not something you’d want to be doing during the hectic environment of a shoot day — it’s best to have extra media on hand. We’ve all made a mistake and deleted a file we didn’t mean to, so it’s best to avoid that mistake by not having to delete or manage files while shooting. Play it safe, and only reformat when you have the time and clear head to do so.

During Your Shoot

On many modern camera systems, you have the option of dual-recording using two different card slots. If your camera offers this option, we cannot recommend it enough. Doubling the media you’re recording onto can overcome a failure in one of the memory cards. While the added cost may be a hard sell, it’s negligible when compared to all the money spent on lights, cameras, actors and lousy pizza for the day. Additionally, develop a system that works for you and keeps everything as organized as possible. Spent media shouldn’t be in the same location as unused media, and your file structure should be consistent throughout the entire shoot. A proper file structure not only saves time but assures that none of the footage goes missing after the shoot, lost in some random folder.

Camera memory cards

Among one of the most critical jobs while on set is the work of a DIT (Digital Imaging Technician) for video, and a DT (Digital Technician) for photography. Essentially, the responsibilities of these positions are to keep the data archived and organized on a set, as well as metadata logging and other technical tasks involved in keeping a shoot organized. While it may not be cost effective to have a DIT/DT on every shoot, if the budget allows for it, I highly recommend you hire one to take on the responsibilities. Having someone on set who is solely responsible for safely backing up and organizing footage helps keep the rest of the crew focused on their obligations to assure nothing goes wrong. When they’re not transferring and archiving data, DIT/DT’s also log metadata, color correct footage and help with the other preliminary editing processes. Even if the budget doesn’t allow for this position to be filled, work to find someone who can solely handle these processes while on set. You don’t want your camera operator to be in charge of also backing up and organizing footage if you can help it.

Ingest Software

If there is one piece of information we’d like for videographers and photographers to take away from this article, it is this: file-moving or ‘offloading’ software is worth the investment and should be used every time you shoot anything. For those who are unfamiliar with offload software, it’s any application that is designed to make it easier for you to back up footage from one location to another, and one shoot to another. In short, to avoid accidents or data corruption, it’s always best to have your media on a MINIMUM of two different devices. The easiest way to do this is to simply dump media onto two separate hard drives, and keep those drives separately stored. Ideally (if the budget allows), you’ll also keep all of your data on the original media for the day as well, making sure you have multiple copies stored in various locations. Many other options are available and recommended if possible, such as RAID arrays or even copying the data over to a cloud service such as Backblaze B2. What offloading software does is just this process, and helps build a platform of automation while verifying all the data as it’s transferred.

There are a few different recommendations I give for offloading software, all at different price points and with unique features. At the highest end of video production, you’ll often see DITs using a piece of software called Silverstack, which offers color grading functionalities, LTO tape support, and basic editing tools for creating daily edits. At a $600 annual price, it is the most expensive in this field and is probably overkill for most users. As for my recommendation, I recommend a tool call Shotput Pro. At $129, Shotput Pro offers all the tools you’d need to build a great archiving process while sacrificing some of the color editing tools. Shotput Pro can simultaneously copy and transfer files to multiple locations, build PDF reports, and verify all transfers. If you’re looking for something even cheaper, there are additional options such as Offload and Hedge. They’re both available for $99 each and give all the tools you’d need within their simple interfaces.

When it comes to photo, the two most obvious choices are Adobe Lightroom and Capture One Pro. While both tools are known more for their editing tools, they also have a lot of archiving functions built into their ingest systems, allowing you to unload cards to multiple locations and make copies on the fly.

workstation with video camera and RAID NAS

When it comes to video, the most crucial feature all of the apps should have is an option called “checksum verification.” This subject can get complicated, but all you really need to know is that larger files are more likely to be corrupted when transferring and copying, so what checksum verification does is verify the file to assure that it’s identical to the original version down to the individual byte. It is by far the most reliable and effective way to ensure that entire volumes of data are copied without corruption or loss of data. Whichever application you choose, make sure checksum verification is an available feature, and part of your workflow every time you’re copying video files. While available on select photo ingesting software, corruption happens less on smaller files and is generally less of an issue. Still, if possible, use it.

Post-Production

Once you’ve completed your shoot and all of your data is safely transfered over to external drives, it’s time to look at how you can store your information long term. Different people approach archiving in different ways because none of us will have an identical workflow. There is no correct way to handle how to archive your photos and videos, but there are a few rules that you’ll want to implement.

The first rule is the most obvious. You’ll want to make sure your media is stored on multiple drives. That way, if one of your drives dies on you, you still have a backup version of the work ready to go. The second rule of thumb is that you’ll want to store these backups in different locations. This can be extremely important if there is a fire in your office, or you’re a victim of a robbery. The most obvious way to do this is to back up or archive into a cloud service such as Backblaze B2. In my production experience I’ve seen multiple production houses implement a system where they store their backup hard drives in a safety deposit box at their bank. The final rule of thumb is especially important when you’re working with significant amounts of data, and that is to keep a working drive separate from an archive drive. The reason for this is an obvious one: all hard drives have a life expectancy, and you can prolong that by minimizing drive use. Having a working drive separate from your archive drives means that your archive drives will have fewer hours on them, thereby extending their practical life.

Ryan Hill’s Workflow

To help visualize what we discussed above, I’ll lay out my personal workflow for you. Please keep in mind that I’m mainly a one-man band, so my workflow is based on me handling everything. I’m also working with a large variety of mediums, so nothing I’m doing is going to be video and camera specific as all of my video projects, photo projects, and graphic projects are organized in the same way. I won’t bore you with details on my file structure, except to say that everything in my root folder is organized by job number, followed by sub-folders with the data classified into categories. I will keep track of which jobs are which, and have a Google Spreadsheet that organizes the job numbers with descriptions and client information. All of this information is secured within my Google account but also allows me to access it from anywhere if needed.

With archiving, my system is pretty simple. I’ve got a 4-drive RAID array in my office that gets updated every time I’m working on a new project. The array is set to RAID 1+0, which means I could lose two of the four hard drives, and still be able to recover the data. Usually, I’ll put 1TB drives in each bay, fill them as I work on projects, and replace them when they’re full. Once they’re full, I label them with the corresponding job numbers and store them in a plastic case on my bookshelf. By no means am I suggesting that my system is a perfect system, but for me, it’s incredibly adaptable to the various projects I work on. In case I was to get robbed, or if my house caught fire, I still have all of my work also archived onto a cloud system, giving me a second level of security.

Finally, to finish up my backup solution, I also keep a two-bay Thunderbolt hard drive dock on my desk as my working drive system. Solid state drives (SSD) and the Thunderbolt connection give me the speed and reliability that I’d need from a drive that I’ll be working from, and rendering outputs off of. For now, there is a single 960gb SSD in the first bay, with the option to extend to the second bay if I need additional storage. I start work by transferring the job file from my archive to the working drive, do whatever I need to do to the files, then replace the old job folder on my archive with the updated one at the end of the day. This way, if I were to have a drive failure, the worst I will lose is a day’s worth of work. For video projects or anything that takes a lot of data, I usually keep copies of all my source files on both my working and archive drive, and just replace the Adobe Premiere project file as I go. Again, this is just my system that works for me, and I recommend you develop one that works for your workflow while keeping your data safe.

The Takeaway

The critical point you should take away is that these sorts of strategies are things you should be thinking about at every step of your production. How does your camera or codec choice affect your media needs? How are you going to ensure safe data backup in the field? How are you going to work with all of this footage in post-production in a way that’s both secure and efficient? Answering all of these questions ahead of time will keep your media safe and your clients happy.

— Zach Sutton and Ryan Hill, lensrentals.com

The post Protecting Your Data From Camera to Archive appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

What’s the Diff: Backup vs Archive

Post Syndicated from Roderick Bauer original https://www.backblaze.com/blog/data-backup-vs-archive/

Whats the Diff: Backup vs Archive

Backups and archives serve different functions, yet it’s common to hear the terms used interchangeably in cloud storage. It’s important to understand the difference between the two to ensure that your data storage methodology meets your needs in a number of key areas:

  1. retained for the period of time you require
  2. protected from loss or unauthorized access
  3. able to be restored or retrieved when needed
  4. structured or tagged to enable locating specific data
  5. kept current according to your requirements

Our two choices can be broadly categorized:

  • backup is for recovery from hardware failure or recent data corruption or loss
  • archive is for space management and long term retention

What Is a Backup?

A backup is a copy of your data that is made to protect against loss of that data. Typically, backups are made on a regular basis according to a time schedule or when the original data changes. The original data is not deleted, but older backups are often deleted in favor of newer backups.

Data backup graphic

Desktop computers, servers, VMs, and mobile devices are all commonly backed up. Backups can include data, OS and application files, or a combination of these according to the backup methodology and purpose.

The goal of a backup is to make a copy of anything in current use that can’t afford to be lost. A backup of a desktop or mobile device might include just the user data so that a previous version of a file can be recovered if necessary. On these types of devices an assumption is often made that the OS and applications can easily be restored from original sources if necessary (and/or that restoring an OS to a new device could lead to significant corruption issues). In a virtual server environment, a backup could include .VMDK files that contain data and the OS as well as both structured (database) and unstructured data (files) so that the system can be put back into service as quickly as possible if something happens to the original VM in a VMware, Hyper-V, or other virtual machine environment.

In the case of a ransomware attack, a solid backup strategy can mean the difference between being able to restore a compromised system and having to pay a ransom in the vague hopes of getting a decryption key to obtain access to files that are no longer available because they were encrypted by the attacker.

Backups can have additional uses. A user might make go to a backup to retrieve an earlier version of a file because it contains something no longer in the current file, or, as is possible with some backup services such as Backblaze Backup, to share a file with a colleague or other person.

What Is an Archive?

An archive is a copy of data made for long-term storage and reference. The original data may or may not be deleted from the source system after the archive copy is made and stored, though it is common for the archive to be the only copy of the data.

Data archive graphic

In contrast to a backup whose purpose is to be able to return a computer or file system to a state it existed in previously, an archive can have multiple purposes. An archive can provide an individual or organization with a permanent record of important papers, legal documents, correspondence, and other matters. Often, an archive is used to meet information retention requirements for corporations and businesses. If a dispute or inquiry arises about a business practice, contract, financial transaction, or employee, the records pertaining to that subject can be obtained from the archive.

An archive is frequently used to ease the burden on faster and more frequently accessed data storage systems. Older data that is unlikely to be needed often is put on systems that don’t need to have the speed and accessibility of systems that contain data still in use. Archival storage systems are usually less expensive, as well, so a strong motivation is to save money on data storage.

Archives are often created based on the age of the data or whether the project the data belongs to is still active. An archiving program might send data to an archive if it hasn’t been accessed in a specified amount of time, when it has reached a certain age, if a person is no longer with the organization, or the files have been marked for storage because the project has been completed or closed.

Archives also can be created using metadata describing the project. An archiving program can automatically add relevant metadata, or the user can tag data manually to aid in future retrieval. Common metadata added can be business information describing the data, or in the case of photos and videos, the equipment, camera settings, and geographical location where the media was created. Artificial intelligence (AI) can be used to identify and catalog subject matter in some data such as photos and videos to make it easier to find the data at a later date. AI tools will become increasingly important as we archive more data and need to be able to find it based on parameters that might not be known at the time the data was archived.

What’s the Diff?

Backup Archive
Data backup graphic Data archive graphic
Enables rapid recovery of live, changing data Stores unchanging data no longer in use but must be retained
One of multiple copies of data Usually only remaining copy of data
Restore speed: crucial Retrieval speed: not crucial
Short Term Retention
Retained for as long as data is in active use
Long Term Retention
Retained for required period or indefinitely
Duplicate copies are periodically overwritten Data cannot be altered or deleted

What’s the Difference Between Restore and Retrieve?

In general backup systems restore and archive systems retrieve. The tools needed to perform these functions are different.

If you are interested in restoring something from a backup, it usually is a single file, a server, or structured data such as a database that needs to be restored to a specific point in time. You need to know a lot about the data, such as where it was located when it was backed up, the database or folder it was in, the name of the file, when it was backed up, and so forth.

When you retrieve data from an archive, the data is connected in some manner, such as date, email recipient, period of time, or other set of parameters that can be specified in a search. A typical retrieval query might be to obtain all files related to a project name, or all emails sent by a person during a specific period of time.

Trying to use a backup for an archive can present problems. You would need to keep rigorous records of where and when the files were backed up, what medium they were backed up to, and myriad other pieces of information that would need to be recorded at the time of backup. By definition, backup systems keep copies of data currently in use, so maintaining backups for lengthy periods of time go beyond the capabilities of backup systems and would require manual management.

The bottom line is don’t use a backup for an archive. Select the approach that suits your needs: a backup to keep additional copies of data currently in use in case something happens to your primary copy, or an archive to keep a permanent (and perhaps only record) of important data you wish to retain for personal, business, or legal reasons.

Why You Need Both Backup and Archive

It’s clear the a backup and an archive have different uses. Do you need both?

If you’re a business, the wise choice is yes. You need to make sure that your active business data is protected from accidental or malicious loss, and that your important records are maintained as long as necessary for business and legal reasons. If you are an individual or a small business with documents, photos, videos, and other media, you also need both backup and archive to ensure that your data is protected both short and long term and available and retrievable when you need it.

Data backup graphic & Data archive graphic

Selecting the right tools and services for backup and archiving is essential. Each have feature sets that make them suited to their tasks. Trying to use backup for archiving, or archiving for backup, is like trying to fit a round peg into a square hole. It’s best to use the right tool and service for the data storage function you require.

The post What’s the Diff: Backup vs Archive appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

B2 Quickstart Guide

Post Syndicated from Roderick Bauer original https://www.backblaze.com/blog/b2-quickstart-guide/

B2 Quickstart Guide
If you’re ready to get started with B2 Cloud Storage, these tips and resources will quickly get you up and running with B2.

What can you do with B2, our low-cost, fast, and easy object cloud storage service?

  • Creative professionals can archive their valuable photos and videos
  • Backup, archive, or sync servers and NAS devices
  • Replace tape backup systems (LTO)
  • Host and serve text, photos, videos, and other web content
  • Build apps that demand affordable cloud storage

B2 cloud storage logo

If you haven’t created an account yet, you can do that here.

Just For Newbies

Are you a beginner to B2? Here’s just what you need to get started.

Saving photos to the cloud

Developer or Old Hat at the Cloud?

If you’re a developer or more experienced with cloud storage, here are some resources just for you.

diagram of how to save files to the cloud

Have a NAS You’d Like to Link to the Cloud?

Would you like to get more out of your Synology, QNAP, or Morro Data NAS? Take a look at these posts that enable you to easily extend your local data storage to the cloud.

diagram of NAS to cloud backup

Looking for an Integration to Super Charge the Cloud?

We’ve blogged about the powerful integrations that work with B2 to provide solutions for a wide-range of backup, archiving, media management, and computing needs. The links below are just a start. You can visit our Integrations page or search our blog for the integration that interests you.

diagram of cloud computing integrations

We hope that gets you started. There’s plenty more about B2 on our blog in the “Cloud Storage” category and in our Knowledge Base.

Didn’t find what you need to get started with B2? Ask away in the comments.

The post B2 Quickstart Guide appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Five Cool Multi-Cloud Backup Solutions You Should Consider

Post Syndicated from Roderick Bauer original https://www.backblaze.com/blog/multi-cloud-backup-solutions/

Multi-Cloud backup

We came across Kevin Soltow’s blog, VMWare Blog, and invited him to contribute a post on a topic that’s getting a lot of attention these days: using multiple public clouds for storing data to increase data redundancy and also to save money. We hope you enjoy the post.
Kevin Soltow
Kevin lives in Zumikon, Switzerland where he works as a Storage Consultant specializing in VMware technologies and storage virtualization. He gradudated from Swiss Federal Institute of Technology in Zurich, and now works on implementing disaster recovery and virtual SAN solutions.

Nowadays, it’s hard to find a company without a backup or DR strategy in place. An organization’s data has become one of its most important assets, so making sure it remains safe and available are key priorities. But does it really matter where your backups are stored? If you follow the “3-2-1” backup rule, you know the answer. You should have at least three copies of your data, two of which are local but on different media, and at least one copy is offsite. That all sounds reasonable.

What about the devices to store your backup data?

Tapes — Tapes were first, had large capacity, the ability to keep data for a long time, but unfortunately they were slow. Historically, they have been less expensive than disks.

Disks — Disks have great capacity, are more durable and faster than tapes, and are improving rapidly in capacity, speed, and cost-per-unit-stored.

In a previous post, Looking for the most affordable cloud storage? AWS vs Azure vs Backblaze B2, I looked at the cost of public cloud storage. With reasonably-priced cloud services available, cloud can be that perfect offsite option for keeping your data safe.

No cloud storage provider can guarantee 100% accessibility and security. Sure, they get close to this number, with claims of 99-point-some-number-of-nines durability, but an unexpected power outage or disaster could knock out their services for minutes, hours, or longer. This happened last year to Amazon S3 when the service suffered from a service disruption. This year, S3 was down due to a power outage. Fortunately, Amazon did not lose their customers’ data. The key words in the 3-2-1 backup rule are at least one copy offsite. More is always better.

Keeping data in multiple clouds provides a clear advantage for reliability, but it also can provide a cost savings, as well. Using multiple cloud providers can simultaneously provide geographically dispersed backups while taking advantage of the lower storage costs available from competitively-priced cloud providers.

In this post, we take a closer look at solutions that support multiple public clouds and allow you to keep several backup copies in different and dispersed clouds.

The Backup Process

The backup process is illustrated in the figure below:

diagram of the backup process from local to cloud

Some solutions create backups and move them to the repository. Data is kept there for a while and then shifted to the cloud where it stays as long as needed.

In this post, I discuss the dedicated software serving as a “data router,” in other words, the software involved in the process of moving data from some local repository to one or more public clouds.

software to send backups to cloud diagram

Let’s have a look at the options we have to achieve this.

1 — Rclone

When I considered solutions that let you back up your data to several clouds, Rclone and CloudBerry were the first solutions that popped into my head. Rclone acts as a data mover synchronizing your local repository with cloud-based object storage. You basically create a backup using something else (e.g. Veeam Backup & Replication), allocate it on-premises, and the solution sends it to several clouds. First developed for Linux, Rclone has a command-line interface to sync files and directories between clouds.

OS Compatibility

The solution can be run on all OS’s using the command-line interface.

Cloud Support

The solution works with most popular public cloud storage platforms, such as Backblaze B2, Microsoft Azure, Amazon S3 and Glacier, Google Cloud Platform, etc.

Feature set

Rclone commands work wonderfully on whatever remote storage system, be it public cloud storage or just your backup server located somewhere else. It also can send data to multiple places simultaneously, but bi-directional sync does not work yet. In other words, changes you make to your files in the cloud do not affect their local copies. The synchronization process is incremental on the file-by-file basis. It should also be noted that Rclone preserves timestamps on files, which helps when searching for the right backup.

This solution provides two options for moving data to the cloud: sync and copy. The first one, sync, allows moving the backups to the cloud automatically as soon as they appear in the specified local directory. The second mode, copy, as expected from its name, allows only copying data from on-premises to the cloud. Deleting your files locally won’t affect the ones stored in the cloud. There’s also an option to verify hash equality.

Learn more about Rclone: https://rclone.org/

2 — CloudBerry Backup

CloudBerry Backup is built from the self-titled backup technology developed for service providers and enterprise IT departments. It is a cross-platform solution. Note that it’s full-fledged backup software, allowing you to not only move backups to the cloud but also create them.

OS compatibility

CloudBerry is a cross-platform solution.

Cloud Support

So far, the solution can talk to Backblaze B2, Microsoft Azure, Amazon S3 and Glacier, Google Cloud Platform, and more.

Feature set

Designed for large IT departments and managed service providers, CloudBerry Backup provides some features that make the solution really handy for the big guys. It offers the opportunity for client customization up to and including the complete rebranding of the solution.

Let’s look at the backup side of this thing. The solution allows backing up files and directories manually. If you prefer, you can sync the selected directory to the root of the bucket. CloudBerry Backup also can schedule backups. Now, you won’t miss them! Another cool thing is backup jobs management and monitoring. Thanks to this feature you are always aware of backup processes on the client machines.

The solution offers AES 256-bit end-to-end encryption to ensure your data safety.

Learn more about CloudBerry Backup: https://www.cloudberrylab.com/managed-backup.aspx

Read Backblaze’s blog post, How To Back Up Your NAS with B2 and CloudBerry.

3 — StarWind VTL

Some organizations still use Virtual Tape Library (VTL), but want to sync their tape objects to the cloud as well.

OS compatibility

This product is available only for Windows.

Cloud Support

So far, StarWind VTL can talk to popular cloud storage platforms like Backblaze B2, AWS S3 and Glacier, and Microsoft Azure.

Feature set

The product has many features for anyone who wants to back up to the cloud. First, it allows sending data to the cloud’s respective tier with their subsequent automatic de-staging. This automation makes StarWind VTL really cool. Second, the product supports both on-premises and public cloud object storage. Third, StarWind VTL supports deduplication and compression, making storage utilization more efficient. This solution also allows client-side encryption.

StarWind supports standard “LTO” (Linear Tape-Open) protocols. This appeals to organizations that have LTO in place since it allows adoption of more scalable, cost efficient cloud storage without having to update the internal backup infrastructure.

All operations in the StarWind VTL environment are done via the Management Console and Web-Console, the web-interface that makes VTL compatible with all browsers.

Learn more about StarWind Virtual Tape Library: https://www.starwindsoftware.com/starwind-virtual-tape-library

Also, see Backblaze’s post on StarWind VTL: Connect Veeam to the B2 Cloud: Episode 2 — Using StarWind VTL

4 — Duplicati

Duplicati is designed for online backups from scratch. It also can send your data directly to multiple clouds or use local storage as a backend.

OS compatibility

It is free and compatible with Windows, macOS, and Linux.

Cloud Support

So far, the solution talks to Backblaze B2, Amazon S3, Mega, Google Cloud Storage, and Microsoft Azure.

Feature set

Duplicati has some awesome features. First, the solution is free. Notably, its team does not restrict using this software for free even for commercial purposes. Second, Duplicati employs decent encryption, compression, and deduplication, making your storage more efficient and safe. Third, the solution adds timestamps to your files, so you can easily find the specific backup. Fourth, the backup scheduler helps make users’ lives simpler. Now, you won’t miss the backup time!

What makes this piece of software special and really handy is backup content verification. Indeed, you never know whether the backup works out until you literally back up from it. Thanks to this feature, you can pinpoint any problems before it is too late.

Duplicati is managed via a web interface, making it possible to access from anywhere and any platform.

Learn more about Duplicati: https://www.duplicati.com/.

Read Backblaze’s post on Duplicati: Duplicati, a Free, Cross-Platform Client for B2 Cloud Storage.

5 — Duplicacy

Duplicacy supports popular public cloud storage services. Apart from the cloud, it can use SFTP servers and NAS boxes as backends.

OS compatibility

The solution is compatible with Windows, Mac OS X, and Linux.

Cloud Support

Duplicacy can offload data to Backblaze B2, Amazon S3, Google Cloud Storage, Microsoft Azure, and more.

Feature set

Duplicacy not only routes your backups to the cloud but also creates them. Note that each backup created by this solution is incremental. Each of them is treated as a full snapshot, allowing simpler restoration, deletion, and backup transition between storage sites. Duplicacy sends your files to multiple cloud storages and uses strong client-side encryption. Another cool thing about this solution is its ability to provide multiple clients with simultaneous access to the same storage.

Duplicacy has a comprehensive GUI that features one-page configuration for quick backup scheduling and managing retention policies. If you are a command-line interface fan, you can manage Duplicacy via the command line.

Learn more about Duplicacy: https://duplicacy.com/

Read Backblaze’s Knowledge Base article, How to upload files to B2 with Duplicacy.

So, Should You Store Your Data in More Than One Cloud?

Undoubtedly, keeping a copy of your data in the public cloud is a great idea and enables you to comply with the 3-2-1 backup rule. By going beyond that and adopting a multi-cloud strategy, it is possible to save money and also gain additional data redundancy and security by having data in more than one public cloud service.

As I’ve covered in this post, there are a number of wonderful backup solutions that can talk to multiple public cloud storage services. I hope this article proves useful to you and you’ll consider employing one of the reviewed solutions in your backup infrastructure.

Kevin Soltow

The post Five Cool Multi-Cloud Backup Solutions You Should Consider appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.