Deduplication is simply the process of eliminating redundant data on disk. Deduplication reduces storage space requirements, improves backup speed, and lowers backup storage costs. The dedup field used to be dominated by a few big-name vendors who sold dedup systems that were too expensive for most of the SMB market. Then an open-source challenger came along in OpenDedup, a project that produced the Space Deduplication File System (SDFS). SDFS provides many of the features of commercial dedup products without their cost.
OpenDedup provides inline deduplication that can be used with applications such as Veeam, Veritas Backup Exec, and Veritas NetBackup.
Features Supported by OpenDedup:
Variable Block Deduplication to cloud storage
Local Data Caching
Fast Cloud Recovery
Windows and Linux Support
Why use Veeam with OpenDedup to Backblaze B2?
With your VMs backed up to B2, you have a number of options to recover from a disaster. If the unexpected occurs, you can quickly restore your VMs from B2 to the location of your choosing. You also have the option to bring up cloud compute through B2’s compute partners, thereby minimizing any loss of service and ensuring business continuity.
Backblaze’s B2 is an ideal solution for backing up Veeam’s backup repository due to B2’s combination of low-cost and high availability. Users of B2 save up to 75% compared to other cloud solutions such as Microsoft Azure, Amazon AWS, or Google Cloud Storage. When combined with OpenDedup’s no-cost deduplication, you’re got an efficient and economical solution for backing up VMs to the cloud.
How to Use OpenDedup with B2
For step-by-step instructions for how to set up OpenDedup for use with B2 on Windows or Linux, see Backblaze B2 Enabled on the OpenDedup website.
Are you backing up Veeam to B2 using one of the solutions we’ve written about in this series? If you have, we’d love to hear from you in the comments.
Microsoft has issued a press release describing the security dangers involved with the Internet of things (“a weaponized stove, baby monitors that spy, the contents of your refrigerator being held for ransom“) and introducing “Microsoft Azure Sphere” as a combination of hardware and software to address the problem. “Unlike the RTOSes common to MCUs today, our defense-in-depth IoT OS offers multiple layers of security. It combines security innovations pioneered in Windows, a security monitor, and a custom Linux kernel to create a highly-secured software environment and a trustworthy platform for new IoT experiences.”
American Public Television was like many organizations that have been around for a while. They were entrenched using an older technology — in their case, tape storage and distribution — that once met their needs but was limiting their productivity and preventing them from effectively collaborating with their many media partners. APT’s VP of Technology knew that he needed to move into the future and embrace cloud storage to keep APT ahead of the game.
Since 1961, American Public Television (APT) has been a leading distributor of groundbreaking, high-quality, top-rated programming to the nation’s public television stations. Gerry Field is the Vice President of Technology at APT and is responsible for delivering their extensive program catalog to 350+ public television stations nationwide.
In the time since Gerry joined APT in 2007, the industry has been in digital overdrive. During that time APT has continued to acquire and distribute the best in public television programming to their technically diverse subscribers.
This created two challenges for Gerry. First, new technology and format proliferation were driving dramatic increases in digital storage. Second, many of APT’s subscribers struggled to keep up with the rapidly changing industry. While some subscribers had state-of-the-art satellite systems to receive programming, others had to wait for the post office to drop off programs recorded on tape weeks earlier. With no slowdown on the horizon of innovation in the industry, Gerry knew that his storage and distribution systems would reach a crossroads in no time at all.
Living the tape paradigm
The digital media industry is only a few years removed from its film, and later videotape, roots. Tape was the input and the output of the industry for many years. As a consequence, the tools and workflows used by the industry were built and designed to work with tape. Over time, the “file” slowly replaced the tape as the object to be captured, edited, stored and distributed. Trouble was, many of the systems and more importantly workflows were based on processing tape, and these have proven to be hard to change.
At APT, Gerry realized the limits of the tape paradigm and began looking for technologies and solutions that enabled workflows based on file and object based storage and distribution.
Thinking file based storage and distribution
For data (digital media) storage, APT, like everyone else, started by installing onsite storage servers. As the amount of digital data grew, more storage was added. In addition, APT was expanding its distribution footprint by creating or partnering with distribution channels such as CreateTV and APT Worldwide. This dramatically increased the number of programming formats and the amount of data that had to be stored. As a consequence, updating, maintaining, and managing the APT storage systems was becoming a major challenge and a major resource hog.
Knowing that his in-house storage system was only going to cost more time and money, Gerry decided it was time to look at cloud storage. But that wasn’t the only reason he looked at the cloud. While most people consider cloud storage as just a place to back up and archive files, Gerry was envisioning how the ubiquity of the cloud could help solve his distribution challenges. The trouble was the price of cloud storage from vendors like Amazon S3 and Microsoft Azure was a non-starter, especially for a non-profit. Then Gerry came across Backblaze. B2 Cloud Storage service met all of his performance requirements, and at $0.005/GB/month for storage and $0.01/GB for downloads it was nearly 75% less than S3 or Azure.
Gerry did the math and found that he could economically incorporate B2 Cloud Storage into his IT portfolio, using it for both program submission and for active storage and archiving of the APT programs. In addition, B2 now gives him the foundation necessary to receive and distribute programming content over the Internet. This is especially useful for organizations that can’t conveniently access satellite distribution systems. Not to mention downloading from the cloud is much faster than sending a tape through the mail.
Adding B2 Cloud Storage to their infrastructure has helped American Public Television address two key challenges. First, they now have “unlimited” storage in the cloud without having to add any hardware. In addition, with B2, they only pay for the storage they use. That means they don’t have to buy storage upfront trying to match the maximum amount of storage they’ll ever need. Second, by using B2 as a distribution source for their programming APT subscribers, especially the smaller and remote ones, can get content faster and more reliably without having to perform costly upgrades to their infrastructure.
The road ahead
As APT gets used to their file based infrastructure and workflow, there are a number of cost saving and income generating ideas they are pondering which are now worth considering. Here are a few:
Program Submissions — New content can be uploaded from anywhere using a web browser, an Internet connection, and a login. For example, a producer in Cambodia can upload their film to B2. From there the film is downloaded to an in-house system where it is processed and transcoded using compute. The finished film is added to the APT catalog and added to B2. Once there, the program is instantly available for subscribers to order and download.
“The affordability and performance of Backblaze B2 is what allowed us to make the B2 cloud part of the APT data storage and distribution strategy into the future.” — Gerry Field
Easier Previews — At any time, work in process or finished programs can be made available for download from the B2 cloud. One place this could be useful is where a subscriber needs to review a program to comply with local policies and practices before airing. In the old system, each “one-off” was a time consuming manual process.
Instant Subscriptions — There are many organizations such as schools and businesses that want to use just one episode of a desired show. With an e-commerce based website, current or even archived programming kept in B2 could be available to download or stream for a minimal charge.
At APT there were multiple technologies needed to make their file-based infrastructure work, but as Gerry notes, having an affordable, trustworthy, cloud storage service like B2 is one of the critical building blocks needed to make everything work together.
In 2015, we announced Backblaze B2 Cloud Storage — the most affordable, high performance storage cloud on the planet. The decision to release B2 as a service was in direct response to customers asking us if they could use the same cloud storage infrastructure we use for our Computer Backup service. With B2, we entered a market in direct competition with Amazon S3, Google Cloud Services, and Microsoft Azure Storage. Today, we have over 500 petabytes of data from customers in over 150 countries. At $0.005 / GB / month for storage (1/4th of S3) and $0.01 / GB for downloads (1/5th of S3), it turns out there’s a healthy market for cloud storage that’s easy and affordable.
As B2 has grown, customers wanted to use our cloud storage for a variety of use cases that required not only storage but compute. We’re happy to say that through partnerships with Packet & ServerCentral, today we’re announcing that compute is now available for B2 customers.
Cloud Compute and Storage
Backblaze has directly connected B2 with the compute servers of Packet and ServerCentral, thereby allowing near-instant (< 10 ms) data transfers between services. Also, transferring data between B2 and both our compute partners is free.
Storing data in B2 and want to run an AI analysis on it? — There are no fees to move the data to our compute partners.
Generating data in an application? — Run the application with one of our partners and store it in B2.
Transfers are free and you’ll save more than 50% off of the equivalent set of services from AWS.
These partnerships enable B2 customers to use compute, give our compute partners’ customers access to cloud storage, and introduce new customers to industry-leading storage and compute — all with high-performance, low-latency, and low-cost.
Is This a Big Deal? We Think So
Compute is one of the most requested services from our customers Why? Because it unlocks a number of use cases for them. Let’s look at three popular examples:
Transcoding Media Files
B2 has earned wide adoption in the Media & Entertainment (“M&E”) industry. Our affordable storage and download pricing make B2 great for a wide variety of M&E use cases. But many M&E workflows require compute. Content syndicators, like American Public Television, need the ability to transcode files to meet localization and distribution management requirements.
There are a multitude of reasons that transcode is needed — thumbnail and proxy generation enable M&E professionals to work efficiently. Without compute, the act of transcoding files remains cumbersome. Either the files need to be brought down from the cloud, transcoded, and then pushed back up or they must be kept locally until the project is complete. Both scenarios are inefficient.
Starting today, any content producer can spin up compute with one of our partners, pay by the hour for their transcode processing, and return the new media files to B2 for storage and distribution. The company saves money, moves faster, and ensures their files are safe and secure.
Backblaze’s heritage is based on providing outstanding backup services. When you have incredibly affordable cloud storage, it ends up being a great destination for your backup data.
Most enterprises have virtual machines (“VMs”) running in their infrastructure and those VMs need to be backed up. In a disaster scenario, a business wants to know they can get back up and running quickly.
With all data stored in B2, a business can get up and running quickly. Simply restore your backed up VM to one of our compute providers, and your business will be able to get back online.
Since B2 does not place restrictions, delays, or penalties on getting data out, customers can get back up and running quickly and affordably.
Saving $74 Million (aka “The Dropbox Effect”)
Ten years ago, Backblaze decided that S3 was too costly a platform to build its cloud storage business. Instead, we created the Backblaze Storage Pod and our own cloud storage infrastructure. That decision enabled us to offer our customers storage at a previously unavailable price point and maintain those prices for over a decade. It also laid the foundation for Netflix Open Connect and Facebook Open Compute.
Dropbox recently migrated the majority of their cloud services off of AWS and onto Dropbox’s own infrastructure. By leaving AWS, Dropbox was able to build out their own data centers and still save over $74 Million. They achieved those savings by avoiding the fees AWS charges for storing and downloading data, which, incidentally, are five times higher than Backblaze B2.
For Dropbox, being able to realize savings was possible because they have access to enough capital and expertise that they can build out their own infrastructure. For companies that have such resources and scale, that’s a great answer.
“Before this offering, the economics of the cloud would have made our business simply unviable.” — Gabriel Menegatti, SlicingDice
The questions Backblaze and our compute partners pondered was “how can we democratize the Dropbox effect for our storage and compute customers? How can we help customers do more and pay less?” The answer we came up with was to connect Backblaze’s B2 storage with strategic compute partners and remove any transfer fees between them. You may not save $74 million as Dropbox did, but you can choose the optimal providers for your use case and realize significant savings in the process.
This Sounds Good — Tell Me More About Your Partners
We’re very fortunate to be launching our compute program with two fantastic partners in Packet and ServerCentral. These partners allow us to offer a range of computing services.
We recommend Packet for customers that need on-demand, high performance, bare metal servers available by the hour. They also have robust offerings for private / customized deployments. Their offerings end up costing 50-75% of the equivalent offerings from EC2.
To get started with Packet and B2, visit our partner page on Packet.net.
ServerCentral is the right partner for customers that have business and IT challenges that require more than “just” hardware. They specialize in fully managed, custom cloud solutions that solve complex business and IT challenges. ServerCentral also has expertise in managed network solutions to address global connectivity and content delivery.
To get started with ServerCentral and B2, visit our partner page on ServerCentral.com.
We’re excited to find out. The combination of B2 and compute unlocks use cases that were previously impossible or at least unaffordable.
“The combination of performance and price offered by this partnership enables me to create an entirely new business line. Before this offering, the economics of the cloud would have made our business simply unviable,” noted Gabriel Menegatti, co-founder at SlicingDice, a serverless data warehousing service. “Knowing that transfers between compute and B2 are free means I don’t have to worry about my business being successful. And, with download pricing from B2 at just $0.01 GB, I know I’m avoiding a 400% tax from AWS on data I retrieve.”
What can you do with B2 & compute? Please share your ideas with us in the comments. And, for those attending NAB 2018 in Las Vegas next week, please come by and say hello!
We’ve spent the last month making changes to Backblaze B2. We’ve reduced the B2 Download Prices in Half, expanded on our Snapshot USB Restore program by offering refunds if the hard drives are shipped back to us, and have built out our Backblaze Fireball program into a self-service model where you can seed 70TBs of data into your Backblaze B2 account. For any other cloud storage company, all of these value-adds would be enough, but we noticed that something was missing.
We kept hearing from our customers that we were simply doing too much and not charging enough. People were worried about our ability to stay in the market, despite our track record over the last 10 years of providing low cost storage, all while operating a cash-flow positive business. Our customers simply couldn’t believe that we could keep this charade going for much longer, and demanded that we do something to bolster our financial stability and to “stop giving everything away — practically for free,” even if it meant that we would make more money.
We listened, and today we are proud to announce a new service that compliments our wildly popular B2 Cloud Storage: Backblaze Bling2 Cloud Storage. It’s very similar to Backblaze B2, identical in fact, except for one minor change. It’s 4x more expensive for both storage and downloads, just like our competitors! We’re confident that the same level of service for 4x the price will appeal to our users who think that we’re simply not charging enough.
If you’re interested in this Bling2, we’ve made a tool to help you calculate your storage costs with Bling2 Cloud Storage, and compare it to leading cloud storage providers such as Backblaze B2, Amazon S3, Google Cloud Service, and Microsoft Azure!
We hope you enjoy this new service from Backblaze. If you think that Backblaze B2 is too affordable, you’ll be happy to know that Bling2 storage prices are available to you at the “industry standard” 4x markup. Why pay less when you can Bling2?!
Backblaze is pleased to announce that, effective immediately, we are reducing the price of Backblaze B2 Cloud Storage downloads by 50%. This means that B2 download pricing drops from $0.02 to $0.01 per GB. As always, the first gigabyte of data downloaded each day remains free.
If some of this sounds familiar, that’s because a little under a year ago, we dropped our download price from $0.05 to $0.02. While that move solidified our position as the affordability leader in the high performance cloud storage space, we continue to innovate on our platform and are excited to provide this additional value to our customers.
This price reduction applies immediately to all existing and new customers. In keeping with Backblaze’s overall approach to providing services, there are no tiers or minimums. It’s automatic and it starts today.
Why Is Backblaze Lowering What Is Already The Industry’s Lowest Price?
Because it makes cloud storage more useful for more people.
When we decided to use Backblaze B2 as our cloud storage service, their download pricing at the time enabled us to offer our broadcasters unlimited audio uploads so they can upload past decades of preaching to our extensive library for streaming and downloading. With Backblaze cutting the bandwidth prices 50% to just one penny a gigabyte, we are excited about offering much higher quality video. — Ian Wagner, Senior Developer, Sermon Audio
Since our founding in 2007, Backblaze’s mission has been to make storing data astonishingly easy and affordable. We have a well documented, relentless pursuit of lowering storage costs — it starts with our storage pods and runs through everything we do. Today, we have over 500 petabytes of customer data stored. B2’s storage pricing already being 1⁄4 that of Amazon’s S3 has certainly helped us get there. Today’s pricing reduction puts our download pricing 1⁄5 that of S3. The “affordable” part of our story is well established.
I’d like to take a moment to discuss the “easy” part. Our industry has historically done a poor job of putting ourselves in our customers’ shoes. When customers are faced with the decision of where to put their data, price is certainly a factor. But it’s not just the price of storage that customers must consider. There’s a cost to download your data. The business need for providers to charge for this is reasonable — downloading data requires bandwidth, and bandwidth costs money. We discussed that in a prior post on the Cost of Cloud Storage.
But there’s a difference between the costs of bandwidth and what the industry is charging today. There’s a joke that some of the storage clouds are competing to become “Hotel California” — you can check out anytime you want, but your data can never leave.1 Services that make it expensive to restore data or place time lag impediments to data access are reducing the usefulness of your data. Customers should not have to wonder if they can afford to access their own data.
When replacing LTO with StarWind VTL and cloud storage, our customers had only one concern left: the possible cost of data retrieval. Backblaze just wiped this concern out of the way by lowering that cost to just one penny per gig. — Max Kolomyeytsev, Director of Product Management, StarWind
Many businesses have not yet been able to back up their data to the cloud because of the costs. Many of those companies are forced to continue backing up to tape. That tape is an inefficient means for data storage is clear. Solution providers like StarWind VTL specialize in helping businesses move off of antiquated tape libraries. However, as Max Kolomyeytsev, Director of Product Management at StarWind points out, “When replacing LTO with StarWind VTL and cloud storage our customers had only one concern left: the possible cost of data retrieval. Backblaze just wiped this concern out of the way by lowering that cost to just one penny per gig.”
Customers that have already adopted the cloud often are forced to make difficult tradeoffs between data they want to access and the cost associated with that access. Surrendering the use of your own data defeats many of the benefits that “the cloud” brings in the first place. Because of B2’s download price, Ian Wagner, a Senior Developer at Sermon Audio, is able to lower his costs and expand his product offering. “When we decided to use Backblaze B2 as our cloud storage service, their download pricing at the time enabled us to offer our broadcasters unlimited audio uploads so they can upload past decades of preaching to our extensive library for streaming and downloading. With Backblaze cutting the bandwidth prices 50% to just one penny a gigabyte, we are excited about offering much higher quality video.”
Better Download Pricing Also Helps Third Party Applications Deliver Customer Solutions
Many organizations use third party applications or devices to help manage their workflows. Those applications are the hub for customers getting their data to where it needs to go. Leaders in verticals like Media Asset Management, Server & NAS Backup, and Enterprise Storage have already chosen to integrate with B2.
With Backblaze lowering their download price to an amazing one penny a gigabyte, our CloudNAS is even a better fit for photographers, videographers and business owners who need to have their files at their fingertips, with an easy, reliable, low cost way to use Backblaze for unlimited primary storage and active archive. — Paul Tian, CEO, Morro Data
For Paul Tian, founder of Ready NAS and CEO of Morro Data, reasonable download pricing also helps his company better serve its customers. “With Backblaze lowering their download price to an amazing one penny a gigabyte, our CloudNAS is even a better fit for photographers, videographers and business owners who need to have their files at their fingertips, with an easy, reliable, low cost way to use Backblaze for unlimited primary storage and active archive.”
If you use an application that hasn’t yet integrated with B2, please ask your provider to add B2 Cloud Storage and mention the application in the comments below.
How Do the Major Cloud Storage Providers Compare on Pricing?
Not only is Backblaze B2 storage 1⁄4 the price of Amazon S3, Google Cloud, or Azure, but our download pricing is now 1⁄5 their price as well.
First 1 TB
Next 9 TB
Next 40 TB
Next 100 TB
Next 350 TB+
Using the chart above, let’s compute a few examples of download costs…
Not only is Backblaze B2 pricing dramatically lower cost, it’s also simple — one price for any amount of data downloaded to anywhere. In comparison, to compute the cost of downloading 500 TB of data with S3 you start with the following formula: (($0.09 * 10) + ($0.085 * 40) + ($0.07 * 100) + ($0.05 * 350)) * 1,000 Want to see this comparison for the amount of data you manage? Use our cloud storage calculator.
Customers Want to Avoid Vendor Lock In
Halving the price of downloads is a crazy move — the kind of crazy our customers will be excited about. When using our Transmit 5 app on the Mac to upload their data to B2 Cloud Storage, our users can sleep soundly knowing they’ll be getting a truly affordable price when they need to restore that data. Cool beans, Backblaze. — Cabel Sasser, Co-Founder, Panic
As the cloud storage industry grows, customers are increasingly concerned with getting locked in to one vendor. No business wants to be fully dependent on one vendor for anything. In addition, customers want multiple copies of their data to mitigate against a vendor outage or other issues.
Many vendors offer the ability for customers to replicate data across “regions.” This enables customers to store data in two physical locations of the customer’s choosing. Of course, customers pay for storing both copies of the data and for the data transfer between regions.
At 1¢ per GB, transferring data out of Backblaze is more affordable than transferring data between most other vendor regions. For example, if a customer is storing data in Amazon S3’s Northern California region (US West) and wants to replicate data to S3 in Northern Virginia (US East), she will pay 2¢ per GB to simply move the data.
However, if that same customer wanted to replicate data from Backblaze B2 to S3 in Northern Virginia, she would pay 1¢ per GB to move the data. She can achieve her replication strategy while also mitigating against vendor risk — all while cutting the bandwidth bill by 50%. Of course, this is also before factoring the savings on her storage bill as B2 storage is 1⁄4 of the price of S3.
How Is Backblaze Doing This?
Simple. We just changed our pricing table and updated our website.
The longer answer is that the cost of bandwidth is a function of a few factors, including how it’s being used and the volume of usage. With another year of data for B2, over a decade of experience in the cloud storage industry, and data growth exceeding 100 PB per quarter, we know we can sustainably offer this pricing to our customers; we also know how better download pricing can make our customers and partners more effective in their work. So it is an easy call to make.
Our pricing is simple. Storage is $0.005/GB/Month, Download costs are $0.01/GB. There are no tiers or minimums and you can get started any time you wish.
Our desire is to provide a great service at a fair price. We’re proud to be the affordability leader in the Cloud Storage space and hope you’ll give us the opportunity to show you what B2 Cloud Storage can enable for you.
Enjoy the service and I’d love to hear what this price reduction does for you in the comments below…or, if you are attending NAB this year, come by to visit and tell us in person!
1 For those readers who don’t get the Eagles reference there, please click here…I promise you won’t regret the next 7 minutes of your life.
StarWind provides “VTL” (Virtual Tape Library) technology that enables users to back up their “VMs” (virtual machines) from Veeam to on-premise or cloud storage. StarWind does this using standard “LTO” (Linear Tape-Open) protocols. This appeals to organizations that have LTO in place since it allows adoption of more scalable, cost efficient cloud storage without having to update the internal backup infrastructure.
Why An Additional Backup in the Cloud?
Common backup strategy, known as 3-2-1, dictates having three copies at a minimum of active data. Two copies are stored locally and one copy is in another location.
Relying solely on on-site redundancy does not guarantee data protection after a catastrophic or temporary loss of service affecting the primary data center. To reach maximum data security, an on-premises private cloud backup combined with an off-site public cloud backup, known as hybrid cloud, provides the best combination of security and rapid recovery when required.
Why Consider a Hybrid Cloud Solution?
The Hybrid Cloud Provides Superior Disaster Recovery and Business Continuity
Having a backup strategy that combines on-premise storage with public cloud storage in a single or multi-cloud configuration is becoming the solution of choice for organizations that wish to eliminate dependence on vulnerable on-premises storage. It also provides reliable and rapidly deployed recovery when needed.
If an organization requires restoration of service as quickly as possible after an outage or disaster, it needs to have a backup that isn’t dependent on the same network. That means a backup stored in the cloud that can be restored to another location or cloud-based compute service and put into service immediately after an outage.
Hybrid Cloud Example: VTL and the Cloud
Some organizations will already have made a significant investment in software and hardware that supports LTO protocols. Specifically, they are using Veeam to back up their VMs onto physical tape. Using StarWind to act as a VTL with Veeam enables users to save time and money by connecting their on-premises Veeam Backup & Replication archives to Backblaze B2 Cloud Storage.
Why Veeam, StarWind VTL, and Backblaze B2?
What are the primary reasons that an organization would want to adopt Veeam + StarWind VTL + B2 as a hybrid cloud backup solution?
You are already invested in Veeam along with LTO software and hardware.
Using Veeam plus StarWind VTL with already-existing LTO infrastructure enables organizations to quickly and cost-effectively benefit from cloud storage.
You require rapid and reliable recovery of service should anything disrupt your primary data center.
Having a backup in the cloud with B2 provides an economical primary or secondary cloud storage solution and enables fast restoration to a current or alternate location, as well as providing the option to quickly bring online a cloud-based compute service, thereby minimizing any loss of service and ensuring business continuity. Backblaze’s B2 is an ideal solution for backing up Veeam’s backup repository due to B2’s combination of low-cost and high availability compared to other cloud solutions such as Microsoft Azure or Amazon AWS.
Using Veeam, StarWind VTL, and Backblaze B2 cloud storage is a superior alternative to tape as B2 offers better economics, instant access, and faster recovery.
Connect Veeam to the Backblaze B2 Cloud using StarWind VTL (graphic courtesy of StarWind)
Veeam is well-known for its easy-to-use software for backing up virtual machines from VMware and Microsoft.
Users of Veeam and Backblaze B2 Cloud Storage have asked for a way to back up a Veeam repository to B2. Backblaze’s B2 is an ideal solution for backing up Veeam’s backup repository due to B2’s combination of low-cost and high availability compared to other cloud solutions such as Microsoft Azure.
This is the first in a series of posts on the topic of backing up Veeam to B2. Future posts will cover other methods.
In this post we provide a step-by-step tutorial on how to configure a Synology NAS as a Veeam backup repository, and in turn use Synology’s CloudSync software to back up that repository to the B2 Cloud.
Our guest contributor, Rhys Hammond, is well qualified to author this tutorial. Rhys is a Senior System Engineer for Data#3 in Australia specializing in Veeam and VMware solutions. He is a VMware vExpert and a member of the Veeam Vanguard program.
If you back up Veeam using Starwind VTL, we have a BETA program for you. Help us with the Starwind VTL to Backblaze B2 integration Beta and test whether you can automatically back up Veeam to Backblaze B2 via Starwind VTL. Motivated beta testers can email email@example.com for details and how to get started.
Over the past several months, B2 Cloud Storage has continued to grow like we planted magic beans. During that time we have added a B2 Java SDK, and certified integrations with GoodSync, Arq, Panic, UpdraftPlus, Morro Data, QNAP, Archiware, Restic, and more. In addition, B2 customers like Panna Cooking, Sermon Audio, and Fellowship Church are happy they chose B2 as their cloud storage provider. If any of that sounds interesting, read on.
The B2 Java SDK
While the Backblaze B2 API is well documented and straight-forward to implement, we were asked by a few of our Integration Partners if we had an SDK they could use. So we developed one as an open-course project on GitHub, where we hope interested parties will not only use our Java SDK, but make it better for everyone else.
There are different reasons one might use the Java SDK, but a couple of areas where the SDK can simplify the coding process are:
Expiring Authorization — B2 requires an application key for a given account be reissued once a day when using the API. If the application key expires while you are in the middle of transferring files or some other B2 activity (bucket list, etc.), the SDK can be used to detect and then update the application key on the fly. Your B2 related activities will continue without incident and without having to capture and code your own exception case.
Error Handling — There are different types of error codes B2 will return, from expired application keys to detecting malformed requests to command time-outs. The SDK can dramatically simplify the coding needed to capture and account for the various things that can happen.
While Backblaze has created the Java SDK, developers in the GitHub community have also created other SDKs for B2, for example, for PHP (https://github.com/cwhite92/b2-sdk-php,) and Go (https://github.com/kurin/blazer.) Let us know in the comments about other SDKs you’d like to see or perhaps start your own GitHub project. We will publish any updates in our next B2 roundup.
What You Can Do with Affordable and Available Cloud Storage
You’re probably aware that B2 is up to 75% less expensive than other similar cloud storage services like Amazon S3 and Microsoft Azure. Businesses and organizations are finding that projects that previously weren’t economically feasible with other Cloud Storage services are now not only possible, but a reality with B2. Here are a few recent examples:
SermonAudio wanted their media files to be readily available, but didn’t want to build and manage their own internal storage farm. Until B2, cloud storage was just too expensive to use. Now they use B2 to store their audio and video files, and also as the primary source of downloads and streaming requests from their subscribers.
Fellowship Church wanted to escape from the ever increasing amount of time they were spending saving their data to their LTO-based system. Using B2 saved countless hours of personnel time versus LTO, fit easily into their video processing workflow, and provided instant access at any time to their media library.
Panna Cooking replaced their closet full of archive hard drives with a cost-efficient hybrid-storage solution combining 45Drives and Backblaze B2 Cloud Storage. Archived media files that used to take hours to locate are now readily available regardless of whether they reside in local storage or in the B2 Cloud.
Leading companies in backup, archive, and sync continue to add B2 Cloud Storage as a storage destination for their customers. These companies realize that by offering B2 as an option, they can dramatically lower the total cost of ownership for their customers — and that’s always a good thing.
If your favorite application is not integrated to B2, you can do something about it. One integration partner told us they received over 200 customer requests for a B2 integration. The partner got the message and the integration is currently in beta test.
Below are some of the partner integrations completed in the past few months. You can check the B2 Partner Integrations page for a complete list.
Archiware — Both P5 Archive and P5 Backup can now store data in the B2 Cloud making your offsite media files readily available while keeping your off-site storage costs predictable and affordable.
Arq — Combine Arq and B2 for amazingly affordable backup of external drives, network drives, NAS devices, Windows PCs, Windows Servers, and Macs to the cloud.
GoodSync — Automatically synchronize and back up all your photos, music, email, and other important files between all your desktops, laptops, servers, external drives, and sync, or back up to B2 Cloud Storage for off-site storage.
QNAP — QNAP Hybrid Backup Sync consolidates backup, restoration, and synchronization functions into a single QTS application to easily transfer your data to local, remote, and cloud storage.
Morro Data — Their CloudNAS solution stores files in the cloud, caches them locally as needed, and syncs files globally among other CloudNAS systems in an organization.
Restic – Restic is a fast, secure, multi-platform command line backup program. Files are uploaded to a B2 bucket as de-duplicated, encrypted chunks. Each backup is a snapshot of only the data that has changed, making restores of a specific date or time easy.
Transmit 5 by Panic — Transmit 5, the gold standard for macOS file transfer apps, now supports B2. Upload, download, and manage files on tons of servers with an easy, familiar, and powerful UI.
UpdraftPlus — WordPress developers and admins can now use the UpdraftPlus Premium WordPress plugin to affordably back up their data to the B2 Cloud.
Getting Started with B2 Cloud Storage
If you’re using B2 today, thank you. If you’d like to try B2, but don’t know where to start, here’s a guide to getting started with the B2 Web Interface — no programming or scripting is required. You get 10 gigabytes of free storage and 1 gigabyte a day in free downloads. Give it a try.
If you want to support Active Directory–aware workloads in AWS and Office 365 simultaneously using a managed Active Directory in the cloud, you need a nonintuitive integration to synchronize identities between deployments. AWS has recently introduced the ability for you to authenticate your Office 365 permissions using AWS Directory Service for Microsoft Active Directory (AWS Managed Microsoft AD) by using a custom configuration of Active Directory Federation Services (AD FS). In this webinar, AWS Directory Service Product Manager Ron Cully shows how to configure your AWS Managed Microsoft AD environment to synchronize with Office 365. He will provide detailed configuration settings, architectural considerations, and deployment steps for a highly available, secure, and easy-to-manage solution in the AWS Cloud.
You also will learn how to:
Deploy AWS Managed Microsoft AD.
Deploy Microsoft Azure AD Connect and AD FS with AWS Managed Microsoft AD.
Authenticate user access to Office 365 by using AWS Managed Microsoft AD.
WordPress is the most popular CMS (Content Management System) for websites, with almost 30% of all websites in the world using WordPress. That’s a lot of sites — over 350 million!
In this post we’ll talk about the different approaches to keeping the data on your WordPress website safe.
Stop the Presses! (Or the Internet!)
As we were getting ready to publish this post, we received news from UpdraftPlus, one of the biggest WordPress plugin developers, that they are supporting Backblaze B2 as a storage solution for their backup plugin. They shipped the update (1.13.9) this week. This is great news for Backblaze customers! UpdraftPlus is also offering a 20% discount to Backblaze customers wishing to purchase or upgrade to UpdraftPlus Premium. The complete information is below.
Your WordPress website data is on a web server that’s most likely located in a large data center. You might wonder why it is necessary to have a backup of your website if it’s in a data center. Website data can be lost in a number of ways, including mistakes by the website owner (been there), hacking, or even domain ownership dispute (I’ve seen it happen more than once). A website backup also can provide a history of changes you’ve made to the website, which can be useful. As an overall strategy, it’s best to have a backup of any data that you can’t afford to lose for personal or business reasons.
Your web hosting company might provide backup services as part of your hosting plan. If you are using their service, you should know where and how often your data is being backed up. You don’t want to find out too late that your backup plan was not adequate.
Sites on WordPress.com are automatically backed up by VaultPress (Automattic), which also is available for self-hosted WordPress installations. If you don’t want the work or decisions involved in managing the hosting for your WordPress site, WordPress.com will handle it for you. You do, however, give up some customization abilities, such as the option to add plugins of your own choice.
Very large and active websites might consider WordPress VIP by Automattic, or another premium WordPress hosting service such as Pagely.com.
This post is about backing up self-hosted WordPress sites, so we’ll focus on those options.
Backup strategies for WordPress can be divided into broad categories depending on 1) what you back up, 2) when you back up, and 3) where the data is backed up.
With server data, such as with a WordPress installation, you should plan to have three copies of the data (the 3-2-1 backup strategy). The first is the active data on the WordPress web server, the second is a backup stored on the web server or downloaded to your local computer, and the third should be in another location, such as the cloud.
We’ll talk about the different approaches to backing up WordPress, but we recommend using a WordPress plugin to handle your backups. A backup plugin can automate the task, optimize your backup storage space, and alert you of problems with your backups or WordPress itself. We’ll cover plugins in more detail, below.
What to Back Up?
The main components of your WordPress installation are:
You should decide which of these elements you wish to back up. The database is the top priority, as it contains all your website posts and pages (exclusive of media). Your current theme is important, as it likely contains customizations you’ve made. Following those in priority are any other files you’ve customized or made changes to.
You can choose to back up the WordPress core installation and plugins, if you wish, but these files can be downloaded again if necessary from the source, so you might not wish to include them. You likely have all the media files you use on your website on your local computer (which should be backed up), so it is your choice whether to back these up from the server as well.
If you wish to be able to recreate your entire website easily in case of data loss or disaster, you might choose to back up everything, though on a large website this could be a lot of data.
Generally, you should 1) prioritize any file that you’ve customized that you can’t afford to lose, and 2) decide whether you need a copy of everything in order to get your site back up quickly. These choices will determine your backup method and the amount of storage you need.
A good backup plugin for WordPress enables you to specify which files you wish to back up, and even to create separate backups and schedules for different backup contents. That’s another good reason to use a plugin for backing up WordPress.
When to Back Up?
You can back up manually at any time by using the Export tool in WordPress. This is handy if you wish to do a quick backup of your site or parts of it. Since it is manual, however, it is not a part of a dependable backup plan that should be done regularly. If you wish to use this tool, go to Tools, Export, and select what you wish to back up. The output will be an XML file that uses the WordPress Extended RSS format, also known as WXR. You can create a WXR file that contains all of the information on your site or just portions of the site, such as posts or pages by selecting: All content, Posts, Pages, or Media. Note: You can use WordPress’s Export tool for sites hosted on WordPress.com, as well.
Many of the backup plugins we’ll be discussing later also let you do a manual backup on demand in addition to regularly scheduled or continuous backups.
Note: Another use of the WordPress Export tool and the WXR file is to transfer or clone your website to another server. Once you have exported the WXR file from the website you wish to transfer from, you can import the WXR file from the Tools, Import menu on the new WordPress destination site. Be aware that there are file size limits depending on the settings on your web server. See the WordPress Codex entry for more information. To make this job easier, you may wish to use one of a number of WordPress plugins designed specifically for this task.
You also can manually back up the WordPress MySQL database using a number of tools or a plugin. The WordPress Codex has good information on this. All WordPress plugins will handle this for you and do it automatically. They also typically include tools for optimizing the database tables, which is just good housekeeping.
A dependable backup strategy doesn’t rely on manual backups, which means you should consider using one of the many backup plugins available either free or for purchase. We’ll talk more about them below.
Which Format To Back Up In?
In addition to the WordPress WXR format, plugins and server tools will use various file formats and compression algorithms to store and compress your backup. You may get to choose between zip, tar, tar.gz, tar.gz2, and others. See The Most Common Archive File Formats for more information on these formats.
Select a format that you know you can access and unarchive should you need access to your backup. All of these formats are standard and supported across operating systems, though you might need to download a utility to access the file.
Where To Back Up?
Once you have your data in a suitable format for backup, where do you back it up to?
We want to have multiple copies of our active website data, so we’ll choose more than one destination for our backup data. The backup plugins we’ll discuss below enable you to specify one or more possible destinations for your backup. The possible destinations for your backup include:
A backup folder on your web server
A backup folder on your web server is an OK solution if you also have a copy elsewhere. Depending on your hosting plan, the size of your site, and what you include in the backup, you may or may not have sufficient disk space on the web server. Some backup plugins allow you to configure the plugin to keep only a certain number of recent backups and delete older ones, saving you disk space on the server.
Email to you
Because email servers have size limitations, the email option is not the best one to use unless you use it to specifically back up just the database or your main theme files.
FTP, SFTP, SCP, WebDAV
FTP, SFTP, SCP, and WebDAV are all widely-supported protocols for transferring files over the internet and can be used if you have access credentials to another server or supported storage device that is suitable for storing a backup.
Sync service (Dropbox, SugarSync, Google Drive, OneDrive)
A sync service is another possible server storage location though it can be a pricier choice depending on the plan you have and how much you wish to store.
Cloud storage (Backblaze B2, Amazon S3, Google Cloud, Microsoft Azure, Rackspace)
A cloud storage service can be an inexpensive and flexible option with pay-as-you go pricing for storing backups and other data.
A good website backup strategy would be to have multiple backups of your website data: one in a backup folder on your web hosting server, one downloaded to your local computer, and one in the cloud, such as with Backblaze B2.
If I had to choose just one of these, I would choose backing up to the cloud because it is geographically separated from both your local computer and your web host, it uses fault-tolerant and redundant data storage technologies to protect your data, and it is available from anywhere if you need to restore your site.
Backup Plugins for WordPress
Probably the easiest and most common way to implement a solid backup strategy for WordPress is to use one of the many backup plugins available for WordPress. Fortunately, there are a number of good ones and are available free or in “freemium” plans in which you can use the free version and pay for more features and capabilities only if you need them. The premium options can give you more flexibility in configuring backups or have additional options for where you can store the backups.
How to Choose a WordPress Backup Plugin
When considering which plugin to use, you should take into account a number of factors in making your choice.
Is the plugin actively maintained and up-to-date? You can determine this from the listing in the WordPress Plugin Repository. You also can look at reviews and support comments to get an idea of user satisfaction and how well issues are resolved.
Does the plugin work with your web hosting provider? Generally, well-supported plugins do, but you might want to check to make sure there are no issues with your hosting provider.
Does it support the cloud service or protocol you wish to use? This can be determined from looking at the listing in the WordPress Plugin Repository or on the developer’s website. Developers often will add support for cloud services or other backup destinations based on user demand, so let the developer know if there is a feature or backup destination you’d like them to add to their plugin.
Other features and options to consider in choosing a backup plugin are:
Whether encryption of your backup data is available
What are the options for automatically deleting backups from the storage destination?
Can you globally exclude files, folders, and specific types of files from the backup?
Do the options for scheduling automatic backups meet your needs for frequency?
Can you exclude/include specific database tables (a good way to save space in your backup)?
WordPress Backup Plugins Review
Let’s review a few of the top choices for WordPress backup plugins.
UpdraftPlus is one of the most popular backup plugins for WordPress with over one million active installations. It is available in both free and Premium versions.
UpdraftPlus just released support for Backblaze B2 Cloud Storage in their 1.13.9 update on September 25. According to the developer, support for Backblaze B2 was the most frequent request for a new storage option for their plugin. B2 support is available in their Premium plugin and as a stand-alone update to their standard product.
Note: The developers of UpdraftPlus are offering a special 20% discount to Backblaze customers on the purchase of UpdraftPlus Premium by using the coupon code backblaze20. The discount is valid until the end of Friday, October 6th, 2017.
XCloner supports B2 Cloud Storage in their free plugin.
BlogVault describes themselves as a “complete WordPress backup solution.” They offer a free trial of their paid WordPress backup subscription service that features real-time backups of changes to your WordPress site, as well as many other features.
BlogVault has announced their intent to support Backblaze B2 Cloud Storage in a future update.
BackWPup is a popular and free option for backing up WordPress. It supports a number of options for storing your backup, including the cloud, FTP, email, or on your local computer.
WPBackItUp has been around since 2012 and is highly rated. It has both free and paid versions.
VaultPress is part of Automattic’s well-known WordPress product, JetPack. You will need a JetPack subscription plan to use VaultPress. There are different pricing plans with different sets of features.
Backup by Supsystic
Backup by Supsystic supports a number of options for backup destinations, encryption, and scheduling.
BackUpWordPress is an open-source project on Github that has a popular and active following and many positive reviews.
BackupBuddy, from iThemes, is the old-timer of backup plugins, having been around since 2010. iThemes knows a lot about WordPress, as they develop plugins, themes, utilities, and provide training in WordPress.
BackupBuddy’s backup includes all WordPress files, all files in the WordPress Media library, WordPress themes, and plugins. BackupBuddy generates a downloadable zip file of the entire WordPress website. Remote storage destinations also are supported.
WordPress and the Cloud
Do you use WordPress and back up to the cloud? We’d like to hear about it. We’d also like to hear whether you are interested in using B2 Cloud Storage for storing media files served by WordPress. If you are, we’ll write about it in a future post.
In the meantime, keep your eye out for new plugins supporting Backblaze B2, or better yet, urge them to support B2 if they’re not already.
The Best Backup Strategy is the One You Use
There are other approaches and tools for backing up WordPress that you might use. If you have an approach that works for you, we’d love to hear about it in the comments.
Office 365 provides different options to support user authentication with identities that come from AD. One common way to do this is to use Azure AD Connect and AD FS together with your AD directory. In this model, you use Azure AD Connect to synchronize user names from AD into Azure AD so that Office 365 can use those identities. To complete this solution, you use AD FS to enable Office 365 to authenticate the identities against your AD directory. Good news: AWS Microsoft AD now supports this model!
In this blog post, we show how to use Azure AD Connect and AD FS with AWS Microsoft AD so that your employees can access Office 365 by using their AD credentials.
Note: You must use RDP and sign in with the AWS Microsoft AD admin account using the password you specified when you created your AWS Microsoft AD directory when performing Steps 3 and 6 in this “Prerequisites” section.
The following diagram illustrates the environment you must have in place to implement the solution in this blog post (the numbers in the diagram correspond to Steps 1–8 earlier in this section). We build on this configuration to install and configure Azure AD Connect and AD FS with Azure AD and Office 365.
Note: In this blog post, we use separate Microsoft Windows Server instances on which to run AD FS and Azure AD Connect. You can choose to combine these on a single server, as long as you use Windows Server 2016. Though it is technically possible to use an on-premises server as the AD FS and Azure AD host, such a configuration is counter to the idea of a Windows environment completely in the cloud. Also, this requires configuration of firewall ports and AWS security groups, which is beyond the scope of this blog.
When you create an AWS Microsoft AD directory, AWS exclusively retains the enterprise administrator account of the forest and domain administrator account for the root domain to deliver the directory as a managed service. When you set up your directory, AWS creates an organizational unit (OU) in the directory and delegates administrative privileges for the OU to your admin account. Within this OU, you administer users, groups, computers, Group Policy objects, other devices, and additional OUs as needed. You perform these actions using standard AD administration tools from a computer that is joined to an AWS Microsoft AD domain. Typically, the administration computer is an EC2 instance that you access using RDP, by logging in with your admin account credentials. From your admin account, you can also delegate permissions to other users or groups you create within your OU.
To use Office 365 with AD identities, you use Azure AD Connect to synchronize the AD identities into Azure AD. There are two commonly supported ways to use Azure AD Connect to support Office 365 use. In one model, you synchronize user names only, and you use AD FS to federate authentication from Office 365 to your AD. In the second model, you synchronize user names and passwords from your AD directory to Azure AD, and you do not have to use AD FS. The model supported by AWS Microsoft AD is the first model: synchronize user names only and use AD FS to authenticate from Office 365 to your AWS Microsoft AD. The AD FS model also enables authentication with SaaS applications that support federated authentication (this topic is beyond the scope of this blog post).
Note: Azure AD Connect now has a pass-through model of authentication. Because this was in a preview status at the time of writing this blog post, this authentication model is beyond the scope of this blog post.
In a default AD FS installation, AD FS uses two containers that require special AD permissions that your AWS Microsoft AD administrative account does not have. To address this, you will create two nested containers in your OU for AD FS to use. When you install AD FS, you tell AD FS where to find the containers through a Windows PowerShell parameter.
As described previously, we will now show you how to use Azure AD Connect and AD FS with AWS Microsoft AD with Azure AD and Office 365 in five steps, as illustrated in the following diagram.
Add two containers to AWS Microsoft AD for use by AD FS.
Install AD FS.
Integrate AD FS with Azure AD.
Synchronize users from AWS Microsoft AD to Azure AD with Azure AD Connect.
Sign in to Office 365 by using your Microsoft AD identities.
Step 1: Add two containers to AWS Microsoft AD for use by AD FS
The following steps show how to create the AD containers required by AD FS in your AWS Microsoft AD directory.
From the Management instance:
Generate a random global unique identifier (GUID) using the following Windows PowerShell command.
Make a note of the GUID output because it will be required later on. In this case, the GUID is 67734c62-0805-4274-b72b-f7171110cd56.
Create a container named ADFS in your OU. The OU is located in the domain root and it has the same name as the NetBIOS name you specified when you created your AWS Microsoft AD directory. In this example, our OU name is AWS, and our domain is DC=awsexample,DC=com. You create the container by running the following Windows PowerShell command. You must replace the names that are in bold text with the names from your AWS Microsoft AD directory.
Create another AD container in your new ADFS container, and use the previously generated GUID as the name. Do this by running the following Windows PowerShell command. Be sure to replace the names in bold text with the names from your AWS Microsoft AD directory and your GUID. In this example, we replace GUID with 67734c62-0805-4274-b72b-f7171110cd56. The other bold items shown match the names in our example AWS Microsoft AD directory.
To verify that you successfully created the ADFS and GUID containers, open Active Directory Users and Computers and navigate to the containers you created. Your root domain, OU name, and GUID name should match your AWS Microsoft AD configuration.
Note: If you do not see the ADFS and GUID containers, turn on Advanced Features by choosing View in the Active Directory Users and Computers tool, and then choosing Advanced Features.
Step 2: Install AD FS
In this section, we show how to install AD FS by using Windows PowerShell commands. First, though, select a federation service name for your AD FS server. You can create your federation service name by adding a short name (for example, sts) followed by your domain name (for example, awsexample.com). In this example, we use sts.awsexample.com as the federation service name.
Using your AWS Microsoft AD admin account, open an RDP session to your ADFS instance, run Windows PowerShell as a local administrator, and complete the following steps:
Install the Windows feature, AD FS, by running the following Windows PowerShell command. This command only adds the components needed to install your ADFS server later.
Now that you have installed AD FS, you must obtain a certificate for use when you configure your ADFS server. The AD FS certificate plays an important role to secure communication between the ADFS server and clients, and to ensure tokens issued by the ADFS server are secured. AWS recommends that you use a certificate from a trusted Certificate Authority (CA).
In our example, we use the SSL certificate, sts.awsexample.com. It is important to note that the common name and subject alternative name (SAN) must include the federation service name we plan to use for the AD FS server. In our example, the name is sts.awsexample.com.
Choose File, choose Add/Remove snap-in, and choose Add.
For Add StandaloneSnap-in, choose Certificates and then choose Add.
For the Certificates snap-in, choose Computer account and then choose Next.
Choose Finish, and then choose OK to load the Certificates snap In.
Expand Certificates (Local Computer).
Right-click Personal, choose All Tasks, and then choose Import.
On the Certificate Import Wizard, choose Next.
Choose Browse to locate and select your certificate that has been given by your CA. Choose Next.
Ensure Certificate store is set to Personal, and choose Next.
Choose Finish and OK to complete the installation of the certificate on the AD FS server.
Next you need to retrieve the Thumbprint value of the newly installed certificate and save it for use when you configure your ADFS server. Follow the remaining steps:
In the Certificates console window, expand Personal, and choose Certificates.
Right-click the certificate, and then choose Open.
Choose the Details tab to locate the Thumbprint
Note: In this case, we will copy our certificate Thumbprint, d096652327cfa18487723ff61040c85c7f57f701, and save it in Windows Notepad.
Open an RDP session to your ADFS server by using the admin account for your AWS Microsoft AD directory. Install AD FS by running the following Windows PowerShell command. You must replace the bold strings in the command with the GUID you created in Step 1 and the names from your AWS Microsoft AD directory.
Enter the AD FS standard user account credentials for the ADFSSVC user and save it in the script variable, $svcCred, by running the following Windows PowerShell command.
$svcCred = (get-credential)
Type the Microsoft AD administrator credentials of the Admin user and save it in the script variable, $localAdminCred, by running the following Windows PowerShell command.
$localAdminCred = (get-credential)
Install the AD FS server by running the following Windows PowerShell command. You must replace the bold items with the Thumbprint ID from your certificate, and replace the federation service name with the federation service name you chose earlier. For our example, the federation service name is awsexample.com and we copy our certificate Thumbprint, d096652327cfa18487723ff61040c85c7f57f701, from where we saved it in Windows Notepad.
Note: Be sure to remove any empty spaces in the certificate Thumbprint value.
Create a DNS A record for use with AD FS. This record resolves the federation service name to the public IP address you assign to your ADFS instance. You must create the DNS A record at the DNS hosting provider that hosts your domain. In the following example, sts.awsexample.com is the federation service name and 54.x.x.x is the public IP address of our AD FS instance.
Enable the AD FS sign-in page by running the following Windows PowerShell command.
To verify that the AD FS sign-in page works, open a browser on the AD FS instance, and sign in on the AD FS sign-in page (https://<myfederation service name>/AD FS/ls/IdpInitiatedSignOn.aspx) by using your AWS Microsoft AD admin account. In our example, the federation service name (<my federation service name> in the sign-in page URL) is sts.awsexample.com.
Step 3: Integrate AD FS with Azure AD
The following steps show you how to connect AD FS with Office 365 by connecting to Azure AD with Windows PowerShell and federating the custom domain.From the ADFS instance, make sure you run Windows PowerShell as a local administrator and complete the following steps:
Connect to Azure AD using Windows PowerShell. Federate the custom domain you added and verified in Azure AD by running the following two Windows PowerShell commands. You must update the items in bold text with the names from your AWS Microsoft AD directory. For our example, our AD FS instance’s Fully Qualified Domain Name (FQDN) is adfsserver.awsexample.com, and our domain name is awsexample.com.
Step 4: Synchronize users from AWS Microsoft AD to Azure AD with Azure AD Connect
The following steps show you how to install and customize Azure AD Connect to synchronize your AWS Microsoft AD identities to Azure AD for use with Office 365.Open an RDP session to your ADSync instance by using your AWS Microsoft AD admin user account:
On the Welcome page of the Azure AD Connect Wizard, accept the license terms and privacy notice, and then choose Continue.
On the Express Settings page, choose Customize.
On the Install required components page, choose Install.
On the User sign-in page, choose Do not configure and then choose Next.
On the Connect to Azure AD page, enter your Office 365 global administrator account credentials and then choose Next.
On the Connect your directories page, choose Active Directory as the Directory Type, and then choose your Microsoft AD Forest as your Forest. Choose Add Directory.
At the prompt, enter your AWS Microsoft AD admin account credentials, and then choose OK.
Now that you have added the AWS Microsoft AD directory, choose Next.
On the Azure AD sign-in configuration page, choose Next.
Note: AWS recommends the userPrincipalName (UPN) attribute for use by AWS Microsoft AD users when they sign in to Azure AD and Office 365. The UPN attribute format combines the user’s login name and the UPN-suffix of an AWS Microsoft AD user. The UPN suffix is the domain name of your AWS Microsoft AD domain and the same domain name you added and verified with Azure AD.
In the following example from the Active Directory Users and Computers tool, the user’s UPN is [email protected], which is a combination of the user’s login name, awsuser, with the UPN-suffix, @awsexample.com.
On the Domain and OU filtering page, choose Sync selected domains and OUs, choose the Users OU under your NetBIOS OU, and then choose Next.
On the Uniquely identifying your users page, choose Next.
On the Filter users and devices page, choose Next.
On the Optional features page, choose Next.
On the Ready to configure page, choose Start the synchronization process when configuration completes, and then choose Install.
The Azure AD Connect installation has now completed. Choose Exit.
Note: By default, the Azure AD Connect sync scheduler runs every 30 minutes to synchronize your AWS Microsoft AD identities to Azure AD. You can tune the scheduler by opening a Windows PowerShell session as an administrator and running the appropriate Windows PowerShell commands. For more information, go to Azure AD Connect Sync Scheduler.
Tip: Do you need to synchronize a change immediately? You can manually start a sync cycle outside the scheduled sync cycle from the Azure AD Connect sync instance. Open a Windows PowerShell session as an administrator and run the following Windows PowerShell commands.
Step 5: Sign in to Office 365 by using your AWS Microsoft AD identities
The following steps show you how to sign in to Office 365 using AD FS as the authentication method with your AWS Microsoft AD user account. In this example, we assign a license to the AWS Microsoft AD user account, [email protected], in the Office 365 admin center. We then sign in to Office 365 by using the AWS Microsoft AD user account UPN, [email protected].
Using a computer on the internet, open a browser and complete the following steps:
When entering the UPN of the AWS Microsoft AD user account, you will be redirected to your ADFS server sign-in page to complete user authentication.
On the AD FS sign-in page, enter your UPN and the password of the AWS Microsoft AD user account.
You have successfully signed in to Office 365 using your AWS Microsoft AD user account!
In this blog post, we showed how to use Azure AD Connect and AD FS with AWS Microsoft AD so that your employees can access Office 365 using their AD credentials. Now that you have Azure AD Connect and AD FS in place, you also might want to explore how to build upon this infrastructure to add sign-in for other Software as a Service (SaaS) applications that are compatible with AD FS. For example, this blog post explains how you can provide your users single sign-on access to Amazon AppStream by using AD FS.
So why do many vendors make it so hard to get information about how much you’re storing and how much you’re being charged?
Cloud storage is fast becoming the central repository for mission critical information, irreplaceable memories, and in some cases entire corporate and personal histories. Given this responsibility, we believe cloud storage vendors have an obligation to be transparent as possible in how they interact with their customers.
In that light we decided to challenge four cloud storage vendors and ask two simple questions:
Can a customer understand how much data is stored?
Can a customer understand the bill?
The detailed results are below, but if you wish to skip the details and the screen captures (TL;DR), we’ve summarized the results in the table below.
Summary of Cloud Storage Pricing Test
Our challenge was to upload 1 terabyte of data, store it for one month, and then download it.
Visibility to Data Stored
Easy to Understand Bill
Accurate, intuitive display of storage information.
Available on demand, and the site clearly defines what has and will be charged for.
Storage is being measured in KiB, but is billed by the GB. With a calculator, it is unclear how much storage we are using.
Available, but difficult to find. The nearly 30 day lag in billing creates business and accounting challenges.
Incomplete. From the file browsing user interface, there is no reasonable way to understand how much data is being stored.
Available on demand. While there are some line items that seem unnecessary for our test, the bill is generally straight-forward to understand.
Google Cloud Service
Incomplete. From the file browsing user interface, there is no reasonable way to understand how much data is being stored.
Available, but provides descriptions in units that are not on the pricing table nor commonly used.
Cloud Storage Test Details
For our tests, we choose Backblaze B2, Microsoft’s Azure, Amazon’s S3, and Google Cloud Storage. Our idea was simple: Upload 1 TB of data to the comparable service for each vendor, store it for 1 month, download that 1 TB, then document and share the results.
Let’s start with most obvious observation, the cost charged by each vendor for the test:
Google Cloud Service
Later in this post, we’ll see if we can determine the different cost components (storage, downloading, transactions, etc.) for each vendor, but our first step is to see if we can determine how much data we stored. In some cases, the answer is not as obvious as it would seem.
Test 1: Can a Customer Understand How Much Data Is Stored?
At the core, a provider of a service ought to be able to tell a customer how much of the service he or she is using. In this case, one might assume that providers of Cloud Storage would be able to tell customers how much data is being stored at any given moment. It turns out, it’s not that simple.
Backblaze B2 Logging into a Backblaze B2 account, one is presented with a summary screen that displays all “buckets.” Each bucket displays key summary information, including data currently stored.
Clicking into a given bucket, one can browse individual files. Each file displays its size, and multiple files can be selected to create a size summary.
Summary: Accurate, intuitive display of storage information.
Moving on to Microsoft’s Azure, things get a little more “exciting.” There was no area that we could find where one can determine the total amount of data, in GB, stored with Azure.
There’s an area entitled “usage,” but that wasn’t helpful.
We then moved on to “Overview,” but had a couple challenges.The first issue was that we were presented with KiB (kibibyte) as a unit of measure. One GB (the unit of measure used in Azure’s pricing table) equates to roughly 976,563 KiB. It struck us as odd that things would be summarized by a unit of measure different from the billing unit of measure.
Summary: Storage is being measured in KiB, but is billed by the GB. Even with a calculator, it is unclear how much storage we are using.
Next we checked on the data we were storing in S3. We again ran into problems.
In the bucket overview, we were able to identify our buckets. However, we could not tell how much data was being stored.
Drilling into a bucket, the detail view does tell us file size. However, there was no method for summarizing the data stored within that bucket or for multiple files.
Summary: Incomplete. From the file browsing user interface, there is no reasonable way to understand how much data is being stored.
Google Cloud Storage (“GCS”)
GCS proved to have its own quirks, as well.
One can easily find the “bucket” summary, however, it does not provide information on data stored.
Clicking into the bucket, one can see files and the size of an individual file. However, no ability to see data total is provided.
Summary: Incomplete. From the file browsing user interface, there is no reasonable way to understand how much data is being stored.
Test 1 Conclusions
We knew how much storage we were uploading and, in many cases, the user will have some sense of the amount of data they are uploading. However, it strikes us as odd that many vendors won’t tell you how much data you have stored. Even stranger are the vendors that provide reporting in a unit of measure that is different from the units in their pricing table.
Inside of the Backblaze user interface, one finds a navigation link entitled “Billing.” Clicking on that, the user is presented with line items for previous bills, payments, and an estimate for the upcoming charges.
One can expand any given row to see the the line item transactions composing each bill.
Summary: Available on demand, and the site clearly defines what has and will be charged for.
Trying to understand the Azure billing proved to be a bit tricky.
On August 6th, we logged into the billing console and were presented with this screen.
As you can see, on Aug 6th, billing for the period of May-June was not available for download. For the period ending June 26th, we were charged nearly a month later, on July 24th. Clicking into that row item does display line item information.
Summary: Available, but difficult to find. The nearly 30 day lag in billing creates business and accounting challenges.
Amazon presents a clean billing summary and enables users to “drill down” into line items.
Going to the billing area of AWS, one can survey various monthly bills and is presented with a clean summary of billing charges.
Expanding into the billing detail, Amazon articulates each line item charge. Within each line item, charges are broken out into sub-line items for the different tiers of pricing.
Summary: Available on demand. While there are some line items that seem unnecessary for our test, the bill is generally straight-forward to understand.
Google Cloud Storage (“GCS”)
This was an area where the GCS User Interface, which was otherwise relatively intuitive, became confusing.
Going to the Billing Overview page did not offer much in the way of an overview on charges.
However, moving down to the “Transactions” section did provide line item detail on all the charges incurred. However, similar to Azure introducing the concept of KiB, Google introduces the concept of the equally confusing Gibibyte (GiB). While all of Google’s pricing tables are listed in terms of GB, the line items reference GiB. 1 GiB is 1.07374 GBs.
Summary: Available, but provides descriptions in units that are not on the pricing table nor commonly used.
Test 2 Conclusions
Clearly, some vendors do a better job than others in making their pricing available and understandable. From a transparency standpoint, it’s difficult to justify why a vendor would have their pricing table in units of X, but then put units of Y in the user interface.
Transparency: The Backblaze Way
Transparency isn’t easy. At Backblaze, we believe in investing time and energy into presenting the most intuitive user interfaces that we can create. We take pride in our heritage in the consumer backup space — servicing consumers has taught us how to make things understandable and usable. We do our best to apply those lessons to everything we do.
This philosophy reflects our desire to make our products usable, but it’s also part of a larger ethos of being transparent with our customers. We are being trusted with precious data. We want to repay that trust with, among other things, transparency.
So “Fast and Furious 8” opened this weekend to world-wide box office totals of $500,000,000. I thought I’d write up some notes on the “hacking” in it. The tl;dr version is this: yes, while the hacking is a bit far fetched, it’s actually more realistic than the car chase scenes, such as winning a race with the engine on fire while in reverse.
[SPOILERS] Car hacking
The most innovative cyber-thing in the movie is the car hacking. In one scene, the hacker takes control of the cars in a parking structure, and makes them rain on to the street. In another scene, the hacker takes control away from drivers, with some jumping out of their moving cars in fear.
How real is this?
Well, today, few cars have a mechanical link between the computer and the steering wheel. No amount of hacking will fix the fact that this component is missing.
With that said, most new cars have features that make hacking possible. I’m not sure, but I’d guess more than half of new cars have internet connections (via the mobile phone network), cameras (for backing up, but also looking forward for lane departure warnings), braking (for emergencies), and acceleration.
In other words, we are getting really close.
As this Wikipedia article describes, there are levels for autonomous cars. At level 2 or 3, cars get automated steering, either for parking or for staying in the lane. Level 3 autonomy is especially useful, as it means you can sit back and relax while your car is sitting in a traffic jam. Higher levels of autonomy are still decades away, but most new cars, even the cheapest low end cars, will be level 3 within 5 years. That they make traffic jams bearable makes this an incredibly attractive feature.
Thus, while this scene is laughable today, it’ll be taken seriously in 10 years. People will look back on how smart this movie was at predicting the future.
Car hacking, part 2 Quite apart from the abilities of cars, let’s talk about the abilities of hackers.
The recent ShadowBrokers dump of NSA hacking tools show that hackers simply don’t have a lot of range. Hacking one car is easy — hacking all different models, makes, and years of cars is far beyond the ability of any hacking group, even the NSA.
I mean, a single hack may span more than one car model, and even across more than one manufacturer, because they buy such components from third-party manufacturers. Most cars that have cameras buy them from MobileEye, which was recently acquired by Intel. As I blogged before, both my Parrot drone and Tesla car have the same WiFi stack, and both could be potential hacked with the same vulnerability. So hacking many cars at once isn’t totally out of the question.
It’s just that hacking all the different cars in a garage is completely implausible. God’s Eye
The plot of the last two movies as been about the “God’s Eye”, a device that hacks into every camera and satellite to view everything going on in the world.
First of all, all hacking is software. The idea of stealing a hardware device in order enable hacking is therefore (almost) always fiction. There’s one corner case where a quantum chip factoring RSA would enable some previously impossible hacking, but it still can’t reach out and hack a camera behind a firewall.
Hacking security cameras around the world is indeed possible, though. The Mirai botnet of last year demonstrated this. It wormed its way form camera to camera, hacking hundreds of thousands of cameras that weren’t protected by firewalls. It used these devices as simply computers, to flood major websites, taking them offline. But it could’ve also used the camera features, to upload pictures and video’s to the hacker controlling these cameras.
However, most security cameras are behind firewalls, and can’t be reached. Building a “Gody’s Eye” view of the world, to catch a target every time they passed in front of a camera, would therefore be unrealistic.
Moreover, they don’t have either the processing power nor the bandwidth to work like that. It takes heavy number crunching in order to detect faces, or even simple things like license plates, within videos. The cameras don’t have that. Instead, cameras could upload the videos/pictures to supercomputers controlled by the hypothetical hacker, but the bandwidth doesn’t exist. The Internet is being rapidly upgraded, but still, Internet links are built for low-bandwidth webpages, not high-bandwidth streaming from millions of sources.
This rapidly changing. Cameras are rapidly being upgraded with “neural network” chips that will have some rudimentary capabilities to recognize things like license plates, or the outline of a face that could then be uploaded for more powerful number crunching elsewhere. Your car’s cameras already have this, for backup warnings and lane departure warnings, soon all security cameras will have something like this. Likewise, the Internet is steadily being upgraded to replace TV broadcast, where everyone can stream from Netflix all the time, so high-bandwidth streams from cameras will become more of the norm.
Even getting behind a firewall to the camera will change in the future, as owners will simply store surveillance video in the cloud instead of locally. Thus, the hypothetical hacker would only need to hack a small number of surveillance camera companies instead of a billion security cameras.
Evil villain lair: ghost airplane
The evil villain in the movie (named “Cipher”, or course) has her secret headquarters on an airplane that flies along satellite “blind spots” so that it can’t be tracked.
This is nonsense. Low resolution satellites, like NOAA satellites tracking the weather, cover the entire planet (well, as far as such airplanes are concerned, unless you are landing in Antartica). While such satellites might not see the plane, they can track the contrail (I mean, chemtrail). Conversely high resolution satellites miss most of the planet. If they haven’t been tasked to aim at something, they won’t see it. And they can’t be aimed at you unless they already know where you are. Sure, there are moving blind spots where even tasked satellites can’t find you, but it’s unlikely they’d be tracking you anyway.
Since the supervillain was a hacker, the airplane was full of computers. This is nonsense. Any compute power I need as a hacker is better left on the Earth’s surface, either by hacking cloud providers (like Amazon AWS, Microsoft Azure, or Rackspace), or by hiding data centers in Siberia and Tibet. All I need is satellite communication to the Internet from my laptop to be a supervillain. Indeed, I’m unlikely to get the bandwidth I need to process things on the plane. Instead, I’ll need to process everything on the Earth anyway, and send the low-bandwidth results to the plane.
In any case, if I were writing fiction, I’d have nuclear-powered airplanes that stayed aloft for months, operating out of remote bases in the Himalayas or Antartica.
EMP pulses Small EMP pulse weapons exist, that’s not wholly fictional.
However, an EMP with the features, power, and effects in the movie is, of course, fictional. EMPs, even non-nuclear ones, are abused in films/TV so much that the Wikipedia pages on them spend a lot of time debunking them.
Since hacking is the goto deus ex machina these days, they could’ve just had the hackers disable the power instead of using the EMP to do it.
Conclusion In the movie, the hero uses his extraordinary driving skills to blow up a submarine. Given this level of willing disbelief, the exaggerated hacking is actually the least implausible bits of the movie. Indeed, as technology changes, making some of this more possible, the movie might be seen as predicting the future.
We are thrilled to announce that, effective immediately, we are reducing the price of Backblaze B2 Cloud Storage downloads from $0.05 to $0.02 per GB. What’s more, the first gigabyte of data downloaded each day is still free.
Backblaze has always aimed to make storing data astonishingly easy and affordable. This price reduction applies immediately to all existing and new customers, and there are no tiers or minimums required to get this pricing. It’s automatic, and it starts today.
Why Does This Matter?
It makes cloud storage useful for more people.
B2 is already known for being reliable, easy-to-use, and affordable – our storage pricing is ¼ that of S3. This allows you to save more thorough backups, keep longer archives, store large data sets needed for machine learning and much more. Reducing the price of downloading data lowers the total cost of using cloud storage. This makes cloud storage a viable solution for organizations where it previously didn’t make financial sense.
For example, Vintage Aerial has over 50 years’ worth of aerial photography of rural America. It’s an American treasure. They scanned and digitized the photos and needed a place to cost effectively store the hi-res image files they would send to their customers. Before B2, the other cloud storage options were simply too expensive, leaving Vintage Aerial in the unenviable position of trying to figure out which of their assets they could offer for sale online. But, as Vintage Aerial CEO Fritz Byers says, “because of B2’s pricing, reliability, and service levels, Vintage Aerial is now able to offer and monetize our complete catalog of over 20 million pictures to anyone that’s interested.”
Today’s reduction in download pricing opens another opportunity for Vintage Aerial – downloading high-res photos as previews to its customers. Customers will soon be able to see in detail what they’re getting and zoom in to request specific parts of photos. B2 is empowering Vintage Aerial to provide new functionality that dramatically improves the customer experience and expands the company’s market.
It gives you access to your data when you need it.
Backblaze B2 removes the need to choose between cost and access when it comes to storing your data in the cloud. When you store data in the cloud, you expect to be able to retrieve it at some point. Some services make it expensive to restore data or place time lag impediments to data access to reduce their cost. That reduces the usefulness of your data. If you need to recover all your data quickly from an archive or backup or want to make your data available in real-time, you don’t want to wait, and you don’t want to be shocked at the price tag.
It ensures that your data is yours.
When it’s expensive to get data out, you feel like your cloud storage provider is holding your files hostage. You can’t switch providers or move data back on-site. Part of Backblaze B2 being easy is ensuring that you can do what you want, when you want, with your information. Reducing the price of downloads ensures you can feel comfortable knowing your data is yours.
It’s another reason for third party applications to integrate with B2.
Many organizations already manage their data backups, archives, and workflows using third party applications that have integrated with B2 Cloud Storage. Applications like CloudBerry, Synology CloudSync, Retrospect, Cantemo, axle Video, CatDV and many others have added B2 support in their products; over the next few months, Transmit and QNAP will release their integrations as well.
For applications that have integrated with B2, users not only get the lowest cost storage but the lowest cost download bandwidth as well. For application providers, integrating B2 offers a differentiated service for their users. If you use an application that doesn’t use B2 Cloud Storage, ask the application provider to add B2 and mention the application in the comments below.
It reduces your bill.
Regardless of how you use B2, the download price reduction matters because it lowers your bill. And a lower bill means you can lower your cost and increase your margins, or lower your prices – each of which makes business better.
How does this compare?
Not only is Backblaze B2 storage 1/4th the price of Amazon S3, Google Cloud, or Azure, but our download pricing is now as little as 1/4th their price as well.
First 1 TB
Next 9 TB
Next 40 TB
Next 100 TB
Next 350 TB+
Using the chart above, let’s compute a few examples of download costs…
Not only is Backblaze B2 pricing dramatically lower cost, it’s also simple. One price for any amount of data downloaded to anywhere. In comparison, to compute the cost of downloading 500 TB of data with S3 you start with the following formula: (($0.09 * 10) + ($0.085 * 40) + ($0.07 * 100) + ($0.05 * 350)) * 1,000. Want to see this comparison for the amount of data you manage? Use our cloud storage calculator.
How did we do this?
Easy, we just lowered the price.
We’ve been reducing the cost of cloud storage for a decade, building and open-sourcing our Storage Pods, developing our Vaults, and more. As a result, we know a fair bit about storing data cost efficiently.
When we announced B2 Cloud Storage, we weren’t totally sure how individuals and companies would use bandwidth, and so we priced it competitively within the market. With a year and a half of B2 usage (and a decade of related experience storing customer data), we’ve determined the patterns are sufficiently stable that we can sustainably reduce our pricing.
To sum up our pricing, downloading data costs $0.02/GB, with the first gigabyte downloaded each day being free. Storage costs are $0.005/GB per month with the first 10 gigabytes being free. We have just one pricing tier so you get the best price we can offer from the start.
Our aim has always been to provide a great service at a fair price. While we’re certainly proud to be the low-cost leader in the space, we’re much happier that we can help customers to be more effective in their businesses.
Enjoy the service, and I’d love to hear in the comments what this price reduction means for you.
The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.