All posts by Jeremy Milk

CORS Correction: Developer Insight on the Backblaze B2 Command Line

Post Syndicated from Jeremy Milk original https://www.backblaze.com/blog/cors-correction-developer-insight-on-the-backblaze-b2-command-line/

We recently spoke with Kristian Kielhofner, a developer and entrepreneur who’s on his third go-round as a startup founder and CEO after two very successful exits. He’s built a next-gen, crypto-centric media asset management platform, Tovera, which launched two days ago.

Developer customers are regularly choosing Backblaze B2 as the cloud storage platform that sits under their products and services. We feel lucky to learn about the innovations they are bringing to this world. Kristian found a clearer path to setting up CORS for B2 Cloud Storage and Cloudflare, so we asked him to share why he started Tovera, how he thought through his cloud storage options, and the exact steps he took to go live with his solution.

—Backblaze

The Tovera Backstory: Fighting Deepfakes

One morning, this story really caught my attention.

Like many technology enthusiasts, I’m familiar with deepfakes. That said, the “Pennsylvania Cheerleading Mom” story told me something: As we’ve seen time and time again, technology rapidly evolves beyond its original intended use. Sometimes for our benefit, and (unfortunately) sometimes not so much…

I realized it would only be a matter of time before this incredibly powerful technology would be in the hands of everyone—for uses good or evil. With more research, I found that (not surprisingly) the current approach to stopping misuse of the technology utilizes the same fundamental machine learning approaches powering the deepfakes themselves. It seems that what we now have is a machine learning arms race: a new model to generate deepfakes, a new model to detect them. Around and around we go.

I began thinking of approaching the deepfake problem from the other side of the coin. What if, instead of using machine learning to guess what is fake, we prove what is real? Deepfake detection models can’t provide 100% certainty today (or ever), but cryptographic authentication can. This simple idea was the genesis for Tovera.

What Does Tovera Do?

Tovera takes digital media you upload and uses existing cryptography and emerging blockchain technology to create a 100% secure validation record. When published on our platform, we can confirm (with 100% certainty) that your digital media assets are yours and haven’t been tampered with.

Tovera asset upload and management page.

After working through the initial proof of concept, I had another revelation: “Hey, while we’re hitting our API whenever and wherever digital media is viewed, why don’t we return some extra stuff?” Now, not only can our users validate that their content is really theirs and hasn’t been modified, they can use the features provided by Tovera Publish to dynamically update their released digital content from our dashboard. With Tovera, any changes you make to your digital media and online presence are updated across social media platforms, websites, and devices globally—instantly.

An image served via Tovera, with authentication dropdown.

In keeping with our mission of ensuring everyone can protect, validate, and control their online presence, we provide this technology for free with a simple sign up and onboarding process.

The Tovera Storage Journey

To provide this service, we needed to host the digital media files somewhere. Of course, you have your go-to juggernauts—Amazon, Google, and Microsoft. The problem is Tovera is a tiny startup. Having some prior startup experience, I know that spending your money and time wisely from the beginning is one of the most important things you can do.

I took one look at pricing from the “big three” cloud providers through the lens of someone who has experience buying bandwidth and storage (long story) and I thought, “Wow, this is a good business.” As has been covered on this blog and elsewhere, the storage and (especially) bandwidth markups from the big providers is, to put it mildly, significant.

Like some of you, I’ve also been a fan of Backblaze for a long time. Since it was announced, I’ve kept an eye on their B2 Cloud Storage product. So, one morning I took it upon myself to give Backblaze B2 a try.

Sign up and initial onboarding couldn’t have been easier. I found myself in the Backblaze B2 user dashboard up and running in no time. Creating application keys for my purposes was also extremely easy.

After deciding B2 Cloud Storage would work in theory, I decided to try it out in practice. As I integrated the service into Tovera, I ran into a few different vexing issues. I thought other devs might be able to benefit from my CORS troubleshooting, and so I’m outlining my experience here.

Checking the Backblaze S3 Compatible API

We make it simple for our users to upload their assets directly to our cloud storage provider. Because B2 Cloud Storage has the Backblaze S3 Compatible API, the use of presigned URLs fits the bill. This way, Tovera users can upload their digital media assets directly to Backblaze, securely, and make them available to the world via our platform.

In case you’re not familiar with the presigned URL process, the overall flow looks something like the structure laid out in this blog post.

After perusing the available documentation, I started off with the following Node.js Javascript code:

With this Javascript function, Tovera API services provide a URL for our user dashboard to instantly (and securely) upload their assets to our Backblaze account. I had read Backblaze B2 has a 100% Amazon S3 Compatible API but I was a little skeptical. Is this really going to work? Sure enough, it worked on the first attempt!

Integrating Cloudflare and Setting Up CORS

Between the Bandwidth Alliance and having dealt with DDoS attacks and shady internet stuff in general before, I’m also a big fan of Cloudflare. Fortunately, Backblaze provides guidance on how to best use B2 Cloud Storage with Cloudflare to make use of their combined power.

Once I set up Cloudflare to work with B2 Cloud Storage and the Tovera API services were returning valid, presigned URLs for clients to do a direct HTTP PUT, I tried it out in our Next.js-powered user dashboard.

Uh-oh. Dreaded CORS errors. I’ll spare you the details, but here’s where things get interesting… I don’t know about you, but CORS can be a little frustrating. LONG story short, I dug in deep, feeling a little like I was wandering around a dark room looking for the light switch.

After fumbling around a bit, I was able to piece together the following with the Backblaze B2 command line utility:

With this usage of the Backblaze B2 command line utility, we’re setting the following CORS rules on our bucket:

  • Allow users to download Backblaze B2 files from anywhere using the native B2 Cloud Storage interfaces.
  • Allow users to use the Backblaze S3 Compatible API to download and upload their files from anywhere with the authenticated presigned URL provided by the server side Javascript function above.

With these rules, Tovera users can use our embeddable verification links across any site they provide them to—existing websites, social media networks, and more. In other applications you may want to limit these CORS rules to what makes sense for your use case.

Focusing on What’s Important

With Backblaze B2, we at Tovera can focus on our mission of putting our digital media security, validation, and publishing functionality in the hands of as many people as possible. Tovera users can take back control of their online presence and address the many threats posed by deepfake technologies that threaten their likeness, reputation, and brand.


Kristian Kielhofner works on overall technical architecture, vision, and strategy for Tovera when he’s not out buying yet another whiteboard to scribble on. Kristian previously built, grew, and exited Star2Star Communications—a leading provider of business productivity solutions.

The post CORS Correction: Developer Insight on the Backblaze B2 Command Line appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Recent Outages: Why We Accelerated Registry Changes

Post Syndicated from Jeremy Milk original https://www.backblaze.com/blog/recent-outages-why-we-accelerated-registry-changes/

Looking Out for Our Team, Customers, and Community

Earlier in February, Backblaze experienced two unexpected service disruptions. We take this very seriously and we apologize for any disruptions that you may have experienced as a result of these instances. We would also like to explain what happened and what steps we have taken to avoid such interruptions in the future.

What Happened:

Backblaze B2 Cloud Storage can serve a variety of use cases, including hosting content. Sometimes cloud storage services are used to host malicious content. The use of malicious content by any of our customers violates Backblaze policy. Any time we receive notice that an account is serving such content involving our cloud storage system, our Compliance team investigates and instigates appropriate actions as needed. (For reference, our preferred reporting channel is an email to reportphishing@backblaze.com.)

Sometimes domain registrars notice or receive these complaints and then reach out to the involved parties with a notice and mandate that the content be removed or the account will be suspended. That is what happened in both outage cases mentioned above: Our registrar notified us of malicious content from certain customers and requested that we remove specific links for these customers from Backblaze B2 in order to avoid suspension of the Backblaze B2 domain.

On February 12, due to a variety of factors—including a notice from a third-party registrar outside of our established communication process—the Backblaze B2 domain was suspended and we worked with our registrar to bring it back online shortly thereafter. This prompted us to review and refine our operating procedures (for example, we expanded our response team and hardened our escalation process) to coordinate more effectively with our third-party registrar, and to move up exploration of developer-friendly registrar solutions more fitting for current and future needs.

On February 17, a similar scenario occurred again. In this case, we acted in line with our new processes, yet despite our removal of the content as they had requested, the domain was still suspended by our third-party registrar. The domain was brought back up shortly thereafter, but for anyone experiencing outages for their Backblaze B2-hosted data who were unable to log in as a result of the suspension, we believe that any outage is too long.

What We’ve Changed

After the second domain suspension, we accelerated our decision to switch registrars—with determination to migrate as fast as possible. We opted to move to Cloudflare, whom we already partner closely with as part of the Bandwidth Alliance and whom we see as offering proven, enterprise-worthy service to a wide array of businesses and organizations. Beyond their technology, one of the key benefits of working with our partners at Cloudflare is that we’ll have a direct line of communication in the event of any future content takedown requests.

We started the process of switching our registration to Cloudflare on February 18 and completed the process on February 23.

We truly appreciate the clear communication and suggestions from our customers and others who were experiencing issues—and yet we aim to make any such future communication unnecessary. If you have any additional questions, please feel free to post them in the comments below and we’ll respond as we’re able!

The post Recent Outages: Why We Accelerated Registry Changes appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Backblaze Mobile Update: iOS and Android Bucket Management

Post Syndicated from Jeremy Milk original https://www.backblaze.com/blog/backblaze-mobile-update-ios-mobile-uploads/

This post was originally published on February 18, 2021 and has been updated to reflect the newest functionality releases for Backblaze Mobile users on both iOS and Android.

Ready to update now? Go to Google Play or the App Store to run updates or download the Backblaze app.

December 20, 2022: Mobile 6.0 Is Available

Today, we’re announcing the arrival of Backblaze Mobile 6.0 featuring an enhanced visual experience, authentication improvements, bug fixes, and many design updates. Check out the specifics below.

What’s New in Backblaze Mobile 6.0?

Backblaze Mobile 6.0 features an overhauled visual experience (so fresh, so clean!).

Before: Android login screen
After: Android login screen

The update also features authentication enhancements for both iOS and Android. We’ve made it easier to log in and opt to see your password in plain text as you enter it. We’ve also optimized the stability of our mobile login flow.

Before: iOS login screen
After: iOS login screen

iOS Updates

  • Design updates: Redesigned login and settings screens, updated icons, and improved upload/download progress animations.
  • Login updates: Email and password now appear on the same screen when logging in, and you can choose to see your password in plain text as you enter it. 
  • Viewing and previewing files: You can now view downloaded files in full-screen mode on iPhones as well as iPads.
  • SwiftUI is here: Much of the iOS code has been migrated to use SwiftUI and The Composable Architecture. 
  • Bug fixes and performance improvements: A lot has been tightened up under the hood, including fixing a file download timeout issue and progress messaging display issues. 
Before
After

Android Updates

  • Design updates: A fresh UI and navigation experience comes courtesy of updated material libraries.
  • Navigation and controls: We’ve also advanced the Android navigation bar, scrollable header and footers, and updated gesture controls for a better Android experience. You can now also see the file path for any file uploaded to Computer Backup or B2 Cloud Storage files.
  • Edit mode and selection capabilities: Navigation and maneuvering inside of edit mode for files, buckets, folders, and downloads has also been improved. We’ve also added multiselection capabilities and swipe-to-delete functionality.
Before
After
Before
After

Backblaze Mobile 6.0 Available Now: Download Today

To get the latest and greatest Backblaze Mobile experience, update your apps or download them today on Google Play or the App Store.

March 28, 2022: Added Folder Creation

Backblaze Mobile users on iOS and Android devices can now create folders directly on their devices with our latest app update. The update is generally available the week of March 27, 2022 for both iOS and Android platforms.

The functionality expands on previous releases to allow users to more easily work from their mobile devices. 

November 30, 2021: Added Bucket Creation and Bucket, Folder, and File Deletion

With this update, Backblaze Mobile users on iOS and Android devices can create buckets and delete buckets, folders, and files directly on their devices.

If you routinely work from your mobile device, this means you’ll be able to better manage your cloud storage while you’re away from your workstation. For media and entertainment pros who regularly shoot images and footage on powerful smart devices, for example, this functionality allows you to create buckets for new projects from the field. And if you need to delete a bucket, file, or folder, you can do that on the go, too. With this functionality at your fingertips, you can focus on shooting, producing, and doing more with ease rather than waiting until you’re back at your desktop or laptop to handle organizational tasks.

The update also included bug fixes and an upgrade to Android 11.

Older Releases

In case you missed the last few releases, Backblaze Mobile allows iOS and Android users to preview and download content through the app and upload files directly to Backblaze B2 Cloud Storage buckets.

The post Backblaze Mobile Update: iOS and Android Bucket Management appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

On-prem to Cloud, Faster: Meet Our Newest Fireball

Post Syndicated from Jeremy Milk original https://www.backblaze.com/blog/on-prem-to-cloud-faster-meet-our-newest-fireball/

We’re determined to make moving data into cloud storage as easy as possible for you, so today we are releasing the latest improvement to our data migration pathways: a bigger, faster Backblaze Fireball.

The new Fireball increases capacity for the rapid ingest service from 70TB to 96TB and connectivity speed from 1 Gb/s to 10 Gb/s so that businesses can move larger data sets and media libraries from on-premises to the Backblaze Storage Cloud faster than before.

What Hasn’t Changed

The service is still drop-dead simple. Data is secure and encrypted during the transfer process, and you gain the benefits of the cloud without having to navigate the constraints (and sluggishness) of internet bandwidth. We’re still happy to send you two, or three, or more Fireballs as needed—you can order whatever you need right from your Backblaze B2 Cloud Storage account. Easy.

How It Works

The customer favorite (of folks like Austin City Limits and Yoga International) service works like this: We ship you the Fireball, you copy on-premises data to it directly or through the transfer tool of your choice, you send the Fireball back to us, and we quickly upload your data into your B2 Cloud Storage account.

The Fireball is not right for everyone—organizations already storing to public clouds now frequently use our cloud to cloud migration solution, while those with small, local data sets often find internet transfer tools more than sufficient. For a refresher, definitely check out this “Pathways to the Cloud” guide.

Don’t Be Afraid to Ask

However you’d like to join us, we’re here to help. So—shameless plug alert—please don’t hesitate to contact our Sales team to talk about how to best start saving with B2 Cloud Storage.

The post On-prem to Cloud, Faster: Meet Our Newest Fireball appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Q&A: Developing for the Data Transfer Project at Facebook

Post Syndicated from Jeremy Milk original https://www.backblaze.com/blog/qa-developing-for-the-data-transfer-project-at-facebook/

Facebook pointing at Backblaze Cloud

In October of 2020, we announced that Facebook integrated Backblaze B2 Cloud Storage as a data transfer destination for their users’ photos and videos. This secure, encrypted service, based on code that Facebook developed with the open-source Data Transfer Project, allows users choices for how and where they manage or archive their media.

We spoke with Umar Mustafa, the Facebook Partner Engineer who led the project, about his team’s role in the Data Transfer Project (DTP) and the development process in configuring the data portability feature for Backblaze B2 Cloud Storage using open-source code. Read on to learn about the challenges of developing data portability including security and privacy practices, coding with APIs, and the technical design of the project.

Q: Can you tell us about the origin of Facebook’s data portability project?

A: Over a decade ago, Facebook launched a portability tool that allowed people to download their information. Since then, we have been adding functionality for people to have more control over their data.

In 2018, we joined the Data Transfer Project (DTP), which is an open-source effort by various companies, like Google, Microsoft, Twitter, and Apple, that aims to build products to allow people to easily transfer a copy of their data between services. The DTP tackles common problems like security, bandwidth limitations, and just the sheer inconvenience when it comes to moving large amounts of data.

And so in connection with this project, we launched a tool in 2019 that lets people port their photos and videos. Google was the first destination and we have partnered with more companies since then, with Backblaze being the most recent one.

Q: As you worked on this tool, did you have a sense for the type of Facebook customer that chooses to copy or transfer their photos and videos over to cloud storage?

A: Yes, we thought of various ways that people could use the tool. Someone might want to try out a new app that manages photos or they might want to archive all the photos and videos they’ve posted over the years in a private cloud storage service.

Q: Would you walk us through the choice to develop it using the open-source DTP code?

A: In order to transfer data between two services, you’d typically use the API from the first service to read data, then transform it if necessary for the second service, and finally use the API from the second service to upload it. While this approach works, you can imagine that it requires a lot of effort every time you need to add a new source or destination. And an API change by any one service would force all its collaborators to make updates.

The DTP solves these problems by offering an open-source data portability platform. It consists of standard data models and a set of service adapters. Companies can create their import and export adapters, or for services with a public API, anyone can contribute the adapters to the project. As long as two services have adapters available for a specific data type (e.g. photos), that data can be transferred between them.

Being open-source also means anyone can try it out. It can be run locally using Docker, and can also be deployed easily in enterprise or cloud-based environments. At Facebook, we have a team that contributes to the project, and we encourage more people from the open-source community to join the effort. More details can be found about the project on GitHub.

Integrating a new service as a destination or a source for an existing data type normally requires adding two types of extensions, an auth extension and a transfer extension. The open-source code is well organized, so you can find all available auth extensions under the extensions/auth module and all transfer extensions under the extensions/data-transfer module, which you can refer to for guidance.

The auth extension only needs to be written once for a service and can be reused for each different data type that the service supports. Some common auth extensions, like OAuth, are already available in the project’s libraries folder and can be extended with very minimal code (mostly config). Alternatively, you can add your own auth extension as long as it implements the AuthServiceExtension interface.

A transfer extension consists of import adapters and export adapters for a service, and each of them is for a single data type. You’ll find them organized by service and data type in the extensions/data-transfer module. In order to add one, you’ll have to add a similar package structure, and write your adapter by implementing the Importer<a extends AuthData, T extends DataModel> interface using the respective AuthData and DataModel classes for the adapter.

For example, in Backblaze we created two import adapters, one for photos and one for videos. Each of them uses the TokenSecretAuthData containing the application key and secret. The photos importer uses the PhotosContainerResource as the DataModel and the videos importer uses the VideosContainerResource. Once you have the boilerplate code in place for the importer or exporter, you have to implement the required methods from the interface to get it working, using any relevant SDKs as you need. As Backblaze offers the Backblaze S3 Compatible APIs, we were able to use the AWS S3 SDK to implement the Backblaze adapters.

There’s a well written integration guide for the project on GitHub that you can follow for further details about integrating with a new service or data type.

Q: Why did you choose Backblaze as a storage endpoint?

A: We want people to be able to choose where they want to take their data. Backblaze B2 is a cloud storage of choice for many people and offers Backblaze S3 Compatible APIs for easy integration. We’re happy to see people using Backblaze to save a copy of their photos and videos.

Q: Can you tell us about the comprehensive security and compliance review you conducted before locking in on Backblaze?

A: Privacy and security is of utmost importance for us at Facebook. When engaging with any partner, we check that they comply with certain standards. Some of the things that help us evaluate a partner include:

  • Information security policies.
  • Privacy policies.
  • Third-party security certifications, as available.

We followed a similar approach to review the security and privacy practices that Backblaze follows, which are also demonstrated by various industry standard certifications.

Q: Describe the process of coding to Backblaze, anything you particularly enjoyed? Anything you found different or challenging? Anything surprising?

A: The integration for the data itself was easy to build. The Backblaze S3 Compatible APIs make coding the adapters pretty straightforward, and Backblaze has good documentation around that.

The only difference between Backblaze and our other existing destinations was with authentication. Most adapters in the DTP use OAuth for authentication, where users log in to each service before initiating a transfer. Backblaze is different as it uses API keys-based authentication. This meant that we had to extend the UI in our tool to allow users to enter their application key details and wire that up as TokenSecretAuthData to the import adapters to transfer jobs securely.

Q: What interested you in data portability?

A: The concept of data portability sparked my interest once I began working at Facebook. Coincidentally, I had recently wondered if it would be possible to move my photos from one cloud backup service to another, and I was glad to discover a project at Facebook addressing the issue. More importantly, I felt that the problem it solves is important.

Facebook is always looking for new ways to innovate, so it comes with an opportunity to potentially influence how data portability will be commonly used and perceived in the future.

Q: What are the biggest challenges for DTP? It seems to be a pretty active project three years after launch. Given all the focus on it, what is it that keeps the challenge alive? What areas are particularly vexing for the project overall?

One major challenge we’ve faced is around technical design—currently the tool has to be deployed and run independently as a single instance to be able to make transfers. This has its advantages and disadvantages. On one hand, any entity or individual can run the project completely and enable transfers to any of the available services as long as the respective credentials are available. On the other hand, in order to integrate a new service, you need to redeploy all the instances where you need that service.

At the moment, Google has their own instance of the project deployed on their infrastructure, and at Facebook we have done the same, as well. This means that a well-working partnership model is required between services to offer the service to their respective users. As one of the maintainers of the project, we try to make this process as swift and hassle-free as possible for new partners.

With more companies investing time in data portability, we’ve started to see increased improvements over the past few months. I’m sure we’ll see more destinations and data types offered soon.

The post Q&A: Developing for the Data Transfer Project at Facebook appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.

Announcing Facebook Photo and Video Transfers Direct to Backblaze B2 Cloud Storage

Post Syndicated from Jeremy Milk original https://www.backblaze.com/blog/facebook-photo-video-transfers-direct-to-cloud-storage/

Facebook pointing to Backblaze

Perhaps I’m dating myself when I say that I’ve been using Facebook for a very long time. So long that the platform is home to many precious photos and videos that I couldn’t imagine losing. And even though they’re mostly shared to Facebook from my phone or other apps, some aren’t. So I’ve periodically downloaded my Facebook albums to my Mac, which I’ve of course set to automatically back up with Backblaze, to ensure they’re safely archived.

And while it’s good to know how to download and back up your social media profile, you might be excited to learn that it’s just become a lot easier: Facebook has integrated Backblaze B2 Cloud Storage directly as a data transfer destination for your photos and videos. This means you can now migrate or copy years of memories in a matter of clicks.

What Data Transfer Means for You

If you use Facebook and want to exercise even greater control over the media you’ve posted there, you’ll find that this seamless integration enables:

  • Personal safeguarding of images and videos in Backblaze.
  • Enhanced file sharing and access control options.
  • Ability to organize, modify, and collaborate on content.

How to Move Your Data to Backblaze B2

Current Backblaze B2 customers can start data transfers within Facebook via Settings & Privacy > Settings / Your Facebook Information / Transfer a Copy of Your Photos or Videos / Choose Destination / Backblaze.

      1. You can find Settings & Privacy listed in the options when you click your profile icon.
      2. Under Settings & Privacy, select Settings.
      3. Go to Your Facebook Information and select “View” next to Transfer a Copy of Your Photos or Videos.

    Transfer a Copy of Your Photos or Videos

      4. Under Choose Destination, simply select Backblaze and your data transfer will begin.

    Transfer a Copy of Your Photos or Videos to Backblaze

If you don’t have a Backblaze B2 account, you can create one here. You’ll need a Key ID and an Application Key when you select Backblaze.

The Data Transfer Project and B2 Cloud Storage

The secure, encrypted data transfer service is based on code Facebook developed through the open-source Data Transfer Project (and you all know we love open-source projects, from our original Storage Pod design to Reed-Solomon erasure coding). Data routed to your B2 Cloud Storage account enjoys our standard $5/TB month pricing with a standard 10GB of free capacity.

Our Co-Founder and CEO, Gleb Budman, noted that this new integration harkens back to our roots: “We’ve been helping people safely store their photos and videos in our cloud for almost as long as Facebook has been providing the means to post content. For people on Facebook who want more choice in hosting their data outside the platform, we’re happy to make our cloud a seamlessly available destination.”

My take: 👍

The post Announcing Facebook Photo and Video Transfers Direct to Backblaze B2 Cloud Storage appeared first on Backblaze Blog | Cloud Storage & Cloud Backup.